Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to set AWS cloudtrail input from another aws account bucket #77

Open
qoovsxp opened this issue May 21, 2018 · 9 comments
Open

How to set AWS cloudtrail input from another aws account bucket #77

qoovsxp opened this issue May 21, 2018 · 9 comments

Comments

@qoovsxp
Copy link

qoovsxp commented May 21, 2018

Hi,

We follow this document to set sharing CloudTrail Log Files Between AWS Accounts.
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-sharing-logs.html
We put B AWS account’s cloudtrail logs to A AWS account bucket.
And we let cloudtrail iam user can access A AWS account bucket.

We set a assume role to input try to get log from A AWS account bucket,but a SQS error occur.
image
I guess it's cause by the assume role cant's access the SQS queue on B AWS account.
And it occur can’t get log from s3 bucket if we don’t set assume role to input.
image

Anyone can help us?

Thanks.

@radykal-com
Copy link
Contributor

Hello,

what version of the graylog-aws-plugin are you using and how is your input configured?

@qoovsxp
Copy link
Author

qoovsxp commented May 21, 2018

Hi radykal,

My graylog version is 2.4.4 and that has integrate graylog-aws-plugin.
I set the assume role on input.
image

image

The sqs set on AWS as follow.
image

@radykal-com
Copy link
Contributor

Well, looks like some kind of permission problems. Can you describe what resources and roles(with permissions) are in each account?

@qoovsxp
Copy link
Author

qoovsxp commented May 22, 2018

A acccount:
s3==>
I build a s3 bucket named "sanderson-cloudtrail" and set the policy for receive cloudtrail from B account.And sure could receive cloudtrail logs on s3 bucket.
image

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-set-bucket-policy-for-multiple-accounts.html

IAM role==>
I set a IAMRole named "sanderson-readlog" for Cross-Account Access and give s3 and SQS permission.
image
image

B account:
IAM User==>
I create a iam user for read sqs and then attach administrator and assumerole's permission.
image

Cloudtrail==>
image

SNS==>
image

SQS==>
I set a sqs named "sandersontest" for subscribe sns notify from cloudtrail.
image

I poll messages from sqs queue "sandersontest" and confirmed that direction is right.
image

These configuration of above will occur error as follow on graylog.
image

Thank you.

@radykal-com
Copy link
Contributor

Well, your setup looks fine for me. I'll try to configure one of my graylog instances with the same setup and check what happens.

@qoovsxp
Copy link
Author

qoovsxp commented May 23, 2018

OK,appreciate your help.I am looking forward to hear about your test result.I have stuck at this error about two month and try this lab for GDPR scenario.I think that could help graylog more suitable in enterprise environment if we resolve this error.

@et304383
Copy link

Would like support for this too. It's AWS best practice to run multiple accounts and centralize CloudTrail logs into a single bucket in a logging account.

It would be incredibly cumbersome to have each account's CloudTrail send SNS notifications to the logging account.

The CloudTrail plugin should support parsing SQS messages sent by S3 event notifications, not just CloudTrail notifications.

The format is defined here:

https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html

Please add support for this format so we can keep things centralized and not needlessly rely on SNS which just inflates the cost of getting the logs to Graylog for no added value.

@et304383
Copy link

@qoovsxp I got around the current limitations by putting S3 events on the CloudTrail bucket, sending them to Lambda, then restructuring them to look like the SNS log delivery messages the plugin wants.

Some python code:

import json

import boto3

sqs_client = boto3.client('sqs')


def handler(event, context):
    s3_event = event['Records'][0]['s3']
    bucket_name = s3_event['bucket']['name']
    object_key = s3_event['object']['key']

    if 'CloudTrail-Digest' in object_key:
        return None

    message = {
        's3Bucket': bucket_name,
        's3ObjectKey': [
            object_key
        ]
    }

    sqs_message = {
        'Message': json.dumps(message)
    }

    sqs_client.send_message(
        QueueUrl='<SQS queue URL>',
        MessageBody=json.dumps(sqs_message)
    )

@naggappan
Copy link

Do SNS support cross account? If yes Then I can provide my aws log account SNS topic to all my other accounts cloud trial.
@et304383 if I use the above come then in cloud trial of different accounts I don't need to select SNS notification right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants