Central Logging CUR with S3

Quick hacky solution because I couldn't be bothered to learn Terragrunt

"Required" Reading

Read it (especially the bottom) for information on how to filter the data once you have it.

CUR SPEC

Perquisites

Please install or update these programs to their latest/indicated versions

Program	Version
GNU Make	> 4.2.1
Terraform	> 1.1.8
gomplate	> 0.0.0
yq	> 4.24.2

Config | Build | Run

1. Run make init
2. Add your SSO / Credential PROFILE names for the member accounts to the memberprofiles.yaml
3. Add the logging account ID and PROFILE name to the logprofile.yaml
4. Add your Identifier and the region to generalconfig.yaml
5. Run make build
6. Run make run

Pain

The Docs lie!!! As of 2022-04 the Official docs describe the wrong permissions required to get cross account replication working!

What it is actually supposed to be (IAM Replication Role Policy - Local)

This was auto generated by the s3 replication wizard

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetReplicationConfiguration",
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging",
        "s3:GetObjectRetention",
        "s3:GetObjectLegalHold"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::local-source",
        "arn:aws:s3:::local-source/*",
        "arn:aws:s3:::remote-destination",
        "arn:aws:s3:::remote-destination/*"
      ]
    },
    {
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags",
        "s3:ObjectOwnerOverrideToBucketOwner"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::local-source/*",
        "arn:aws:s3:::remote-destination/*"
      ]
    }
  ]
}

What it is actually supposed to be (s3 Bucket Policy - Remote)

Inferred from the above

{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : [
            account1.role.arn,
            account2.role.arn,
            ...
          ]
        },
        "Action" : [
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags",
            "s3:ObjectOwnerOverrideToBucketOwner"
            ],
        "Resource" : "arn:aws:s3:::local-destination/*"
      },
      {
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : [
            account1.role.arn,
            account2.role.arn,
            ...
          ]
        },
        "Action" : [
            "s3:ListBucket",
            "s3:GetReplicationConfiguration",
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging",
            "s3:GetObjectRetention",
            "s3:GetObjectLegalHold"
            ],
        "Resource" : [
            "arn:aws:s3:::local-destination",
            "arn:aws:s3:::local-destination/*"
        ]
      }
    ]
}

Tech Dept

• If the schema turns out to be stable look to fiddling with the crawler settings
• If the schema turns out to be stable (or glue crawler become too expensive), look to implement a Lambda based solution to adding partitions to the 3 catalogs [SO inspiration]
• When the CUR Terraform Resource doesn't require a special us-east-1 provider JUST for global, You can remove that weirdness
• When AWS CUR updates and allow pointing to an external bucket, remove ALL the bucket creations for member accounts
• When AWS CUR updates (to allow more in the prefix regex) then change the prefix to something like account="value" for the proper Hive Style Partitioning

TODO

• Investigate static schema derived from the CUR SPEC

• Investigate cost performance / discoverability and maybe do the SNS->SQS thing for crawlers

• Enable Bucket, Replication and Athena Metrics

• Add prefix 'filter' to replication rules on the cur buckets (probably a good idea)

• Add lifecycle policies to the member cur buckets (should be 2 months minimum)

• Move to Terragrunt...

• ...if not, figure out how to do gomplate templating correctly across templates