Oreilly main rebase [DRAFT]

[corybekk]

Description

I wanted to use our original oreilly/aws-nuke fork, but found that we would lack write permissions on ekristen/aws-nuke because we aren't forked from it. I created this fork aws-nuke-v2 which will eventually be renamed to aws-nuke once we replace the old one.

I attempted to rebase oreilly/aws-nuke onto ekristen/aws-nuke (see below steps) with a few goals in mind.

• maintain our commit history.
• maintain any customizations we had in oreilly-main that we were excluding from upstream.

I am not sure if this is the best approach. I have reached out to the team to discuss our path forward. Here is the slack thread.

EDIT: We have decided to just start from scratch instead of rebasing. My findings show that a lot our modules were rewritten to follow a new structure and, as a result, most, if not all, of our changes were stomped on.

Git steps

Step 1: Create a carbon copy of the current main branch in aws-nuke-v2

• git checkout -b oreilly-main

Step 2: Add the remote repository for the old fork

• git remote add old-fork https://github.com/oreillymedia/aws-nuke.git
• git fetch old-fork

Step 3: Rebase changes from oreillymedia/aws-nuke's oreilly-main branch onto aws-nuke-v2:oreilly-main
Use the -Xours strategy to prioritize aws-nuke-v2:oreilly-main changes in conflicts

• git rebase -Xours old-fork/oreilly-main

Step4: Manually fix any merge issues, delete old files, fix Go linting and static analysis errors.

Required Code Changes (not an extensive list)

• Our modules don't conform to the new module structure. They need to be updated. I have updated CL-1069 to account for this.

Required config.yml changes

• We need to update config.yml features flags to use settings.
• We need to Update config.yml resource types to use includes
• We need to add a blocklist to prevent the below errors on each run:

time="2024-11-05T23:42:49Z" level=error msg="Listing OpsWorksUserProfile failed:\n    AccessDeniedException: AWS OpsWorks Stacks is no longer accepting new customers\n    \tstatus code: 400, request id: 39c087b6-a9a6-4e9a-8609-4341eba19fb0" error="AccessDeniedException: AWS OpsWorks Stacks is no longer accepting new customers\n\tstatus code: 400, request id: 39c087b6-a9a6-4e9a-8609-4341eba19fb0"
time="2024-11-05T23:42:50Z" level=info msg="TagOptionNotMigratedException: TagOption Migration not complete"
time="2024-11-05T23:42:50Z" level=error msg="Listing OpsWorksApp failed:\n    AccessDeniedException: AWS OpsWorks Stacks is no longer accepting new customers\n    \tstatus code: 400, request id: e686baa6-13ba-46bc-b567-f5b6d9d842df" error="AccessDeniedException: AWS OpsWorks Stacks is no longer accepting new customers\n\tstatus code: 400, request id: e686baa6-13ba-46bc-b567-f5b6d9d842df"

• We need to update config.yaml account-blocklist to blocklist
• We need to Update control-tower to alternatives
• We need to add a cloudtrail control tower filter to config.yml

 CloudTrailTrail:
      - type: glob
        value: aws-controltower-*

issues

• Control-tower resources are outputting these errors:

time="2024-11-06T00:41:43Z" level=error msg="lister for resource type not found: AWS::Bedrock::Agent"
time="2024-11-06T00:41:43Z" level=error msg="lister for resource type not found: AWS::Bedrock::Flow"
time="2024-11-06T00:41:43Z" level=error msg="lister for resource type not found: AWS::Bedrock::Guardrail"
time="2024-11-06T00:41:43Z" level=error msg="lister for resource type not found: AWS::Bedrock::Prompt"

CLI changes

• We must use "run" to execute aws nuke, i.e run aws-nuke
• Instead --force and --force-sleep we will now use --no-prompt and --prompt-delay

Testing

• I tests this fork locally with the below setup.
• Make sure your aws-nuke devcontainer image is using Go 1.21 or later
• Make sure your env vars have the necessary AWS keys
• If you want to debug the aws-nuke service, you can use my launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [ 
        {
        "name": "Run Service",
        "type": "go",
        "request": "launch",
        "program": "main.go",
        "args": ["run", "--config", "config.yml", "--no-prompt", "--prompt-delay", "3", "--no-dry-run"],
        "envFile": "${workspaceFolder}/.env"
      },
    ]
}

• You can use the below config.yaml. This config includes all the updates mentioned above. We can use this config.yml as a template for updating cl-cloud-user-creator.

blocklist:
- '993122482293'
- '495548951594'
- 068377897844
accounts:
  '299719426765':
    filters:
      AWS::ECS::CapacityProvider:
      - FARGATE
      - FARGATE_SPOT
      CloudFormationStack:
      - type: glob
        value: StackSet-AWSControlTowerBP-*
      CloudTrailTrail:
      - type: glob
        value: aws-controltower-*
      CloudWatchEventsRule:
      - type: contains
        value: ConfigComplianceChangeEventRule
      CloudWatchEventsTarget:
      - type: contains
        value: Compliance-Change-Topic
      CloudWatchLogsLogGroup: !!python/tuple
      - /aws/lambda/aws-controltower-NotificationForwarder
      IAMPolicy:
      - property: tag:owner
        value: oreilly-cloudlabs
      IAMRole:
      - aws-controltower-AdministratorExecutionRole
      - aws-controltower-ConfigRecorderRole
      - aws-controltower-ForwardSnsNotificationRole
      - aws-controltower-ReadOnlyExecutionRole
      - AWSControlTowerExecution
      - AWSControlTower_VPCFlowLogsRole
      - property: tag:owner
        value: oreilly-cloudlabs
      - type: glob
        value: AWSReservedSSO_*
      - type: regex
        value: .*cdk-.*-cfn-exec-role.*
      - type: regex
        value: .*cdk-.*-file-publishing-role.*
      - type: regex
        value: .*cdk-.*-image-publishing-role.*
      - type: regex
        value: .*cdk-.*-lookup-role.*
      IAMRolePolicy:
      - property: tag:role:owner
        value: oreilly-cloudlabs
      - property: role:RoleName
        value: aws-controltower-ForwardSnsNotificationRole
      IAMRolePolicyAttachment:
      - property: tag:role:owner
        value: oreilly-cloudlabs
      - property: RoleName
        value: aws-controltower-AdministratorExecutionRole
      - property: RoleName
        value: aws-controltower-ConfigRecorderRole
      - property: RoleName
        value: aws-controltower-ForwardSnsNotificationRole
      - property: RoleName
        value: aws-controltower-ReadOnlyExecutionRole
      - property: RoleName
        value: AWSControlTowerExecution
      - property: RoleName
        value: AWSControlTower_VPCFlowLogsRole
      - property: RoleName
        type: glob
        value: AWSReservedSSO_*
      - property: RoleName
        type: regex
        value: .*cdk-.*-cfn-exec-role.*
      - property: RoleName
        type: regex
        value: .*cdk-.*-file-publishing-role.*
      - property: RoleName
        type: regex
        value: .*cdk-.*-image-publishing-role.*
      - property: RoleName
        type: regex
        value: .*cdk-.*-lookup-role.*
      IAMSAMLProvider:
      - type: glob
        value: arn:aws:iam::*:saml-provider/AWSSSO_*_DO_NOT_DELETE
      LambdaFunction:
      - aws-controltower-NotificationForwarder
      SNSEndpoint:
      - type: contains
        value: aws-controltower-SecurityNotifications
      SNSPlatformApplication:
      - type: contains
        value: aws-controltower-SecurityNotifications
      SNSSubscription:
      - type: contains
        value: aws-controltower-SecurityNotifications
      SNSTopic:
      - type: contains
        value: aws-controltower-SecurityNotifications
settings:
  CloudformationStack:
    DisableDeletionProtection: true
  EC2Instance:
    DisableDeletionProtection: true
  ELBv2:
    DisableDeletionProtection: true
  RDSInstance:
    DisableDeletionProtection: true
regions:
- us-west-2
- us-east-1
- global
resource-types:
  excludes:
    - S3Object # Excluded because S3 bucket removal handles removing all S3Objects
    - ServiceCatalogTagOption # Excluded due to https://github.com/rebuy-de/aws-nuke/issues/515
    - ServiceCatalogTagOptionPortfolioAttachment # Excluded due to https://github.com/rebuy-de/aws-nuke/issues/515
    - FMSNotificationChannel # Excluded because it's not available
    - FMSPolicy # Excluded because it's not available
    - MachineLearningMLModel # Excluded due to ML being unavailable
    - MachineLearningDataSource # Excluded due to ML being unavailable
    - MachineLearningBranchPrediction # Excluded due to ML being unavailable
    - MachineLearningEvaluation # Excluded due to ML being unavailable
    - RoboMakerDeploymentJob # Deprecated Service
    - RoboMakerFleet # Deprecated Service
    - RoboMakerRobot # Deprecated Service
    - RoboMakerSimulationJob
    - RoboMakerRobotApplication
    - RoboMakerSimulationApplication
    - OpsWorksApp # Deprecated service
    - OpsWorksInstance # Deprecated service
    - OpsWorksLayer # Deprecated service
    - OpsWorksUserProfile # Deprecated service
    - OpsWorksCMBackup # Deprecated service
    - OpsWorksCMServer # Deprecated service
    - OpsWorksCMServerState # Deprecated service
    - CodeStarProject # Deprecated service
    - CodeStarConnection # Deprecated service
    - CodeStarNotification # Deprecated service
    - Cloud9Environment # Deprecated service
    - CloudSearchDomain # Deprecated service
    - RedshiftServerlessSnapshot # Deprecated service
    - RedshiftServerlessNamespace # Deprecated service
    - RedshiftServerlessWorkgroup # Deprecated service
  alternatives:
   - AWS::Bedrock::Agent
   - AWS::Bedrock::Flow
   - AWS::Bedrock::Guardrail
   - AWS::Bedrock::Prompt
   - AWS::CloudWatch::MetricStream
   - AWS::Comprehend::Flywheel
   - AWS::CodeGuruProfiler::ProfilingGroup
   - AWS::DynamoDB::GlobalTable
   - AWS::EC2::NetworkInsightsAccessScope
   - AWS::EC2::NetworkInsightsAccessScopeAnalysis
   - AWS::EC2::NetworkInsightsAnalysis
   - AWS::EC2::NetworkInsightsPath
   - AWS::ECS::CapacityProvider
   - AWS::EC2::PrefixList
   - AWS::ECR::PublicRepository
   - AWS::ECR::PullThroughCacheRule
   - AWS::ECR::RegistryPolicy
   - AWS::ECR::ReplicationConfiguration
   - AWS::EFS::AccessPoint
   - AWS::ElastiCache::ServerlessCache
   - AWS::ElasticBeanstalk::ApplicationVersion
   - AWS::ElasticBeanstalk::ConfigurationTemplate
   - AWS::Events::ApiDestination
   - AWS::Events::Archive
   - AWS::Events::Connection
   - AWS::Events::Endpoint
   - AWS::FIS::ExperimentTemplate
   - AWS::Glue::Registry
   - AWS::Glue::Schema
   - AWS::Lightsail::Alarm
   - AWS::Lightsail::Bucket
   - AWS::Lightsail::Certificate
   - AWS::Lightsail::Container
   - AWS::Lightsail::Database
   - AWS::Lightsail::Distribution
   - AWS::Logs::QueryDefinition
   - AWS::Rekognition::StreamProcessor
   - AWS::Route53::CidrCollection
   - AWS::Route53::DNSSEC
   - AWS::Route53::KeySigningKey
   - AWS::Route53Resolver::ResolverConfig
   - AWS::Route53Resolver::ResolverDNSSECConfig
   - AWS::Route53Resolver::ResolverQueryLoggingConfig
   - AWS::S3::MultiRegionAccessPoint
   - AWS::S3::StorageLens
   - AWS::SES::ContactList
   - AWS::SES::DedicatedIpPool
   - AWS::Signer::ProfilePermission
   - AWS::Signer::SigningProfile
   - AWS::StepFunctions::Activity
   - AWS::XRay::ResourcePolicy
  includes:
   - AccessAnalyzer
   - ACMCertificate
   - APIGatewayAPIKey
   - APIGatewayClientCertificate
   - APIGatewayDomainName
   - APIGatewayRestAPI
   - APIGatewayUsagePlan
   - APIGatewayV2API
   - APIGatewayV2VpcLink
   - APIGatewayVpcLink
   - ApplicationAutoScalingScalableTarget
   - AthenaDataCatalog
   - AthenaNamedQuery
   - AthenaPreparedStatement
   - AthenaWorkGroup
   - AutoScalingGroup
   - AWS::Backup::Framework
   - AWSBackupPlan
   - AWSBackupRecoveryPoint
   - AWSBackupSelection
   - AWSBackupVault
   - AWSBackupVaultAccessPolicy
   - BedrockAgentAlias
   - BedrockFlowAlias
   - CloudFormationStack
   - CloudFormationStackSet
   - CloudFormationType
   - CloudTrailTrail
   - CloudWatchAlarm
   - CloudWatchAnomalyDetector
   - CloudWatchDashboard
   - CloudWatchEventsBuses
   - CloudWatchEventsRule
   - CloudWatchEventsTarget
   - CloudWatchInsightRule
   - CloudWatchLogsDestination
   - CloudWatchLogsLogGroup
   - CloudWatchLogsResourcePolicy
   - CodeArtifactDomain
   - CodeArtifactRepository
   - CodeBuildBuild
   - CodeBuildBuildBatch
   - CodeBuildProject
   - CodeBuildReportGroup
   - CodeBuildSourceCredential
   - CodeCommitRepository
   - CodeDeployApplication
   - CodeDeployDeploymentConfig
   - CodeDeployDeploymentGroup
   - CodeGuruReviewerRepositoryAssociation
   - CodePipelineCustomActionType
   - CodePipelinePipeline
   - CodePipelineWebhook
   - CognitoIdentityPool
   - CognitoIdentityProvider
   - CognitoUserPool
   - CognitoUserPoolClient
   - CognitoUserPoolDomain
   - ComprehendDocumentClassifier
   - ComprehendDominantLanguageDetectionJob
   - ComprehendEndpoint
   - ComprehendEntitiesDetectionJob
   - ComprehendEntityRecognizer
   - ComprehendEventsDetectionJob
   - ComprehendKeyPhrasesDetectionJob
   - ComprehendPiiEntititesDetectionJob
   - ComprehendSentimentDetectionJob
   - ComprehendTargetedSentimentDetectionJob
   - DAXCluster
   - DAXParameterGroup
   - DAXSubnetGroup
   - DynamoDBBackup
   - DynamoDBTable
   - DynamoDBTableItem
   - EC2Address
   - EC2ClientVpnEndpoint
   - EC2ClientVpnEndpointAttachment
   - EC2CustomerGateway
   - EC2DHCPOption
   - EC2EgressOnlyInternetGateway
   - EC2Image
   - EC2Instance
   - EC2InstanceConnectEndpoint
   - EC2InternetGateway
   - EC2InternetGatewayAttachment
   - EC2KeyPair
   - EC2LaunchTemplate
   - EC2NATGateway
   - EC2NetworkACL
   - EC2NetworkInterface
   - EC2PlacementGroup
   - EC2RouteTable
   - EC2SecurityGroup
   - EC2Snapshot
   - EC2SpotFleetRequest
   - EC2Subnet
   - EC2TGW
   - EC2TGWAttachment
   - EC2Volume
   - EC2VPC
   - EC2VPCEndpoint
   - EC2VPCEndpointServiceConfiguration
   - EC2VPCPeeringConnection
   - EC2VPNConnection
   - EC2VPNGateway
   - EC2VPNGatewayAttachment
   - ECRRepository
   - ECSCluster
   - ECSClusterInstance
   - ECSService
   - ECSTask
   - ECSTaskDefinition
   - EFSFileSystem
   - EFSMountTarget
   - EKSCluster
   - EKSFargateProfiles
   - EKSNodegroups
   - ElasticacheCacheParameterGroup
   - ElasticacheCacheCluster
   - ElasticacheReplicationGroup
   - ElasticacheSubnetGroup
   - ElasticacheUser
   - ElasticacheUserGroup
   - ElasticBeanstalkApplication
   - ElasticBeanstalkEnvironment
   - ElasticTranscoderPipeline
   - ElasticTranscoderPreset
   - ELB
   - ELBv2
   - ELBv2TargetGroup
   - ESDomain
   - FirehoseDeliveryStream
   - GlueBlueprint
   - GlueClassifier
   - GlueConnection
   - GlueCrawler
   - GlueDatabase
   - GlueJob
   - GlueMLTransform
   - GlueSecurityConfiguration
   - GlueSession
   - GlueTrigger
   - GlueWorkflow
   - IAMGroup
   - IAMGroupPolicy
   - IAMGroupPolicyAttachment
   - IAMInstanceProfile
   - IAMInstanceProfileRole
   - IAMLoginProfile
   - IAMOpenIDConnectProvider
   - IAMPolicy
   - IAMRole
   - IAMRolePolicy
   - IAMRolePolicyAttachment
   - IAMSAMLProvider
   - IAMServerCertificate
   - IAMServiceSpecificCredential
   - IAMSigningCertificate
   - IAMUser
   - IAMUserAccessKey
   - IAMUserGroupAttachment
   - IAMUserPolicy
   - IAMUserPolicyAttachment
   - IAMUserSSHPublicKey
   - IAMVirtualMFADevice
   - ImageBuilderComponent
   - ImageBuilderDistributionConfiguration
   - ImageBuilderImage
   - ImageBuilderInfrastructureConfiguration
   - ImageBuilderPipeline
   - ImageBuilderRecipe
   - KinesisAnalyticsApplication
   - KinesisSignalingChannels
   - KinesisVideoProject
   - KMSAlias
   - KMSKey
   - lambdaLayer
   - LambdaFunction
   - LambdaEventSourceMapping
   - LaunchConfiguration
   - LifecycleHook
   - LightsailDisk
   - LightsailDomain
   - LightsailKeyPair
   - LightsailInstance
   - LightsailLoadBalancer
   - LightsailStaticIP
   - MQBroker
   - OSDomain
   - OSPackage
   - OSVPCEndpoint
   - PollyLexicons
   - RekognitionCollection
   - RekognitionDataset
   - RekognitionProject
   - RDSDBCluster
   - RDSDBClusterParameterGroup
   - RDSClusterSnapshot
   - RDSInstance
   - RDSEventSubscription
   - RDSOptionGroup
   - RDSDBParameterGroup
   - RDSProxy
   - RDSSnapshot
   - RDSDBSubnetGroup
   - Route53HealthCheck
   - Route53HostedZone
   - Route53ResolverEndpoint
   - Route53ResolverRule
   - Route53ResourceRecordSet
   - Route53TrafficPolicy
   - S3AccessPoint
   - S3Bucket
   - S3MultipartUpload
   - S3Object
   - SecretsManagerSecret
   - ServiceDiscoveryInstance
   - ServiceDiscoveryNamespace
   - ServiceDiscoveryService
   - SESConfigurationSet
   - SESIdentity
   - SESReceiptFilter
   - SESReceiptRuleSet
   - SESTemplate
   - SFNStateMachine
   - SignerSigningJob
   - SNSEndpoint
   - SNSPlatformApplication
   - SNSSubscription
   - SNSTopic
   - SQSQueue
   - SSMActivation
   - SSMAssociation
   - SSMDocument
   - SSMMaintenanceWindow
   - SSMParameter
   - SSMPatchBaseline
   - SSMResourceDataSync
   - TranscribeCallAnalyticsCategory
   - TranscribeCallAnalyticsJob
   - TranscribeLanguageModel
   - TranscribeMedicalTranscriptionJob
   - TranscribeMedicalVocabulary
   - TranscribeTranscriptionJob
   - TranscribeVocabulary
   - TranscribeVocabularyFilter
   - XRayGroup
   - XRaySamplingRule
   - AWS::Bedrock::Agent
   - AWS::Bedrock::Flow
   - AWS::Bedrock::Guardrail
   - AWS::Bedrock::Prompt
   - AWS::CloudWatch::MetricStream
   - AWS::Comprehend::Flywheel
   - AWS::CodeGuruProfiler::ProfilingGroup
   - AWS::DynamoDB::GlobalTable
   - AWS::EC2::NetworkInsightsAccessScope
   - AWS::EC2::NetworkInsightsAccessScopeAnalysis
   - AWS::EC2::NetworkInsightsAnalysis
   - AWS::EC2::NetworkInsightsPath
   - AWS::ECS::CapacityProvider
   - AWS::EC2::PrefixList
   - AWS::ECR::PublicRepository
   - AWS::ECR::PullThroughCacheRule
   - AWS::ECR::RegistryPolicy
   - AWS::ECR::ReplicationConfiguration
   - AWS::EFS::AccessPoint
   - AWS::ElastiCache::ServerlessCache
   - AWS::ElasticBeanstalk::ApplicationVersion
   - AWS::ElasticBeanstalk::ConfigurationTemplate
   - AWS::Events::ApiDestination
   - AWS::Events::Archive
   - AWS::Events::Connection
   - AWS::Events::Endpoint
   - AWS::FIS::ExperimentTemplate
   - AWS::Glue::Registry
   - AWS::Glue::Schema
   - AWS::Lightsail::Alarm
   - AWS::Lightsail::Bucket
   - AWS::Lightsail::Certificate
   - AWS::Lightsail::Container
   - AWS::Lightsail::Database
   - AWS::Lightsail::Distribution
   - AWS::Logs::QueryDefinition
   - AWS::Rekognition::StreamProcessor
   - AWS::Route53::CidrCollection
   - AWS::Route53::DNSSEC
   - AWS::Route53::KeySigningKey
   - AWS::Route53Resolver::ResolverConfig
   - AWS::Route53Resolver::ResolverDNSSECConfig
   - AWS::Route53Resolver::ResolverQueryLoggingConfig
   - AWS::S3::MultiRegionAccessPoint
   - AWS::S3::StorageLens
   - AWS::SES::ContactList
   - AWS::SES::DedicatedIpPool
   - AWS::Signer::ProfilePermission
   - AWS::Signer::SigningProfile
   - AWS::StepFunctions::Activity
   - AWS::XRay::ResourcePolicy