EKS Deployment - Deployment Plan for EKS in WRDS VPC TI/UAT/Prod

[DrixTabligan-NOAA]

Fernando has given the go ahead to create a plan to deploy EKS across the three environments for use by the HydroVIS Team and its applications. This ticket is to track the progress of the plan creation and the actual plan itself.

EKS Blueprint Development and Workspace Setup Checklist

Initial Repository Setup

• Create Git repository

  https://gitlab.nws.noaa.gov/hydrovis/visual-processing-pipeline-vpp/hydrovis-eks-deployment.git

• Set up branch protection rules
• Configure repository structure

  .
  ├── environments/
  │   ├── ti/
  │   ├── uat/
  │   └── prod/
  ├── modules/
  ├── addons/
  └── terraform/
  

• Create .gitignore file
• Set up terraform backend configuration

Infrastructure as Code Development

Base Configuration

• Create provider configurations
  • AWS provider
  • Kubernetes provider
  • Helm provider
• Set up remote state configuration
• Create workspace configuration

  terraform workspace new ti_us-east-1
  terraform workspace new uat_us-east-1
  terraform workspace new prod_us-east-1
  terraform workspace new prod_us-east-2

VPC and Networking Module

• Define VPC configuration
  • CIDR blocks for each environment
  • Subnet configurations
  • NAT gateway setup
  • VPC endpoints
• Configure routing tables
  • Public subnets
  • Private subnets
  • Transit gateway (if needed)
• Security groups
  • Cluster security group
  • Node group security group
  • Additional service security groups

EKS Blueprint Core Configuration

• Define cluster configuration
  • Cluster version
  • Control plane logging
  • KMS encryption
  • Network policies
• Configure node groups
  • Instance types
  • Auto-scaling settings
  • Taints and labels
• IAM configurations
  • Cluster role
  • Node role
  • Service accounts

Add-ons Configuration Checklists

Core Add-ons

• VPC CNI

  • Version specification
  • Custom networking settings
  • Secondary IP configuration
  • Network policy settings

• CoreDNS

  • Version configuration
  • Custom DNS settings
  • Cache configuration
  • Metrics enablement

• kube-proxy

  • Version specification
  • Proxy mode configuration
  • IPVS settings (if used)

Monitoring and Logging

• AWS Distro for OpenTelemetry (ADOT)

  • Collector configuration
  • Sampling rules
  • Export destinations
  • Resource attributes

• CloudWatch Container Insights

  • Metrics collection
  • Log group configuration
  • Performance monitoring
  • Custom metrics

• Prometheus

  • Storage configuration
  • Retention settings
  • Alert rules
  • Service monitors
  • Recording rules

• Grafana

  • Datasource configuration
  • Dashboard provisioning
  • Alert configuration
  • User authentication

Networking Add-ons

• AWS Load Balancer Controller

  • SSL certificate configuration
  • Subnet tagging
  • Shield configuration
  • WAF integration

• ExternalDNS

  • Route53 configuration
  • DNS record settings
  • Policy configuration
  • Sync settings

• Nginx Ingress Controller

  • SSL configuration
  • Default backend
  • Custom headers
  • Rate limiting

Security Add-ons

• AWS Security Groups for Pods

  • Security group configuration
  • Pod networking policy
  • ENI configuration

• Cert Manager

  • Certificate issuers
  • ACME configuration
  • DNS validation
  • Certificate rotation

• AWS Private CA

  • CA configuration
  • Certificate templates
  • Validity periods
  • Key usage settings

Storage Add-ons

• AWS EBS CSI Driver

  • Volume configuration
  • Snapshot settings
  • Encryption configuration
  • IAM roles

• AWS EFS CSI Driver

  • File system configuration
  • Access points
  • Backup settings
  • Mount options

Development Tools

• ArgoCD

  • Repository configuration
  • Application sets
  • Sync policies
  • RBAC settings

• AWS App Mesh

  • Mesh configuration
  • Virtual nodes
  • Virtual services
  • Traffic routing

Cost Management

• Karpenter

  • Provisioner configuration
  • Node templates
  • Instance types
  • Scaling settings

• Kubecost

  • Cost allocation
  • Alert configuration
  • Savings recommendations
  • Report configuration

Environment-Specific Variables

• Create variable files (possibly utilize the sensitive repo for variable storage so we might skip this)

  terraform.tfvars.ti_us-east-1
  terraform.tfvars.uat_us-east-1
  terraform.tfvars.prod_us-east-1
  terraform.tfvars.prod_us-east-2
  

• Define environment-specific values
  • Node counts
  • Instance types
  • CIDR ranges
  • Add-on configurations

Documentation

• Create README files
• Document variables
• Create architecture diagrams
• Document deployment procedures
• Create troubleshooting guides

Testing Setup

• Create test cases
• Set up terraform validate in CI
• Configure pre-commit hooks
• Set up integration tests

Would you like me to:

1. Add more detail to any specific addon configuration?
2. Include specific configuration examples?
3. Add more security-related checks?
4. Include environment-specific addon variations?

EKS Multi-Environment/Region Deployment Checklist

TI Environment (us-east-1)

Deployment Steps

• Switch to TI workspace

  terraform workspace select ti_us-east-1

• Initialize Terraform

  terraform init

• Apply TI configuration

  terraform apply -var-file="terraform.tfvars.ti_us-east-1"

• Configure kubectl

  aws eks update-kubeconfig --name eks-blueprint-ti --region us-east-1

• Verify cluster

  kubectl get nodes
  kubectl get pods -A

• Check add-ons status

  kubectl get pods -n kube-system
  kubectl get pods -n monitoring

UAT Environment (us-east-1)

Deployment Steps

• Switch to UAT workspace

  terraform workspace select uat_us-east-1

• Review UAT plan

  terraform plan -var-file="terraform.tfvars.uat_us-east-1"

• Apply UAT configuration

  terraform apply -var-file="terraform.tfvars.uat_us-east-1"

• Configure kubectl

  aws eks update-kubeconfig --name eks-blueprint-uat --region us-east-1

• Verify UAT cluster

  kubectl get nodes
  kubectl get pods -A

• Run integration tests
• Verify monitoring setup

Production Environment (us-east-1)

Pre-Deployment Checks

• Review us-east-1 production configuration
• Verify production VPC and subnet configurations
• Check service quotas for production scale
• Review security group configurations
• Verify backup configurations

Deployment Steps

• Switch to us-east-1 production workspace

  terraform workspace select prod_us-east-1

• Review production plan

  terraform plan -var-file="terraform.tfvars.prod_us-east-1"

• Schedule maintenance window
• Apply production configuration

  terraform apply -var-file="terraform.tfvars.prod_us-east-1"

• Configure kubectl

  aws eks update-kubeconfig --name eks-blueprint-prod --region us-east-1

• Verify production cluster

  kubectl get nodes
  kubectl get pods -A

Production Environment (us-east-2)

Pre-Deployment Checks

• Review us-east-2 production configuration
• Verify cross-region networking setup
• Check service quotas in us-east-2
• Review security group configurations
• Verify backup and DR configurations
• Configure cross-region IAM roles

Deployment Steps

• Switch to us-east-2 production workspace

  terraform workspace select prod_us-east-2

• Review production plan

  terraform plan -var-file="terraform.tfvars.prod_us-east-2"

• Schedule maintenance window
• Apply production configuration

  terraform apply -var-file="terraform.tfvars.prod_us-east-2"

• Configure kubectl

  aws eks update-kubeconfig --name eks-blueprint-prod --region us-east-2

• Verify production cluster

  kubectl get nodes
  kubectl get pods -A

Production Cross-Region Verification

• Test cross-region networking
• Verify DNS setup across regions
• Test cross-region service discovery
• Verify backup replication
• Test disaster recovery procedures
• Validate global load balancing
• Check cross-region monitoring
• Verify alerting for both regions

Environment-Specific Post-Deployment Verification

TI (us-east-1)

• Verify CI/CD pipeline integration
• Check developer access
• Test deployment workflows

UAT (us-east-1)

• Run full integration test suite
• Verify test data configuration
• Test scaling configurations
• Validate monitoring alerts

Production (Both Regions)

• Verify high availability setup
• Check cluster autoscaling
• Test load balancer failover
• Verify backup schedules
• Test alerting system
• Run security scans
• Validate compliance requirements
• Test cross-region failover
• Verify global DNS routing
• Test regional service endpoints

Common Post-Deployment Tasks (All Environments)

• Tag all resources correctly
  • Environment tag
  • Region tag
  • Cost center tag
• Verify cost allocation tags
• Check logging configuration
• Verify metric collection
• Test alert notifications
• Document environment-specific configurations
• Update access documentation
• Review resource quotas