Skip to content

Commit

Permalink
GROW-662: Capture K8 related sub-counts for AWS (#9)
Browse files Browse the repository at this point in the history
  • Loading branch information
laverion authored Apr 3, 2024
1 parent 48557cb commit 3b3824b
Show file tree
Hide file tree
Showing 5 changed files with 219 additions and 10 deletions.
36 changes: 32 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Cloud Resource Counter (v0.7.0) running with:
Activity
* Retrieving Account ID...OK (240520192079)
* Retrieving EC2 counts...................OK (5)
* Retrieving EC2 K8 related VMs Sub-instance counts...................OK (1)
* Retrieving Spot instance counts...................OK (4)
* Retrieving EBS volume counts...................OK (9)
* Retrieving Unique container counts...................OK (3)
Expand All @@ -151,7 +152,7 @@ As you can see above, no command line arguments were necessary: it used my defau
Here is what the CSV file looks like. It is important to mention that this tool was run TWICE to collect the results of two different accounts/profiles.

```csv
Account ID,Timestamp,Region,# of EC2 Instances,# of Spot Instances,# of EBS Volumes,# of Unique Containers,# of Lambda Functions,# of RDS Instances,# of Lightsail Instances,# of S3 Buckets,# of EKS Nodes
Account ID,Timestamp,Region,# of EC2 Instances,# of EC2 K8 related VMs Sub-instances,# of Spot Instances,# of EBS Volumes,# of Unique Containers,# of Lambda Functions,# of RDS Instances,# of Lightsail Instances,# of S3 Buckets,# of EKS Nodes
896149672290,2020-10-20T16:29:39-04:00,ALL_REGIONS,2,3,7,3,2,3,2,2,2
240520192079,2020-10-21T16:24:06-04:00,ALL_REGIONS,5,4,9,3,12,7,0,13,0
```
Expand Down Expand Up @@ -267,9 +268,10 @@ The `aws-resource-counter` examines the following resources:
1. **EC2**. We count the number of EC2 **running** instances (both "normal" and Spot instances) across all regions.
* For EC2 instances, we only count those _without_ an Instance Lifecycle tag (which is either `spot` or `scheduled`).
* For EC2 K8 related VMs sub-instances, we only count those with a tag of `aws:eks:cluster-name`.
* For Spot instance, we only count those with an Instance Lifecycle tag of `spot`.
* This is stored in the generated CSV file under the "# of EC2 Instances" and "# of Spot Instances" columns.
* This is stored in the generated CSV file under the "# of EC2 Instances", "# of EC2 K8 related VMs Sub-instances", and "# of Spot Instances" columns.
1. **EBS Volumes.** We count the number of "attached" EBS volumes across all regions.
Expand Down Expand Up @@ -348,7 +350,7 @@ To collect the total number of EC2 instances across all regions, we will need to
```bash
$ aws ec2 describe-regions $aws_p \
--filters Name=opt-in-status,Values=opt-in-not-required,opted-in \
--region us-east-1 --output text --query Regions[].RegionName
--region us-east-1 --output text --query 'Regions[].RegionName'
eu-north-1 ap-south-1 eu-west-3 ...
```
Expand All @@ -364,7 +366,7 @@ We will be using the results of this command to "iterate" over all regions. To m
```bash
$ ec2_r=$(aws ec2 describe-regions $aws_p \
--filters Name=opt-in-status,Values=opt-in-not-required,opted-in \
--region us-east-1 --output text --query Regions[].RegionName )
--region us-east-1 --output text --query 'Regions[].RegionName' )
```
You can show the list of regions for your account by using the `echo` command:
Expand Down Expand Up @@ -417,6 +419,32 @@ The second and third lines are our call to `describe-instances` (as shown above)
In the fourth line, we paste all of the values into a long addition and use `bc` to sum the values.
#### EC2 K8 Related VMs Subcount Instances
Here is the command to count the number of _EC K8 related VMs subcount_ instances for a given region:
```bash
$ aws ec2 describe-instances $aws_p --no-paginate --region us-east-1 \
--filters Name=instance-state-name,Values=running \
--filters Name=tag-key,Values='aws:eks:cluster-name' \
--query 'length(Reservations[].Instances[?!not_null(InstanceLifecycle)].InstanceId[])'
1
```
This command is similar to the normal EC2 query, but now explicitly checks for EC2 instances whose `Tags` have that key `aws:eks:cluster-name`.
We will need to run this command over all regions. Here is what it looks like:
```bash
$ for reg in $ec2_r; do \
aws ec2 describe-instances $aws_p --no-paginate --region $reg \
--filters Name=instance-state-name,Values=running \
--filters Name=tag-key,Values='aws:eks:cluster-name' \
--query 'length(Reservations[].Instances[?!not_null(InstanceLifecycle)].InstanceId[])' ; \
done | paste -s -d+ - | bc
5
```
#### Spot Instances
Here is the command to count the number of _Spot_ instances for a given region:
Expand Down
85 changes: 85 additions & 0 deletions ec2_k8_subcount.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
/******************************************************************************
Cloud Resource Counter
File: spot.go
Summary: Provides a count of all Spot EC2 instances.
******************************************************************************/

package main

import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
color "github.com/logrusorgru/aurora"
)

// SpotInstances retrieves the count of all EC2 spot instances
// either for all regions (allRegions is true) or the region
// associated with the session.
// This method gives status back to the user via the supplied
// ActivityMonitor instance.
func EC2K8SubInstances(sf ServiceFactory, am ActivityMonitor, allRegions bool) int {
// Indicate activity
am.StartAction("Retrieving EC2 K8 related VMs Sub-instance counts")

// Should we get the counts for all regions?
instanceCount := 0
if allRegions {
// Get the list of all enabled regions for this account
regionsSlice := GetEC2Regions(sf.GetEC2InstanceService(""), am)

// Loop through all of the regions
for _, regionName := range regionsSlice {
// Get the EC2 counts for a specific region
instanceCount += ec2K8SubInstancesForSingleRegion(sf.GetEC2InstanceService(regionName), am)
}
} else {
// Get the EC2 counts for the region selected by this session
instanceCount = ec2K8SubInstancesForSingleRegion(sf.GetEC2InstanceService(""), am)
}

// Indicate end of activity
am.EndAction("OK (%d)", color.Bold(instanceCount))

return instanceCount
}

func ec2K8SubInstancesForSingleRegion(ec2is *EC2InstanceService, am ActivityMonitor) int {
// Indicate activity
am.Message(".")

// Construct our input to find ONLY RUNNING EC2 instances that also have the "aws:eks:cluster-name" tag
input := &ec2.DescribeInstancesInput{
Filters: []*ec2.Filter{
{
Name: aws.String("tag-key"),
Values: []*string{
aws.String("aws:eks:cluster-name"),
},
},
{
Name: aws.String("instance-state-name"),
Values: []*string{
aws.String("running"),
},
},
},
}

// Invoke our service
instanceCount := 0
err := ec2is.InspectInstances(input, func(dio *ec2.DescribeInstancesOutput, lastPage bool) bool {
// Loop through each reservation
for _, reservation := range dio.Reservations {
// We assume that the AWS Service has properly filtered the list of returned instances
instanceCount += len(reservation.Instances)
}

return true
})

// Check for error
am.CheckError(err)

return instanceCount
}
74 changes: 74 additions & 0 deletions ec2_k8_subcount_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
/******************************************************************************
Cloud Resource Counter
File: ec2_k8_subcount_test.go
Summary: The Unit Test for EC2 K8 sub-count.
******************************************************************************/

package main

import (
"testing"

"github.com/expel-io/aws-resource-counter/mock"
)

// =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
// Unit Test for EC2 K8 sub-count.
// =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

func TestEC2K8SubInstances(t *testing.T) {
// Describe all of our test cases: 1 failure and 4 success cases
cases := []struct {
RegionName string
AllRegions bool
ExpectedCount int
ExpectError bool
}{
{
RegionName: "us-east-1",
ExpectedCount: 1,
}, {
RegionName: "us-east-2",
ExpectedCount: 0,
}, {
RegionName: "af-south-1",
ExpectedCount: 0,
}, {
RegionName: "undefined-region",
ExpectError: true,
}, {
AllRegions: true,
ExpectedCount: 1,
},
}

// Loop through each test case
for _, c := range cases {
// Create our fake service factory
sf := fakeEC2ServiceFactory{
RegionName: c.RegionName,
DRResponse: ec2Regions,
}

// Create a mock activity monitor
mon := &mock.ActivityMonitorImpl{}

// Invoke our EC K8 Subcount Instances function
actualCount := EC2K8SubInstances(sf, mon, c.AllRegions)

// Did we expect an error?
if c.ExpectError {
// Did it fail to arrive?
if !mon.ErrorOccured {
t.Error("Expected an error to occur, but it did not... :^(")
}
} else if mon.ErrorOccured {
t.Errorf("Unexpected error occurred: %s", mon.ErrorMessage)
} else if actualCount != c.ExpectedCount {
t.Errorf("Error: EC K8 SubcountInstances returned %d; expected %d", actualCount, c.ExpectedCount)
} else if mon.ProgramExited {
t.Errorf("Unexpected Exit: The program unexpected exited with status code=%d", mon.ExitCode)
}
}
}
32 changes: 27 additions & 5 deletions ec2_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ package main
import (
"errors"
"reflect"
"slices"
"strings"
"testing"

Expand Down Expand Up @@ -50,7 +51,7 @@ var ec2Regions *ec2.DescribeRegionsOutput = &ec2.DescribeRegionsOutput{
// This is our map of regions and the instances in each
var ec2InstancesPerRegion = map[string][]*ec2.DescribeInstancesOutput{
// US-EAST-1 illustrates a case where DescribeInstancesPages returns two pages of results.
// First page: 2 different reservations (1 running instance, then 2 instances [1 is spot])
// First page: 2 different reservations (1 running instance, then 3 instances [1 is k8 related vm, 1 is a spot instance])
// Second page: 1 reservation (2 instances, 1 of which is stopped)
"us-east-1": {
&ec2.DescribeInstancesOutput{
Expand All @@ -71,6 +72,14 @@ var ec2InstancesPerRegion = map[string][]*ec2.DescribeInstancesOutput{
Name: aws.String("running"),
},
},
{
Tags: []*ec2.Tag{
{Key: aws.String("aws:eks:cluster-name"), Value: aws.String("cluster-name")},
},
State: &ec2.InstanceState{
Name: aws.String("running"),
},
},
{
InstanceLifecycle: aws.String("spot"),
State: &ec2.InstanceState{
Expand Down Expand Up @@ -276,13 +285,26 @@ func instanceSatisfiesFilter(reflectStruct reflect.Value, filter *ec2.Filter) bo

// Get our field value from the path
fieldValue, ok := resolvePathByReflection(reflectStruct, fieldNamePath)
if !ok {
if !ok && !slices.Contains(fieldNamePath, "TagKey") {
return false
}

// Does this match one of the filter values?
for _, value := range filter.Values {
// Does it match?
// if we get a filter for "tag-key", check the "Tags" portion of an ec2 instance
// loop through the tags slice checking each of the "Kay" values in the tags struct
// return true if we find any matching values
if slices.Contains(fieldNamePath, "TagKey") {
tags := reflectStruct.FieldByName("Tags")
if !tags.IsValid() || tags.IsNil() || tags.Kind() != reflect.Slice {
return false
}
for i := 0; i < tags.Len(); i++ {
if tagKey := tags.Index(i).Elem().FieldByName("Key").Elem(); tagKey.IsValid() && tagKey.Kind() == reflect.String && tagKey.String() == *value {
return true
}
}
}
if *value == fieldValue {
return true
}
Expand Down Expand Up @@ -455,7 +477,7 @@ func TestEC2Counts(t *testing.T) {
}{
{
RegionName: "us-east-1",
ExpectedCount: 3,
ExpectedCount: 4,
}, {
RegionName: "us-east-2",
ExpectedCount: 5,
Expand All @@ -467,7 +489,7 @@ func TestEC2Counts(t *testing.T) {
ExpectError: true,
}, {
AllRegions: true,
ExpectedCount: 8,
ExpectedCount: 9,
},
}

Expand Down
2 changes: 1 addition & 1 deletion main.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ var date string = "<<never built>>"
//
// This command requires access to a valid AWS Account. For now, it is assumed that
// this is stored in the user's ".aws" folder (located in $HOME/.aws).
//
func main() {
/* =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
* Command line processing
Expand Down Expand Up @@ -92,6 +91,7 @@ func main() {
results.Append("Timestamp", time.Now().Format(time.RFC3339))
results.Append("Region", displayRegion)
results.Append("# of EC2 Instances", EC2Counts(serviceFactory, monitor, settings.allRegions))
results.Append("# of EC2 K8 related VMs Sub-instances", EC2K8SubInstances(serviceFactory, monitor, settings.allRegions))
results.Append("# of Spot Instances", SpotInstances(serviceFactory, monitor, settings.allRegions))
results.Append("# of EBS Volumes", EBSVolumes(serviceFactory, monitor, settings.allRegions))
results.Append("# of Unique Containers", UniqueContainerImages(serviceFactory, monitor, settings.allRegions))
Expand Down

0 comments on commit 3b3824b

Please sign in to comment.