Skip to content

Getting started

Jump to bottom
Peter Simms edited this page May 2, 2024 · 4 revisions

Starting from scratch

Creating your properties project:

Getting properties from on a service

To get your properties on an instance follow the below example

curl -v http://properties.resource.local:9100/v2/conqueso/YOUR_SERVICE_NAME/{optional.nested.property}/{optional.nested.property.2}

Step 1: Creating your config.yml

  1. Create your top level directory for your property project.

    mkdir example-properties

  2. Create a config directory and inside a config.yml file.

cd example-properties/
mkdir config
cd config
touch config.yml

The config.yml will define essential configurations that the generator needs to create the property json files. In the config.yml file we need three keys set to explain our project to the generator.

  1. The environments key. This key will define a array of the environments that we are setting properties for. Environments must be unique and have a one to one mapping with amazon aws regions.
  2. The accounts key. This key will define a array of amazon accounts properties will be uploaded to and served in.
  3. The environment_configs key. This key will define three things.
    • The one to one mapping or aws regions to environments.
    • The account a environment lives in.
    • Interpolations for a given environment. Interpolations will be explained in a separate section.

Here is a example config.yml

environments:
  - 'my-test-env1'
  - 'my-test-env2'

accounts:
  - 123456789012
  - 987654321098

environment_configs:
  my-test-env1:
    region: 'us-east-1'
    account: 123456789012
    interpolations:
      region: 'us-east-1'
      cloud: 'test-cloud-1'
      domain: 'my1.com'
  my-test-env2:
    region: 'eu-central-1'
    account: 987654321098
    interpolations:
      region: 'eu-central-1'
      cloud: 'test-cloud-2'
      domain: 'my2.com'

Step 2: Creating your globals

Globals are properties that get mixed in with service definitions. The hierarchy of the global definition sets the ruling for what that property can supersede. The globals supersedence order is as follows. Any Environmental globals will overwrite account globals. The resultant merge of environmental and account globals overwrite definitions in the top level globals.yml. In short Superscedence from least to greatest is globals.yml, account globals, environment globals. To define globals follow the steps below.

Top level globals
  1. Create a globals folder
cd example-properties/
mkdir globals
  1. If you would like top level globals then create a globals.yml in your globals folder.
cd globals/
touch globals.yml
  1. Define your yaml values in the globals.yml
Account globals
  1. Create a folder called the account id of your aws account.
  2. In the folder created above create an YAML file named after the account id.
  3. Define your account level globals in the YAML file you created above.
Environment Globals
  1. Create a folder called environments inside your folder named after your account.
  2. Inside the environments folder create a yaml file named after the environment you would like to define globals for. Only environments defined in your config are supported. The environment must also be mapped in the config to the account the sharing the same name as the folder the environment global yaml file lives in.
  3. In the newly created environment's yaml file you may define your globals.
Step 3: Creating your service definitions

Service definitions have the highest level of supersedence and will overwrite matching global definitions. To create you service definitions you need to create the services folder and then the services yaml files.

cd example-properties/
mkdir services
cd services
touch my-service-name.yml

Service definitions consist of three parts default, environments, and encrypted. Encrypted definitions overwrite environment definitions which will overwrite default definitions. Here is a example of a service file. The name of your service file MUST be the same as your service.

default:
  database.host: 'my.database.{domain}'
  database.port: 3306

environments:
  my-test-env-1:
    thread.pool.size: 12
  my-test-env-2:
    thread.pool.size: 8

encrypted:
  my-test-env-1:
    my.encrypted.property:
      $ssm:
        region: my_region
        encrypted: stuff
Adding interpolations

An interpolation is a value that will be dynamically substituted during generation with the correct value for the environment being generated. Interpolations are declared in the config for a given environment. Once declared an interpolation can be used in a property definition by referencing it in braces. If we were to reference the domain interpolation from the example config above we would use {domain}.

Adding an encrypted secret (using ckrt)
To add an encrypted secret you must first download ckrt.

  1. Navigate to https://github.com/rapid7/ckrt/releases

  2. wget/curl the latest release for your platfom. The project is pretty stable at this point, but you may want to check in once in awhile to get the latest features.

  3. chmod +x ckrt

  4. Move ckrt to somewhere in your path or just use ./ckrt

  5. Running the command without any arguments gives you the available subcommands:

    ckrt - A tool for managing encrypted SSM properties

Usage:
  ckrt [flags]
  ckrt [command]

Available Commands:
  get         Gets an encrypted property
  help        Help about any command
  ls          Lists all encrypted property keys
  put         Puts an encrypted property in ssm
  version     Prints the version

Flags:
  -h, --help   help for ckrt

Use "ckrt [command] --help" for more information about a command.
Now you can add a secret.

Note. Secret properties are created here in the AWS console - AWS Systems Manager --> Parameter Store

  1. Export your awsaml creds
  2. Get the yaml you will need for your property. There are two ways:
  • Global secret (recommended)
    • You still need to add the property to each service, the property is just available to be inserted in any service. The property goes into SSM's global scope.
    • Ensure to add a label to the property so we can distinguish between environments, example "--label ipims-staging-1". Be very careful here to use the correct label as not to break another environment in same account.
    • To update an existing property or add an additional label, need to pass in "--overwrite" this will overwrite the existing property. Note multiple labeled values can co-exist under the same property.
    • This is useful when you want to share a property among different services. Like a shared database, for example.
    • Examples:

      • ./ckrt put my.database.password --secret uy6WZP+fQPBEadC7Q --region=us-east-1 --label ipims-staging-1

      • This will give you an output like this:

        my.database.password:
  $ssm:
    region: us-east-1
    label: ipims-staging-1
    encrypted: AQICAHjkwn9XuixxAOCL40UpIe1fF7AZiA4PBLOtIrrb1uE2OwGfFlFV7pN2d1Py76siTFrWAAAAZjBkBgkqhkiG9w0BBwagVzBVAgEAMFAGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMQ4qS8ow4eb5KukHYAgEQgCOZFhQae6ybcrmNvYnk+2UQ83bFYfSuiDxPcgQpNHvHLKmu/g==

      • See Step 3 for an example of where to put the returned output

  • Service-local secret (use only if required)
    • These are useful when different services need to have the same secret key with different values. We have run across this use case with kong tokens.
    • Ensure to add a label to the property so we can distinguish between environments, example "-l ipims-staging-1". Be very careful here to use the correct label as not to break another environment in same account.
    • To update an existing property or add an additional label, need to pass in "--overwrite" this will overwrite the existing property. Note multiple labeled values can co-exist under the same property.
    • Examples:

      • ./ckrt put my.kong.token --service=my-service --secret e21e1e6f-dd72-4a79-a923-8049963b09b4 --region=us-east-1 ---label ipims-staging-1

      • This will give you an output like this:

        my.kong.token:
  $ssm:
    service: my-service
    region: us-east-1
    label: ipims-staging-1
    encrypted: AQICAHjkwn9XuixxAOCL40UpIe1fF7AZiA4PBLOtIrrb1uE2OwGDK4XC9gp9H1sHkux/CZLBAAAAgzCBgAYJKoZIhvcNAQcGoHMwcQIBADBsBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDLAwh75FgfexNFpQ3QIBEIA/x0F+ED7q6qZtUC2HGnpf0ibAzv/xmf/GkWKoP4SaZ6avcrN0WdTWdDQhNtecQjaBQ1eSXy4xF3RLxKmutE13

      • It looks much the same, but now we have an additional key service which tells cps to do it's service-local magic. Adding it is the same as with global secrets.

      • NOTE: --service must match the name of your service file, minus the .yaml.

Make sure you set up the linter or you will run into problems

Jenkins in EKS

Jenkinsfile

env.PROJECT_PATH = "CPS" # Path to CPS props
cpsPipeline()

Commonci

A GitHub pull request example can be found here. You should just be able to add your repo to the array up top.

Migrating from propsd

This repo can be used to migrate from propsd properties to CPS

https://github.com/rapid7/propsd-migrator

Clone this wiki locally