REQ: external secret management

[j-zimnowoda]

Introduction

**Note: The final requirements derived from all comments are presented here: **

The multi-tenant environment puts extra requirements on configuration management and data access control. At otomi we promote git operations and infrastructure as a code design patterns. At the same time we want to ensure that customer sensitive data is perfectly safe. Data encryption and access control does not play nicely with git. It often leads to merge conflicts and there is no access control at all. Therefore secret values need to be abstracted from git repository in a way that otomi can still benefit from mentioned design patters.

The goal of this document is to provide a deep insight into secret management issue for multitenant environments. This document describes user stories, system requirements, and system architecture. This document aims to give a guideline for the implementation.

Some open-source solutions has been already selected and taken into account in this text. These are: vault, bank-vaults and kubernetes-external-secrets. A reader is encouraged to get familiar with them firstly.

Note: related code change can be found on abstract-secret-management branch.

---

Github issue: #263
Requirements: (#279 (comment))

[Morriz]

Ver elaborate starting point. Thanks a lot!

Can the category/ part be removed from the proposed key name convention? its the same in every example.

[j-zimnowoda]

Thanks. Yes, removing category make sense.

[Morriz]

Also, I would propose a master cluster that runs a Vault HA setup, to be replicated to a slave cluster. Other clusters can pull from the master Vault, and we can failover to another one if need be.

[Morriz]

Actually, forget about the replication. We can always dump bucket folders if we don't trust cloud object storage. A failover vault can attach to the bucket without a problem if a cluster goes down.

[j-zimnowoda]

There are some issues with Vault HA that is deployed into many clusters:

• inter cluster latency (The latency between availability zones should be less than 8 milliseconds for a round trip [1]
• traffic failover (we do not have a LoadBalancer that performs failover between clusters)

[Morriz]

Why is latency an issue? I don't see that being a problem. Vault HA will probably only go down when the cluster goes down, which is either due to a BIG PROBLEM or intentional maintenance. I don't see Vault HA going down easily in a healthy cluster. And even when it does go down it should not affect running services, only new replicas. So I don't see Vault as critical in that regard.

So when we want to failover we can use dns for the time being and just update vault domain to point to other cluster loadbalancer ip. If we want to be fancy we can let the user opt in for an extra loadbalancer so it would be fully automated.

[j-zimnowoda]

If Vault HA is deployed at the one cluster then there is no problem.
We have already talked about having master cluster anyway.

[0-sv]

Well done with this elaboration!

Unfortunately it is not possible to deploy many vault standalone instances that share the same storage. It would would lead to data inconstancy

I think you are confusing this with distributed and integrated storage. In the stand-up I was thinking about a solution like this: https://learn.hashicorp.com/tutorials/vault/raft-storage. The idea would be to expose the Vault deployment to the end-user so there doesn't need to be any configuration of external storage, in my experience that leads to a hassle.

It's not that this doesn't have it's own challenges (CAP theorem and all that), but it might take complexity away from the end-user into our hands.

[j-zimnowoda]

As far as I understand raft replicates data for each Vault instance, it is good solution if your storage does not support HA mode, e.g. S3 [1]. However you still need a storage to persist your data. But thanks, you just made me aware that S3 and Azure Object Storage does not support HA mode.

[0-sv]

However you still need a storage to persist your data.

I think I am sounding a bit stubborn by now but I actually think this is not the case. ETCD is based on raft (as the most concrete example we both know of) and it doesn't need "a storage to persist data". Or maybe we have a different definition of "storage to persist data".

So in case there is doubt about that, I imagine an external storage provider is NOT needed with this, and you think it still does (are we agreeing or disagreeing on this?). This is because it could be based on the raft consensus algorithm.

[j-zimnowoda]

I investigated ExternalSecrets and possible race conditions:
My findings are there: external-secrets/kubernetes-external-secrets#154
Long story short: By defining proper timeout for helm we shouldn't experience any problems. Of course as long as Vault operates properly.

[Morriz]

No timeout dependent solutions, everrrrrr. I think we need a job to query the secret for the right annotation. The solution is likely writing back something like a done flag.

[j-zimnowoda]

It would be enough to check if Secret exists. Anyway your solution is also timeout based.

[Morriz]

Huh? No it is not. I explicitly mention the solution to inspect for a ready state.

But yes, checking for secret existence is even better.

[j-zimnowoda]

[j-zimnowoda]

• solve issue with policies that enforce multi-tenancy (done)
• configure KMS (will be part of regular task)

[Morriz]

I love all that this has to offer. I swerve in two directions now:

First thought: we should ditch current "generic" secret creation where we let the teams create them in the console. Than we will instead only use bank-vaults injection. We can then also skip etcd encryption at rest (what else is sensitive?).

Second thought: kubernetes-external-secrets approach offers a real advantage: once secrets are created, and their TTL has not expired, vault is not needed (it might have hickups or we break it) and thus it is possible to create a maintenance window. I suggest a TTL of one day, and re-fetch secrets at preferred local time. (Let's say 5am.)

Suddenly I start favoring the second idea above all else. We then also have to keep our current setup where we select keys from a shared secret, as it would be preferable to not generate one k8s secret per envVar or file. Instead of inputting values during "generic" secret creation, we let users give the vault key.

Nightly dialogue with myself paid off ;)

[j-zimnowoda]

Thanks for feedback !

I am in favour of Kubernetes-external-secrets because:

1. it is scalable solution. Once secret is created all allowed Pod can use it. In case of mutation webhook container from each Pod will ask for a secret from vault.
2. it does not introduce additional delay until Pod is ready (except one time when script is created by external-secret controller

[Morriz]

Ok, then we would only use the operator from bank-vaults because you think is is so handy? Can we avoid it with some work?

[j-zimnowoda]

No, we can't.

[j-zimnowoda]

I am trying to define some user scenarios for secrets creation:

My design goals are:

• to not keep secrets in otomi-api at all
• to avoid need for communication between otomi-api and vault.
• to come up with scenarios that are user friendly.

Below there are two proposals:

A user can create am external-secret in otomi-console

1. A user vistis to vault UI and define a new secret at a specific path
2. Next a user visits Team Secrets page in otomi-console.
3. Then he creates a new secrets by specifying its anew, path at vault and secret version
4. Later a user clicks deploy button, so otomi-api pushes it to the git repository
5. The deployment pipeline is triggered
6. An ExternalSecret CR is created
7. The ExternalSecret controller creates Secret by fetching it from vault

A user can change secret value:

1. A user goes to Team Secrets page in in otomi-console.
2. A user can see secret name vault path and its version. There is also a hyperlink to that secret at vault webpage
3. A user may click on that hyperlink in order manage value of a given secret

A user can change external-secret version:

1. A user goes to Team Secrets page in in otomi-console.
2. A user clicks on a given secret and is able to select a new version of the secret

@Morriz I would be thankful if you could help with that.

[0-sv]

Yes, I agree that this is a good way to start defining user stories.

The actual workflow seems reasonable to me as well! My only concern would be to add good documentation, because we think this workflow is obvious, but maybe an end-user wouldn't. I found this recently: https://documentation.divio.com/how-to-guides/. There is actually a lot more helpful information about big-picture documentation here.

[j-zimnowoda]

We have just started this way of working and it is too early to say if it is really helpful. Let's give us some before defining a structure and process. For time being let's keep it very simple:

1. Discussions are for collecting requirements and preparing overall design
2. During that time we may also perform a very first PoC with a demo. The demo should help in making decisions about target solution.

First two steps shouldn't take more that 2 weeks.
3) Prepare user stories in Github and introduce them into the team.
4) PO decides about priorities with consultancy with the team

[0-sv]

Well I'm glad you have some thoughts about the process too. I've spoken to @Morriz about this and the result is the discussion in #300, I'm curious to hear what you have to say about it.

[Morriz]

2. During that time we may also perform a very first PoC with a demo. The demo should help in making decisions about target solution.

I disagree here. I think a PoC should be an MVP delivered as a story. A PoC is not always needed, nor is it needed to involve others. We expect devs to be self reliant and only ask for input when they need it. We already have a demo phase at the end of a sprint. It seems nice to be treated with demos left and right but we need to work more efficiently than we do now.

[j-zimnowoda]

This why I said "may".
To me demo and PoC is equal.

[Morriz]

Can I ask where the story is for these requirements? Can you add a link to that in the top @j-zimnowoda ?

[Morriz]

@j-zimnowoda can I voice some frustrations?

• First of all, I saw your tiny link abbreviation under the description. I don't think that is very helpful, and it would be better to introduce it with a sentence.
• Another issue I see is the mixing of FR/Story templates, which we want to avoid, and which I have mentioned to you some times before.

I think our process docs should make it very clear how to use our tools: FR voices a need, and might propose solutions, but will NOT contain any process steps. The FR should ONLY serve as input, and a story may be created to reflect intended work.

Of course it is up to outsiders to do this differently, but they are not part of our scrum team.

Also, I think it is limiting the outcome to choose hashicorp "vault" as a solution. We have a business need that is and are exploring still.

[j-zimnowoda]

@Morriz, Thanks for your feedback.
I agree that discussion should not drift away from setting requirements. We have just started to work like that and there are some lessons learned! I am pretty sure that next discussion will be less chaotic.

Regarding choosing HashiCorp vault,
First of all I love exploring and trying new things, so if your gut feeling is to keep exploring, then I am fine with that.
While exploring, I am also trying to provide workable PoC, so you have look&feel experience and decide if it meet business requirements or not.

I would like to get to know your concerns about vault. Let's talk today.

[j-zimnowoda]

An admin can bootstrap chart secrets (not covered by MVP 1)

Let’s start from simple statement:

Most of the chart secrets cannot be provisioned automatically, as they are derived from 3rd party providers.

Below there are examples of the secret jsonpaths that otomi-core is using:

alerts.email.nonCritical
alerts.slack.url
azure.resourceGroup
azure.subscriptionId
azure.tenantId
charts.drone.adminToken 
charts.drone.adminUser
charts.drone.githubAdmins.token
charts.drone.sharedSecret
charts.harbor.adminPassword
charts.harbor.persistence.imageChartStorage.gcs
charts.harbor.secretKey
charts.keycloak.idp.clientSecret
charts.loki.adminPassword
charts.oauth2-proxy.config.cookieSecret
charts.otomi-api.git.email
charts.otomi-api.git.password
charts.otomi-api.git.user
charts.otomi-api.tools.gcloudServiceKey (to be removed)
charts.prometheus-operator.alertmanager.slack
charts.prometheus-operator.grafana.adminPassword
google.cloudDnsKey
google.kmsAccount
home.email.critical
home.slack.url
oidc.clientID
oidc.clientSecret
otomi.pullSecret
smtp.auth_password

Note: this list may not be complete and it is responsibility of task owner to come up with full list of secrets

A guideline for secrets migration in otomi-core:

There are two use cases that we have to take into account:

1. Charts allows to reference an existing secret
2. Charts only allow to create a secret by providing secret values

Case 1:

A developer needs to:

• define ExternalSecret CR in values/<chart-name>/<chart-name>-raw.gotmpl file at otomi-core project
• provide a reference name to a Secret in values/<chart-name>/<chart-name>.gotmpl Note: the Secret will be created by ExternalSecret controller after deploying ExternalSecret CR
• Instruct helmfile to deploy <chart-name>-artifacts release in helmfile.d/

Case 2:

It is similar to use case 1 but and additional effort is required:

• change chart so reference to existing secret is allowed
• Make chart Secret templating optional, so if secret reference is provided then Secret is not deployed by that chart.

Cleanup:

Take care about cleaning up the old secrets from values repository. Mind, that it can be done only if all clusters from a given customer (value repo) are using a new secret solution

[Morriz]

What is this sudden extra blob of text doing here? Is it intended to add to the requirements? Don't you notice that it is totally unclear and out of context?

[Morriz]

I get it now, it is to extract core secrets from the values repo. Can we create a follow up story for that? With separate requirements (from what you just put above)?

[j-zimnowoda]

yes :)

[Morriz]

@j-zimnowoda we want to add the following requirements:

• support Bring Your Own Vault: external-secrets does that, it works with all vaults, great. We can use

otomi:
  vault:
    exists: false
    provider: (azure|aws|hashicorp)
    aws: ...
    azure: ...
    hashicorp: ...

• when exists: false we provide ootb hashicorp vault install: install bank-vault operator and configure accordingly

So you are now aware that we should have an open configuration that allows swapping vaults.

[j-zimnowoda]

That's challenging but I can imagine use case for that.

Few question are poping up:

• how about access control based on JWT?
• how about otomi-core internal secrets?

[Morriz]

• how about access control based on JWT?

Out of scope with Bring Your Own IDP? I don't know how external-secrets handles different backends, but it involves plumbing and adds overhead, so lets keep that also for a next story.

• how about otomi-core internal secrets?

We can still use what we have now, as there is no need to move them out of the repo.

[j-zimnowoda]

sure

[j-zimnowoda]

Added,
I think we are finally aligning on that there are:

• otomi-core internal secrets
• and team secrets

[j-zimnowoda]

User stories

1. A user can define external-secret in values repo: Facilitate teams with ExternalSecrets and Vault #317
2. A user can define external-secret in otomi-console: https://github.com/redkubes/otomi-console/issues/58 https://github.com/redkubes/otomi-api/issues/131
3. An admin can bootstrap otomi-core internal secrets

System requirements

Vault security

1. The Vault seal is configurable
2. The vault encryption key is periodically rotated
3. The Vault defines access polices that restrict access to certain paths
4. The vault perform periodically encryption key rotation
5. The Kubernetes Service Account token is used for authentication at Vault
6. The Kubernetes Service Account name has associated vault policy
7. The Kubernetes Service Account name is unique for each deployed chart/app (Pod)
8. The JWT is used for authentication at Vault
9. The JWT contains groups claim
10. A group name from JWT has associated Vault policy
11. A Cloud Service Account is allowed to manage KMS and Object Storage cloud resource
12. A Cloud Service Account credentials are stored as a Secret in vault namespace
13. A Cloud Service Account credentials .secrets file and needed only for initial otomi platform deployment
14. The bank-vaults, kubernetes-external-secrets have istio sidecar
15. Each Pod deployed by bank-vaults has an istio sidecar
16. Each cluster has its own vault instance

Vault storage

1. The Vault stores secrets in HA object storage ([GCS](https://www.vaultproject.io/docs/configuration/storage/google-cloud-storage), S3, ABS), Raft
2. Velero performs backup for Vault operating with Raft velero-vault

Vault HA

1. Vault is deployed in HA mode

Secrets

1. A secret value can be either defined by a user or generated by vault
2. A secret value generated by vault can be rotated
3. A secret is team and cluster scoped
4. The following secret types are supported Opaque, kubernetes.io/service-account-token, kubernetes.io/dockerconfigjson, kubernetes.io/tls.
5. Pods are aware of secret value change
6. A secret can be injected directly into container
7. A secret can be created in kubernetes by the kubernetes-external-secrets controller
8. A team secrets can be derived from customer vault (Bring Your Own Vault) - TBD

Architecture

Structure of secrets

Mount path is used for organising secrets in a structured way. The can be also used to define polices and to restrict access.

For multitenant cluster the following path naming convetion is proposed: secret/teams//category// For example:

• secret/teams/team-mercury/category/services/aws/dev/servicA/secret1
• secret/teams/team-mars/category/services/aws/demo/servicA/secret1
• secret/teams/team-admin/category/charts/keycloak/idp/clientSecret

For multi organization solution above paths can be prefixed with org/<organization name>.

Vault secret engines

Secrets engines are components which store, generate, or encrypt data. [1]

At otomi platform the following secrets engine are going to be used:

• KV Secrets Engine - storing arbitrary secrets within the configured physical storage
• Google Cloud KMS - encryption and key management
• Identity - fine grained and dynamic access authorisation

secret engine can be used in production environment. For onprem setup the secret engine is not defined yet.

For platform development purpose the [Google Cloud KMS] can be disabled used.

Vault policies

Properly organized structure of secrets enables to apply vault policies efficiently. The vault policy is a collection of paths, each has associated capabilities. A policies can be associated with vault roles. The Vault roles can assigned with one or many authentication method

Some policy examples are presented below:

path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/teams/team-admin/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Vault authentication

In otomi platform the vault Kubernetes and OIDC authentication methods are going to be supported.

Vault authorization policies

The vault also introduces concept of groups. Any group can contain many vault policies.
In our case a group can contain one policy that indicated path to a team secrets, e.g. secret/teams/<team-name>/*

In otomi we leverage groups claim for OIDC scope that can contain many team names. In vault we map OIDC groups claim to vault group by creating identity group-alias

Note: we are not going to use vault templated policies, as they are difficult to define declarative and we can achive multi tenancy just by assigning policies to groups.

References:

• https://www.vaultproject.io/docs/secrets/identity#identity-groups

Multicluster support

Each cluster has its own standalone Vault instance.

References:

• https://banzaicloud.com/docs/bank-vaults/overview/
• https://github.com/banzaicloud/bank-vaults/tree/master/charts/vault-secrets-webhook
• https://banzaicloud.com/docs/bank-vaults/mutating-webhook/