Developer-first secrets specification

[cosmicBboy]

User story: As a flytekit user, I'd like to securely mount secrets from my local environment variables so that I can easily use secrets like API tokens for various external services (e.g. OpenAI, Huggingface, etc)

@flytekit.task(secrets=[Secret(group=SECRET_GROUP, key=SECRET_NAME)])
def my_task(...):
    context = flytekit.current_context()
    secret_val = context.secrets.get(SECRET_GROUP, SECRET_NAME)

Which can be specified in pyflyte run (maybe also pyflyte register?), with something like:

pyflyte run ... --secret "SECRET_GROUP:SECRET_NAME=<value>"

• Secrets are mounted from the user’s local environment variables
• They’re encrypted by flytekit and stored in flyte DB

[EngHabu]

Why do we need to store them in the DB? can't we just pass them as env vars on the execution?

I guess the logic in secrets.get() would be that if the equivalent env var is available, then go decrypt the secret, if it's not available then fallback to existing secrets logic?

[cosmicBboy]

Why do we need to store them in the DB? can't we just pass them as env vars on the execution?

Wanna address this @kumare3 ?

[davidmirror-ops]

2023-11-09 Contributor's meetup notes: implementation already in progress.