Add local env runtime variables override for local testing

[yuli-han]

Summary

Currently Chronon set the runtime environment variables from several sources based on the below priority order:
- Environment variables existing already.
- Environment variables derived from args (like app_name)
- conf.metaData.modeToEnvMap for the mode (set on config)
- team environment per context and mode set on teams.json
- default team environment per context and mode set on teams.json
- Common Environment set in teams.json

The team environment per context mode set is a team level environment setting which defined in the json file teams.json. Team could put the environment settings(like EMR clusters and queue) based on their need, and these settings will be applied to all the jobs within the team folder.

However, sometimes our users may want to test their new job locally, the local tests may use different environment settings comparing with the prod run on airflow. Currently they need to add their job config into a different folder(like the test folder) other than the production folder(which will be parsed as context), and add context specific environment variables, but this may bring new complexity to the testing process since they still have to copy their changes to the production config to make it work in production environment.

This PR is trying to solve this problem by introducing a new arg env whose default value is dev. Users can define runtime environment variables for different environment in teams.json file, for example, for production environment and dev environment:

"production": {
    "backfill": {
        "EMR_CLUSTER": "backfill-shared-prod",
        "EMR_QUEUE": "backfill"
    },
    "upload": {
        "EMR_CLUSTER": "backfill-shared-prod",
        "EMR_QUEUE": "backfill"
    }
},
"dev": {
    "backfill": {
        "EMR_CLUSTER": "homes-shared-prod-ap",
        "EMR_QUEUE": "staging"
    }
},

1. For production run on airflow, the job will be scheduled with --env=production, so the production environment will be used.
2. For local testing, the default env is dev, so if the dev environment is provided in teams.json file, dev environment will be used, otherwise production environment will be used.

Why / Goal

Test Plan

• Added Unit Tests
• Covered by existing CI
• Integration tested

1. dev environment is provided in teams.json, env=production
   Local tests with
   python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20 --env=production

2. dev environment is provided in teams.json, env=dev
   Local tests with
   python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20 --env=dev

3. dev environment is provided in teams.json, env is not used
   Local tests with
   python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20

4. dev environment is not provided in teams.json, env = dev
   Local tests with
   python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20 --env=dev

Checklist

• Documentation update

Reviewers

@better365

[pengyu-hou]

we want to leave the Airbnbism code within our data repo. So when we are ready to open source, it is more general here.

[yuli-han]

Got it! If we cannot add the airflow related logic in chronon code, I am thinking on some other options:

1. Ask users to add an extra test flag when they run the python job for local testing, like
   python3 ~/.local/bin/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2022-07-02 --test
   And then in run.py file we can prioritize the dev_team_env if the test flag is set here.

2. Ask users to put the compiled config files which they want to test into the test folder. And they can add environment variables as key test in the teams.json file so that run.py will automatically pick the test parameter. No code change is needed in run.py

3. Nothing needed to be changed from the user side, but we add something in data repo code, like add --env=prod in airflow scheduling code. In run.py we try to get env from the command, if env is set to be prod, then use production env otherwise use dev_team_env. This will help the airbnb internal users but will introduce extra complexity for outside users(since they may not use airflow and they may need to add this env parameter)

Both option 1 and 2 needs users to change the way they do the testing. Wander do you have any other suggestions here? @better365

[pengyu-hou]

should we put production_team_env before team_env？

Suggested change

	order = ["conf_env", "team_env", "production_team_env", "default_env", "common_env", "cli_args"]
	order = ["conf_env", "production_team_env"， "team_env", "default_env", "common_env", "cli_args"]

[yuli-han]

Hi @better365 the PR has been moved to here #698
For this change if we put production_team_env over team_env, then even for local testing, as long as production environment is provided in team.json file it will always be picked right? We want to use dev environment if we set env=dev.

The logic is:
env=dev: production_team_env = production environment, team_env = dev team environment. and we want to use dev team environment.

[yuli-han]

PR has been moved to #698