REQ: backwards compatibility transformation interface in otomi-tasks

[0-sv]

Splits: #259.

Proposal for interface in otomi-tasks

This REQ discussion is my proposal for a decoupled interface with respect to this #259. My idea is to provide a CLI with positional arguments in otomi-tasks. The advantage of creating such a flexible tool is that it isn't coupled to otomi-core, promotes reuse (as a tool for otomi-core and as a potentially stand-alone application).

Basically, the requirements for #259 dictate that we should have a mechanism/tool to apply some transformations to .yaml files:

locations:
    charts.bla.someProp: someNewRootProp.someProp
deletions:
  - charts.bla.someOtherProp
forwardMutations:
  # image tag went from semver to glob
  charts.bla.image.tag: 'printf "v%s"'
backwardMutations:
  charts.bla.image.tag: 'replace "v" ""'

To achieve this, I want to start with the end in mind. I've mocked an interface with the yargs library. The preview I present is split up between code and a quick demo.

If you remember my demo, I've already showed one of the transformations (locations) as a proof of concept. There is definitely a lot of code re-use possible, but my concern is that it is too coupled with an otomi-values repository.

Code

import yargs from 'yargs/yargs'
import { hideBin } from 'yargs/helpers'

const argv = yargs(hideBin(process.argv))
  .command('delete <.yaml> <key>', `Delete nested property in .yaml file`)
  .command('move <.yaml> <key> <new-location>', `Move property in .yaml file to <new-location>`)
  .command('transform <.yaml> <key> <gotmpl>', `Apply transformation to a nested key's value`)
  .help()
  .usage('$0 <command>', 'Executes <command> on .yaml file(s)')
  // End every yargs call with `.parse()` or `.argv`
  .parse()

alert(argv)

Demo

npm run tasks:yq-recipes-dev -- --help

> [email protected] tasks:yq-recipes-dev /home/sv/projects/otomi-tasks
> ts-node-dev ./src/tasks/yq-recipes/yq-recipes.ts "--help"

Using ts-node version 8.10.2, typescript version 3.9.7
yq-recipes.ts <command>

Executes <command> on .yaml file(s)

Commands:
  yq-recipes.ts delete <.yaml> <key>        Delete nested property in .yaml file
  yq-recipes.ts move <.yaml> <key>          Move property in .yaml file to
  <new-location>                            <new-location>
  yq-recipes.ts transform <.yaml> <key>     Apply transformation to a nested
  <gotmpl>                                  key's value
  yq-recipes.ts <command>                   Executes <command> on .yaml file(s)
                                                                       [default]

Options:
  --version  Show version number                                       [boolean]
  --help     Show help                                                 [boolean]

Explanation

• $0 is the name of the executable. Right now it's just calling the .ts file, but imagine an executable like yq-recipes.
• I call it yq-recipes, because it basically implements advanced yq commands. Yes, it could be implemented with yq and bash, but I suppose it's quite convoluted for generalising certain commands, e.g.: https://stackoverflow.com/questions/42214984/jq-how-to-move-the-child-members-to-parent provides an example how to transform a to b. But a.b.c.d -> a.b.c.e is another matter, and other transformations, become tricky to maintain without abstracting the plumbing.

Use in otomi-core

I imagine we will use the executable to apply the changes yaml. E.g.:

yq-recipes move values-schema.yaml "charts.bla.someProp" "someNewRootProp.someProp"

We need the forward script to:

• bring a repo to the latest version
• push that to git

the backward logic:

• to bring the structure to its required version
• be invoked by every otomi command that works with values

otomi migrate-values $SEMVER

tl;dr

• Main benefit: decoupled interface, flexible code, reuse
• If OK, this will split Create values-schema mechanism to update values to conform to schema changes #259 into its otomi-tasks story and otomi-core story. Both will be re-estimated.

[Morriz]

but my concern is that it is too coupled with an otomi-values repository.

What is too coupled with an otomi-values repository?

Right now it's just calling the .ts file, but imagine an executable like yq-recipes.

Maybe give it a meaningful name: yaml-mutate

To parse .gotmpl for the backwardMutations and forwardMutations, we would need to parse Go in Typescript. I'm taking a course in compilers at the moment so I will be the last to say it can be done. However, a suggestion or lead into this topic would be great.

We will hopefully not do any gotmpl parsing in typescript, as that now involves helmfile and all of our gotmpl architecture. I also don't share your conclusion to do so.

This would apply all changes within the git tree to otomi-values, ie. for each revision "migrate" otomi-values to the following version.

I am not following you here. We just want:

• the forward script to:
  • bring a repo to the latest version
  • push that to git
• the backward logic:
  • to bring the structure to its required version
  • be invoked by every otomi command that works with values

As an enhancement, we can also do stuff like otomi migrate-values --interactive and move up and down the git tree, because it's supposed to go both ways. Both otomi-core and otomi-tasks can be easily extended without modifying the other to support different transformation.

No need for anything interactive.

otomi-tasks can be extended indefinitely, however, every "task" will use all the same packages as the other tasks. I wonder if this is OK for its dependency management. E.g. upgrading a package to use another package while the previous has been written for a certain package is going to break stuff. Also, one can't really upgrade any package without a chance of destroying the other dependencies.

That is not a concern now. We are one team and have no issues with this mono repo setup. Actually only benefits.

[Dunky13]

I agree with @Morriz on the Go in TS. It seems overcomplicated, as why wouldn't we - if we want to change the order of values - change the gotmpl itself? If we want to change the otomi-values then it is not bound to Go(tmpl) anyway right?

[0-sv]

gotmpl parsing in typescript

We decided on applying transformations in otomi-tasks. You expect backwardMutations and forwardMutations based on a .gotmpl, e.g.: 'printf "v%s"'. What is your suggestion to implement this?

[Dunky13]

Do I understand correctly from:

This would apply all changes within the git tree to otomi-values, ie. for each revision "migrate" otomi-values to the following version.

That you would migrate the values throughout the full git history?

[Morriz]

No, that is hopefully not the case. This has nothing to do with git history I hope. I believe we settled in the reqs for a meta property that holds the version changes, right? So we don't need git log?

[0-sv]

We settled in the reqs on plucking the changes & version properties from the git history.

[Morriz]

that is not workable...as everything should be contained in the latest repo commit, as that is the one driving the automation, and as such should have all the information readily available for performance reasons

where are those reqs?

[0-sv]

that is not workable...as everything should be contained in the latest repo commit, as that is the one driving the automation, and as such should have all the information readily available for performance reasons

Please just realise that I'm doing my best to prevent going in the wrong direction and doing a lot of work for nothing. It is my intention to take a careful approach. It is true, we have spoken about this issue a lot. But I also hear you talk about it being some sort of magic bullet, or solve world hunger, while I still see some assumptions and misconceptions.

where are those reqs?

The reqs were discussed during a conversation. I want to flesh them out in one or more requirements discussions, in writing. It is my intention to take a very flexible approach, hence I started this discussion for one of the requirements of which I'm more sure what the interface should be (ie. the transformations themselves).

So, if desirable, let's discuss otomi-core logic in this thread. The original intent of this discussion is JUST to split the story and finish work on the side of otomi-tasks.

otomi migrate-values bases ALL mutations to be applied to otomi-values on a changes key/property (ie. values-schema.yaml). We will maintain a full list of all possible mutations in changes. In consequence, any otomi-values repository can be upgraded, in full, for all matched mutations in changes. The changes property will, at some point, have a lot of entries, but at least it has all the information to do a full upgrade from any otomi-values repository with some keys that have been moved/deleted/mutated.

In other words; or, if you want to look at the issue from another perspective, the otomi-values repository has some property according to some values-schema.yaml version -- a more recent values-schema.yaml. This property needs to move somewhere/be deleted/be mutated, to work with a more recent version of otomi-core.

So practically, a developer's workflow looks like this:

Assume there is a values-schema.yaml with a changes property and some version property. We also assume we have a to-be-upgraded otomi-values repository. We can take the following steps to upgrade a new feature f:

1. Modify values-schema.yaml for all mutations that f needs
2. Add mutations to values-schema.yaml's changes property

Now for any otomi-values repository: otomi migrate-values $version applies all mutations from the changes property to the $version.

[Morriz]

Thanks, that is the upgrade path indeed. So to summarize all:

Upgrade path

1. values-repo is upgraded by a newer version otomi-core: successively apply all the changes between versions
2. Newly changed values are committed
   (I don't care much if those steps can be combined)

Rollback path

1. Any command ran in an older otomi-core container that works on a newer values repo invokes an ON THE FLY transformation from the newer structure to the older format before consumption. Where that transformation takes place remains to be seen, but I would like you to consider that helmfile is tightly coupled to the files it reads in.

[Morriz]

I don't see a solution to this yet. I have a hard time wrapping my head around this, and advise you to think about this up front, as it might be the case that we can't deliver a solution to our problem at all @svatwork.