Meltano Plugin SDK: First iteration contract and spec proposal

[aaronsteers]

Status: Stub, WIP Draft, Reviews and Alternatives much appreciated.

Spec Proposal Overview

Each plugin may have one or more declared capabilities and commands which correspond to specific behaviors and expectations when invoked by Meltano.

Command Name Contracts

Command names may be exactly named by convention (in the case of start and stop) or they can be pattern-matched by convention (test, test-foo, and test_bar all match to meltano test). When a command name matches to a known behavior, Meltano may "smartly" invoke the specified command when the behavior is implemented.

Capability-to-Command Declarations

In the plugin definition yaml (and/or in --about command output), the plugins can advertise which contract(s) they subscribe to.

Note: Since commands are statically detectable, we may not need to rely much on capabilities.

Setup and Cleanup Hooks

Setup and Cleanup Hooks would be defined by commands, for instance setup_hook and cleanup_hook for example. Meltano could either detect these via a simple scan of command names in yaml, or else they could be 'advertised' by specific capabilities that the plugin would advertise.

Templates and SDKs

1. We (Meltano and Community) likely would create a Python-based SDK to expedite plugin development, in a similar vein as the Meltano SDK for Taps and Targets (sdk.meltano.com).
2. Except that Python-based accelerator would be created to streamline new plugin development, the spec itself would not presuppose or make special accommodations for the Python language.

Compatibility with Docker-based plugins

Because this approach is command-driven, the spec itself is 100% compatible with container-based plugins.

[pandemicsyn]

Few quick initial questions/thoughts about your first pass @aaronsteers:

Command Name Contracts

they can be pattern-matched by convention (test, test-foo, and test_bar all match to meltano test). When a command name matches to a known behavior, Meltano may "smartly" invoke the specified command when the behavior is implemented.

I can see people just creating basic wrappers for existing CLI's and commands that forward commands/args - and doing pattern matched invocations could be a really nice shortcut there, but could also lead to unexpected invocation.

I'm a little torn, I was going to suggest that to start with, it might be worth only inferring what to do or invoke via explicit means and then supporting pattern-matched by convention behavior for later. But, on the other hand, having pattern-matched invocations out of the gate early on prevents unexpected behavior changes in a later version - so probably is good to have this sooner rather than later.

Capability-to-Command Declarations

In the plugin definition yaml (and/or in --about command output), the plugins can advertise which contract(s) they subscribe to.

Small pet-peeve, whatever we call this command flag - could we call it something other than about ? Unless, we're trying to mirror the singer plugins very closely? It's always bothered me that a flag named --about produces output primarily designed for machine consumption.

Note: Since commands are statically detectable, we may not need to rely much on capabilities.

@aaronsteers can you say more on this? Do you just mean because we'll have a yaml config describing the capabilities?

Thinking out loud: Wonder if this would evolve overtime. As this framework gets more prevalent, would we want to become better at dynamically detecting a plugins capabilities and supported behaviors? Not sure, if it's even that useful beyond saving some keystrokes at config time.

Setup and Cleanup Hooks

Setup and Cleanup Hooks would be defined by commands, for instance setup_hook and cleanup_hook for example.

Just want to clarify and be explicit. When you say "by commands" do you mean commands like the actual cli command's like "airflow" and "dbt" or sub-commands like "airflow scheduler" and "dbt run"? Or both, I suppose.

Meltano could either detect these via a simple scan of command names in yaml, or else they could be 'advertised' by specific capabilities that the plugin would advertise.

With a contributor hat of "im just quickly trying to get some 3rd party command into the meltano ecosystem" on, the "via a simple scan of command names in yaml" approach sounds really nice, quick, and easy because I'm probably already very comfortable using yaml to drive work.

If I'm the maintainer of a complex plugin though, being able to drive that from capabilities might be nice - but not having this ability probably wouldn't be a deal breaker since I can instruct users how to drive this from the yaml config.

Compatibility with Docker-based plugins

Because this approach is command-driven, the spec itself is 100% compatible with container-based plugins.

Generally true, with the caveat that there's almost certainly going to be gotchas and issues for plugins that themselves's expect to leverage docker. Accidental Docker-In-Docker or Container-In-Container style scenarios. Which actually makes me wonder if plugins should advertise how they'll be executing things or perhaps advertise how they're allowed to be called: e.g.

allowed_execution_modes: ["native", "docker", "oci"]

Versioning

We should bake this in now, I'm willing to bet that we'll want some way to signal versioning of the spec a plugin supports sooner rather than later.

[pandemicsyn]

@aaronsteers @edgarrmondragon @pnadolny13 curious what sort of hooks do you think we'll need? Do we need to essentially copy meltano's internal hook's? That feels like too big an interface surface.

Is a basic set of pre/post hooks like @pnadolny13 did during his hack-sploration sufficient? That meshes nice with the block pattern, which currently also just uses a basic set of pre/post hooks.

If the basic pre/post pattern is sufficient to start with, are there any special hooks we should consider supporting out the gate. Like a post_install hook to call at install time?

[pandemicsyn]

@pnadolny13 🙏 yea actually helped me reset and remember some ideas i had when i was thinking about this last year https://gitlab.com/groups/meltano/-/epics/134#note_764611563, especially around what plugins like a webhook, slack, databricks, random cloud ml api's might enabled.

@aaronsteers before we get too far down path of collecting additional plugin ideas (tbh between Pat's and a few of my own feel like we have pretty diverse set use cases). I wasn't sure if you wanted to drive this, but I've got a few possible architecture patterns in mind based on the premise of a plugin being a callable cli.

How about I write up a brief synopsis of those, and we can see if we like either of them and go from there.

[pandemicsyn]

How about I write up a brief synopsis of those, and we can see if we like either of them and go from there.

Heh, actually, let me just be proactive and write them up real quick.

[aaronsteers]

Thanks, @pandemicsyn and @pnadolny13 for the detailed thoughts and use cases. A couple points:

1. The pre and post hooks' cli signature should probably allow for some amount of context args, like --command-name for example. Or alternatively, we could send a generic --context which expects a generic json string (escaped of course). This is more future proof than having n CLI args.
   • Note that the plugin is free to ignore the arg(s) if it doesn't need them. It would just not be okay to crash upon receiving whatever named or positional args we establish as the hooks' (generic) CLI signature.
2. I expect that the way plugins will get data from Meltano will be via CLI commands - following the Airflow pattern of invoking meltano schedule/job list --format=json. Our Python SDK may have helper functions to execute those, but we should avoid any special or private interfaces and we should avoid baking in Meltano logic into the SDK itself.
   • In future, the CLI access path can be migrated to an API path, although there may be cases where the CLI approach is still the more stable one.

[pandemicsyn]

@aaronsteers There's a whole big world of awesome sauce integrations that would could support - with varying levels of effort. In an ideal world, meltano wouldn't just be a cli - and we'd be able to expose a language agnostic API to plugins (i.e. through a background/rest api or socket/rpc/etc), that would allow for clean callbacks and deep integration. That doesn't feel like something we can pull off though in a reasonable time frame. Definitely think we can transition to that later, and agree that initially providing a CLI access path might be the shortest route.

The a "a plugin is a cli we call" pattern can still get us pretty far though and can progress incrementally. I think the pattern, to start with, is one that looks a bit like a context-based HTTP request chain.

Assume we create an execution context at the start of a meltano invocation.

This context evolves and grows as each plugin (and its various hooks) is invoked and executed. Importantly, this context doesn't need to hold everything. The context is fairly lightweight and describes the current state of the meltano invocation.

It just details:

• What's already happened
• What's going to happen
• In the future - maaaaaybe it signals meltano capabilities TO plugins (i.e. advertises a path to secret's store/interface or a path to a socket for RPC/API calls)

Each plugin gets passed the context, and the context is updated after each plugin call before being passed to the next. The context gives each plugin awareness about what's already happened and what's going to happen. That enables things like notification plugins, which need to know what happened, and provisioning plugins...a dbx plugin might adjust what sort of cluster capacity to provision based on the pipeline to follow.

At the end of an invocation (whether hook or actual command call) a plugin can optionally return its own updated context object. To start with, that might just be as simple as a {"state": "someblob that represents my state that meltano should persists"} object. But perhaps could also evolve to be things like config write backs.

The context is really just a way to pass info - the types of hooks and capabilities we support largely dictate and control how plugins can integrate and what awesome-sauce folks can develop.

For example, a monitoring plugin could advertise a ignores_upstream_failures capability to signal that it should be executed... no matter what failures occurred upstream. So if you invoke: meltano run tap-gitalb target-postgres datadog, the datadog plugin is always invoked and allowed to gather what happened from the context passed to it.

Not saying that capability is something we should have out of the gate (or even at all), just trying to illustrate that capabilities from plugin could also influence meltano execution, it's not just a one-way street.

[pandemicsyn]

Small example (we can bikeshed on schema's, naming, specific contents later, mostly just trying to illustrate what it might look like)

For example when starting to execute the datadog portion of this statement: meltano run dbt:run something-that-failed datadog the context might look like:

InvocationContext: {[
   "SomeFinishedPluginCall": {
      "result": 0,
      "cmd": "run",
      "last_hook_called": "cleanup",
      "state": {"some": "state"}
      "config_hint": "how to see the config for this plugin"
      "meltano_cli_path":  "dot/meltano"
      "workdir": "where i should do work"
      "started_at": 191919191
      "finished_at": 202020202
   },
   "SomeFailedPluginCall": {
     "result": 1,
     "last_hook_called": "pre_run",
     "state": {"some": "blob with some failure info"},
     "config_hint": "how to see the config for this plugin"
     "meltano_cli_path":  "dot/meltano"
     "workdir": "where i should do work"
   }
   "DatadogAboutToBeStartedPlugin": {
     "state": {},
     "config_hint": "how to find the config for myself"
     "meltano_cli_path":  "dot/meltano"
     "workdir": "where i should do work"
     "always_call": true,
   }
]}

And the end of a plugin invocation (whether a hook or actual invoke) would be something small like:

"neat_webapp_plugin": {
  result: 0 // explicitly signal that i failed or succeeded
  state: {"state": "blob-representing saved dashboards or something"}
  err: "" //maybe allow for a nice human readable error we pass along
  config: {
   "restore_dashboard": true
   "dashboard_source": "meltano.state",
  } //maybe a config value i want to change because a user edit something in my plugins ui? 
}

[pandemicsyn]

@aaronsteers closing in on a draft spec (with some working examples). My last big missing piece is how or if we want to implement passing context to plugins in the first version. This is purely passing a read-only context from meltano to the plugin, its not bidirectional or anything.

In my current version, I'm passing a very basic context around that looks essentially like this for a simple meltano invoke call:

"invocation_id": "eca7bf08-845f-40e0-aa2c-317dd8de1fc8"  # used by the plugins to find themself in the invocation context
"meltano_invocation_context": [{
   "id": "eca7bf08-845f-40e0-aa2c-317dd8de1fc8"
   "plugin_name": "meltano-echo",
   "command": "test"
   "args": [],
   "last_hook_called": "pre-invoke",
   "meltano_cli_path": ".meltano/run/bin",
   "workdir": ".",
   "status": "inflight"
}] # array describing what in the execution chain

As part of a meltano run the list would include all the commands/blocks that would be run and status just reflects whether the invocation was completed or failed or is in-flight.

So two explicit questions.

1. Should we pass a simple invocation context describing the entire pipeline being run to each plugin in this first version?
   • My vote: It would be nice to have the mechanic baked in from day one, even if the context supplied at first is minimal. Also not opposed to dropping this for now though working it in later.
2. How do we want to implement it (--context-file=path, env var, stdin?).
   • My vote: plugins should accept --context-file option or env var

[aaronsteers]

@pandemicsyn - because any change later will be a breaking change, I'd lean towards providing only the bare minimum context for which already fits into the 'necessary' bucket of the plugin's role.

More specifically, I would prefer if plugins did not know anything about what preceded them or what is coming afterwards, just on principle of having each command's behavior atomic and deterministic.

So, according to that design objective, the behavior of the echo command in these three executions would be guaranteed to be identical:

meltano invoke meltano-echo:run
meltano run meltano-echo:run
meltano run tap-foo target-bar meltano-echo:run

While I definitely see some use cases 'wanting to know' the surrounding context, the problem with this is that then we don't get the same reproducible results, and we may end up in a situation where a bug prevents a command from running next to other commands run while it runs perfectly fine on its own via invoke (for instance).

[pandemicsyn]

@aaronsteers ok draft spec as promised:

---

Initial required commands and syntax

plugin pre-invoke --context-file=path  # can noop but must be callable
plugin invoke <command> --context-file=path
plugin post-invoke --context-file=path # can noop but must be callable
plugin describe --format=[default=json, yaml?]
plugin --help

pre-invoke & post-invoke

pre-invoke and post-invoke are called IMMEDIATELY before and after invoke is called. Both pre-invoke and post-invoke will be passed an invocation context describing what command is about to be invoked (see invocation context below). These commands may NOOP, but they must always be present, callable, and return a 0 exit code.

Error conditions:

• Any non-0 exit code will be reported to the user
• A non-0 exit code in pre-invoke will halt execution and the requested command will NOT be invoked

invoke

Analogous to a meltano invoke <command> [flags...] , the plugin invoke <command> call is expected to behave similarly and invoke the requested command. i.e.

$ plugin invoke sync-ui # may sync with some dashboard ui
$ plugin invoke deploy # may perform some other action

While a service pattern is not yet supported, future additional top-level commands are likely, and so plugin developers not using the SDK may want to keep that in mind. i.e.

# possible future patterns
plugin service ui start
plugin service scheduler start
plugin service all stop

describe

The describe command allows for auto discovery/configuration and generally describes the execution requirements of the plugin. In v1 the payload is limited. Consisting only of commands which describes what the commands the plugin supports.

Expected describe output, jsonspec pending:

$ plugin describe
{
  commands: [list of invocable commands]
}

Exit codes

• 0 in all success cases
• All non 0 indicate unsuccessful invocation and in most cases will halt the execution chain

Special/reserved command keywords

Some commands may trigger special handling behavior upstream within Meltano. Currently, limited to:

• test* (test, test_a_thing, test-other-thing, testAllThings) which maybe triggered during meltano test.

If a plugin exposes multiple matching commands, they're invoked in the order they're encountered. pre-invoke/post-invoke will be called for each command present.

Stdio behavior (Errors, Logging, and Output)

IO

• stdin is currently unused but reserved. Its unused at this time but maybe used for message passing in the future.
• stdout is currently unused but reserved. Its unused at this time but maybe used for message passing in the future. Plugins should avoid needlessly writing to stdout.
• stderr is available for structured plugin output intended for end users and operators.

Errors & Output

Plugins should emit plain (uncolored) structured log's to stderr. In the first version of this spec the structure isn't yet set, but developers are advised to follow the KISS principle and use a basic format. In the future, plugins may be required to support an optional JSON output format.

The following 3 fields should be present, with limited other adhoc key/value pairs to supply context where needed:

• ISO formatted timestamp
• log level
• event (which should encapsulate any forwarded log lines from underlying calls the plugin has made. e.g. airflow output)
• limited kv pairs for additional context as deemed useful

A small note on log level usage:

Debug and Info are generally the two most used and abused levels, but in the future, meltano will likely optimize around Warn and Error for passing on messaging to end users. With that in mind, it's important to call errors out explicitly via messages at the Error level on stderr. You should assume that an end user will be the primary consumer of a such a message - and so you should co-locate important context in the same log line or concisely explain what went wrong in the message if possible.

"good" example:

$ plugin invoke scheduler
2022-07-26 17:30.32 [INFO] message="Starting scheduler" important=context key=variable
<lots of airflow logs>
2022-07-26 17:30.32 [ERROR] message="Failed to start airflow: couldn't create db" error="/some/path doesn't exist" important=context

"bad" example:

$ plugin invoke scheduler
2022-07-26 17:30.32 [INFO] message="Starting airflow"
<lots of airflow logs>
<bare exceptions/trace backs>
2022-07-26 17:30.32 [ERROR] message=Failed to start airflow

In the future the spec will likely evolve to report errors in an explicitly structured form.

Invocation context

Each plugin is passed a context concisely describing its own execution context. This is purely passing a read-only context from meltano to the plugin, its not bidirectional at this time.

Each plugin is passed the path to a per plugin context file as a --context-file=path config option OR via env var. The plugin is not yet required to consume or utilize the context.

Context file info, jsonspec pending:

"meltano_invocation_context": {
   "env_prefix": "EXAMPLE_PLUGIN_",
   "command": "test"
   "args": [],
   "meltano_cli_path": ".meltano/run/bin",
   "workdir": ".",
} 

Config and env handling

Config should be sourced from the environment searching for ENV key prefixes. The context payload includes a field env_prefix that plugins can use to search for env based config options.

[pandemicsyn]

@aaronsteers in a convo you also mentioned possibly supporting a --context-string style option in addition file path. I think we could definitely include that or even start with that only (base64 encoded?).

[pandemicsyn]

Ok @aaronsteers @pnadolny13 updated draft spec based on our convo. Important bits at the top. Let me know if I missed anything.

---

Note - Spec changes

• We're dropping pre-invoke, post-invoke in the draft since we feel like the wrapper approach lends itself to implementing these directly - saving excess calls and decreasing complexity in the process. Some form of pre/post hooks will likely be added later but during this early period pre/post invocation tasks are expected to be controlled directly by the extension. As a result context is also being dropped, as it was only present in its current form to support pre/post hooks.
• We're explicitly allowing command :splat pass through pattern's to extension invoke and signaling of that via the describe command. This allows scenarios where authors want to only support specific commands, or scenario's where authors want to allow transparent pass through in a straight up "extension wraps another cli" scenario.
• Extensions should bundle required files within the extension so that a extension is essentially a complete "usable" package. Requiring file-bundles should be avoided.
• Rolling with "extension" as the term in this doc to prevent confusion during discussions in relations to existing plugins (e.g. Orchestrator, Utility, Singer, etc).
• No changes need to pip/python dependencies management (e.g. apache-airflow). Will continue to be managed as today at install time at the plugin level.

Initial required commands and syntax

extension invoke <specific-command> 
extension invoke <:splat> 
extension describe --format=[default=json, yaml?]
extension --help

pre-invoke & post-invoke

Warning
Dropped for draft spec

pre-invoke and post-invoke are called IMMEDIATELY before and after invoke is called. Both pre-invoke and post-invoke will be passed an invocation context describing what command is about to be invoked (see invocation context below). These commands may NOOP, but they must always be present, callable, and return a 0 exit code.~~

Error conditions:

• Any non-0 exit code will be reported to the user
• A non-0 exit code in pre-invoke will halt execution and the requested command will NOT be invoked~~

invoke

Analogous to a meltano invoke <command> [flags...], the extension invoke <command> call is expected to behave similarly and invoke the requested command. i.e.

$ extension invoke sync-ui # may sync with some dashboard ui
$ extension invoke deploy # may perform some other action

While a service pattern is not yet supported, future additional top-level commands are likely, and so extension developers not using the SDK may want to keep that in mind. i.e.

# possible future patterns
extension service ui start
extension service scheduler start
extension service all stop

describe

The describe command allows for auto discovery/configuration and generally describes the execution requirements of the extension. In v1 the payload is limited. Consisting only of commands which describes what the commands the extension supports.

Expected describe output, jsonspec pending:

$ extension describe
{
  commands: {description/list of invocable commands}
}

Exit codes

• 0 in all success cases
• All non 0 indicate unsuccessful invocation and in most cases will halt the execution chain

Special/reserved command keywords

Some commands may trigger special handling behavior upstream within Meltano. Currently, limited to:

• test* (test, test_a_thing, test-other-thing, testAllThings) which maybe triggered during meltano test.

If a extension exposes multiple matching commands, they're invoked in the order they're encountered. pre-invoke/post-invoke will be called for each command present.

Stdio behavior (Errors, Logging, and Output)

IO

• stdin is currently unused but reserved. It's unused at this time, but maybe used for message passing in the future.
• stdout is currently unused but reserved. It's unused at this time, but might be used for message passing in the future. extensions should avoid needlessly writing to stdout.
• stderr is available for structured extension output intended for end users and operators.

Errors & Output

extensions should emit plain (uncolored) structured log's to stderr. In the first version of this spec the structure isn't yet set, but developers are advised to follow the KISS principle and use a basic format. In the future, extensions may be required to support an optional JSON output format.

The following 3 fields should be present, with limited other ad hoc key/value pairs to supply context where needed:

• ISO formatted timestamp
• log level
• event (which should encapsulate any forwarded log lines from underlying calls the extension has made. e.g. airflow output)
• limited KV pairs for additional context as deemed useful

A small note on log level usage:

Debug and Info are generally the two most used and abused levels, but in the future, meltano will likely optimize around Warn and Error for passing on messaging to end users. With that in mind, it's important to call errors out explicitly via messages at the Error level on stderr. You should assume that an end user will be the primary consumer of a such a message - and so you should co-locate important context in the same log line or concisely explain what went wrong in the message if possible.

"Good" example:

$ extension invoke scheduler
2022-07-26 17:30.32 [INFO] message="Starting scheduler" important=context key=variable
<lots of airflow logs>
2022-07-26 17:30.32 [ERROR] message="Failed to start airflow: couldn't create db" error="/some/path doesn't exist" important=context

"bad" example:

$ extension invoke scheduler
2022-07-26 17:30.32 [INFO] message="Starting airflow"
<lots of airflow logs>
<bare exceptions/trace backs>
2022-07-26 17:30.32 [ERROR] message=Failed to start airflow

In the future, the spec will likely evolve to report errors in an explicitly structured form.

Invocation context

Warning
Dropped for draft spec

Each extension is passed a context concisely describing its own execution context. This is purely passing a read-only context from meltano to the extension, it's not bidirectional at this time.

Each extension is passed the path to a per extension context file as a --context-file=path config option OR via env var. The extension is not yet required to consume or utilize the context.

Context file info, jsonspec pending:

"meltano_invocation_context": {
   "env_prefix": "EXAMPLE_extension_",
   "command": "test"
   "args": [],
   "meltano_cli_path": ".meltano/run/bin",
   "workdir": ".",
} 

[pandemicsyn]

headsup - I transferred this discussion to the EDK repo.

@aaronsteers now that edk has landed - I need to update the spec, I can do it as another comment like we have in this discussion in the past but was thinking it might be nice to write up the current spec and implementer expectations and check them into the EDK repo as a living SPECIFICATION.md file alongside the README.md, wdyt?

[aaronsteers]

@pandemicsyn - Yes, good call!

Just so we're on the same page though, which of these would the primary bulk of content?

1. Meltano Plugin Specification: What a Meltano extension must do in terms of behaviors and contracts.
   • We didn't really change anything here so this might not be needed.
2. EDK dev specs: What the developer must do to properly implement the EDK for their own extension.
3. Implementation reference: What the EDK does behind the scenes from an "implementation choices" perspective. (The intersection/glue between 1 and 2.)

Some of these things (certainly not all) might have a good home in the Cookiecutter or in a Dev Guide, or in docstrings within the classes/methods.

[pandemicsyn]

@aaronsteers I was just referring to number 1 The extension spec. 2/3 seem more like they belong in a dev guide.

I suspect you're asking because I mentioned "implementer expectations"? I was referring to expectations we have for folks implementing the spec (i.e. like us as writers of the EDK package) so that if some enterprising developer comes along and wants to implement the EDK in Go or Rust they have some guidelines (like the default log format , fields, and flags, where to allow command shadowing and where not).

We didn't really change anything here so this might not be needed.

A lot changed during the PR review. Off the top of my head, these are probably the 4 big things we need to go back and update/reconcile:

• Our whole invocation pattern changed and wrappers now ship two commands instead of a single command.
• The initialize() command seems like it might be something we should codify or at least mention as an emerging pattern and likely future spec addition.
• The naming conventions we've settled on are worth mentioning.
• We should definitely at least mention that the log format and related options will probably become part of the spec eventually.

[aaronsteers]

@pandemicsyn - Sounds great. I definitely understand the benefit of compiling the spec work into a repo artifact at this point.