What should variant names be for EDK utilities?

[pnadolny13]

Related to discussion in #1135 (comment) and #1169 (review)

@tayloramurphy related to variant names and ^^ I still think Superset is still a weird case. We have apache for the direct pip install and meltano for the extension version, both would be apache if we went by the source code naming. I also think theres a chance that we will get multiple EDK variants that have different features, what do you think about doing - (or a similar combo of the 2 names) as the variant so we'd have:

• apache and apache-meltano for superset
• dbt-snowflake -> dbt-labs-meltano
• dbt-duckdb -> jwills-meltano

[tayloramurphy]

My goals with the current suggestion listed in the description are to avoid the assumption that a meltano variant of Superset is a fork of the superset codebase.

The downside of the concatenation approach is that it's hard to automatically parse which is the main repo and which is the EDK.

Perhaps we can get away with having the variant still be based on the EDK repo name but we just handle it appropriately on the Hub side. Since these currently will only be for Tools/Utilities we could mainly update the language on the Hub side to be less about variants and more about EDK alternatives.

For now though this is a lesser priority as we don't have anything aside from Superset that has multiple variants.

cc @aaronsteers

[aaronsteers]

I have always felt for utilities like dbt that the maintainer is the one maintaining the plugin definition and extension code, not that they are maintaining the wrapped tool.

So, I would be fine with saying all of our extensions are maintained by meltano or meltanolabs unless the extension itself is contributed by a community member - and then we can just use the normal rules for naming variants based on the parent GitHub org/owner. Changing variant would be a breaking change though at this point, so I'm hesitant to suggest anything different that what we have now for already-existing utilities.

[aaronsteers]

I just checked the utilities list, and it looks like only cron is shown as a meltano variant. Just as we understand cron itself isn't owned or maintained by Meltano, I think it's appropriate and understandable to say the Airflow integration itself (the plugin/extension/etc.) are maintained by meltano and not apache.

While not causing us pain today, I think this is important in the long-run because the implementation choices in the extension level can make a big impact on usability. Superset and Airflow are great examples where the glue code and implementation choices can vary widely - and if Pat as a community member does fork and create a variant of any of these, it would be misleading to have one variant listed as apache and one as pnadolny - that implies that the apache variant is maintained by the vendor and Pat's is not, whereas actually neither are.

[aaronsteers]

Regarding breaking changes when modifying the variant...

Modifying the variant on already-published plugins will disconnect the user from being able to update or re-lock the plugin. On closer consideration, maybe this is isn't a huge deal - as long as they have a lock file installed, it shouldn't break existing workloads... but still, it's worth considering if renaming variants is worth disconnecting the users from the future upgrade path of their plugin definitions.

[tayloramurphy]

@aaronsteers your arguments make sense to me. I think with our broader shift to having just connectors and tools/utilities we can make the UX clearer on the tool side about what a variant means.

I believe your concern about breaking existing installations can be overcome by having metadata versioning on the Hub. At a certain point we can create a new version for every bit of metadata on the hub and use that where needed and then just opt everyone into versioning their plugins on the Meltano side. Likely at a major Meltano release. But, like I said - we don't need to solve this immediately.

[aaronsteers]

@aaronsteers your arguments make sense to me. I think with our broader shift to having just connectors and tools/utilities we can make the UX clearer on the tool side about what a variant means.

👍 👍

I believe your concern about breaking existing installations can be overcome by having metadata versioning on the Hub.

Just wanted to call out metadata versioning most likely would not help here - as that versioning would be within a specific plugin ID, and the change I'm calling out above as breaking would specifically be the change to the ID itself. So, superset--apache could indeed be versioned, and superset--meltanolabs could be a new thing that is also versioned - but we wouldn't want to maintain definitions for both superset--apache and also superset--meltanolabs if they are representing the same plugin - and versioning would not on its own provide a bridge between plugins with different IDs.

I don't really see many any strong cases for metadata versioning as of now - except as an audit trail but git already gives us that. Users will have their lock file locally - and on-demand, they can update their lock file with the latest from the Hub. At some point before 3.0 we'll start failing or warning loudly for any plugins that don't have lock files.

I think the next most important requirement is that they can bump and freeze their pip URLs, which we're discussing in:

• Discussion on options to pin Python package versions and manage python version bump lifecycles meltano#7297

Agreed no rush to solve right away. I do just want to call out that I don't see metadata versioning offering relief for variant changes specifically.