Add support for Secret Manager usage in JdbcToBigQuery

[philippebailer]

No description provided.

[bvolpato]

Thanks for the contribution, LGTM! Running the checks

[bvolpato]

Can you please mvn spotless:apply and commit? Other than that, it's looking good, thanks!

[brianhuangee]

@bvolpato I pushed up Spotless changes

[philippebailer]

@bvolpato any chance we could get another review?

[philippebailer]

@bharadwaj-aditya was looking at other PRs and saw you were active recently any chances you could review this and trigger the tests?

@bvolpato if your around as well just trying to get this merged if possibel

[philippebailer]

@shreyakhajanchi @bharadwaj-aditya @bvolpato

Just trying to get this reviewed I'm not sure what the correct way to get attention is so I've just been looking at recently reviewed PRs and pulling ppl sorry for being a bother but any guidance would be appreciated

[liferoad]

Will merge this later after the tests are done.

[philippebailer]

thank you! @liferoad

[liferoad]

@philippebailer can you check the failed tests?

[philippebailer]

Looking into it @liferoad, thanks for running the tests

[liferoad]

Can we also update the doc with one example to show how to use this?

[philippebailer]

Sure I can update the docs, do you mind pointing me to the docs that need updating? @liferoad

sorry we tried looking into the test failures last week and were pretty confused as to how they were happening and then things picked up at work and this got put on hold but I'll be circling back to this next week and hopefully will be able to resolve this.

I know that other templates that use the secrets manage have a selector so I am planning to follow those examples since I the current solution seems to have issues with the get and I'm just not familiar enough with all this to understand why.

[liferoad]

The doc is https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/jdbc-to-googlecloud/README_Jdbc_to_BigQuery_Flex.md

[geoffreylemaire]

Any update ?
I am also very interested in this feature

[zachberger]

Any update?

[damccorm]

It looks like this PR had failing checks which blocked merging. Logs for those tests have since expired, so not sure if its a false positive or real problem. @philippebailer could you please pull in the latest changes from master onto your branch? That will cause the checks to be rerun and will make sure no weird conflicts have come in since this PR was opened

EDIT: I tried this in #1937 - TBD on results

[damccorm]

I tried these changes out, and they don't work. If you try to stage this template, it errors with:

An exception occured while executing the Java class. Value only available at runtime, but accessed from a non-runtime context: RuntimeValueProvider{propertyName=connectionURL, default=null}

The underlying problem here is that we're trying to access the ValueProvider value as we construct the pipeline/template, but that value doesn't exist yet. The proper approach here would be to either:

1. (preferred) Extend the KMSEncryptedNestedValueProvider to handle this functionality directly
2. Implement a new value provider which handles this check as part of its functionality and shells out to KMSEncryptedNestedValueProvider

My only hesitation with (1) is that it impacts more templates than just this, so we need to make extra sure we're not doing something weirdly incompatible with other template options.