Port #1278 Add support for Secret Manager usage in JdbcToBigQuery

[damccorm]

Since #1278 is rather old and the author is no longer at Google, porting the PR here to make progress and hopefully unblock a use case

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.

Project coverage is 44.84%. Comparing base (0fa3bbf) to head (039e27d).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...oogle/cloud/teleport/templates/JdbcToBigQuery.java	0.00%	8 Missing ⚠️
.../java/org/apache/beam/it/gcp/TemplateTestBase.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1937      +/-   ##
============================================
+ Coverage     44.80%   44.84%   +0.03%     
- Complexity     3562     3879     +317     
============================================
  Files           832      832              
  Lines         49479    49484       +5     
  Branches       5196     5196              
============================================
+ Hits          22168    22189      +21     
+ Misses        25659    25642      -17     
- Partials       1652     1653       +1     

Components	Coverage Δ
spanner-templates	65.67% <ø> (+0.11%)	⬆️
spanner-import-export	63.80% <ø> (ø)
spanner-live-forward-migration	74.86% <ø> (ø)
spanner-live-reverse-replication	75.55% <ø> (ø)
spanner-bulk-migration	84.51% <ø> (+0.31%)	⬆️

Files with missing lines	Coverage Δ
.../java/org/apache/beam/it/gcp/TemplateTestBase.java	0.00% <0.00%> (ø)
...oogle/cloud/teleport/templates/JdbcToBigQuery.java	0.00% <0.00%> (ø)

... and 4 files with indirect coverage changes

[damccorm]

Running this, I see a bunch of failures like:

2024-10-14T23:53:07.3695559Z [ERROR] testMySqlToBigQueryClassic(com.google.cloud.teleport.templates.JdbcToBigQueryIT)  Time elapsed: 506.037 s  <<< ERROR!
2024-10-14T23:53:07.3698453Z org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Error staging template
2024-10-14T23:53:07.3700840Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2086)
2024-10-14T23:53:07.3702564Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache.get(LocalCache.java:4012)
2024-10-14T23:53:07.3704346Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4922)
2024-10-14T23:53:07.3705968Z 	at org.apache.beam.it.gcp.TemplateTestBase.getSpecPath(TemplateTestBase.java:272)
2024-10-14T23:53:07.3707362Z 	at org.apache.beam.it.gcp.TemplateTestBase.setUpBase(TemplateTestBase.java:219)
2024-10-14T23:53:07.3708665Z 	at jdk.internal.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
2024-10-14T23:53:07.3710409Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-10-14T23:53:07.3711827Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2024-10-14T23:53:07.3713306Z 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2024-10-14T23:53:07.3714917Z 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2024-10-14T23:53:07.3716386Z 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2024-10-14T23:53:07.3717923Z 	at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
2024-10-14T23:53:07.3719517Z 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
2024-10-14T23:53:07.3720953Z 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2024-10-14T23:53:07.3722461Z 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
2024-10-14T23:53:07.3723717Z 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2024-10-14T23:53:07.3725647Z 	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2024-10-14T23:53:07.3726963Z 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2024-10-14T23:53:07.3728900Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2024-10-14T23:53:07.3730703Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2024-10-14T23:53:07.3731974Z 	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2024-10-14T23:53:07.3733287Z 	at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:410)
2024-10-14T23:53:07.3734949Z 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2024-10-14T23:53:07.3736276Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2024-10-14T23:53:07.3738243Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2024-10-14T23:53:07.3740073Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2024-10-14T23:53:07.3741571Z 	at java.base/java.lang.Thread.run(Thread.java:829)
2024-10-14T23:53:07.3742684Z Caused by: java.lang.IllegalArgumentException: Error staging template
2024-10-14T23:53:07.3744445Z 	at org.apache.beam.it.gcp.TemplateTestBase.lambda$getSpecPath$0(TemplateTestBase.java:321)
2024-10-14T23:53:07.3747181Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4927)
2024-10-14T23:53:07.3749941Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3571)
2024-10-14T23:53:07.3752048Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2313)
2024-10-14T23:53:07.3754652Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2190)
2024-10-14T23:53:07.3756787Z 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2080)
2024-10-14T23:53:07.3757902Z 	... 26 more
2024-10-14T23:53:07.3758622Z Caused by: java.lang.RuntimeException: Error staging template, check Maven logs.
2024-10-14T23:53:07.3760118Z 	at org.apache.beam.it.gcp.TemplateTestBase.lambda$getSpecPath$0(TemplateTestBase.java:310)
2024-10-14T23:53:07.3761082Z 	... 31 more

[damccorm]

there are issues with this approach. I won't have time to handle them now, so I'm going to suggest them in the original pr and close here.