Seamlessly propagate schema changes made after pipelines starts running for Spanner Change Streams to BigQuery template #1678

ChangyuLi28 · 2024-06-21T16:07:09Z

This pr supports schema updates after pipeline starts running.

Before you add a new column to an existing tracked Cloud Spanner table, first add the column to the BigQuery changelog table. The new column must be NULLABLE. Then add the column to the Cloud Spanner table. The new column is automatically populated when the pipeline receives a new record with that column. It's recommended to wait for a couple of minutes after making schema updates in BigQuery to start the load including the new column because of this issue.
To add a new table, add the table in the Cloud Spanner database first. The table is automatically created when the pipeline receives a record for the new table.
The template doesn't drop tables or columns from BigQuery. If a column is dropped from the Cloud Spanner table, then null values are populated to these changelog columns for records generated after the columns are dropped from the Cloud Spanner table, unless you manually drop the column from BigQuery.
The template doesn't support column type updates or column nullable mode updates.

Follow this guide to deploy pipelines with this template. More specific instructions.

google-cla · 2024-06-21T16:07:13Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

ChangyuLi28 · 2024-06-21T16:11:17Z

Integration tests will be added in a future PR. Do not merge now.

codecov · 2024-07-18T17:27:34Z

Codecov Report

Attention: Patch coverage is 51.20275% with 142 lines in your changes missing coverage. Please review.

Project coverage is 42.92%. Comparing base (6b094c2) to head (1e87547).
Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
...tobigquery/schemautils/SpannerToBigQueryUtils.java	53.52%	24 Missing and 9 partials ⚠️
...eamstobigquery/SpannerChangeStreamsToBigQuery.java	0.00%	28 Missing ⚠️
...reamstobigquery/schemautils/SchemaUpdateUtils.java	60.00%	16 Missing and 6 partials ⚠️
...hangestreamstobigquery/schemautils/TypesUtils.java	58.33%	17 Missing and 3 partials ⚠️
...erchangestreamstobigquery/model/ModColumnType.java	33.33%	18 Missing ⚠️
...bigquery/FailsafeModJsonToTableRowTransformer.java	0.00%	12 Missing ⚠️
...streamstobigquery/BigQueryDynamicDestinations.java	0.00%	4 Missing ⚠️
...ates/spannerchangestreamstobigquery/model/Mod.java	62.50%	2 Missing and 1 partial ⚠️
...gestreamstobigquery/model/TrackedSpannerTable.java	83.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1678      +/-   ##
============================================
+ Coverage     42.83%   42.92%   +0.09%     
- Complexity     3424     3453      +29     
============================================
  Files           824      827       +3     
  Lines         48147    48275     +128     
  Branches       5168     5196      +28     
============================================
+ Hits          20624    20724     +100     
- Misses        25847    25864      +17     
- Partials       1676     1687      +11

Components	Coverage Δ
spanner-templates	`62.81% <ø> (-0.01%)`	⬇️
spanner-import-export	`63.88% <ø> (-0.03%)`	⬇️
spanner-live-forward-migration	`75.05% <ø> (ø)`
spanner-live-reverse-replication	`67.63% <ø> (ø)`
spanner-bulk-migration	`83.75% <ø> (ø)`

Files with missing lines	Coverage Δ
...ngestreamstobigquery/schemautils/OptionsUtils.java	`61.90% <ø> (-1.74%)`	⬇️
...igquery/schemautils/SpannerChangeStreamsUtils.java	`80.08% <100.00%> (+9.18%)`	⬆️
...gestreamstobigquery/model/TrackedSpannerTable.java	`61.53% <83.33%> (+13.39%)`	⬆️
...ates/spannerchangestreamstobigquery/model/Mod.java	`48.07% <62.50%> (+48.07%)`	⬆️
...streamstobigquery/BigQueryDynamicDestinations.java	`0.00% <0.00%> (ø)`
...bigquery/FailsafeModJsonToTableRowTransformer.java	`0.00% <0.00%> (ø)`
...erchangestreamstobigquery/model/ModColumnType.java	`33.33% <33.33%> (ø)`
...hangestreamstobigquery/schemautils/TypesUtils.java	`58.33% <58.33%> (ø)`
...reamstobigquery/schemautils/SchemaUpdateUtils.java	`60.00% <60.00%> (ø)`
...eamstobigquery/SpannerChangeStreamsToBigQuery.java	`0.00% <0.00%> (ø)`
... and 1 more

... and 2 files with indirect coverage changes

nancyxu123 · 2024-08-02T15:49:27Z

...leport/v2/templates/spannerchangestreamstobigquery/FailsafeModJsonToTableRowTransformer.java


        // For "INSERT" mod, we can get all columns from mod.
-        if (mod.getModType() == ModType.INSERT) {
+        // For "DELETE" mod, we can simply populate null for all tracking columns.


"For "DELETE" mod, we only set the key columns. For all non-key columns, we already populated "null".

Fixed the comment.

nancyxu123 · 2024-08-02T15:52:39Z

...leport/v2/templates/spannerchangestreamstobigquery/FailsafeModJsonToTableRowTransformer.java

@@ -181,7 +183,7 @@ public void setUp() {
                  "Caught exception when setting up FailsafeModJsonToTableRowFn, message: %s,"
                      + " cause: %s",
                  Optional.ofNullable(e.getMessage()), e.getCause()));
-          seenException = true;


Why was this removed?

Because we still want the first error message to be printed at later stage when process the element here.

nancyxu123 · 2024-08-02T16:08:28Z

@ShuranZhang could you also add thiago as reviewer?

thiagotnunes

LGTM, few suggestions

thiagotnunes · 2024-08-06T01:24:51Z

...oud/teleport/v2/templates/spannerchangestreamstobigquery/SpannerChangeStreamsToBigQuery.java

@@ -453,7 +476,7 @@ private static DeadLetterQueueManager buildDlqManager(
            ? tempLocation + "dlq/"
            : options.getDeadLetterQueueDirectory();

-    LOG.info("Dead letter queue directory: {}", dlqDirectory);
+    LOG.info("Dead letter queue directory: {}" + dlqDirectory);


This should be as it was ({} is replaced by the parameter)

thiagotnunes · 2024-08-06T01:26:15Z

...in/java/com/google/cloud/teleport/v2/templates/spannerchangestreamstobigquery/model/Mod.java

    this.modType = modType;
    this.valueCaptureType = valueCaptureType;
    this.numberOfRecordsInTransaction = numberOfRecordsInTransaction;
    this.numberOfPartitionsInTransaction = numberOfPartitionsInTransaction;
+    this.rowTypeAsMap = rowType == null ? Collections.emptyMap() : new HashMap<>();


nit: can't we always assign a new HashMap<>? Or do you want it to be unmodifiable when rowType is null?

I am also not sure on the intention as I was not the original author here.
@ChangyuLi28 Could you please take a look here? Thanks!

thiagotnunes · 2024-08-06T03:07:58Z

...va/com/google/cloud/teleport/v2/templates/spannerchangestreamstobigquery/TypesUtilsTest.java

@@ -0,0 +1,43 @@
+/*


I say we would need more tests here to improve the coverage of TypesUtils since it does a lot of String manipulation

Done. I added tests to cover all string manipulation branches inTypesUtils.java .

…ng for Spanner Change Streams to BigQuery template.

pull-request-size bot added the size/XXL label Jun 21, 2024

ChangyuLi28 marked this pull request as draft June 21, 2024 17:35

ShuranZhang force-pushed the main branch from d546c15 to b50ed03 Compare June 22, 2024 05:01

ShuranZhang force-pushed the main branch 2 times, most recently from ac239fc to c8c5336 Compare July 16, 2024 01:51

ChangyuLi28 marked this pull request as ready for review July 17, 2024 17:17

ShuranZhang force-pushed the main branch from 49f968a to 7a8f5dd Compare July 30, 2024 19:13

nancyxu123 approved these changes Aug 2, 2024

View reviewed changes

ShuranZhang force-pushed the main branch from 7a8f5dd to cf7912e Compare August 2, 2024 17:25

thiagotnunes approved these changes Aug 6, 2024

View reviewed changes

ChangyuLi28 and others added 4 commits August 26, 2024 22:54

Seamlessly propagate schema changes made after pipelines starts runni…

f2d6c5c

…ng for Spanner Change Streams to BigQuery template.

Resolve merge conflicts

ca2d649

Fix missing columns in bq when initial record is UPDATE or DELETE

9c53b3e

Improve test coverage for TypesUtils

1e87547

ShuranZhang force-pushed the main branch from de9b2a6 to 1e87547 Compare August 27, 2024 01:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Seamlessly propagate schema changes made after pipelines starts running for Spanner Change Streams to BigQuery template #1678

Seamlessly propagate schema changes made after pipelines starts running for Spanner Change Streams to BigQuery template #1678

ChangyuLi28 commented Jun 21, 2024

google-cla bot commented Jun 21, 2024

ChangyuLi28 commented Jun 21, 2024 •

edited

Loading

codecov bot commented Jul 18, 2024 •

edited

Loading

nancyxu123 Aug 2, 2024

ShuranZhang Aug 2, 2024

nancyxu123 Aug 2, 2024

ShuranZhang Aug 2, 2024

nancyxu123 commented Aug 2, 2024

thiagotnunes left a comment

thiagotnunes Aug 6, 2024

ShuranZhang Aug 6, 2024

thiagotnunes Aug 6, 2024

ShuranZhang Aug 6, 2024

thiagotnunes Aug 6, 2024

ShuranZhang Aug 6, 2024

Seamlessly propagate schema changes made after pipelines starts running for Spanner Change Streams to BigQuery template #1678

Are you sure you want to change the base?

Seamlessly propagate schema changes made after pipelines starts running for Spanner Change Streams to BigQuery template #1678

Conversation

ChangyuLi28 commented Jun 21, 2024

google-cla bot commented Jun 21, 2024

ChangyuLi28 commented Jun 21, 2024 • edited Loading

codecov bot commented Jul 18, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nancyxu123 commented Aug 2, 2024

thiagotnunes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChangyuLi28 commented Jun 21, 2024 •

edited

Loading

codecov bot commented Jul 18, 2024 •

edited

Loading