feat: add BigQueryJob library #14403

scotthart · 2024-06-28T21:59:29Z

As we need to keep bigquery_job separate from the existing bigquery target to avoid introducing REST dependencies, this required some massaging of our ci builds and generated inputs into those builds.

This PR is split into several commits to group related changes for easier reviewing.

This change is

codecov · 2024-06-28T22:35:03Z

Codecov Report

Attention: Patch coverage is 33.19328% with 318 lines in your changes missing coverage. Please review.

Project coverage is 93.44%. Comparing base (6505814) to head (5d79418).

Files	Patch %	Lines
...gquery/job/v2/internal/job_rest_connection_impl.cc	7.31%	76 Missing ⚠️
...le/cloud/bigquery/job/v2/internal/job_rest_stub.cc	5.79%	65 Missing ⚠️
...bigquery/job/v2/internal/job_tracing_connection.cc	6.66%	42 Missing ⚠️
...uery/job/v2/internal/job_rest_logging_decorator.cc	15.15%	28 Missing ⚠️
google/cloud/bigquery/job/v2/job_client.cc	17.64%	28 Missing ⚠️
...ery/job/v2/internal/job_rest_metadata_decorator.cc	21.87%	25 Missing ⚠️
...gquery/job/v2/job_connection_idempotency_policy.cc	15.78%	16 Missing ⚠️
google/cloud/bigquery/job/v2/job_connection.cc	6.66%	14 Missing ⚠️
google/cloud/bigquery/job/v2/job_connection.h	58.33%	10 Missing ⚠️
...igquery/job/v2/internal/job_rest_connection_impl.h	20.00%	8 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #14403      +/-   ##
==========================================
- Coverage   93.58%   93.44%   -0.14%     
==========================================
  Files        2313     2332      +19     
  Lines      206885   207361     +476     
==========================================
+ Hits       193617   193776     +159     
- Misses      13268    13585     +317

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

snippet-bot · 2024-06-29T00:26:48Z

Here is the summary of possible violations 😱

There is a possible violation for not having product prefix.

google/cloud/bigquery/quickstart/quickstart_job.cc:15, tag bigqueryjob_quickstart

The end of the violation section. All the stuff below is FYI purposes only.

Here is the summary of changes.

You are about to add 1 region tag.

google/cloud/bigquery/quickstart/quickstart_job.cc:15, tag bigqueryjob_quickstart

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

coryan

I would wait for @dbolduc to review the CMake and Bazel files.

coryan · 2024-06-29T13:40:49Z

google/cloud/bigquery/quickstart/quickstart_job.cc

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! [START bigqueryjob_quickstart] [all]


Consider

Suggested change

//! [START bigqueryjob_quickstart] [all]

//! [all]

or

//! [START bigquery_job_quickstart] [all]

coryan · 2024-06-29T13:41:04Z

google/cloud/bigquery/quickstart/quickstart_job.cc

+  std::cerr << "google::cloud::Status thrown: " << status << "\n";
+  return 1;
+}
+//! [END bigqueryjob_quickstart] [all]


Remember to make a matching change.

dbolduc

I am not sure I agree with where we are placing the library. I think the point is that adding the jobs/ directory makes globbing easier when we introduce bigquery_admin.

It still feels like it'd be easier for us (and our customers build scripts) to combine them into one library.

I really think we should consider google/cloud/bigquery_job/¹ with all the scaffolding that comes with it. Code-wise bigquery_job is independent of bigquery.

Pro:

all of the customization goes away. i.e. the library is no different than sql.

Con:

in v3, we will probably combine the bigquery libraries? If we want to move it later, we can leave forwarding headers, though.
underscore in directory name is not cool with vcpkg. The fix is not too bad though.

Neutral:

docs are separate

WDYT?

or google/cloud/bigquery_rest/ ↩

dbolduc · 2024-07-01T04:16:57Z

generator/generator_config.textproto

@@ -445,6 +445,15 @@ service {
  emitted_rpcs: ["ScheduleTransferRuns"]
 }

+service {
+  service_proto_path: "google/cloud/bigquery/v2/job.proto"
+  product_path: "google/cloud/bigquery/job/v2"


Why not mimic the googleapis structure: google/cloud/bigquery/v2 ?

It would not conflict with our google/cloud/bigquery/v2/minimal

dbolduc · 2024-07-01T04:24:33Z

generator/standalone_main.cc

@@ -182,6 +182,14 @@ int WriteInstallDirectories(
                                    "_mocks");
    }
  }
+
+  // TODO(#14402): Consider enhancing the generator_config.proto to allow custom


Not that this is awesome, but the cmake-install script has a place to enumerate special directions. We use it for storage_grpc:

google-cloud-cpp/ci/cloudbuild/builds/cmake-install.sh

Lines 145 to 159 in e9352ce

./lib64/cmake/google_cloud_cpp_bigquery_rest

./lib64/cmake/google_cloud_cpp_common

./lib64/cmake/google_cloud_cpp_compute_protos

./lib64/cmake/google_cloud_cpp_googleapis

./lib64/cmake/google_cloud_cpp_grafeas

./lib64/cmake/google_cloud_cpp_grpc_utils

./lib64/cmake/google_cloud_cpp_logging_type

./lib64/cmake/google_cloud_cpp_iam_v2

./lib64/cmake/google_cloud_cpp_mocks

./lib64/cmake/google_cloud_cpp_oauth2

./lib64/cmake/google_cloud_cpp_opentelemetry

./lib64/cmake/google_cloud_cpp_rest_internal

./lib64/cmake/google_cloud_cpp_rest_protobuf_internal

./lib64/cmake/google_cloud_cpp_storage_grpc

./lib64/cmake/google_cloud_cpp_storage_grpc_mocks

dbolduc · 2024-07-01T04:35:18Z

ci/cloudbuild/builds/checkers.sh

@@ -211,6 +211,7 @@ time {

  mapfile -t libraries < <(features::libraries)
  for library in "${libraries[@]}" opentelemetry; do
+    [[ $library == "bigquery_job" ]] && continue


Will this library have a README?

Where are we going to say how to build the BQ job library? Something needs to say "Add bigquery_job to the GOOGLE_CLOUD_CPP_ENABLE=... option."

Storage does this:

google-cloud-cpp/google/cloud/storage/doc/storage-grpc.dox

Line 74 in e9352ce

`google-cloud-cpp::%experimental-storage_grpc` instead

but it doesn't mention GOOGLE_CLOUD_CPP_ENABLE=... :/

I guess we rely on:

google-cloud-cpp/doc/compile-time-configuration.md

Line 38 in e9352ce

- `experimental-storage_grpc` enables the GCS+gRPC plugin. Contact your account

dbolduc · 2024-07-01T04:45:36Z

google/cloud/bigquery/google_cloud_cpp_bigquery_job.bzl

@@ -0,0 +1,37 @@
+# Copyright 2024 Google LLC


Remove unused file. We use globs in the build scripts.

dbolduc · 2024-07-01T04:51:13Z

google/cloud/bigquery/google_cloud_cpp_bigquery_job.cmake

+    ${proto_list} PROTO_PATH_DIRECTORIES "${EXTERNAL_GOOGLEAPIS_SOURCE}"
+    "${PROTO_INCLUDE_DIR}")
+external_googleapis_set_version_and_alias(bigquery_v2_protos)
+target_link_libraries(google_cloud_cpp_bigquery_v2_protos PUBLIC ${proto_deps})


just looking ahead, bigquery_admin will need this too. We cannot recreate the target.

We will probably want to factor out a common function like:

if (NOT TARGET google_cloud_cpp_bigquery_v2_protos) # define proto library endif ()

dbolduc · 2024-07-01T04:59:15Z

cmake/GoogleCloudCppLibrary.cmake

@@ -198,6 +198,18 @@ function (google_cloud_cpp_add_gapic_library library display_name)
        endif ()
    endforeach ()

+    foreach (dir IN LISTS _opt_ADDITIONAL_DOXYGEN_DIRS)


So the point of this is for bigquery_job docs to show up under cloud.google.com/cpp/docs/reference/bigquery/latest

Do the docs look okay?

e.g. we won't have a bigquery_v2_protos here. I am not sure what that results in.

google-cloud-cpp/cmake/GoogleCloudCppLibrary.cmake

Line 215 in a96b6d9

"google-cloud-cpp::${library}_protos")

dbolduc · 2024-07-01T05:01:54Z

google/cloud/bigquery/quickstart/quickstart_job.cc

+#include "google/cloud/bigquery/job/v2/job_client.h"
+#include <iostream>
+
+namespace {}  // namespace


dbolduc · 2024-07-01T05:03:10Z

google/cloud/bigquery/quickstart/quickstart_job.cc

+
+  bigquery_proto::ListJobsRequest list_request;
+  list_request.set_project_id(project_id);
+  std::vector<std::string> job_ids;


unused? We must not be running clang-tidy on this

I think you want to add bigquery_job here:

google-cloud-cpp/ci/lib/shard.sh

Lines 41 to 44 in 54274c0

readonly BIGQUERY_SHARD=(

bigquery

experimental-bigquery_rest

)

dbolduc · 2024-07-01T05:31:32Z

google/cloud/bigquery/quickstart/CMakeLists.txt

@@ -30,3 +30,10 @@ endif ()
 # Once the bigquery_client package is found, define new targets.
 add_executable(quickstart quickstart.cc)
 target_link_libraries(quickstart google-cloud-cpp::bigquery)
+
+find_package(google_cloud_cpp_bigquery_job)


find_package(google_cloud_cpp_bigquery_job CONFIG)

I think we should consider google/cloud/bigquery_foo/

scotthart temporarily deployed to internal June 28, 2024 21:59 — with GitHub Actions Inactive

scotthart force-pushed the bigquery_job_service branch from c334d4d to 55c7658 Compare June 28, 2024 23:14

scotthart temporarily deployed to internal June 28, 2024 23:14 — with GitHub Actions Inactive

scotthart temporarily deployed to internal June 28, 2024 23:55 — with GitHub Actions Inactive

scotthart force-pushed the bigquery_job_service branch from 791d672 to 80e648d Compare June 28, 2024 23:55

scotthart temporarily deployed to internal June 28, 2024 23:56 — with GitHub Actions Inactive

scotthart added 5 commits June 28, 2024 19:57

feat: add BigQueryJob library

049a267

update builds to accommodate bigquery_job as a feature

a96b6d9

add bazel and cmake support

79d644f

add quickstart

9f2b0de

generated code

5d79418

scotthart force-pushed the bigquery_job_service branch from 80e648d to 5d79418 Compare June 28, 2024 23:57

scotthart temporarily deployed to internal June 28, 2024 23:58 — with GitHub Actions Inactive

scotthart marked this pull request as ready for review June 29, 2024 00:26

scotthart requested a review from a team as a code owner June 29, 2024 00:26

coryan previously approved these changes Jun 29, 2024

View reviewed changes

dbolduc reviewed Jul 1, 2024

View reviewed changes

scotthart closed this Jul 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add BigQueryJob library #14403

feat: add BigQueryJob library #14403

scotthart commented Jun 28, 2024 •

edited by coryan

Loading

codecov bot commented Jun 28, 2024 •

edited

Loading

snippet-bot bot commented Jun 29, 2024

coryan left a comment

coryan Jun 29, 2024

coryan Jun 29, 2024

dbolduc left a comment •

edited

Loading

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

dbolduc Jul 1, 2024

	./lib64/cmake/google_cloud_cpp_bigquery_rest
	./lib64/cmake/google_cloud_cpp_common
	./lib64/cmake/google_cloud_cpp_compute_protos
	./lib64/cmake/google_cloud_cpp_googleapis
	./lib64/cmake/google_cloud_cpp_grafeas
	./lib64/cmake/google_cloud_cpp_grpc_utils
	./lib64/cmake/google_cloud_cpp_logging_type
	./lib64/cmake/google_cloud_cpp_iam_v2
	./lib64/cmake/google_cloud_cpp_mocks
	./lib64/cmake/google_cloud_cpp_oauth2
	./lib64/cmake/google_cloud_cpp_opentelemetry
	./lib64/cmake/google_cloud_cpp_rest_internal
	./lib64/cmake/google_cloud_cpp_rest_protobuf_internal
	./lib64/cmake/google_cloud_cpp_storage_grpc
	./lib64/cmake/google_cloud_cpp_storage_grpc_mocks

	readonly BIGQUERY_SHARD=(
	bigquery
	experimental-bigquery_rest
	)

feat: add BigQueryJob library #14403

feat: add BigQueryJob library #14403

Conversation

scotthart commented Jun 28, 2024 • edited by coryan Loading

codecov bot commented Jun 28, 2024 • edited Loading

Codecov Report

snippet-bot bot commented Jun 29, 2024

coryan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbolduc left a comment • edited Loading

Choose a reason for hiding this comment

Footnotes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scotthart commented Jun 28, 2024 •

edited by coryan

Loading

codecov bot commented Jun 28, 2024 •

edited

Loading

dbolduc left a comment •

edited

Loading