Load files from GCS to BigQuery - Possibleways

[PratapKumarChowdary]

Hi Team,

I'm looking for possible ways to load our GCS files which were stored in a way month/day/hour to bigquery. And we would like to remove duplicates by scanning the record based on id and timestamp in json file and move the latest record to bigquery. Could you please help the possible ways to implement this.

[sydney-munro]

Hi while we do not have any managed solutions here are a couple of resources for you to take a look at.

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv

https://stackoverflow.com/questions/10671266/load-data-from-google-cloud-storage-to-bigquery-using-java

Please open/reopen this ticket if you have any further storage specific questions.

Bigquery specific questions can go here:
https://github.com/googleapis/java-bigquery