VGN Departure Extractor

The VGN Departure Extractor is a data pipeline that extracts departure information from Nuremberg's public transport company VGN and saves it to Google Cloud Storage which is then read by an external BigQuery table. The whole pipeline can easily be deployed to Google Cloud Platform (GCP) with Terraform.

Architecture

GCP resources used in this are Cloud Scheduler, Cloud Functions, Cloud Tasks, Cloud Storage, and BigQuery.

Deployment

You need a GCP project with billing enabled. Also create a Service Account with the role Owner. Export a Key as JSON file and refer it in the following environment variables.

export TF_VAR_credentials_file=path/to/service/account.json
export TF_VAR_project_id=my-gcp-project

terraform init
terraform validate
terraform plan
terraform apply

After terraform apply was executed successfully, the pipeline is executing automatically as defined in the Cloud Scheduler.