Skip to content

Latest commit

 

History

History
158 lines (115 loc) · 5.12 KB

RELEASING.md

File metadata and controls

158 lines (115 loc) · 5.12 KB
Raw

Google Cloud Platform - Python Runtime Docker Image

build.sh

There is a shell script called build.sh that builds everything in this repository.

Environment variables for build.sh

DOCKER_NAMESPACE : The prefix applied to all images names created. To push images to Google Container Registry (GCR), this should be gcr.io/YOUR-PROJECT-NAME.

TAG : The suffix applied to all images created. This should be unique. If not specified, the current time will be used (timestamp format YYYY-mm-dd-HHMMSS).

GOOGLE_APPLICATION_CREDENTIALS_FOR_TESTS : (System test only) Path to service account credentials in JSON format.

GOOGLE_CLOUD_PROJECT_FOR_TESTS : (System test only) Name of the Google Cloud Platform project to run the system tests under.

Building and Releasing

A custom Jenkins job builds and releases this repository using scripts and job configurations that are not yet available publicly. The control flow is as follows:

  1. Jenkins job python/release is invoked by a. Manually running the script build_and_release.py with arguments b. Manually invoking the job from the GUI
  2. The job runs the script release.sh a. Service account credentials are read b. gcloud auth activate-service-account is performed c. gcloud config set project is performed
  3. The script invokes build.sh in this repository
  4. build.sh invokes Google Cloud Build with the cloudbuild-*.yaml config files.

Building interpreters

The interpreters used are now built in a separate step, and stored on GCS. This allows the runtime images to be build more rapidly.

To build the interpreters, run:

gcloud builds submit . --config=cloudbuild_interpreters.yaml

Building outside Jenkins

To build this repository outside Jenkins, authenticate and authorize yourself with gcloud auth, set the variables listed above, and run:

./build.sh

This assumes an environment similar to the internal Jenkins environment (Linux, Debian or Ubuntu-like).

Building locally

To build this repository using local Docker commands instead of the Google Cloud Build service, add the --local flag as shown:

./build.sh --local

To open an interactive shell session to this image after building it, do the following:

docker run -it --entrypoint /bin/bash YOUR-IMAGE-NAME

Running tests against a released image

To run compatibility tests against an existing image, such as gcr.io/google-appengine/python:latest, run:

DOCKER_NAMESPACE=gcr.io/google-appengine TAG=latest ./build.sh --nobuild --test

Running benchmarks

There is a benchmark suite which compares the performance of interpreters against each other.

**Benchmark different versions of interpreter in the same release

DOCKER_NAMESPACE=DOCKER_NAMESPACE_EXAMPLE TAG=TAG_EXAMPLE ./build.sh --nobuild --benchmark

**Benchmark same versions of interpreter from release to release

DOCKER_NAMESPACE=DOCKER_NAMESPACE_EXAMPLE TAG1=TAG1_EXAMPLE TAG2=TAG2_EXAMPLE ./benchmark_between_releases.sh

Since these benchmarks are run on cloud instances, the timings may vary from run to run.

Running system tests

TAKE NOTE: You will incur charges for use of Google Cloud Platform services!

System tests perform mutating operations against the real Google Cloud services. Since these system tests may fail or be flaky for outside reasons such as netorking issues, configuration errors, or services outages, they are run separately from building the images, and should be run in their own project.

To run the system tests, you need a Google Cloud Project with a service account. From the Google Cloud Console, either create a new project or switch to an existing one. Next, create a service account that will be used to run the system tests. Once you have a service account, create and download a service account key.

In the IAM & Admin section, grant the Owner role to the service account you created above. Also grant the Editor role to the [email protected] service account.

Then, follow the system test setup instructions. It describes various steps, including running some scripts to populate and/or delete datastore example data and indexes (populate_datastore.py, clear_datastore.py, and gcloud preview datastore create-indexes system_tests/data/index.yaml).

From the cloud console, you will need to enable at least the following APIs for your project:

  • Bigquery API
  • Cloud Bigtable Admin API
  • Cloud Spanner API
  • Google Cloud Natural Language API
  • Google Cloud Pub/Sub API
  • Google Cloud Speech API
  • Google Cloud Storage JSON API
  • Google Cloud Translation API
  • Google Cloud Vision API
  • Stackdriver Logging API
  • Stackdriver Monitoring API

Once all the setup has been done, run the following:

./build.sh --nobuild --system_tests