Isolated sign #41
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -20,11 +20,14 @@ on: | |||||
| - '' # without this, it's technically "required" 🙃 | ||||||
| - 2022 | ||||||
| - 2019 | ||||||
| pruneArtifact: | ||||||
| required: true | ||||||
| type: boolean | ||||||
| default: true | ||||||
| run-name: '${{ inputs.bashbrewArch }}: ${{ inputs.firstTag }} (${{ inputs.buildId }})' | ||||||
| permissions: | ||||||
| contents: read | ||||||
| actions: write # for https://github.com/andymckay/cancel-action (see usage below) | ||||||
| concurrency: | ||||||
| group: ${{ github.event.inputs.buildId }} | ||||||
| cancel-in-progress: false | ||||||
|
|
@@ -34,9 +37,20 @@ defaults: | |||||
| env: | ||||||
| BUILD_ID: ${{ inputs.buildId }} | ||||||
| BASHBREW_ARCH: ${{ inputs.bashbrewArch }} | ||||||
|
|
||||||
| # the image we'll run to access the signing tool | ||||||
| # https://explore.ggcr.dev/?repo=docker/image-signer-verifier | ||||||
| IMAGE_SIGNER: 'docker/image-signer-verifier:0.6.9@sha256:e38be2b9e6f010cf3432a2772b0e800feee572f7733c6df81e21293cb3e977e0' | ||||||
|
|
||||||
| # Docker Hub repository we'll push the (signed) attestation artifacts to | ||||||
| REFERRERS_REPO: oisupport/referrers | ||||||
| jobs: | ||||||
| build: | ||||||
| name: Build ${{ inputs.buildId }} | ||||||
| outputs: | ||||||
| json: ${{ steps.json.outputs.json }} | ||||||
| artifactUrl: ${{ steps.oci.outputs.artifact-url }} | ||||||
| artifactSha256: ${{ steps.sha256.outputs.sha256 }} | ||||||
| runs-on: ${{ inputs.bashbrewArch == 'windows-amd64' && format('windows-{0}', inputs.windowsVersion) || 'ubuntu-latest' }} | ||||||
| steps: | ||||||
|
|
||||||
|
|
@@ -148,135 +162,243 @@ jobs: | |||||
| fi | ||||||
| eval "$shell" | ||||||
|
|
||||||
| # save the build as an "artifact" so we can sign it in a separate step (and minimize the exposure of the signing credentials); also save a checksum so we can be sure it transmits between the steps accurately | ||||||
| # TODO what do we do here for "classic" builds like Windows? (no "temp" OCI layout currently) | ||||||
| - name: Generate Artifact | ||||||
| id: sha256 | ||||||
| run: | | ||||||
| tar -cvf temp.tar -C build temp | ||||||
| sha256="$(sha256sum temp.tar | cut -d' ' -f1)" | ||||||
| echo "sha256=$sha256" >> "$GITHUB_OUTPUT" | ||||||
| - name: Upload Artifact | ||||||
| id: oci | ||||||
| uses: actions/upload-artifact@v4 | ||||||
| with: | ||||||
| name: build-oci | ||||||
| path: | | ||||||
| temp.tar | ||||||
| retention-days: 5 | ||||||
|
|
||||||
| sign: | ||||||
| name: Sign | ||||||
| needs: build | ||||||
| if: fromJSON(needs.build.outputs.json).shouldSign | ||||||
| runs-on: ubuntu-latest | ||||||
| permissions: | ||||||
| contents: read | ||||||
| id-token: write # for AWS KMS signing (see usage below) | ||||||
| steps: | ||||||
| - uses: actions/checkout@v4 | ||||||
| with: | ||||||
| sparse-checkout-cone-mode: 'false' | ||||||
| sparse-checkout: | | ||||||
| .scripts/oci.jq | ||||||
| .scripts/provenance.jq | ||||||
|
Comment on lines
+194
to
+196
There was a problem hiding this comment. In this repository, On the other hand, I'm not personally super concerned about this, and think a checkout just like the one above would be fine. If we want something like the intent of this, I think we should front-load it to the previous step instead, which is really the step that performed the "build" anyhow so arguably should be validating the build and generating Edit: with my following suggested changes, this becomes irrelevant and we can drop |
||||||
| - name: Download Artifact | ||||||
| uses: actions/download-artifact@v4 | ||||||
| with: | ||||||
| name: build-oci | ||||||
| - name: Extract Artifact | ||||||
| env: | ||||||
| sha256: ${{ needs.build.outputs.artifactSha256 }} | ||||||
| run: | | ||||||
| sha256sum <<<"$sha256 *temp.tar" --strict --check - | ||||||
| tar -xvf temp.tar | ||||||
| [ -d temp ] # basic "valid JSON" check | ||||||
| - name: Configure AWS (for signing) | ||||||
| # https://github.com/aws-actions/configure-aws-credentials/releases | ||||||
| uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4.0.1 | ||||||
| with: | ||||||
| aws-region: ${{ github.ref_name == 'main' && secrets.AWS_KMS_PROD_REGION || secrets.AWS_KMS_STAGE_REGION }} | ||||||
| role-to-assume: ${{ github.ref_name == 'main' && secrets.AWS_KMS_PROD_ROLE_ARN || secrets.AWS_KMS_STAGE_ROLE_ARN }} | ||||||
| - name: Generate Provenance | ||||||
|
|
||||||
| env: | ||||||
| json: ${{ needs.build.outputs.json }} | ||||||
| GITHUB_CONTEXT: ${{ toJson(github) }} | ||||||
| run: | | ||||||
| image-digest() { | ||||||
| local dir="$1/blobs" | ||||||
| img=$( | ||||||
| grep -R --include "*" '"mediaType":\s"application/vnd.oci.image.layer.' "$dir" \ | ||||||
| | head -n 1 \ | ||||||
| | cut -d ':' -f1 | ||||||
| ) | ||||||
| [ "$(cat $img | jq -r '.mediaType')" = "application/vnd.oci.image.manifest.v1+json" ] || exit 1 | ||||||
| echo $img | rev | cut -d '/' -f2,1 --output-delimiter ':' | rev | ||||||
| } | ||||||
|
|
||||||
| digest=$(image-digest temp) | ||||||
|
|
||||||
| echo $json | jq -L.scripts --argjson github '${{ env.GITHUB_CONTEXT }}' --argjson runner '${{ toJson(runner) }}' --arg digest ${digest} ' | ||||||
| include "provenance"; | ||||||
| github_actions_provenance($github; $runner; $digest) | ||||||
| ' >> provenance.json | ||||||
| - name: Sign | ||||||
| env: | ||||||
| AWS_KMS_REGION: ${{ github.ref_name == 'main' && secrets.AWS_KMS_PROD_REGION || secrets.AWS_KMS_STAGE_REGION }} | ||||||
| AWS_KMS_KEY_ARN: ${{ github.ref_name == 'main' && secrets.AWS_KMS_PROD_KEY_ARN || secrets.AWS_KMS_STAGE_KEY_ARN }} | ||||||
|
|
||||||
| DOCKER_HUB_USERNAME: ${{ secrets.DOCKER_HUB_SIGNING_USERNAME }} | ||||||
| DOCKER_HUB_PASSWORD: ${{ secrets.DOCKER_HUB_SIGNING_PASSWORD }} | ||||||
| run: | | ||||||
| validate-oci-layout() { | ||||||
|
There was a problem hiding this comment. We can drop this entirely now (handled elsewhere). |
||||||
| local dir="$1" | ||||||
| jq -L.scripts -s ' | ||||||
| include "oci"; | ||||||
| validate_oci_layout | true | ||||||
| ' "$dir/oci-layout" "$dir/index.json" || return "$?" | ||||||
| local manifest | ||||||
| manifest="$dir/blobs/$(jq -r '.manifests[0].digest | sub(":"; "/")' "$dir/index.json")" || return "$?" | ||||||
| jq -L.scripts -s ' | ||||||
| include "oci"; | ||||||
| if length != 1 then | ||||||
| error("unexpected image index document count: \(length)") | ||||||
| else .[0] end | ||||||
| | validate_oci_index | ||||||
|
|
||||||
| # TODO more validation? | ||||||
| ' "$manifest" || return "$?" | ||||||
| } | ||||||
|
|
||||||
| dockerArgs=( | ||||||
| --interactive | ||||||
| --rm | ||||||
| --read-only | ||||||
| --workdir /tmp # see "--tmpfs" below (TODO the signer currently uses PWD as TMPDIR -- something to fix in the future so we can drop this --workdir and only keep --tmpfs perhaps adding --env TMPDIR=/tmp if necessary) | ||||||
| ) | ||||||
| if [ -t 0 ] && [ -t 1 ]; then | ||||||
| dockerArgs+=( --tty ) | ||||||
| fi | ||||||
|
|
||||||
| user="$(id -u)" | ||||||
| dockerArgs+=( --tmpfs "/tmp:uid=$user" ) | ||||||
| user+=":$(id -g)" | ||||||
| dockerArgs+=( --user "$user" ) | ||||||
|
|
||||||
| awsEnvs=( "${!AWS_@}" ) | ||||||
| dockerArgs+=( "${awsEnvs[@]/#/--env=}" ) | ||||||
|
|
||||||
| validate-oci-layout temp | ||||||
|
|
||||||
| # Login to Docker Hub | ||||||
| export DOCKER_CONFIG="$PWD/.docker" | ||||||
| mkdir "$DOCKER_CONFIG" | ||||||
| trap 'find "$DOCKER_CONFIG" -type f -exec shred -fuvz "{}" + || :; rm -rf "$DOCKER_CONFIG"' EXIT | ||||||
| docker login --username "$DOCKER_HUB_USERNAME" --password-stdin <<<"$DOCKER_HUB_PASSWORD" | ||||||
| unset DOCKER_HUB_USERNAME DOCKER_HUB_PASSWORD | ||||||
|
|
||||||
| # Create signatures | ||||||
| dockerArgs+=( | ||||||
| --mount "type=bind,src=$PWD/temp,dst=/doi-build/image,ro" # TODO this currently assumes normalized_builder == "buildkit" and !should_use_docker_buildx_driver -- we need to factor that in later (although this signs the attestations, not the image, so buildkit/buildx is the only builder whose output we *can* sign right now) | ||||||
| --mount "type=bind,src=$PWD/provenance.json,dst=/doi-build/provenance.json,ro" | ||||||
| --mount "type=bind,src=$DOCKER_CONFIG,dst=/docker-config,ro" | ||||||
| --env DOCKER_CONFIG=/docker-config # TODO verify that image-signer supports this environment variable correctly | ||||||
|
|
||||||
| # https://explore.ggcr.dev/?repo=docker/image-signer-verifier | ||||||
| "$IMAGE_SIGNER" | ||||||
|
There was a problem hiding this comment. Since this is the only place it's referenced, we could move this back here (ie, no
Suggested change
There was a problem hiding this comment. Same as previous one. It will be used more than once when we add VSA. |
||||||
| ) | ||||||
|
|
||||||
| kmsArg=( | ||||||
| # kms key used to sign attestation artifacts | ||||||
| --kms="AWS" | ||||||
| --kms-region="$AWS_KMS_REGION" | ||||||
| --kms-key-ref="$AWS_KMS_KEY_ARN" | ||||||
|
|
||||||
| --referrers-dest="$REFERRERS_REPO" # repo to store attestation artifacts and provenance | ||||||
| ) | ||||||
|
|
||||||
| # Sign buildkit statements | ||||||
| signArgs=( | ||||||
| "${kmsArg[@]}" | ||||||
| --input=oci:///doi-build/image | ||||||
| --keep=true # keep preserves the unsigned attestations generated by buildkit | ||||||
|
There was a problem hiding this comment. I recall discussions around not needing this anymore because nothing we're doing munges the input -- is that accurate? Is this still necessary? There was a problem hiding this comment. @kipz do you know? There was a problem hiding this comment. The |
||||||
| ) | ||||||
|
|
||||||
| docker run "${dockerArgs[@]}" sign "${signArgs[@]}" | ||||||
|
|
||||||
| # Attach and sign provenance | ||||||
| provArgs=( | ||||||
| "${kmsArg[@]}" | ||||||
| --image=oci:///doi-build/image | ||||||
| --statement="/doi-build/provenance.json" | ||||||
| ) | ||||||
| docker run "${dockerArgs[@]}" attest "${provArgs[@]}" | ||||||
|
|
||||||
| push: | ||||||
| name: Push | ||||||
| needs: | ||||||
| - build | ||||||
| - sign | ||||||
LaurentGoderre marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| # - verify | ||||||
| if: ${{ always() }} | ||||||
|
There was a problem hiding this comment. We don't want this to still run on failure, right? I guess this is how you're managing the issue with the There was a problem hiding this comment. We want to clear the artifact on failure and there is an extra setting to not clear it for debugging (that's what the of does) There was a problem hiding this comment. Ok, collapsing both comments into this one, I was able to use |
||||||
| runs-on: ubuntu-latest | ||||||
| steps: | ||||||
| - run: ${{ (needs.sign.result == 'skipped' && needs.build.result == 'success') || needs.verify.result == 'success' || 'exit 1' }} | ||||||
| - name: Download Artifact | ||||||
| uses: actions/download-artifact@v4 | ||||||
| with: | ||||||
| name: build-oci | ||||||
| - name: Extract Artifact | ||||||
| env: | ||||||
| sha256: ${{ needs.build.outputs.artifactSha256 }} | ||||||
| run: | | ||||||
| sha256sum <<<"$sha256 *temp.tar" --strict --check - | ||||||
| tar -xvf temp.tar | ||||||
| [ -d temp ] # basic "valid JSON" check | ||||||
| - name: Tools | ||||||
| run: | | ||||||
| mkdir .gha-bin | ||||||
| echo "$PWD/.gha-bin" >> "$GITHUB_PATH" | ||||||
|
|
||||||
| case "${RUNNER_ARCH}" in \ | ||||||
| X64) ARCH='amd64';; \ | ||||||
| esac | ||||||
|
|
||||||
| _download() { | ||||||
| local target="$1"; shift | ||||||
| local url="$1"; shift | ||||||
| wget --timeout=5 -O "$target" "$url" --progress=dot:giga | ||||||
| } | ||||||
|
|
||||||
| # https://doi-janky.infosiftr.net/job/wip/job/crane | ||||||
| _download ".gha-bin/crane" "https://doi-janky.infosiftr.net/job/wip/job/crane/lastSuccessfulBuild/artifact/crane-$ARCH" | ||||||
| # TODO checksum verification ("checksums.txt") | ||||||
| chmod +x ".gha-bin/crane" | ||||||
| ".gha-bin/crane" version | ||||||
| - name: Push | ||||||
| env: | ||||||
| DOCKER_HUB_USERNAME: ${{ secrets.DOCKER_HUB_USERNAME }} | ||||||
| DOCKER_HUB_PASSWORD: ${{ secrets.DOCKER_HUB_PASSWORD }} | ||||||
| json: ${{ needs.build.outputs.json }} | ||||||
| run: | | ||||||
| export DOCKER_CONFIG="$PWD/.docker" | ||||||
| mkdir "$DOCKER_CONFIG" | ||||||
| trap 'find "$DOCKER_CONFIG" -type f -exec shred -fuvz "{}" + || :; rm -rf "$DOCKER_CONFIG"' EXIT | ||||||
| docker login --username "$DOCKER_HUB_USERNAME" --password-stdin <<<"$DOCKER_HUB_PASSWORD" | ||||||
| unset DOCKER_HUB_USERNAME DOCKER_HUB_PASSWORD | ||||||
|
|
||||||
| shell="$(jq <<<"$json" -r '.commands.push')" | ||||||
| eval "$shell" | ||||||
|
|
||||||
| clean: | ||||||
| name: Cleanup | ||||||
| needs: | ||||||
| - build | ||||||
| - sign | ||||||
| - push | ||||||
| if: ${{ always() && inputs.pruneArtifact }} | ||||||
| runs-on: ubuntu-latest | ||||||
| steps: | ||||||
| - name: Clean Up Artifact | ||||||
| env: | ||||||
| ARTIFACT_URL: ${{ needs.build.outputs.artifactUrl }} | ||||||
| TOKEN: ${{ github.token }} | ||||||
| run: | | ||||||
| url="${ARTIFACT_URL/\/\/github\.com///api.github.com/repos}" | ||||||
| url="${url/runs\/*\/artifacts/artifacts}" # Translate web URL to API url | ||||||
| curl -L \ | ||||||
| -X DELETE \ | ||||||
| -H "Accept: application/vnd.github+json" \ | ||||||
| -H "Authorization: Bearer $TOKEN" \ | ||||||
| -H "X-GitHub-Api-Version: 2022-11-28" \ | ||||||
| "$url" | ||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These should move down to be part of
jobs.sign.envor better yet evenjobs.sign.steps[].envfor the specific step that needs them (given there's only the one).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is only one of the iteration but it would be used more than once once we have the VSA in. It's not as obvious here but I wanted to do it like this to minimize the changrs