feat: store artifacts in cache by default #399

knqyf263 · 2024-10-04T13:53:22Z

Description

Store DBs and checks bundle in the cache by default by using actions/cache to mitigate 429 TOOMANYREQUESTS.

Rewrite a docker action to a composite action
- Remove Dockerfile
Use actions/cache to store DBs and checks bundle in cache
Update inputs
- Set the default value to cache-dir
- Add version
- Delete unused artifact-type
Fix tests

knqyf263 · 2024-10-05T16:44:55Z

It took me a while to understand why this test failed, while the run of entrypoint.sh succeeded, so I'm leaving it as a note: the checks bundle download fails due to GHCR's rate limit and falls back to the embedded checks. However, there is a difference between the embedded checks and the checks in GHCR, which also results in a difference in the results and causes this failure.

If the download fails, it should not fall back so that the execution fails and the failure of the test can be more easily understood.

entrypoint.sh

action.yaml

DmitriyLewen · 2024-10-07T11:10:00Z

action.yaml

Since we are adding GitHub cache - some users may want to change the update interval (e.g. update trivy-db every 2 days).

I think the skip-db-update (and flags for other DBs) flag can be added for this.

On the other hand, we have these flags in config file.
But then perhaps we should mention these flags in the documents.

I've also added an example for cronjob.
a8b935f

simar7 · 2024-10-07T20:23:12Z

It took me a while to understand why this test failed, while the run of entrypoint.sh succeeded, so I'm leaving it as a note: the checks bundle download fails due to GHCR's rate limit and falls back to the embedded checks. However, there is a difference between the embedded checks and the checks in GHCR, which also results in a difference in the results and causes this failure.

You're right but this is a one off case for the difference between what it downloads vs what's embedded as we're just transitioning to using the v1 checks bundle. This was just updated in the action.

simar7 · 2024-10-07T22:59:01Z

@knqyf263 do you have a test run where we can see the output?

action.yaml

knqyf263 · 2024-10-08T04:41:00Z

@knqyf263 do you have a test run where we can see the output?

Yes, I have.
https://github.com/knqyf263/trivy-action-test/actions/runs/11214873483/job/31170976150

However, this PR is making changes that will rewrite this action almost from scratch, and I would like to ask you guys (@simar7 and @DmitriyLewen ) to test it carefully in your own repositories. Since I'm not familiar with trivy-action, I'm really afraid of breaking something.

Signed-off-by: knqyf263 <[email protected]>

https://bats-core.readthedocs.io/en/stable/writing-tests.html#when-not-to-use-run Signed-off-by: knqyf263 <[email protected]>

Signed-off-by: knqyf263 <[email protected]>

DmitriyLewen

@knqyf263 I played with your changes in my test repository and didn't find issues.

simar7 · 2024-10-08T20:20:12Z

@knqyf263 do you have a test run where we can see the output?

Yes, I have. https://github.com/knqyf263/trivy-action-test/actions/runs/11214873483/job/31170976150

However, this PR is making changes that will rewrite this action almost from scratch, and I would like to ask you guys (@simar7 and @DmitriyLewen ) to test it carefully in your own repositories. Since I'm not familiar with trivy-action, I'm really afraid of breaking something.

I also tested it. Seems to be fine, you can see some tests here https://github.com/simar7/trivy-action-composite-test/actions - which indicate the cache was hit.

mattnakama-skytap · 2024-10-08T21:30:24Z

action.yaml

+  using: 'composite'
+  steps:
+    - name: Install Trivy
+      shell: bash
+      run: curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sudo sh -s -- -b /usr/local/bin ${{ inputs.version }}


Why was this switched from docker to running on the local system? It's not a good practice to grant sudo access to github action runners. I just had some actions fail because they attempted to use sudo. Unless I misunderstand what "composite" means?

It also means curl is a requirement where it wasn't previously.

Hello @mattnakama-skytap @danielnitsche
We migrated to local system to add the ability to store databases in cache.

@DmitriyLewen You might want to consider installing via something like https://github.com/jaxxstorm/action-install-gh-release

@rvesse We just published.
https://github.com/aquasecurity/setup-trivy

Bad request - jaxxstorm/[email protected] is not allowed to be used in apache/pulsar.
https://github.com/apache/pulsar/actions/runs/11267174066/job/31332113823?pr=23429#step:1:45
Addressing this for Apache Pulsar in this way: apache/pulsar#23431

aquasecurity/trivy-action#399

SimonRelu · 2024-10-10T10:11:38Z

Hey, This PR seems to cause a small breaking change for us. for with: template we used '@/github/workspace/.github/workflows/trivy_markdown_template.tpl' but we had to rename this to "@${{ github.workspace }}/.github/workflows/trivy_markdown_template.tpl"

knqyf263 · 2024-10-10T11:22:03Z

I've tested it with v0.25.0, but got an error.

2024-10-10T11:17:55Z	FATAL	Fatal error	report error: unable to write results: failed to initialize template writer: error retrieving template from path: open /github/workspace/.github/workflows/html.tpl: no such file or directory

      - uses: actions/checkout@v4
      - uses: aquasecurity/[email protected]
        with:
          scan-type: 'image'
          image-ref: 'jenkins:2.60.3'
          format: 'template'
          template: '@/github/workspace/.github/workflows/html.tpl'

Where does your template (Ptrivy_markdown_template.tpl) come from? If it's from your repository, how is it located under /github/workspace/.github/workflows`?

SimonRelu · 2024-10-10T12:04:01Z

Yes, it's something that's located in our repo under .github/workflows/trivy_markdown_template.tpl.

We use the following template:

{{- if . }}
{{- range . }}
<h3>Target <code>{{ escapeXML .Target }}</code></h3>
{{- if (and (eq (len .Vulnerabilities) 0) (eq (len .Misconfigurations) 0) (eq (len .Secrets) 0)) }}
<h4>Nothing found</h4>
{{- else }} 
{{- if (gt (len .Vulnerabilities) 0) }}
<h4>Vulnerabilities ({{ len .Vulnerabilities }})</h4>
<table>
    <tr>
        <th>Package</th>
        <th>ID</th>
        <th>Severity</th>
        <th>Installed Version</th>
        <th>Fixed Version</th>
    </tr>
    {{- range .Vulnerabilities }}
    <tr>
        <td><code>{{ escapeXML .PkgName }}</code></td>
        <td>{{ escapeXML .VulnerabilityID }}</td>
        <td>{{ escapeXML .Severity }}</td>
        <td>{{ escapeXML .InstalledVersion }}</td>
        <td>{{ escapeXML .FixedVersion }}</td>
    </tr>
    {{- end }}
</table>
{{- end }}
{{- if (gt (len .Misconfigurations ) 0) }}
<h4>Misconfigurations ({{ len .Misconfigurations }})</h4>
<table>
    <tr>
        <th>Type</th>
        <th>ID</th>
        <th>Check</th>
        <th>Severity</th>
        <th>Message</th>
    </tr>
    {{- range .Misconfigurations }}
    <tr>
        <td>{{ escapeXML .Type }}</td>
        <td>{{ escapeXML .ID }}</td>
        <td>{{ escapeXML .Title }}</td>
        <td>{{ escapeXML .Severity }}</td>
        <td>
          {{ escapeXML .Message }}
          <br><a href={{ escapeXML .PrimaryURL | printf "%q" }}>{{ escapeXML .PrimaryURL }}</a></br>
        </td>
    </tr>
    {{- end }}
</table>
{{- end }}
{{- if (gt (len .Secrets ) 0) }}
<h4>Secrets ({{ len .Secrets }})</h4>
<table>
    <tr>
        <th>Type</th>
        <th>ID</th>
        <th>Severity</th>
        <th>Lines</th>
        <th>Match</th>
    </tr>
    {{- range .Secrets }}
    <tr>
        <td>{{ escapeXML (toString .Category) }}</td>
        <td>{{ escapeXML .RuleID }}</td>
        <td>{{ escapeXML .Severity }}</td>
        <td>{{ escapeXML (toString .StartLine) }}-{{ escapeXML (toString .EndLine) }}</td>
        <td>{{ escapeXML .Match }}</td>
    </tr>
    {{- end }}
</table>
{{- end }}
{{- end }}
{{- end }}
{{- else }}
<h3>Trivy Returned Empty Report</h3>
{{- end }}

knqyf263 · 2024-10-10T12:11:43Z

Honestly, I don't understand why it used to work in your environment. I tested it, but the directory doesn't exist.

ls: cannot access '/github/workspace': No such file or directory

Could you share a small snippet that has started to cause errors after this change?

SimonRelu · 2024-10-10T12:47:04Z

Hey, thnx for looking into this. I just did another test to confirm and this script works on 0.25.0 but doesn't on 0.26.0.

name: Scan image
on:
  push:
    branches:
      - bug/trivy-broke
jobs:
  build-and-push-image:
    runs-on: ubuntu-latest
    # These permissions are needed to interact with GitHub's OIDC Token endpoint.
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: REDACTED
          aws-region: eu-north-1

      - name: Login to Amazon ECR Private
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Vulnerability scan (fixed and non-fixed)
        uses: aquasecurity/[email protected]
        env:
          TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2
        with:
          image-ref: 'REDACTED'
          format: 'template'
          # Manually downloaded the template from the Trivy GitHub repo since it is somehow not found by default
          template: '@/github/workspace/.github/workflows/trivy_html_template.tpl'
          output: 'report.html'
          severity: 'HIGH,CRITICAL'
          timeout: '12m'

      - uses: actions/upload-artifact@v4
        id: artifact-upload-step
        with:
          name: vulnerabilities-report
          path: report.html

      - name: Vulnerability scan (only fixed)
        uses: aquasecurity/[email protected]
        env:
          TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2
        with:
          image-ref: 'REDACTED'
          format: 'template'
          # Manually downloaded this custom template from this PR: https://github.com/aquasecurity/trivy/issues/3201
          template: '@/github/workspace/.github/workflows/trivy_markdown_template.tpl'
          output: 'report.md'
          severity: 'HIGH,CRITICAL'
          ignore-unfixed: true # Only show the ones that we are able to fix. The full report is still available in HTML format.
          timeout: '12m'

      - name: Publish scan summary
        run: |
          if [[ -s report.md ]]; then
            {
              echo "### Trivy Vulnerability Scan 🔍"
              echo "[Full report (fixed and non-fixed vulnerabilities)](${{ steps.artifact-upload-step.outputs.artifact-url }})"
              echo "<details><summary>Summarized report (only fixed vulnerabilities)</summary>"
              echo ""
              cat report.md
              echo "</details>"
            } >> $GITHUB_STEP_SUMMARY
          fi

I took a look at your run and I think it failed while running the following command and thus didn't get to the trivy command. Perhaps you could try rerunning the action without this step?

pwd
  ls -l
  ls -l /github/workspace
  ls -l /github/workspace/.github/workflows

knqyf263 · 2024-10-10T12:54:05Z

Perhaps you could try rerunning the action without this step?

Trivy failed, then I added them for debugging.

SimonRelu · 2024-10-10T12:56:51Z

Strange, so for me the configuration I gave works with [email protected] but not with [email protected]. It's not too important for me too get fixed since it works now fine after renaming @/github/workspace/ to @${{ github.workspace }}

SimonRelu · 2024-10-10T13:24:09Z

After some googling I found that github actions set the WORKDIR in a container automatically to /github/workspace/ https://stackoverflow.com/questions/77871301/how-to-solve-error-cannot-find-module-in-custom-dockerfile-based-github-actions

Maybe that's the reason?

markbaird · 2024-10-10T17:13:12Z

Trivy is a verified creator on GitHub Actions, so organizations that only allow verified actions in their repositories can use Trivy. Except now Trivy depends on jaxxstorm/action-install-gh-release which is not an action from a verified creator, so now any repositories that require actions to be from verified creators are encountering this error when trying to run the Trivy action:

Error: Bad request - jaxxstorm/[email protected] is not allowed to be used in <repository name>. Actions in this workflow must be: within a repository owned by <repository account name>, created by GitHub, or verified in the GitHub Marketplace.

By including a dependency on jaxxstorm/action-install-gh-release the Trivy action has broken the trust relationship established by their verified creator status, and broken any GitHub Action pipelines that relied on that.

DmitriyLewen · 2024-10-11T05:17:06Z

Hello @markbaird
We are already working on setup-trivy v0.2.0 without using jaxxstorm/action-install-gh-release

aquasecurity/trivy-action#399 (comment) Signed-off-by: knqyf263 <[email protected]>

knqyf263 · 2024-10-11T07:00:31Z

@SimonRelu I copied your workflow file and deleted some unnecessary steps.
https://github.com/knqyf263/trivy-action-test/blob/main/.github/workflows/test.yaml

I also put the template file.
https://github.com/knqyf263/trivy-action-test/blob/main/.github/workflows/trivy_markdown_template.tpl

However, it still failed with v0.25.0.

2024-10-11T06:56:20Z	FATAL	Fatal error	report error: unable to write results: failed to initialize template writer: error retrieving template from path: open /github/workspace/.github/workflows/trivy_html_template.tpl: no such file or directory

https://github.com/knqyf263/trivy-action-test/actions/runs/11287781686/job/31394298976

Any ideas? Do aws-actions/configure-aws-credentials@v4 or aws-actions/amazon-ecr-login@v2 happen to do something?

knqyf263 self-assigned this Oct 4, 2024

knqyf263 force-pushed the composite branch from d52f156 to 646184e Compare October 4, 2024 18:32

knqyf263 marked this pull request as ready for review October 7, 2024 03:26

DmitriyLewen reviewed Oct 7, 2024

View reviewed changes

knqyf263 force-pushed the composite branch from d9b7c11 to 5c43f68 Compare October 7, 2024 11:52

simar7 mentioned this pull request Oct 7, 2024

Ratelimiting error when downloading vulnerability db from ghcr.io #389

Open

simar7 reviewed Oct 7, 2024

View reviewed changes

action.yaml Outdated Show resolved Hide resolved

knqyf263 added 13 commits October 8, 2024 11:18

feat: migrate to a composite action

bad351b

Signed-off-by: knqyf263 <[email protected]>

Fix tests

b937d02

Signed-off-by: knqyf263 <[email protected]>

Delete an unused input

a1e9b97

Signed-off-by: knqyf263 <[email protected]>

test: expect status code 0

59d3056

Signed-off-by: knqyf263 <[email protected]>

test: not use run

ccf6e0f

https://bats-core.readthedocs.io/en/stable/writing-tests.html#when-not-to-use-run Signed-off-by: knqyf263 <[email protected]>

feat: add 'cache' input

4f4b6e0

Signed-off-by: knqyf263 <[email protected]>

docs: update README

c19d38f

Signed-off-by: knqyf263 <[email protected]>

feat: pin Trivy version

40d20e1

Signed-off-by: knqyf263 <[email protected]>

fix: bump trivy version

84a6ffe

Signed-off-by: knqyf263 <[email protected]>

feat: use date for cache key

5dd4e7e

Signed-off-by: knqyf263 <[email protected]>

chore: delete a comment

330fdca

Signed-off-by: knqyf263 <[email protected]>

docs: update README

a8b935f

Signed-off-by: knqyf263 <[email protected]>

refactor: resolve conflicts and use envs

e94275c

Signed-off-by: knqyf263 <[email protected]>

knqyf263 force-pushed the composite branch from 77c2289 to e94275c Compare October 8, 2024 07:19

DmitriyLewen approved these changes Oct 8, 2024

View reviewed changes

simar7 self-requested a review October 8, 2024 20:20

simar7 approved these changes Oct 8, 2024

View reviewed changes

simar7 merged commit a20de54 into aquasecurity:master Oct 8, 2024
2 checks passed

torredil mentioned this pull request Oct 8, 2024

Bump GH actions (trivy, codeql, upload-artifact) kubernetes-sigs/aws-ebs-csi-driver#2175

Merged

mattnakama-skytap reviewed Oct 8, 2024

View reviewed changes

mattnakama-skytap mentioned this pull request Oct 8, 2024

trivy-action attempts to run sudo on the local machine (!?) #403

Closed

Tiryoh added a commit to Tiryoh/docker-mkdocs-builder that referenced this pull request Oct 9, 2024

fix(ci): Try latest action to use cache

ca197e7

aquasecurity/trivy-action#399

Tiryoh mentioned this pull request Oct 9, 2024

fix(ci): Fix Trivy TOOMANYREQUESTS errors Tiryoh/docker-mkdocs-builder#244

Merged

knqyf263 deleted the composite branch October 9, 2024 09:52

cbandy mentioned this pull request Oct 9, 2024

license scanner not possible via action? #219

Open

Tiryoh mentioned this pull request Oct 9, 2024

chore(ci): Save Trivy cache to avoid Trivy TOOMANYREQUESTS errors Tiryoh/docker-ros2-desktop-vnc#182

Merged

knqyf263 added a commit to knqyf263/trivy-action-test that referenced this pull request Oct 11, 2024

test

5acd2d4

aquasecurity/trivy-action#399 (comment) Signed-off-by: knqyf263 <[email protected]>

knqyf263 added a commit to knqyf263/trivy-action-test that referenced this pull request Oct 11, 2024

test

e8dbe61

aquasecurity/trivy-action#399 (comment) Signed-off-by: knqyf263 <[email protected]>

Kieffer87 mentioned this pull request Oct 15, 2024

Bump aquasecurity/trivy-action from 0.24.0 to 0.27.0 tkhom3/gha-workflows#37

Merged

marcrohlfs mentioned this pull request Oct 16, 2024

trivy-config input is not respected anymore #416

Open

coilysiren mentioned this pull request Oct 16, 2024

Cache trivy HHS/simpler-grants-gov#2496

Closed

DmitriyLewen mentioned this pull request Oct 24, 2024

fix: bump setup-trivy and add new contrib directory path info #424

Merged

EricAverittTMNA mentioned this pull request Dec 19, 2024

Text File Busy #441

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: store artifacts in cache by default #399

feat: store artifacts in cache by default #399

knqyf263 commented Oct 4, 2024 •

edited

Loading

knqyf263 commented Oct 5, 2024

DmitriyLewen Oct 7, 2024

DmitriyLewen Oct 7, 2024

knqyf263 Oct 8, 2024

simar7 commented Oct 7, 2024

simar7 commented Oct 7, 2024

knqyf263 commented Oct 8, 2024

DmitriyLewen left a comment

simar7 commented Oct 8, 2024

mattnakama-skytap Oct 8, 2024

danielnitsche Oct 9, 2024

DmitriyLewen Oct 9, 2024

rvesse Oct 9, 2024

knqyf263 Oct 9, 2024

lhotari Oct 10, 2024

SimonRelu commented Oct 10, 2024 •

edited

Loading

knqyf263 commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

knqyf263 commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

knqyf263 commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

markbaird commented Oct 10, 2024

DmitriyLewen commented Oct 11, 2024

knqyf263 commented Oct 11, 2024

feat: store artifacts in cache by default #399

feat: store artifacts in cache by default #399

Conversation

knqyf263 commented Oct 4, 2024 • edited Loading

Description

knqyf263 commented Oct 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simar7 commented Oct 7, 2024

simar7 commented Oct 7, 2024

knqyf263 commented Oct 8, 2024

DmitriyLewen left a comment

Choose a reason for hiding this comment

simar7 commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonRelu commented Oct 10, 2024 • edited Loading

knqyf263 commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

knqyf263 commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

knqyf263 commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

SimonRelu commented Oct 10, 2024

markbaird commented Oct 10, 2024

DmitriyLewen commented Oct 11, 2024

knqyf263 commented Oct 11, 2024

knqyf263 commented Oct 4, 2024 •

edited

Loading

SimonRelu commented Oct 10, 2024 •

edited

Loading