Skip to content

Commit

Permalink
Seems to work...make this v1.0.0
Browse files Browse the repository at this point in the history
  • Loading branch information
keithlayne committed Jul 4, 2022
1 parent 09d6f2a commit ba1d448
Show file tree
Hide file tree
Showing 4 changed files with 148 additions and 48 deletions.
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
.yarn/
dist/
node_modules/
3 changes: 2 additions & 1 deletion .prettierignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
.yarn/
dist/
node_modules/
node_modules/
179 changes: 137 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ The comment will have a summary that shows project-wide coverage changes, and an
table that shows changes on a per-file basis. Only files which have coverage changes will be
included in the table.

**Contents**

- [What It Does](#what-it-does)
- [Important Notes](#important-notes)
- [Inputs](#inputs)
- [Outputs](#outputs)
- [Example Usage](#example-usage)
- [How Do I Manage the Baseline?](#how-do-i-manage-the-baseline)
- [First run on push to main branch](#first-run-on-push-to-main-branch)
- [First run on pull request](#first-run-on-pull-request)
- [TODO](#todo)
- [Example Output](#example-output)

## What It Does

```mermaid
Expand All @@ -27,9 +40,39 @@ graph TD

## Important Notes

- I made this for a work project, so the choices I've made reflect my personal preferences.
- This will fail on any workflow that's not a pull request.
- The baseline is required - how you get that into the workflow context is up to you (see below).
I made this for a work project, so the choices I've made reflect my personal preferences and needs.
This initial version solves _my_ problems. I will make this available on the GH marketplace, but I
have no clue if anyone else will ever use it. However, if people find it useful and want more
configurability or features, I am happy to discuss it via GH issues and pull requests.

Things that will cause this action to fail:

- running on a non-PR workflow - since this action adds issue comments, that's the only thing that
makes sense to me
- missing the baseline resultset - the main idea of this action is to display coverage diffs

The underlying ruby coverage libaray (as far as I can tell) uses absolute paths for coverage entries
on each file. When the coverage report is generated, we remove `$GITHUB_WORKSPACE` from the start of
the path, keeping a relative path from the repo root. (This is something that could potentially be
configured in the future.) When comparing files for the diff, we have some options:

1. Keep an extra file along with the baseline that records the absolute path of the local checkout
when the coverage was generated
2. Postprocess `.resultset.json` and remove the absolute path of the local checkout from each of the
file entries
3. Do nothing and hope for the best

I rejected option 1 because it would be more painful to use, and it _appears_ so far that
`$GITHUB_WORKSPACE` is stable across runs and takes the form
`/home/runner/work/<repo name>/<repo name>`. I have only tested with the Ubuntu 22.04 runner at this
point. Option 2 seemed bad also - too much work, and potentially problematic. So I opted for option
3, which is in line with my overall laziness, and hopefully it just won't be a problem.

If for some reason the `$GITHUB_WORKSPACE` turns out not to be stable, I will have to revisit this.
If your first run of the action is on your main branch (see below in
[How Do I Manage the Baseline?](#how-do-i-manage-the-baseline)), then you _should_ have no problems
at all. If for some reason the absolute paths don't match, the overall diff should be correct, but
the per-file diff will include all the files, which is just noisy.

## Inputs

Expand Down Expand Up @@ -61,18 +104,12 @@ on:
jobs:
test:
name: Lint & Test
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3
- uses: ruby/setup-ruby@v1
with: # your options
- run: bin/rails test
- # set up, run linter, etc., and generate coverage
- name: Set up coverage baseline
run: echo 'This is up to you...see below for my suggestion'
- name: Report coverage
if: github.event_name == 'pull_request'
uses: keithlayne/simplecov-reporter-action@v0.1.0
uses: keithlayne/simplecov-reporter-action@v1.0.0
with:
# these are all the defaults
github-token: ${{ github.token }}
Expand All @@ -83,41 +120,22 @@ jobs:
run: echo 'On pushes to master, update the baseline'
```
This example amends the initial commit and force-pushes so that you only ever have a single commit
at `refs/coverage/baseline`, because I don't care about the history of that file. You can do
whatever you like.

## How Do I Manage the Baseline?

This is completely up to you. However, I have a suggestion: use git.
This is completely up to you -- but read this section anyway, as there may be some considerations
regardless of where you store the baseline. One reasonable approach is to store the single file in
some kind of cloud storage. However, I have a suggestion: use git.

I store the baseline in a ref that doesn't start with `refs/heads/`. This has a couple of nice
benefits, but for our purposes, it's a convenient place to stash files, and it's pretty cheap to
access in an action. It should never get in the way of your day-to-day git usage either.

You need to do some initial setup to push your initial baseline result set:

```sh
# Make sure you're in the repo root
cd $(git rev-parse --show-toplevel)
# The baseline should represent the current state of your main branch
git checkout main
# Generate coverage, I'm assuming it's at coverage/.resultset.json
rake test
# Create an orphan branch so we have a clean HEAD
git checkout --orphan temp
# Copy the json file to the root
cp coverage/.resultset.json .
# You'll probably have a whole bunch of stuff staged...clear that out
git rm --cached -rf .
# Create an initial commit
git add .resultset.json
git commit --message='Update coverage baseline'
git push origin HEAD:refs/coverage/baseline
# go back to main
git checkout main
```

Now you have a special ref that has a single file holding your coverage baseline.

Now, in your workflow, before you run the coverage report, you can check this file out, and update
it after running tests when you're on the main branch:
Here's how you can use git to store the baseline resultset in a workflow (I use
`refs/coverage/baseline` for this):

```yml
jobs:
Expand Down Expand Up @@ -147,9 +165,86 @@ jobs:
git push --force origin HEAD:refs/coverage/baseline
```

This example amends the initial commit and force-pushes so that you only ever have a single commit
at `refs/coverage/baseline`, because I don't care about the history of that file. You can do
whatever you like.
> **IMPORTANT:** The checkout step will not work unless that git ref exists in `origin`. This means
> if you use this method, it needs to be set up before this workflow runs the first time.

How you initially set up your baseline ref depends on if your first workflow run will be in the main
branch or in a PR.

### First run on push to main branch

This is simpler in theory, but I haven't tried it yet during development of this action. You should
be able to create an empty ref like this:

```sh
# Create an orphan branch
git checkout --orphan temp
# Add a single empty commit
git commit --message='Initial commit' --allow-empty
# Push it
git push origin HEAD:refs/coverage/baseline
```

Then clean up:

```sh
git checkout main
git branch --delete --force temp
```

At this point, the ref is set up with nothing in it but an empty commit. If you add this action to
your workflow in a PR, it will fail, but if you push the update to main first, it should simply
update the baseline ref, and future PRs from that point should show coverage diffs.

### First run on pull request

You need to do some initial setup to push your initial baseline result set:

```sh
# Make sure you're in the repo root
cd $(git rev-parse --show-toplevel)
# The baseline should represent the current state of your main branch in the origin
git checkout origin/main
# Generate coverage, I'm assuming it's at coverage/.resultset.json
rake test
# Create an orphan branch so we have a clean HEAD
git checkout --orphan temp
# clean up the branch (assuming you have ignored coverage dir in git)
git reset --hard
# Copy the json file to the root
cp coverage/.resultset.json .
```

At this point, you very likely need to munge the resultset file as discussed in
[Important Notes](#important-notes) so that your initial file diffs are correct. You can do this
however you want, but I have tested this (assuming an environment with GNU sed):

```sh
REPO=$(basename $(git remote get-url origin) .git)
sed -i "s|$(pwd)|/home/runner/work/${REPO}/${REPO}|" .resultset.json
```

This should leave you with a locally-generated, actions-compatible baseline. Now, push the munged
baseline resultset to `refs/coverage/baseline`:

```sh
# Create an initial commit
git add .resultset.json
git commit --message='Update coverage baseline'
git push origin HEAD:refs/coverage/baseline
```

Now you can clean up the temporary orphan branch:

```sh
git checkout main
git branch --delete --force temp
```

Now you have a special ref that has a single file holding your coverage baseline.

Now, in your workflow, before you run the coverage report, you can check this file out, and update
it after running tests when you're on the main branch:

## TODO

Expand Down
Loading

0 comments on commit ba1d448

Please sign in to comment.