"All checks have passed" only taking latest commit into account

[sj14]

Description

When you have Github Workflows which only run on specific file changes (on.push.paths) and a commit leads to a failed workflow, this failed workflow is ignored (in the PR overview and Checks tab) when there are follow-up commits which will not trigger the same (failed) workflow.

Example

1. Commit

Overview

Commits

2. Commit

Overview

Commits

Conclusion and suggestion

Thus, you could easily think all checks were successful and merge a PR but it only reflects the checks from the latest commit in this PR.

I would suggest always showing the latest results from all workflows that were run in a PR in the overview and Checks tab.

[sigprof]

You probably need to change the problematic workflow to run on pull_request — then it would be rerun on any changes to the PR, even when the last commit does not change the specified paths, but some previous ones do.

Also there is a subtle but very important difference between push and pull_request — workflows running on push test exactly the code that was pushed, but workflows running on pull_request test the result of attempted merge between the PR target and head branches. So the push workflows will never detect that the code in the PR head branch is not actually compatible with some other changes made in the PR target branch — only the pull_request workflows can detect that.

[sj14]

Unfortunately, it was a conscious decision to use push to decrease the time the pipeline takes (many executed jobs were unnecessary when unrelated files changed). Except for the given issue, it seems to work fine so far.

[sj14]

Ah, I see. on can be used with pull_request too. I will test it 👍

[shaneDubbs]

This solution does not fit at all for situations where the pull request trigger is limited by paths. Some of our checks take a rather long time and we only want to run them when certain files and their dependencies change not when some different file changes and it so happened to be in a previous commit.

[farrukh-taqveem-carv]

@sj14 Did you find a solution.

[sj14]

@farrukh-taqveem-carv I use on.pull_request.paths with the trade off @shaneDubbs described, and it only triggers for pull requests but not branches without a correspondign pull requests.