Chess incorrectly report success on failed stage if junit is enabled and not failing

[sodul]

Jenkins and plugins versions report

Environment

Jenkins: 2.401.1
OS: Linux - 5.10.184-175.731.amzn2.x86_64
Java: 11.0.14.1 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
ansicolor:1.0.2
ant:487.vd79d090d4ea_e
antisamy-markup-formatter:159.v25b_c67cd35fb_
anything-goes-formatter:19.v3e2b_1b_3e0ee5
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
apache-httpcomponents-client-5-api:5.2.1-1.0
artifact-manager-s3:822.vf129d4836c31
atlassian-jira-software-cloud:2.0.7
audit-trail:333.vb_e1b_b_0f1238c
authentication-tokens:1.53.v1c90fd9191a_b_
aws-credentials:191.vcb_f183ce58b_9
aws-global-configuration:108.v47b_fd43dfec6
aws-java-sdk:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-cloudformation:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-codebuild:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ec2:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ecr:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ecs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-efs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-elasticbeanstalk:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-iam:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-kinesis:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-logs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-minimal:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-sns:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-sqs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ssm:1.12.481-392.v8b_291cfcda_09
basic-branch-build-strategies:71.vc1421f89888e
blueocean:1.27.4
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.4
blueocean-commons:1.27.4
blueocean-config:1.27.4
blueocean-core-js:1.27.4
blueocean-dashboard:1.27.4
blueocean-display-url:2.4.2
blueocean-events:1.27.4
blueocean-git-pipeline:1.27.4
blueocean-github-pipeline:1.27.4
blueocean-i18n:1.27.4
blueocean-jira:1.27.4
blueocean-jwt:1.27.4
blueocean-personalization:1.27.4
blueocean-pipeline-api-impl:1.27.4
blueocean-pipeline-editor:1.27.4
blueocean-pipeline-scm-api:1.27.4
blueocean-rest:1.27.4
blueocean-rest-impl:1.27.4
blueocean-web:1.27.4
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.0-1
bouncycastle-api:2.28
branch-api:2.1109.vdf225489a_16d
build-monitor-plugin:1.14-717.v3efcdffe8d58
build-time-blame:70.v5031def3c2e3-2.1.0
build-timeout:1.31
build-timestamp:1.0.3
build-user-vars-plugin:1.9
build-with-parameters:76.v9382db_f78962
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
cctray-xml:59.v22fa_7b_8c8fe5
checks-api:2.0.0
chucknorris:1.4
cloudbees-bitbucket-branch-source:809.vc1d904b_30426
cloudbees-folder:6.815.v0dd5a_cb_40e0e
cobertura:1.17
code-coverage-api:4.7.0
command-launcher:100.v2f6722292ee8
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
conditional-buildstep:1.4.2
config-file-provider:938.ve2b_8a_591c596
configuration-as-code:1647.ve39ca_b_829b_42
configurationslicing:548.ve92d48e66b_f8
copyartifact:705.v5295cffec284
credentials:1254.vb_96f366e7b_a_d
credentials-binding:604.vb_64480b_c56ca_
dark-theme:315.va_22e7d692ea_a
data-tables-api:1.13.4-2
datadog:5.4.1
display-url-api:2.3.7
docker-commons:419.v8e3cd84ef49c
docker-java-api:3.3.1-79.v20b_53427e041
docker-workflow:563.vd5d2e5c4007f
durable-task:507.v050055d0cb_dd
ec2:2.0.7
ec2-fleet:2.6.0
echarts-api:5.4.0-5
email-ext:2.98
embeddable-build-status:385.vc95f94e91fb_3
envinject:2.901.v0038b_6471582
envinject-api:1.199.v3ce31253ed13
environment-script:1.2.6
extended-choice-parameter:373.v1a_ecea_fdf2a_a_
extensible-choice-parameter:1.8.0
external-monitor-job:203.v683c09d993b_9
favorite:2.4.2
folder-auth:1.4
font-awesome-api:6.4.0-1
forensics-api:2.3.0
ghprb:1.42.2
git:5.1.0
git-changelog:3.30
git-client:4.4.0
git-parameter:0.9.18
git-server:99.va_0826a_b_cdfa_d
github:1.37.1
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1728.v859147241f49
github-checks:545.v79a_a_68b_ca_682
google-oauth-plugin:1.0.8
gradle:2.8
groovy:453.vcdb_a_c5c99890
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
hashicorp-vault-plugin:360.v0a_1c04cf807d
htmlpublisher:1.31
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
ivy:2.5
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:233.vdc1a_ec702cff
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:66.vd8fa_64ee91b_d
jenkins-design-language:1.27.4
jersey2-api:2.39.1-2
jira:3.10
jira-steps:2.0.165.v8846cf59f3db
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.17-1
job-dsl:1.84
job-import-plugin:3.6
jobConfigHistory:1212.vd4470d08ff12
jobcacher:387.v41e893017398
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.7.0-1
jsch:0.2.8-65.v052c39de79b_2
junit:1207.va_09d5100410f
kubernetes:3909.v1f2c633e8590
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
ldap:682.v7b_544c9d1512
list-git-branches-parameter:0.0.13
lockable-resources:1156.v5e9f897ece02
mailer:457.v3f72cb_e015e5
mapdb-api:1.0.9-28.vf251ce40855d
markdown-formatter:77.vf4787624a_796
mask-passwords:150.vf80d33113e80
matrix-auth:3.1.8
matrix-project:789.v57a_725b_63c79
maven-plugin:3.22
mercurial:1260.vdfb_723cdcc81
metrics:4.2.18-439.v86a_20b_a_8318b_
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
momentjs:1.1.1
monitoring:1.94.1
next-build-number:1.8
node-iterator-api:49.v58a_8b_35f8363
oauth-credentials:0.645.ve666a_c332668
okhttp-api:4.11.0-145.vcb_8de402ef81
pam-auth:1.10
parameterized-scheduler:1.2
parameterized-trigger:2.45
pipeline-build-step:496.v2449a_9a_221f2
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2141.v5402e818a_779
pipeline-model-definition:2.2141.v5402e818a_779
pipeline-model-extensions:2.2141.v5402e818a_779
pipeline-rest-api:2.32
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2141.v5402e818a_779
pipeline-stage-view:2.32
pipeline-utility-steps:2.15.4
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
prism-api:1.29.0-7
publish-to-bitbucket:0.4
pubsub-light:1.17
rebuild:320.v5a_0933a_e7d61
resource-disposer:0.22
run-condition:1.5
saferestart:0.7
saml:4.418.vdfa_7489a_b_a_2d
scm-api:672.v64378a_b_20c60
script-security:1251.vfe552ed55f8d
simple-theme-plugin:160.vb_76454b_67900
skip-notifications-trait:198.vc48f25a_18f41
slack:664.vc9a_90f8b_c24a_
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.26
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.303.vefc7119b_ec23
startup-trigger-plugin:2.9.3
structs:324.va_f5d6774f3a_d
subversion:2.17.2
test-results-analyzer:0.4.0
theme-manager:193.vcef22f6c5f2b_
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
validating-string-parameter:2.8
variant:59.vf075fe829ccb
violation-comments-to-stash:1.130
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1213.v646def1087f9
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3691.v28b_14c465a_b_b_
workflow-durable-task-step:1247.v7f9dfea_b_4fd0
workflow-job:1308.v58d48a_763b_31
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Off the shelf linux containers under kubernetes, but not relevant to the bug.

Reproduction steps

Have a Jenkinsfile with a stage that fails by returning a non zero exit code, but with junit() enabled and configured to allow missing junit reports.

steps {
  sh 'exit 1'
}
post {
  always {
    junit("**/report.xml.xml", allowEmptyResults: true)
  }
}

Expected Results

In GH the PR check for the stage should match the results seen in Jenkins itself: fail.

Actual Results

The check in the PR view is showing success.

Anything else?

We will update our internal jenkins library to remove junit() calls when not needed and for the ones that need it we will remove allowEmptyResults: true as a workaround.