Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skipped tests are counted as Passed tests #539

Open
pknoblich-impinj opened this issue Oct 23, 2024 · 1 comment
Open

Skipped tests are counted as Passed tests #539

pknoblich-impinj opened this issue Oct 23, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@pknoblich-impinj
Copy link

Describe the bug

I have an XML file with 2 skipped tests; however, they are showing up as passed tests in my test report in GitHub.

To Reproduce

Steps to reproduce the behavior:

  1. Run a series of behave tests and ensure the first test fails (e.g., assert False): behave --junit --stop <path_to_tests>
  2. Loop through the generated XML files and change each untested test to a skipped test (testcase), and update the corresponding overall skipped count at the top of each XML file (testsuite)
  3. Run this test reporter on those updated XML files

Expected behavior

All skipped tests should show up in the Skipped column, not the Passed column.

Example

Here are my test logs showing the XML file as it was generated by behave and then as it was when I did my post-processing as mentioned above.

2024-10-23 23:05:29,110 | INFO    | airstream-test                       | junit/TESTS-vehicle-manager.peripheral_management.xml contents before processing:
2024-10-23 23:05:29,110 | INFO    | airstream-test                       | <testsuite name="vehicle-manager.peripheral_management.Peripheral Management" tests="2" errors="0" failures="0" skipped="0" time="0.0" timestamp="2024-10-23T23:03:54.623307" hostname="octane-gh-gkopma-017"><testcase classname="vehicle-manager.peripheral_management.Peripheral Management" name="The misload alarms are triggered when a misload occurs" status="untested" time="0"><system-out>
<![CDATA[
@scenario.begin

  @smoke
  Scenario: The misload alarms are triggered when a misload occurs
    Given the reader is not in commissioning ... untested in 0.000s
    And the misload alarms are off ... untested in 0.000s
    When the vehicle is in load ... untested in 0.000s
    And there is not a misload ... untested in 0.000s
    And I wait for 1 second ... untested in 0.000s
    Then the misload alarms should be off ... untested in 0.000s
    When there is a misload ... untested in 0.000s
    Then the misload alarms should be on ... untested in 0.000s

@scenario.end
--------------------------------------------------------------------------------
]]>
</system-out></testcase><testcase classname="vehicle-manager.peripheral_management.Peripheral Management" name="EAS receives PIR motion events" status="untested" time="0"><system-out>
<![CDATA[
@scenario.begin

  @regression
  Scenario: EAS receives PIR motion events
    When motion is detected ... untested in 0.000s
    Then krest-eas should have received a PIR motion event ... untested in 0.000s

@scenario.end
--------------------------------------------------------------------------------
]]>
</system-out></testcase></testsuite>
2024-10-23 23:05:29,112 | INFO    | airstream-test                       | junit/TESTS-vehicle-manager.peripheral_management.xml contents after processing:
2024-10-23 23:05:29,112 | INFO    | airstream-test                       | <?xml version='1.0' encoding='utf-8'?>
<testsuite name="vehicle-manager.peripheral_management.Peripheral Management" tests="2" errors="0" failures="0" skipped="2" time="0.0" timestamp="2024-10-23T23:03:54.623307" hostname="octane-gh-gkopma-017"><testcase classname="vehicle-manager.peripheral_management.Peripheral Management" name="The misload alarms are triggered when a misload occurs" status="skipped" time="0"><system-out>

@scenario.begin

  @smoke
  Scenario: The misload alarms are triggered when a misload occurs
    Given the reader is not in commissioning ... untested in 0.000s
    And the misload alarms are off ... untested in 0.000s
    When the vehicle is in load ... untested in 0.000s
    And there is not a misload ... untested in 0.000s
    And I wait for 1 second ... untested in 0.000s
    Then the misload alarms should be off ... untested in 0.000s
    When there is a misload ... untested in 0.000s
    Then the misload alarms should be on ... untested in 0.000s

@scenario.end
--------------------------------------------------------------------------------

</system-out></testcase><testcase classname="vehicle-manager.peripheral_management.Peripheral Management" name="EAS receives PIR motion events" status="skipped" time="0"><system-out>

@scenario.begin

  @regression
  Scenario: EAS receives PIR motion events
    When motion is detected ... untested in 0.000s
    Then krest-eas should have received a PIR motion event ... untested in 0.000s

@scenario.end
--------------------------------------------------------------------------------

</system-out></testcase></testsuite>

Yet those tests are showing up as Passed:

dorny-test-results
@pknoblich-impinj pknoblich-impinj added the bug Something isn't working label Oct 23, 2024
@jozefizso
Copy link
Collaborator

A pull request with the fix is welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants