Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "Complex Example" testset for Julia 1.9+ #99

Merged
merged 3 commits into from
Apr 29, 2024

Conversation

omus
Copy link
Member

@omus omus commented Apr 26, 2024

I noticed this failure on Julia 1.9.4:

┌ Info: Reference Test for "complexexample.txt" failed:
│ - DIFF ------------------------
│ """
│   <?xml version=\"1.0\" encoding=\"UTF-8\"?>
│   <testsuites tests=\"19\" failures=\"6\" errors=\"4\"><testsuite name=\"Math/Multiplication\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"0\"><testcase name=\"1 * 3 == 3\" id=\"1\" classname=\"Math/Multiplication\" time=\"0.0\"/><testcase name=\"1 * 2 == 5\" id=\"2\" classname=\"Math/Multiplication\" time=\"0.0\"><failure message=\"2 == 5\" type=\"test\">Test Failed
│     Expression: 1 * 2 == 5
│ -    Evaluated: 2 == 5</failure></testcase><testcase name=\"1 * 4 == 4\" id=\"3\" classname=\"Math/Multiplication\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition/negative addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"1\"><testcase name=\"1 + -1 == 0\" id=\"1\" classname=\"Math/addition/negative addition\" time=\"0.0\"/><testcase name=\"1 + -2 == 1\" id=\"2\" classname=\"Math/addition/negative addition\" time=\"0.0\"><failure message=\"-1 == 1\" type=\"test\">Test Failed
│ +    Evaluated: 2 == 5
│ + </failure></testcase><testcase name=\"1 * 4 == 4\" id=\"3\" classname=\"Math/Multiplication\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition/negative addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"1\"><testcase name=\"1 + -1 == 0\" id=\"1\" classname=\"Math/addition/negative addition\" time=\"0.0\"/><testcase name=\"1 + -2 == 1\" id=\"2\" classname=\"Math/addition/negative addition\" time=\"0.0\"><failure message=\"-1 == 1\" type=\"test\">Test Failed
│     Expression: 1 + -2 == 1
│ -    Evaluated: -1 == 1</failure></testcase><testcase name=\"10 + -5 == 5\" id=\"3\" classname=\"Math/addition/negative addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"2\"><testcase name=\"1 + 1 == 2\" id=\"1\" classname=\"Math/addition\" time=\"0.0\"/><testcase name=\"1 + 2 == 5\" id=\"2\" classname=\"Math/addition\" time=\"0.0\"><failure message=\"3 == 5\" type=\"test\">Test Failed
│ +    Evaluated: -1 == 1
│ + </failure></testcase><testcase name=\"10 + -5 == 5\" id=\"3\" classname=\"Math/addition/negative addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"2\"><testcase name=\"1 + 1 == 2\" id=\"1\" classname=\"Math/addition\" time=\"0.0\"/><testcase name=\"1 + 2 == 5\" id=\"2\" classname=\"Math/addition\" time=\"0.0\"><failure message=\"3 == 5\" type=\"test\">Test Failed
│     Expression: 1 + 2 == 5
│ -    Evaluated: 3 == 5</failure></testcase><testcase name=\"1 + 4 == 5\" id=\"3\" classname=\"Math/addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/other\" tests=\"4\" failures=\"0\" errors=\"3\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"3\"><testcase name=\"sqrt(-1)\" id=\"1\" classname=\"Math/other\" time=\"0.0\"><skip/></testcase><error message=\"Expression evaluated to non-Boolean\" type=\"Expression evaluated to non-Boolean\" classname=\"Math/other\" time=\"0.0\" id=\"2\"></error><error message=\"Nooo\" type=\"ErrorException\" classname=\"Math/other\" time=\"0.0\" id=\"3\">Nooo
│ +    Evaluated: 3 == 5
│ + </failure></testcase><testcase name=\"1 + 4 == 5\" id=\"3\" classname=\"Math/addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/other\" tests=\"4\" failures=\"0\" errors=\"3\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"3\"><testcase name=\"sqrt(-1)\" id=\"1\" classname=\"Math/other\" time=\"0.0\"><skip/></testcase><error message=\"Expression evaluated to non-Boolean\" type=\"Expression evaluated to non-Boolean\" classname=\"Math/other\" time=\"0.0\" id=\"2\"></error><error message=\"Nooo\" type=\"ErrorException\" classname=\"Math/other\" time=\"0.0\" id=\"3\">Nooo
│   </error><error message=\"Got correct result, please change to @test if no longer broken.\" type=\"Unexpected Pass\" classname=\"Math/other\" time=\"0.0\" id=\"4\"></error></testsuite><testsuite name=\"Math/Error outside of tests\" tests=\"0\" failures=\"0\" errors=\"1\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"4\"><error message=\"Got exception outside of a @test\" type=\"ErrorException\" classname=\"Math/Error outside of tests\" time=\"0.0\">Outside of tests
│   </error></testsuite><testsuite name=\"Math/Different failures\" tests=\"2\" failures=\"2\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"5\"><testcase name=\"throw(ArgumentError(&quot;1&quot;))\" id=\"1\" classname=\"Math/Different failures\" time=\"0.0\"><failure message=\"Wrong exception type thrown\" type=\"test_throws_wrong\">Test Failed
│     Expression: throw(ArgumentError(\"1\"))
│       Expected: DimensionMismatch
│ -       Thrown: ArgumentError</failure></testcase><testcase name=\"true\" id=\"2\" classname=\"Math/Different failures\" time=\"0.0\"><failure message=\"No exception thrown\" type=\"test_throws_nothing\">Test Failed
│ +       Thrown: ArgumentError
│ + </failure></testcase><testcase name=\"true\" id=\"2\" classname=\"Math/Different failures\" time=\"0.0\"><failure message=\"No exception thrown\" type=\"test_throws_nothing\">Test Failed
│     Expression: true
│       Expected: DimensionMismatch
│ -   No exception thrown</failure></testcase></testsuite><testsuite name=\"Math/using function from a module\" tests=\"1\" failures=\"0\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"6\"><testcase name=\"nthreads() &gt; 0\" id=\"1\" classname=\"Math/using function from a module\" time=\"0.0\"/></testsuite><testsuite name=\"Math\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"7\"><testcase name=\"4 % 2 == 0\" id=\"1\" classname=\"Math\" time=\"0.0\"/><testcase name=\"sqrt(20) == 5\" id=\"2\" classname=\"Math\" time=\"0.0\"><failure message=\"4.47213595499958 == 5\" type=\"test\">Test Failed
│ +   No exception thrown
│ + </failure></testcase></testsuite><testsuite name=\"Math/using function from a module\" tests=\"1\" failures=\"0\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"6\"><testcase name=\"nthreads() &gt; 0\" id=\"1\" classname=\"Math/using function from a module\" time=\"0.0\"/></testsuite><testsuite name=\"Math\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"7\"><testcase name=\"4 % 2 == 0\" id=\"1\" classname=\"Math\" time=\"0.0\"/><testcase name=\"sqrt(20) == 5\" id=\"2\" classname=\"Math\" time=\"0.0\"><failure message=\"4.47213595499958 == 5\" type=\"test\">Test Failed
│     Expression: sqrt(20) == 5
│ -    Evaluated: 4.47213595499958 == 5</failure></testcase><testcase name=\"16 == 16\" id=\"3\" classname=\"Math\" time=\"0.0\"/></testsuite></testsuites>
│ +    Evaluated: 4.47213595499958 == 5
│ + </failure></testcase><testcase name=\"16 == 16\" id=\"3\" classname=\"Math\" time=\"0.0\"/></testsuite></testsuites>
│
│   """
│ -------------------------------
│   reference = "/Users/cvogt/.julia/dev/TestReports/test/references/complexexample.txt"
└   actual = "/var/folders/j4/g03w3zw11_74xnzmms8hqq4w0000gp/T/jl_vkpCeI/complexexample.txt"
Complex Example: Error During Test at /Users/cvogt/.julia/dev/TestReports/test/reportgeneration.jl:14
  Got exception outside of a @test
  To update the reference images either run the tests interactively with 'include("test/runtests.jl")',
  or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE`
  to "true" and re-run the tests via Pkg.

@omus
Copy link
Member Author

omus commented Apr 26, 2024

Also fixed the Documenter workflow which was running into this issue:

[ Info: CheckDocument: running document checks.
┌ Error: 10 docstrings not included in the manual:
│
│     TestReports.compatible :: Tuple{VersionNumber, VersionNumber}
│     TestReports.getdeps :: Tuple{Any}
│     TestReports.check_manifest :: NTuple{4, Any}
│     TestReports.check_manifest :: Tuple{Any, Any, Any}
│     TestReports.check_testreports_compatability :: Tuple{Any, Any, Any}
│     TestReports.hasdeps :: Tuple{Any}
│     TestReports.check_project :: Tuple{Any, Any, Any, Vector}
│     TestReports.check_project :: Tuple{Nothing, Vararg{Any}}
│     TestReports.check_env :: Tuple{Pkg.Types.EnvCache, Any, Any, Any}
│     TestReports.get_dep_entries
│
│ These are docstrings in the checked modules (configured with the modules keyword)
│ that are not included in canonical @docs or @autodocs blocks.
└ @ Documenter ~/.julia/packages/Documenter/pA5Sa/src/utilities/utilities.jl:44

@omus
Copy link
Member Author

omus commented Apr 26, 2024

I just noticed locally that I forgot to commit the new test/reference/complexexample.txt file. It's unfortunate that the CI checks still passed here as this could have resulted in this PR being merged with the reference tests being in a broken state.

Copy link
Member

@oxinabox oxinabox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@oxinabox oxinabox merged commit f13d00a into master Apr 29, 2024
10 checks passed
@omus omus deleted the cv/fix-complex-example-1.9 branch April 29, 2024 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants