Contributing
`TestReports.jl follows the ColPrac guide for collaborative practices. New contributors should make sure to read that guide
Click here to go to the documentation for the latest stable release.'; + div.appendChild(closer); + document.body.appendChild(div); +} + +if (document.readyState === "loading") { + document.addEventListener("DOMContentLoaded", maybeAddWarning); +} else { + maybeAddWarning(); +} diff --git a/previews/PR112/contributing/index.html b/previews/PR112/contributing/index.html new file mode 100644 index 0000000..bbfaee2 --- /dev/null +++ b/previews/PR112/contributing/index.html @@ -0,0 +1,2 @@ + +
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 21 May 2024. Using Julia version 1.10.3.
TestReports.jl
A JUnit XML test report generator for Julia
This package produces JUnit XML test reports. It is for use with your CI tooling of choice, for example a CI tool like GoCD consumes reports in this format and gives back HTML reports.
Getting Started
The reporting is designed to enable you to write your unit tests in the standard Julia way, that is using test/runtests.jl as the entry point to your tests and with default TestSet types. In theory, it should also work with custom TestSet types - see the Manual for further information.
To test and generate a report for your package:
julia> TestReports.test("MyPackage")
+# Unit tests run, report saved to testlog.xml in current working directoryTo add to CI:
$ julia -e 'using Pkg; Pkg.add("TestReports"); using TestReports; TestReports.test("MyPackage")'Additionally, properties can be associated with your testsets or tests. To do this, use the record_testset_property or record_test_property functions within a testset:
using Test
+using TestReports
+
+@testset "MyTests" begin
+ record_testset_property("ID", 1)
+ @test 1 == 1
+endThe JUnit XML Schema
The JUnit XML schema is a format used by many CI tools. For a definition of the schema, see here.
For reports generated by TestReports.jl, the timestamp attribute of testsuite elements will be in the local system timezone.
Example of Use
Below is a screen shot of TestReports being used with GoCD, to report an test failure in DataDepsGenerators.jl.
The corresponding testlog.xml file (produced with an earlier version of TestReports, and therefore missing some of the new features, and after pretty printing) is below.
<?xml version="1.0" encoding="UTF-8"?>
+<testsuites name="/UCI" id="now" tests="7" failures="1" errors="0">
+ <testsuite name="" id="_id_" tests="1" failures="0" errors="0">
+ <testcase name="pass (info lost)" id="_testcase_id_"/>
+ </testsuite>
+ <testsuite name="" id="_id_" tests="1" failures="0" errors="0">
+ <testcase name="pass (info lost)" id="_testcase_id_"/>
+ </testsuite>
+ <testsuite name="" id="_id_" tests="1" failures="0" errors="0">
+ <testcase name="pass (info lost)" id="_testcase_id_"/>
+ </testsuite>
+ <testsuite name="" id="_id_" tests="1" failures="0" errors="0">
+ <testcase name="pass (info lost)" id="_testcase_id_"/>
+ </testsuite>
+ <testsuite name="" id="_id_" tests="1" failures="0" errors="0">
+ <testcase name="pass (info lost)" id="_testcase_id_"/>
+ </testsuite>
+ <testsuite name="ForestFires" id="_id_" tests="2" failures="1" errors="0">
+ <testcase name="contains(registration_block, "A Data Mining Approach to Predict Forest Fires using Meteorological Data")" id="_testcase_id_">
+ <failure message="nothing" type="test">Test Failed
+ Expression: contains(registration_block, "A Data Mining Approach to Predict Forest Fires using Meteorological Data")</failure>
+ </testcase>
+ <testcase name="pass (info lost)" id="_testcase_id_"/>
+ </testsuite>
+</testsuites>Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 21 May 2024. Using Julia version 1.10.3.
Library
Contents
Public
Documentation for TestReports.jl's public interface.
TestReports.test — FunctionTestReports.test(; kwargs...)
+TestReports.test(pkg::Union{AbstractString, Vector{AbstractString}; kwargs...)Keyword arguments:
coverage::Bool=false: enable or disable generation of coverage statistics.julia_args::Union{Cmd, Vector{String}}: options to be passed to the test process.test_args::Union{Cmd, Vector{String}}: test arguments (ARGS) available in the test process.logfilepath::AbstractString=pwd(): file path where test reports are saved.logfilename::Union{AbstractString, Vector{AbstractString}}: name(s) of test report file(s).
Generates a JUnit XML for the tests of package pkg, or for the current project (which thus needs to be a package) if no positional argument is given to TestReports.test. The test report is saved in the current working directory and called testlog.xml if both logfilepath and logfilename are not supplied. If pkg is of type Vector{String}, the report filenames are prepended with the package name, for example Example_testlog.xml.
If logfilename is supplied, it must match the type (and length, if a vector) of pkg.
The tests are run in the same way as Pkg.test.
TestReports.record_testset_property — Functionrecord_testset_property(name::AbstractString, value)Associates a property with the current testset. The name and value will be turned into a <property> element within the corresponding <testsuite> element within the JUnit XML report.
Multiple properties can be added to one testset and child testsets inherit the properties defined by their parents. If a child testset records a property which is already set both will be present in the resulting report.
The suggested use of this function is to place it inside a testset with unspecified type (see Examples). This will ensure that Pkg.test is unnaffected, but that the properties are added to the report when TestReports.test is used. This is because properties are only added when the Testset type has a TestReports.testset_properties method defined, as does the ReportingTestSet used by TestReports. TestReports.testset_properties can be extended for custom TestSets.
If a child testset has this method defined but its parent doesn't, the property should be in the report when TestReport.test is used, assuming that the parent testset type doesn't do anything to affect the reporting behaviour. However this is not tested functionality.
The value argument must be serializable by EzXML, which gives quite a lot of freedom.
Examples
Using the default testset for compatiblity with Pkg.test and TestReports.test:
using TestReports
+
+# Default testset used, record property calls are ignored by `Pkg.test` but will be used
+# when generating JUnit XML.
+@testset "MyTestSet" begin
+ record_testset_property("ID", 42)
+ record_test_property("Bool", true)
+ @test 1 == 1
+ @test 2 == 2
+endRendering JUnit reports on the REPL:
using Test, TestReports, EzXML
+
+ts = @testset ReportingTestSet "Root" begin # `<testsuite name="Root">` has property "foo"
+ record_testset_property("foo", 1)
+ record_test_property("bar", 2)
+
+ @testset "Inner" begin # `<testsuite name="Root/Inner">` has property "foo"
+ @test 1 == 1 # `<testcase>` has the property "bar"
+ end
+
+ @test 2 != 1 # `<testcase>` has the property "bar"
+end;
+
+prettyprint(report(ts))See also: record_test_property and testset_properties.
TestReports.record_test_property — Functionrecord_test_property(name::AbstractString, value)Associates a property with the tests contained with the testset. The name and value will be turned into a <property> element with the corresponding <testcase> element within the JUnit XML report.
Multiple test properties can be assigned within a testset and child testsets will inherit the test properties defined by their parents. If a child testset records a test property with an already used name both properties will be present in the resulting report.
For more details and examples see the documentation for record_testset_property.
See also: record_testset_property and test_properties.
TestReports.ReportingTestSet — TypeReportingTestSetCustom AbstractTestSet type designed to be used by TestReports.jl for creation of JUnit XMLs.
Does not throw an error when a test fails or has an error. Upon finishing, a ReportingTestSet will display the default test output, and then flatten to a structure that is suitable for report generation.
It is designed to be wrapped around a package's runtests.jl file and this is assumed when both the test results are displayed and when the TestSet is flatted upon finish. See bin/reporttests.jl for an example of this use. ReportingTestSets are not designed to be used directly in a package's tests, and this is not recommended or supported.
A ReportingTestSet has the description and results fields as per a DefaultTestSet, and has the following additional fields:
testset_properties: used to record testset properties to be inserted into the report.test_properties: used to record test properties to be inserted into the report.start_time::DateTime: the start date and time of the testing (local system time).time_taken::Millisecond: the time taken in milliseconds to run theTestSet.last_record_time::DateTime: the time whenrecordwas last called.hostname::String: the name of host on which the testset was executed.
See also: flatten_results!, record_testset_property, record_testset_property, and report.
TestReports.any_problems — Functionany_problems(ts)Checks a testset to see if there were any problems (Errors or Fails). Note that unlike the DefaultTestSet, the ReportingTestSet does not throw an exception on a failure. Thus to set the exit code from the runner code, we check it using exit(any_problems(top_level_testset)).
TestReports.report — Functionreport(ts::AbstractTestSet) -> XMLDocumentProduce an JUnit XML report details about the contained TestSets and Results. As the JUnit XML schema does not allow nested testsuite elements the report will flatten the hierarchical TestSet structure. Each TestSet will become a testsuite element and each Result will become a testcase element.
A Result will only be reported once within its parent TestSet to avoid having duplicate entries within the report and avoid problems with total test counts not matching Julia output.
All AbstractTestSets contained within ts must have a description::AbstractString field and an iterable results field.
Private
Package internals documentation.
Report Generation
TestReports.TESTS_FAILED — ConstantExit code for runner when tests fail
TestReports.checkexitcode! — Methodcheckexitcode!(errs, proc, pkg, logfilename)Checks proc.exitcode and acts as follows:
- If 0, displays tests passed info message
- If equal to
TESTS_FAILEDconst, warning is displayed andpkgadded toerrs - If anything else, throws a
PkgTestError
TestReports.gen_command — Methodgen_command(runner_code, julia_args, coverage)Returns Cmd which will run the runner code in a new Julia instance.
See also: gen_runner_code
TestReports.gen_runner_code — Methodgen_runner_code(testfilename, logfilename, test_args)Returns runner code that will run the tests and generate the report in a new Julia instance.
TestReports.get_deps! — Methodget_deps!(deps, manifest, pkg)Push dependencies for pkg found in manifest into deps.
TestReports.get_deps — Methodget_deps(manifest, pkg) = get_deps!(String[], manifest, pkg)Get list of dependencies for pkg found in manifest
TestReports.get_manifest — Methodget_manifest()Return the parsed manifest that has TestReports as a dependency.
TestReports.get_testreports_environment — Methodget_testreports_environment()Returns new environment to be pushed to LOAD_PATH to ensure TestReports, Test and their dependencies are available for report generation.
TestReports.gettestfilepath — Methodgettestfilepath(ctx::Context, pkgspec::Pkg.Types.PackageSpec)Gets the testfile path of the package. Code for each Julia version mirrors that found in Pkg/src/Operations.jl.
TestReports.isinstalled! — Methodisinstalled!(ctx::Context, pkgspec::Pkg.Types.PackageSpec)Checks if the package is installed by using ensure_resolved from Pkg/src/Types.jl. This function fails if the package is not installed, but here we wrap it in a try-catch as we may want to test another package after the one that isn't installed.
For Julia versions V1.4 and later, the first arguments of the Pkg functions used is of type Pkg.Types.Context. For earlier versions, they are of type Pkg.Types.EnvCache.
TestReports.make_testreports_environment — Methodmake_testreports_environment(manifest)Make a new environment in a temporary directory, using information from the parsed manifest provided.
TestReports.runtests! — Methodruntests!(errs::Vector, pkg, cmd, logfilename)Runs cmd which will run the tests of pkg. The exit code of the process is then checked.
TestReports.test! — Methodtest!(pkg::AbstractString,
+ errs::Vector{AbstractString},
+ nopkgs::Vector{AbstractString},
+ notests::Vector{AbstractString},
+ logfilename::AbstractString;
+ coverage::Bool=false,
+ julia_args::Union{Cmd, AbstractVector{<:AbstractString}}=``,
+ test_args::Union{Cmd, AbstractVector{<:AbstractString}}=``)Tests pkg and save report to logfilename. Tests are run in the same way as Pkg.test.
If tests error pkg is added to nopkgs. If pkg has no testfile it is added to notests. If pkg is not installed it is added to nopkgs.
TestSets
TestReports.ReportingResult — TypeReportingResult{T}Wraps a Result of type T so that additional metadata can be saved.
fields
result::T:Resultthat is being wrapped.time_taken::Millisecond: the time taken (milliseconds) for this test to be run.
TestReports._flatten_results! — Method_flatten_results!(ts::AbstractTestSet)::Vector{<:AbstractTestSet}Recursively flatten ts to a vector of TestSets.
TestReports._flatten_results! — Method_flatten_results!(rs::Result)Return vector containing rs so that when iterated through, rs is added to the results vector.
TestReports.add_to_ts_default! — Methodadd_to_ts_default!(ts_default::DefaultTestSet, result::Result)
+add_to_ts_default!(ts_default::DefaultTestSet, result::ReportingResult)
+add_to_ts_default!(ts_default::DefaultTestSet, ts::AbstractTestSet)
+add_to_ts_default!(ts_default::DefaultTestSet, ts::ReportingTestSet)Populate ts_default with the supplied variable. If result is a Result or an AbstractTestSet (but not a ReportingTestSet) then it is recorded. If it is a ReportingTestSet then a new DefaultTestSet with matching description is created, populated by recursively calling this function and then added to the results of ts_default. If result is a ReportingResult, the Result contained by the ReportingResult is added to ts_default.
TestReports.display_reporting_testset — Methoddisplay_reporting_testset(ts::ReportingTestSet)Displays the test output in the same format as Pkg.test by using a DefaultTestSet.
TestReports.flatten_results! — Methodflatten_results!(ts::AbstractTestSet)Returns a flat vector of TestSets which only contain Results. This is necessary for writing a JUnit XML report the schema does not allow nested XML testsuite elements.
TestReports.has_description — Methodhas_description(ts::AbstractTestSet) -> BoolDetermine if the testset has been provided a description.
TestReports.hostname — Methodhostname(ts::ReportingTestSet)
+hostname(ts::AbstractTestSet)Get the hostname of a ReportingTestSet, returns gethostname() for an AbstractTestSet. Can be extended for custom TestSets, must return a string.
TestReports.ispass — Methodispass(result)Return true if result is a Pass or a ReportingResult{Pass}, otherwise return false.
TestReports.record_test_property! — Methodrecord_test_property!(ts::AbstractTestSet, name, value)Associate a property with the tests within the testset. Can be extended for custom testsets.
TestReports.record_testset_property! — Methodrecord_testset_property!(ts::AbstractTestSet, name, value)Associate a property with the testset. Can be extended for custom testsets.
TestReports.set_start_time! — Methodset_start_time!(ts::ReportingTestSet, start_time)
+set_start_time!(ts::AbstractTestSet, start_time)Sets the start time field of a ReportingTestSet. This is used when flattening ReportingTestSets for report generation and an be extended for custom TestSets.
TestReports.set_time_taken! — Methodset_time_taken!(ts::ReportingTestSet, time_taken)
+set_time_taken!(ts::AbstractTestSet, time_taken)Sets the time taken field of a ReportingTestSet. This is used when flattening ReportingTestSets for report generation and an be extended for custom TestSets.
TestReports.start_time — Methodstart_time(ts::ReportingTestSet)
+start_time(ts::AbstractTestSet)Get the start time of a ReportingTestSet, returns Dates.now() for an AbstractTestSet. Can be extended for custom TestSets, must return a DateTime.
TestReports.test_properties — Methodtest_properties(ts::AbstractTestSet) -> Union{Vector{Pair{String, Any}}, Nothing}Get the properties associated with tests within a testset. Can be extended for custom testsets. Will return nothing for testsets which do not support test properties.
When generating a JUnit XML report these will be the properties associated with all testcase elements contained within a testsuite element.
See also: testset_properties.
TestReports.testset_properties — Methodtestset_properties(ts::AbstractTestSet) -> Union{Vector{Pair{String, Any}}, Nothing}Get the properties associated with a testset. Can be extended for custom testsets. Defaults to nothing for testsets which do not support testset properties.
When generating a JUnit XML report these will be the properties associated with a testsuite element.
See also: test_properties.
TestReports.time_taken — Methodtime_taken(ts::ReportingTestSet)
+time_taken(ts::AbstractTestSet)Get the time taken of a ReportingTestSet, returns Dates.Millisecond(0) for an AbstractTestSet. Can be extended for custom TestSets, must return a Dates.Millisecond.
TestReports.time_taken — Methodtime_taken(result::ReportingResult)
+time_taken(result::Result)For a ReportingResult, return the time taken for the test to run. For a Result, return Dates.Millisecond(0).
TestReports.update_properties! — Methodupdate_properties!(childts::AbstractTestSet, ts::AbstractTestSet)Adds properties of ts to childts.
If the types of ts and/or childts do not a method defined for properties, this is handled as follows:
- If method not defined for
typeof(ts), it has no properties to add tochildtsand therefore nothing happens. - If method not defined for
typeof(childts)andtshas properties, then a warning is shown.
See also: testset_properties and test_properties.
XML Writing
TestReports.add_properties! — Methodadd_properties!(x_element, properties)Add all key value pairs defined within the properties to the referenced XML element.
TestReports.check_ts — Methodcheck_ts(ts::AbstractTestSet)Throws an exception if ts does not have the right structure for report or if the results of ts do not have both description or results fields.
See also: report
TestReports.error_xml — Methodfailure_xml(message, test_type, content)Create an error node (which will be the child of a testcase).
TestReports.failure_xml — Methodfailure_xml(message, test_type, content)Create a failure node (which will be the child of a testcase).
TestReports.format_period_for_xml — Methodformatperiodfor_xml(time::Millisecond)
Formats a millisecond value into a string in seconds with 3 decimal places.
@sprintf is used to ensure that value does not use scientific notification, which is not allowed in an XML decimal.
Methods for other Periods have not been written as are not currently required.
TestReports.get_error_info — Methodget_error_info(v::Error)Return message and type of error for testcase attribute, and number of tests (either 1 or 0). Uses test_type field to determine what caused the original error.
TestReports.get_failure_message — Methodgetfailuremessage(v::Fail)
Return message for failed test testcase attribute. Uses test_type field to determine what caused the original failure.
TestReports.set_attribute! — Methodset_attribute!(node, attr, val)
+set_attribute!(node, attr, val::Period)Add the attritube with name attr and value val to node.
If val is of type Period, format with format_period_for_xml.
TestReports.skip_xml — Methodskip_xml()Create a skip node (which will be the child of a testcase).
TestReports.testcase_xml — Methodtestcase_xml(name, id, x_children)Create a testcase element of a JUnit XML.
This is the generic form (with name, id and children) that is used by other methods.
TestReports.testcase_xml — Methodtestcase_xml(v::Result, childs)Create a testcase element of a JUnit XML for the result v.
The original expression of the test is used as the name, whilst the id is defaulted to testcaseid_.
TestReports.testsuite_xml — Methodtestsuite_xml(name, ntests, nfails, nerrors, x_children)Create a testsuite element of a JUnit XML.
TestReports.testsuites_xml — Methodtestsuites_xml(ntests, nfails, nerrors, x_children)Create the testsuites element of a JUnit XML.
TestReports.to_xml — Methodto_xml(ts::AbstractTestSet)Create a testsuite node from a AbstractTestSet, by creating nodes for each result in ts.results. For creating a JUnit XML, all results must be AbstractResults, that is they cannot be AbstractTestSets, as the XML cannot have one testsuite nested inside another.
TestReports.to_xml — Methodto_xml(res::Pass)
+to_xml(res::Fail)
+to_xml(res::Broken)
+to_xml(res::Error)
+to_xml(res::ReportingResult)Create a testcase node from the result and return it along with information on number of tests.
Index
TestReports.TESTS_FAILEDTestReports.ReportingResultTestReports.ReportingTestSetTestReports._flatten_results!TestReports._flatten_results!TestReports.add_properties!TestReports.add_to_ts_default!TestReports.any_problemsTestReports.check_tsTestReports.checkexitcode!TestReports.display_reporting_testsetTestReports.error_xmlTestReports.failure_xmlTestReports.flatten_results!TestReports.format_period_for_xmlTestReports.gen_commandTestReports.gen_runner_codeTestReports.get_depsTestReports.get_deps!TestReports.get_error_infoTestReports.get_failure_messageTestReports.get_manifestTestReports.get_testreports_environmentTestReports.gettestfilepathTestReports.has_descriptionTestReports.hostnameTestReports.isinstalled!TestReports.ispassTestReports.make_testreports_environmentTestReports.record_test_propertyTestReports.record_test_property!TestReports.record_testset_propertyTestReports.record_testset_property!TestReports.reportTestReports.runtests!TestReports.set_attribute!TestReports.set_start_time!TestReports.set_time_taken!TestReports.skip_xmlTestReports.start_timeTestReports.testTestReports.test!TestReports.test_propertiesTestReports.testcase_xmlTestReports.testcase_xmlTestReports.testset_propertiesTestReports.testsuite_xmlTestReports.testsuites_xmlTestReports.time_takenTestReports.time_takenTestReports.to_xmlTestReports.to_xmlTestReports.update_properties!
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 21 May 2024. Using Julia version 1.10.3.
Manual
Contents
Testing Packages
TestReports.jl provides two methods to test and generate JUnit XMLs for a package. The suggested approach is to use TestReports.test, but additionaly a runner script is also supplied.
TestReports.test can be used in the same way as Pkg.test. It will run the package unit tests, and, by default, create a testlog.xml file in the current directory:
julia> TestReports.test("MyPackage")It is intended that it runtests.jl will not need to be changed to generate a report (unless properties are being added).
This does assume, however, that no custom TestSets are being used. In the case of custom TestSets, please see the discussion below.
The typical use in a CI process would be:
$ julia -e 'using Pkg; Pkg.add("TestReports"); using TestReports; TestReports.test("MyPackage")'It is possible for multiple packages to be tested at once (with one report generated per package), or for the current project to be tested:
julia> TestReports.test(["MyPackage1", "MyPackage2"]) # Multiple packages
+julia> TestReports.test() # Current projectTestReports.test will display the same output as Pkg.test, and accepts the same keyword arguments, namely:
coverage- enable or disable generation of coverage statistics.julia_args- options to be passed to the test process.test_args- test arguments (ARGS) available in the test process.
and has two additional keyword arguments to control the name and location of the report(s):
logfilepath- directory in which test reports are saved.logfilename- name(s) of test report file(s).
Customising Testfile Name and Location
When testing a single package, the testfile name defaults to testlog.xml and is saved in the current working directory. The keyword arguments logfilepath and logfilename can be used to modify the name and/or path of the testfile, for example:
julia> TestReports.test("MyPackage", logfilename="MyPackage.xml")
+# File path is joinpath(pwd(), "MyPackage.xml")
+julia> TestReports.test("MyPackage", logfilepath="path")
+# File path is joinpath("path", "testlog.xml")
+julia> TestReports.test("MyPackage", logfilename="MyPackage.xml", logfilepath="path")
+# File path is joinpath("path", "MyPackage.xml")When testing multiple packages, the testfiles default to being called PackageName_testlog.xml for each package. The keyword arguments can again be used to customise this, for example:
julia> TestReports.test(["MyPackage1", "MyPackage2"], logfilename=["1.xml", "2.xml"])
+# File paths are joinpath(pwd(), "1.xml") and joinpath(pwd(), "2.xml")
+julia> TestReports.test(["MyPackage1", "MyPackage2"], logfilename=["1.xml", "2.xml"], logfilepath="path")
+# File paths are joinpath("path", "1.xml") and joinpath("path", "2.xml")Associating Properties
Properties can be associated to a testsets and/or tests by using the respective record_testset_property and record_test_property functions within a testset:
using Test
+using TestReports
+
+@testset "MyTests" begin
+ record_testset_property("ID", 1)
+ @test 1 == 1
+endThe record_*_property functions will have no affect during normal unit testing.
The associated properties will be added to the corresponding testsuite or testcase in the generated report. Multiple properties can be added, and a properties added to a parent testset will be applied to all child testsets. Duplicate properties are allowed to be set.
The property name must always be a String, but the value can be anything that is serializable by EzXML.jl. In practice this means that Strings, Numbers, Exprs and Symbols can be used, as well as other types.
This example shows a more complete example:
using Test
+using TestReports
+
+@testset "TopLevelTests" begin
+ # This will be added to all child testsets in report. May not be appropriate when using
+ # `include`.
+ record_testset_property("TestFile", @__FILE__)
+
+ @testset "MiddleLevelTests" begin
+ record_testset_property("Testsuite", 100)
+ record_test_property("TestSubject", "Example")
+
+ @testset "Inner" begin
+ record_testset_property("Testsuite", 101) # Associate with both testsuite 100 and 101
+ @test 1 == 1
+ end
+
+ @testset "Types" begin
+ record_test_property("Prop1", :Val1)
+ @test 1 == 1
+ end
+ end
+endRunner Script
Alternatively to TestReports.test, bin/reporttests.jl is a script that runs tests and reports the results. Use it via:
$ julia bin/reporttests.jl tests/runtests.jlReplacing tests/runtests.jl with the path to the package test file. This script creates a file testlog.xml in the current directory.
Known Limitations
Single Nesting
Julia TestSets can be nested to an arbitary degree, but this is not allowed by the JUnit XML schema. Therefore, nested TestSets are flatted in the report.
For example, the following runtests.jl file:
using Test
+
+@testset "TopLevel" begin
+ @testset "Middle1" begin
+ @test 1==1
+ @test 2==2
+ end
+ @testset "Middle2" begin
+ @testset "Inner1" begin
+ @test 1==1
+ end
+ end
+ @test 1==1
+endWill generate the following XML (when pretty printed):
<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="4" failures="0" errors="0">
+ <testsuite name="TopLevel/Middle1" tests="2" failures="0" errors="0" time="0.156" timestamp="2022-03-30T07:55:12.173" hostname="hostname" id="0">
+ <testcase name="1 == 1" id="1" classname="TopLevel/Middle1" time="0.028"/>
+ <testcase name="2 == 2" id="2" classname="TopLevel/Middle1" time="0.000"/>
+ </testsuite>
+ <testsuite name="TopLevel/Middle2/Inner1" tests="1" failures="0" errors="0" time="0.000" timestamp="2022-03-30T07:55:12.329" hostname="hostname" id="1">
+ <testcase name="1 == 1" id="1" classname="TopLevel/Middle2/Inner1" time="0.000"/>
+ </testsuite>
+ <testsuite name="TopLevel" tests="1" failures="0" errors="0" time="0.156" timestamp="2022-03-30T07:55:12.173" hostname="hostname" id="2">
+ <testcase name="1 == 1" id="1" classname="TopLevel" time="0.000"/>
+ </testsuite>
+</testsuites>Each test is recorded in a separate testsuite with the name showing the original nesting.
Custom TestSet Types
TestReports.jl has not been tested significantly with custom TestSet types, please raise an issue if you find any problems/have a request.
However at a minimum, for a custom TestSet type to work with TestReports it must:
- Push itself onto its parent when finishing, if it is not at the top level
- Have
descriptionandresultsfields as per aDefaultTestSet
The following information in a JUnit XML relies on the functionality of ReportingTestSets but can be added to your own custom TestSet as described in the table.
| Information | Description |
|---|---|
| testcase time | This is extracted from a ReportingResult by the TestReports.time_taken function. For standard Results, rather than ReportingResults, this function returns Dates.Millisecond(0). This function can be extended for other custom Result types. |
| testsuite time | This is extracted from a TestSet by the TestReports.time_taken function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to Dates.Millisecond(0). |
| testsuite timestamp | This is extracted from a TestSet by the TestReports.start_time function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to Dates.now(). |
| testsuite hostname | This is extracted from a TestSet by the TestReports.hostname function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to gethostname(). |
| testsuite properties | This is extracted from a TestSet by the TestReports.properties function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to nothing. |
For further details on extending these fuctions, see the docstrings in TestSets.
The source code of TestReports can be used as a starting point for including this behaviour in your custom TestSets.
If no TestSet types are specified (as per the standard Test approach), TestSet functionality will ensure that all child TestSets inherit the ReportingTestSet type.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 21 May 2024. Using Julia version 1.10.3.