ReferenceTests.jl is a Julia package that adds a couple of
additional macros to your testing toolbox. In particular, it
focuses on functionality for testing values against reference
files, which in turn the package can help create and update if
need be. ReferenceTests.jl is build on top of
FileIO.jl and designed
to be used alongside Base.Test.
| Package Status | Package Evaluator | Build Status |
|---|---|---|
  |
  |
  |
Note: This package is still in early development. So far most
attention went into comparing images and strings. The FileIO
fallbacks are thus yet not fully fleshed out and may change over
time.
It is very common for Julia packages to test the functionality of
their exported functions against known input-to-output
combinations. We will refer to such kind of tests as reference
tests. In most cases these will be quite simple; something along
the line of @test f(x) == y, where f is a function of the
user package and x is some interesting input value for which
the desired output y is known.
For testing the output of more complex functions, for which the
expected output is more complicated (e.g. anything image
processing related), using @test can be a little cumbersome to
work with. To that end this package provides the
@test_reference macro, which expects a filename (relative to
the file that invokes the macro) and an expression that evalutes
to the value of interest.
using ReferenceTests
@test_reference "stringtest1.txt" string(collect(1:20))If you put the above code into your test/runtests.jl and
execute the file in an interactive julia session (i.e. with
include), then it will trigger an interactive dialog if the
results don't match or the file does not yet exist. This dialog
allows the user to create and/or update the reference files.
The given file stringtest1.txt is assumed to be the relative
path to the file that contains the macro invocation. This likely
means that the path is relative to the test/ folder of your
package.
The file-extension of the filename (here txt), as well as the
type of the result of evaluating the expression (here String),
determine how the actual value is compared to the reference
value. The default implementation will do a simple equality check
with the result of FileIO.load. This means that it is the
user's responsibility to have the required IO package installed.
Colorant arrays (i.e.) receive special treatment. If the
extension of the filename is txt then the package
ImageInTerminal.jl
will be used to create a string-based crude approximation of the
image. This will have low storage requirements and also allows to
view the reference file in a simple terminal using cat.
using ReferenceTests, TestImages
@test_reference "imagetest1.txt" testimage("cameraman")
Note that while a text-based storage of reference images can be
convenient, proper image formats (e.g. png) are also supported
by the package. Those, however, will require the proper FileIO
backends to be installed.
Another special file extension is sha256 which will cause the
hash of the result of the given expression to be stored and
compared as plain text. This is useful for a convenient
low-storage way of making sure that the return value doesn't
change for selected test cases.
Check out the latest documentation
Additionally, you can make use of Julia's native docsystem.
The following example shows how to get additional information
on @test_reference within Julia's REPL:
?@test_referenceThis package is registered in METADATA.jl and can be installed
as usual.
Pkg.add("ReferenceTests")If you intend to use it for testing on CI, make sure to add the
package name ReferenceTests to your test/REQUIRE file.
Further note, that depending on what file-format you use to store
your references, you may need to add additional dependencies to
your test/REQUIRE file.
This code is free to use under the terms of the MIT license.