Testing

clFFT Tests

The clFFT test suite is based on the googletest framework. The test sources are located in ./src/tests/library. When the test code is built, there is a single test executable that is created called Test. This is the program to run in order to verify the functional correctness of the library.

Setting up the tests to run

The cmake generated build projects define an INSTALL target that can be built, which in addition to compiling the sources also goes through the extra work of creating a ./bin/clFFT/develop/vs10x64/package subdirectory and copying all built executables and libraries together into the same directory. This is convenient for performance measurement and testing, as typically the build tree has the executables and libraries scattered and built in their own build directories.

The test executable uses the FFTW library to validate answers. FFTW needs to be installed to successfully build and run the tests. Find below the help message given by the test program. The test program takes in various parameters to control the scope of test cases to be executed.

C:\clFFT\bin\package\bin64>Test.exe -h

clFFT client API version:  2.1.0
clFFT runtime version:     2.1.0

OpenCL platform [ 0 ]:
. . .

OpenCL devices [ 0 ]:
. . .

clFFT Runtime Test command line options:
  -h [ --help ]                         produces this help message
  -v [ --verbose ]                      print out detailed information for the
                                        tests
  --noVersion                           Don't print version information from
                                        the clFFT library
  --noInfoCL                            Don't print information from the OpenCL
                                        runtime
  -c [ --cpu ]                          Run tests on a CPU device
  -g [ --gpu ]                          Run tests on a GPU device (default)
  -p [ --pointwise ]                    Do a pointwise comparison to determine
                                        test correctness (default: use root
                                        mean square)
  -t [ --tolerance ] arg (=0.00100000005)
                                        tolerance level to use when determining
                                        test pass/fail
  -r [ --numRandom ] arg (=2000)        number of random tests to run
  --seed arg (=66780601)                seed to use for the random test.
                                        defaults to time(NULL)
  -s [ --short ]                        Run radix 2 tests; no random testing
  -m [ --medium ]                       Run all radices; no random testing

The exact time it takes for the test executables to finish is dependent on the hardware under test, and varies widely. Since the test framework is based on gtest, gtest filters can be applied to narrow testing to only a fraction of the overall tests. The gtest_filter flag takes a regular expression that it matches to the test name, and each test name is unique.

There are 2 groups of tests that the program runs. The first group is the individually named unit tests that tests exactly the same transform problem every time. The second group is the random tests that we will cover later. The first group of tests can be filtered using gtest filters. Example of running only tests that run power-of-2 transform lengths is shown below.

C:\clFFT\bin\package\bin64>Test.exe --gtest_filter=*pow2*

clFFT client API version:  2.1.0
clFFT runtime version:     2.1.0

OpenCL platform [ 0 ]:
. . .

OpenCL devices [ 0 ]:
. . .

Result comparison tolerance is 0.001
Random test's seed is 66783095
Note: Google Test filter = *pow2*
[==========] Running 436 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 218 tests from accuracy_test_pow2_single
[ RUN      ] accuracy_test_pow2_single.normal_1D_forward_in_place_complex_planar_to_complex_planar
[       OK ] accuracy_test_pow2_single.normal_1D_forward_in_place_complex_planar_to_complex_planar (424 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_backward_in_place_complex_planar_to_complex_planar
[       OK ] accuracy_test_pow2_single.normal_1D_backward_in_place_complex_planar_to_complex_planar (356 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_forward_in_place_complex_interleaved_to_complex_interleaved
[       OK ] accuracy_test_pow2_single.normal_1D_forward_in_place_complex_interleaved_to_complex_interleaved (329 ms)
[ RUN      ] accuracy_test_pow2_single.len65536_1D_forward_in_place_complex_interleaved_to_complex_interleaved
[       OK ] accuracy_test_pow2_single.len65536_1D_forward_in_place_complex_interleaved_to_complex_interleaved (327 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_backward_in_place_complex_interleaved_to_complex_interleaved
[       OK ] accuracy_test_pow2_single.normal_1D_backward_in_place_complex_interleaved_to_complex_interleaved (328 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_forward_out_of_place_complex_planar_to_complex_planar
[       OK ] accuracy_test_pow2_single.normal_1D_forward_out_of_place_complex_planar_to_complex_planar (362 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_backward_out_of_place_complex_planar_to_complex_planar
[       OK ] accuracy_test_pow2_single.normal_1D_backward_out_of_place_complex_planar_to_complex_planar (360 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_forward_out_of_place_complex_interleaved_to_complex_interleaved
[       OK ] accuracy_test_pow2_single.normal_1D_forward_out_of_place_complex_interleaved_to_complex_interleaved (334 ms)
[ RUN      ] accuracy_test_pow2_single.normal_1D_backward_out_of_place_complex_interleaved_to_complex_interleaved

Negative test filters are also available with the '-' operator to filter out tests. A filter that avoids running power-of-3 tests would look like --gtest_filter=-pow3 with similar output as above. A complicated expression can be created using both positive and negative filters, separated by the ':' character. This is just using standard googletest filter notation.

Randomized tests

When the test program is run without any options, it runs standard unit tests available and also a set of random test cases. These random tests choose the parameters of FFT transform randomly and tests them. By default, the program runs 2000 random tests. But the number of random tests can be controlled. Also, using the verbose switch gives useful information on the kind of transform that is tested. Here is an example:

C:\clFFT\bin\package\bin64>Test.exe -r 5 -v --gtest_filte

clFFT client API version:  2.1.0
clFFT runtime version:     2.1.0

OpenCL platform [ 0 ]:
. . .

OpenCL devices [ 0 ]:
. . .

Result comparison tolerance is 0.001
Random test's seed is 66783469
Note: Google Test filter = *random*
[==========] Running 5 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 5 tests from clAmdFft_RandomTest/accuracy_test_random
[ RUN      ] clAmdFft_RandomTest/accuracy_test_random.random_transform/0
transform parameters as seen by clAmdFft:
2 dimension(s): 108 750
batch: 5
double precision
in-place
hermitian interleaved -> real
input stride(s): 1 166
output stride(s): 3 332
input distance: 124582
output distance: 249164
[       OK ] clAmdFft_RandomTest/accuracy_test_random.random_transform/0 (4520 ms)
[ RUN      ] clAmdFft_RandomTest/accuracy_test_random.random_transform/1
transform parameters as seen by clAmdFft:
3 dimension(s): 12 3750 2
batch: 1
double precision
out-of-place
complex interleaved -> complex interleaved
input stride(s): 4 50 187500
output stride(s): 1 15 56250
input distance: 375011
output distance: 112560
[       OK ] clAmdFft_RandomTest/accuracy_test_random.random_transform/1 (2099 ms)
[ RUN      ] clAmdFft_RandomTest/accuracy_test_random.random_transform/2
transform parameters as seen by clAmdFft:
2 dimension(s): 1200 540
batch: 2
double precision
out-of-place
real -> hermitian planar
input stride(s): 1 1200
output stride(s): 2 2401
input distance: 648000
output distance: 1296662
[       OK ] clAmdFft_RandomTest/accuracy_test_random.random_transform/2 (10036 ms)
[ RUN      ] clAmdFft_RandomTest/accuracy_test_random.random_transform/3
transform parameters as seen by clAmdFft:
3 dimension(s): 5400 18 2
batch: 1
single precision
out-of-place
complex interleaved -> complex interleaved
input stride(s): 4 21601 388820
output stride(s): 1 5400 97200
input distance: 777757
output distance: 194400
[       OK ] clAmdFft_RandomTest/accuracy_test_random.random_transform/3 (3813 ms)
[ RUN      ] clAmdFft_RandomTest/accuracy_test_random.random_transform/4
transform parameters as seen by clAmdFft:
3 dimension(s): 15 2025 24
batch: 1
double precision
in-place
complex interleaved -> complex interleaved
input stride(s): 1 18 36454
output stride(s): 1 18 36454
input distance: 874989
output distance: 874989
[       OK ] clAmdFft_RandomTest/accuracy_test_random.random_transform/4 (6868 ms)
[----------] 5 tests from clAmdFft_RandomTest/accuracy_test_random (27342 ms total)

[----------] Global test environment tear-down
[==========] 5 tests from 1 test case ran. (27347 ms total)
[  PASSED  ] 5 tests.