Skip to content

Latest commit

 

History

History
104 lines (79 loc) · 4.08 KB

conformance.md

File metadata and controls

104 lines (79 loc) · 4.08 KB
Raw

Conformance test harness

The protovalidate conformance tests provide a rigorous set of tests to ensure the correct behavior of any protovalidate implementation. It is expected that all implementations in this project satisfy all conformance tests.

Usage

The conformance test harness can be built using make:

# builds the conformance test harness into .tmp/bin
make conformance

# run the conformance tests against an executor
.tmp/bin/protovalidate-conformance my-executor

Flags

You can customize the behavior of the tests using the following flags:

Flag Description Default Value
--suite <regex> Filter suites using the provided regular expression. None
--case <regex> Filter cases using the provided regular expression. None
--timeout <duration> Set a per-suite timeout. 5 seconds
--verbose, -v Enable verbose output. false
--strict Enable strict mode. false
--json Return results as JSON to stdout. false
--proto Return results as binary serialized proto to stdout. false
--dump Output the expected results, without a command. false
--expected-failures .yaml file containing list of expected failures None

Components

Harness

This binary is the protovalidate conformance test harness. The harness is responsible for building the test suites, shepherding them to an executor, and collating the results.

Executor

Each target library should provide a conformance test executor, an executable responsible for reading a Protobuf-serialized TestConformanceRequest over stdin, executing the provided test case validations, and respond with a Protobuf-serialized TestConformanceResponse over stdout.

It is recommended that the executor should be as type-agnostic as possible, preferably constructing the descriptors at runtime from the provided FileDescriptorSet instead of using a local resolver.

For a reference implementation, see Go's conformance test executor.

Test suites & cases

The test messages are defined in .proto files. These are utilized to make concrete message test cases which are organized into groups known as test suites. Which tests are passed to the executor can be controlled through the -suite and -case flags mentioned above.

Expected Failures

The conformance test harness can be configured to expect certain failures. This is useful for testing the behavior of a library when it encounters an invalid message. To use this feature, pass the --expected-failures flag with a path to a .yaml file containing a list of expected failures. The format of the file is as follows:

<test_suite>:
  - <test_case>

For example, if we wanted to test the behavior of a library when it encounters an invalid recursive message, we could create a file called expected_failures.yaml with the following contents:

standard_constraints/map:
  - recursive/invalid
standard_constraints/well_known_types/duration:
  - gte_lte/invalid/above
  - lte/invalid
  - not in/valid
standard_constraints/well_known_types/timestamp:
  - gte_lte/invalid/above
  - lte/invalid

Then we could run the conformance tests with the following command:

.tmp/bin/protovalidate-conformance \
  --expected-failures=expected_failures.yaml \
  my-executor