This repository contains tests generated by Ferret to test DNS nameserver implementations like Bind, NSD, Knot, PowerDNS, and others. Ferret is the first automatic test generator for DNS nameserver implementation RFC compliance. The tests in this repository were used successfully to find multiple bugs, including crashes, in many famous open-source implementation. The list can be found in the Ferret GitHub README section — Bugs Found 🐛.
DNS OARC also tweeted about Ferret saying: "The automation tool received great compliments from the DNS experts".
A test case in DNS consists of a query and a zone file. Zone files are highly structured objects with various syntactic and semantic well-formedness requirements, and the query must be related to the zone file for the test even to reach the core DNS name resolution logic.
Ferret jointly generates zone files and the corresponding queries, does so in a way that is targeted towards covering many different RFC behaviors, and is applicable to black-box DNS implemenetations. The key insight underlying Ferret is using the existing RFCs to build a much simpler, logical model of DNS resolution and then use this model to guide test generation. More details can be found in Ferret's Test Generation Module README.
The dataset is organized as follows:
FerretDataset ├── ValidZoneFileTests │ ├── ZoneFiles │ │ ├── 0.txt │ │ ├── 1.txt │ │ ├── ... │ │ └── 12673.txt │ └── ExpectedResponses │ ├── 0.json │ ├── 1.json │ ├── ... │ └── 12673.json └── InvalidZoneFileTests ├── FalseCond_1 │ ├── ZoneFiles │ │ ├── 0.txt │ │ ├── ... │ │ └── 99.txt │ ├── PreprocessorOutputs │ │ ├── 0.json │ │ ├── ... │ │ └── 99.json │ ├── EquivalenceClassNames │ │ ├── 0.txt │ │ ├── ... │ │ └── 99.txt │ └── ExpectedResponses │ ├── 0.json │ ├── ... │ └── 99.json ... └── FalseCond_9
Ferret generated 12,673 tests where the zone file in each test case follows the syntactic and semantic requirements mentioned in RFCs. All those zone files are in the ValidZoneFileTests/ZoneFiles directory.
The query that was jointly generated with each zone file is in the ValidZoneFileTests/ExpectedResponses directory, along with the responses for each query from four popular open-source DNS implementations — Bind, Knot, NSD, and PowerDNS. Each implementation link points to the code base at which it was cloned for response generation.
The corresponding zone file and query file are named with the same name but with a different extension — 0.txt in ZoneFiles is the zone file, and 0.json in ExpectedResponses is the corresponding query file along with the response from the above four implementations.
While testing with valid zone files is critical, bugs can also lurk in implementations' handling of ill-formed zones. Ferret also generated zone files that violated one of the zone validity conditions systematically, and to test using such zone files, leveraged GRoot to generate queries. We formulated 9 zone semantic validity conditions and Ferret generated 100 zone files with one of them violated, but the other eight satisfied.
InvalidZoneFileTests/FalseCond_1 directory contains all the info related to the tests where the zone files in those tests violate validity condition one, but all the other conditions are satisfied. The subdirectories in it are:
ZoneFiles- All the Ferret generated zone files that do not satisfy the first zone validity conditionPreprocessorOutputs- The output from checking each of the zone files with each implementation's preprocessor: named-checkzone for Bind, kzonecheck for Knot, nsd-checkzone for NSD, and pdnsutil for PowerDNS.EquivalenceClassNames- The query equivalence class names generated for zone files using GRoot. Ferret uses the names in these files as query names and pairs them with different query types to test the implementations with the invalid zone files.ExpectedResponses- The responses from the four implementations for the queries if the implementation's preprocessor accepted the zone file.
The tests in this dataset repository are licensed under the MIT License.