Paper (preprint): https://arxiv.org/abs/1911.10621

DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning

Testing Deep Neural Network (DNN) models has become more important than ever with the increasing usage of DNN models in safety-critical domains such as autonomous cars. The traditional approach of testing DNNs is to create a test set, which is a random subset of the dataset about the problem of interest. This kind of approach is not enough for testing most of the real-world scenarios since these traditional test sets do not include corner cases, while a corner case input is generally considered to introduce erroneous behaviors. Recent works on adversarial input generation, data augmentation, and coverage-guided fuzzing (CGF) have provided new ways to extend traditional test sets. Among those, CGF aims to produce new test inputs by fuzzing existing ones to achieve high coverage on a test adequacy criterion (i.e. coverage criterion). Given that the subject test adequacy criterion is a well-established one, CGF can potentially find error inducing inputs for different underlying reasons. In this paper, we propose a novel CGF solution for structural testing of DNNs. The proposed fuzzer employs Monte Carlo Tree Search to drive the coverage-guided search in the pursuit of achieving high coverage. Our evaluation shows that the inputs generated by our method result in higher coverage than the inputs produced by the previously introduced coverage-guided fuzzing techniques.

1) Install Dependencies

pip install -r requirements.txt

2) Usage

usage: run_experiment.py [-h] [--params_set [PARAMS_SET [PARAMS_SET ...]]]
                         [--dataset {MNIST,CIFAR10}]
                         [--model {LeNet1,LeNet4,LeNet5,CIFAR_ORIGINAL}]
                         [--implicit_reward [IMPLICIT_REWARD]]
                         [--coverage {neuron,kmn,nbc,snac,tfc}]
                         [--input_chooser {random,clustered_random}]
                         [--runner {mcts,mcts_clustered,deephunter,tensorfuzz}]
                         [--batch_size BATCH_SIZE]
                         [--nb_iterations NB_ITERATIONS]
                         [--random_seed RANDOM_SEED] [--verbose [VERBOSE]]
                         [--image_verbose [IMAGE_VERBOSE]]

Experiments Script For DeepSmartFuzzer

optional arguments:
  -h, --help            show this help message and exit
  --params_set [PARAMS_SET [PARAMS_SET ...]]
                        see params folder
  --dataset {MNIST,CIFAR10}
  --model {LeNet1,LeNet4,LeNet5,CIFAR_ORIGINAL}
  --implicit_reward [IMPLICIT_REWARD]
  --coverage {neuron,kmn,nbc,snac,tfc}
  --input_chooser {random,clustered_random}
  --runner {mcts,mcts_clustered,deephunter,tensorfuzz}
  --batch_size BATCH_SIZE
  --nb_iterations NB_ITERATIONS
  --random_seed RANDOM_SEED
  --verbose [VERBOSE]
  --image_verbose [IMAGE_VERBOSE]

Copyright Notice

DeepSmartFuzzer Copyright (C) 2019 Bogazici University

DeepSmartFuzzer is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

DeepSmartFuzzer is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with DeepSmartFuzzer. If not, see https://www.gnu.org/licenses/.

mail: [email protected], [email protected]