Adding muttfuzz and requesting an experiment

[kjain14]

Adding MuttFuzz (a binary fuzzing alternative to our "First, Fuzz the Mutants" paper). This will likely integrate better with fuzzbench and thus work on experiments, as we are mutating binaries rather than source code.

[DonggeLiu]

Hi, @kjain14, long time no see!
Thanks for updating the experiment request file.

We have changed our way of running experiments, hope this suits you:

1. Would you mind making a trivial modification to service/gcbrun_experiment.py?
   This will allow me to launch experiments in this PR before merging.
   Here is an example to add a dummy comment, thanks!

2. Could you please write your experiment request in this format?
   You can swap the --experiment-name and --fuzzers parameters with your values.
   For example, in your case, the request command this time is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-aflpp-muttfuzz --fuzzers aflplusplus_muttfuzz

3. You no longer have to edit service/experiment-requests.yaml in the future : )

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-aflpp-muttfuzz --fuzzers aflplusplus_muttfuzz

[DonggeLiu]

Experiment 2024-04-23-aflpp-muttfuzz data and results will be available later at:
The experiment data.
The experiment report.

[kjain14]

We accidentally only ran muttfuzz, we also want to compare against aflplusplus

[kjain14]

Hi Dongge, I added another comment for another experiment. Let me know if there is anything else I need to do (mainly we need to compare against aflplusplus). Thanks, Kush

…

On Mon, Apr 22, 2024, 9:14 PM Dongge Liu ***@***.***> wrote: Experiment 2024-04-23-aflpp-muttfuzz data and results will be available later at: The experiment data <https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-04-23-aflpp-muttfuzz/> . The experiment report <https://storage.googleapis.com/www.fuzzbench.com/reports/2024-04-23-aflpp-muttfuzz/index.html> . — Reply to this email directly, view it on GitHub <#1967 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEOTTIU7PEALLX5OGJ2VZTY6WYYNAVCNFSM6AAAAABGFNJK6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZRGIYTQNBRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kjain14]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflplusplus aflsmart_muttfuzz aflsmart libfuzzer libfuzzer_muttfuzz libafl libafl_muttfuzz honggfuzz honggfuzz_muttfuzz

[DonggeLiu]

Thanks for providing us with the experiment request command, @kjain14!

I noticed that the experiment requires 10 fuzzers, which may overwhelm FuzzBench due to its current bottleneck in the measurement stage. Empirically, it's safer to keep the number under 8.
Meanwhile, this config enables FuzzBench to merge fuzzers' results from past experiments into your report when your experiment completes so that we don't have to re-run them in every new experiment.
For example, your previous experiment report should have data of other fuzzers (including afl++) once it completes.
Here is another example command and its report. The report merges core fuzzers' results from past experiments even though they were not explicitly mentioned in the command.

I will revise the new command below, please let me know if they work for you.

[kjain14]

Thanks! Yes merging them should work!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

[DonggeLiu]

The request failed because the fuzzer name is libafl_muttfuzz but its directory name is libafll_muttfuzz (with two ls in afll), is this a typo?
Would you like to change it or it's ok to use that name?

[kjain14]

Apologies for the typo, this should be fixed now

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-24-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

[DonggeLiu]

Experiment 2024-04-24-full-muttfuzz data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[kjain14]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-02-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

[DonggeLiu]

Hi @kjain14, would you mind if I start this experiment next week?

We were trying to investigate this failure and it would be great if we could avoid starting another large experiment at the meantime.

Thanks!

[kjain14]

Sure, we can start it next week

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-06-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

[DonggeLiu]

Experiment 2024-05-06-full-muttfuzz data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[vanhauser-thc]

@kjain14 fyi I saw your experiments. if you did not integrate muttfuzz on the same commit as aflplusplus that you fuzzbench is using, you will be comparing apples with oranges because there were significant advances. same if your muttfuzz afl++ commit is newer than the one on fuzzbench. I didnt check your setup. just in case: better add an aflplusplus variant that has the exact same commit id you have based your muttfuzz integration on.
comparing against honggfuzz looks good, so I would expect it should also improve afl++.