-
Notifications
You must be signed in to change notification settings - Fork 264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding muttfuzz and requesting an experiment #1967
base: master
Are you sure you want to change the base?
Conversation
Hi, @kjain14, long time no see! We have changed our way of running experiments, hope this suits you:
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-aflpp-muttfuzz --fuzzers aflplusplus_muttfuzz |
Experiment |
We accidentally only ran muttfuzz, we also want to compare against aflplusplus |
Hi Dongge,
I added another comment for another experiment. Let me know if there is
anything else I need to do (mainly we need to compare against aflplusplus).
Thanks,
Kush
…On Mon, Apr 22, 2024, 9:14 PM Dongge Liu ***@***.***> wrote:
Experiment 2024-04-23-aflpp-muttfuzz data and results will be available
later at:
The experiment data
<https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-04-23-aflpp-muttfuzz/>
.
The experiment report
<https://storage.googleapis.com/www.fuzzbench.com/reports/2024-04-23-aflpp-muttfuzz/index.html>
.
—
Reply to this email directly, view it on GitHub
<#1967 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACEOTTIU7PEALLX5OGJ2VZTY6WYYNAVCNFSM6AAAAABGFNJK6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZRGIYTQNBRGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflplusplus aflsmart_muttfuzz aflsmart libfuzzer libfuzzer_muttfuzz libafl libafl_muttfuzz honggfuzz honggfuzz_muttfuzz |
Thanks for providing us with the experiment request command, @kjain14! I noticed that the experiment requires 10 fuzzers, which may overwhelm FuzzBench due to its current bottleneck in the measurement stage. Empirically, it's safer to keep the number under 8. I will revise the new command below, please let me know if they work for you. |
Thanks! Yes merging them should work! |
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz |
The request failed because the fuzzer name is |
Apologies for the typo, this should be fixed now |
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-24-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz |
Experiment |
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-02-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz |
Hi @kjain14, would you mind if I start this experiment next week? We were trying to investigate this failure and it would be great if we could avoid starting another large experiment at the meantime. Thanks! |
Sure, we can start it next week |
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-06-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz |
Experiment |
@kjain14 fyi I saw your experiments. if you did not integrate muttfuzz on the same commit as aflplusplus that you fuzzbench is using, you will be comparing apples with oranges because there were significant advances. same if your muttfuzz afl++ commit is newer than the one on fuzzbench. I didnt check your setup. just in case: better add an aflplusplus variant that has the exact same commit id you have based your muttfuzz integration on. |
Adding MuttFuzz (a binary fuzzing alternative to our "First, Fuzz the Mutants" paper). This will likely integrate better with fuzzbench and thus work on experiments, as we are mutating binaries rather than source code.