Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto-generate benchmarks with genthat, and run them #22

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open

Auto-generate benchmarks with genthat, and run them #22

wants to merge 82 commits into from

Commits on Aug 3, 2021

  1. Tooling for benchmarks generated by genthat.

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    bad7b51 View commit details
    Browse the repository at this point in the history

  2. Keep some generated tests (from yaml package) in the repo.

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    dee0ff7 View commit details
    Browse the repository at this point in the history

  3. Quote entry name for genthat benchmarks in rebench.conf.

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    23033e7 View commit details
    Browse the repository at this point in the history

  4. Add script to download and install R packages.

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    798fb9b View commit details
    Browse the repository at this point in the history

  5. Download and install in a single script.

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    ec641dc View commit details
    Browse the repository at this point in the history

  6. Commit packages.txt file

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    cafbeb8 View commit details
    Browse the repository at this point in the history

  7. Specify repo when downloading devtools

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    617f4c1 View commit details
    Browse the repository at this point in the history

  8. Do not depend on genthat for setup_R_packages.sh

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    0d47711 View commit details
    Browse the repository at this point in the history

  9. Add genthat-CRAN in the suits in rebench.conf

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    eee62cd View commit details
    Browse the repository at this point in the history

  10. Add prodlim tests in the repo.

    @vogr
    vogr committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    640490b View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2021

  1. Improve script to install packages: only use stdlib functions.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    b15a4b3 View commit details
    Browse the repository at this point in the history

  2. Remove freeze.sh script: no needed since we use MRAN.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    079d6f0 View commit details
    Browse the repository at this point in the history

  3. intall_pkgs.sh: make saving the source optionnal.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    a2bc055 View commit details
    Browse the repository at this point in the history

  4. Script to extract testcases from the installed packages.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    ac3bbb7 View commit details
    Browse the repository at this point in the history

  5. Add parameter to change number of inner iterations.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    079420b View commit details
    Browse the repository at this point in the history

  6. Fix check in number of arguments.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    45d74f2 View commit details
    Browse the repository at this point in the history

  7. Add README.md explaining how to extract testcases using the Dockerfile.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    5108f1d View commit details
    Browse the repository at this point in the history

  8. Install dependancies for extract_testcases.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    54c8b5c View commit details
    Browse the repository at this point in the history

  9. Add Dockerfile to automate test extraction.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    6d843d7 View commit details
    Browse the repository at this point in the history

  10. Add main entrypoint script for Dockerfile

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    93647dc View commit details
    Browse the repository at this point in the history

  11. Typos in README and run.sh

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    536b228 View commit details
    Browse the repository at this point in the history

  12. Split installation of genthat and of pkgs in two scripts.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    973e519 View commit details
    Browse the repository at this point in the history

  13. Typo in README.md

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    11ede08 View commit details
    Browse the repository at this point in the history

  14. Also pin CRAN snapshot for installation of genthat.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    94bdf27 View commit details
    Browse the repository at this point in the history

  15. Do not try to extract source to null directory.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    c6046ca View commit details
    Browse the repository at this point in the history

  16. Optionnaly, do not specify install directory

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    793ed35 View commit details
    Browse the repository at this point in the history

  17. Do not put class name to lower in genthat benchmarks

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    b58a7a3 View commit details
    Browse the repository at this point in the history

  18. Count number of inner-iterations necessary to reach 200ms of runtime … 

    …per outer-iteration
    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    b5d7d32 View commit details
    Browse the repository at this point in the history

  19. Correctly generate the benchmarks: only keep function that can be run by 

    GNU R and prune.
    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    3f7518d View commit details
    Browse the repository at this point in the history

  20. Read number of inner-iterations from the file previously generated

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    14240c9 View commit details
    Browse the repository at this point in the history

  21. Correctly read nb iteration from dictionnary.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    81513b9 View commit details
    Browse the repository at this point in the history

  22. Keep number of inner-iteration saved in the repo.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    7a39153 View commit details
    Browse the repository at this point in the history

  23. Shorter names fir inner-it determination scripts.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    8b35f25 View commit details
    Browse the repository at this point in the history

  24. Cleanup extract_testcases script.

    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    dd2f454 View commit details
    Browse the repository at this point in the history

  25. Use file specifying number of inner it for each benchmark when 

    generating rebench.conf.
    @vogr
    vogr committed Aug 5, 2021
    Configuration menu
    Copy the full SHA
    2dc80f3 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2021

  1. Check result for each inner iteration (out of the timed loop)

    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    7c2b52c View commit details
    Browse the repository at this point in the history

  2. Wrap function_to_run() result in a list() to prevent NULL results from 

    being assigned to the results list, which would actually remove a cell
from the list instead of assigning NULL to it.
    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    1929d1b View commit details
    Browse the repository at this point in the history

  3. Remove old yaml tests.

    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    79b6de3 View commit details
    Browse the repository at this point in the history

  4. More documentation.

    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    17a2bbc View commit details
    Browse the repository at this point in the history

  5. Save retv and check it as a separate step (ie not directly in genthat… 

    …) to prevent state related issues.
    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    24de287 View commit details
    Browse the repository at this point in the history

  6. Add the recording and checking steps to the container main command

    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    cc2d980 View commit details
    Browse the repository at this point in the history

  7. Remove old versions of generated prodlim benchmarks

    @vogr
    vogr committed Aug 6, 2021
    Configuration menu
    Copy the full SHA
    3c04605 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2021

  1. Record_retv: run a new R session for every testcase

    @vogr
    vogr committed Aug 9, 2021
    Configuration menu
    Copy the full SHA
    68e6971 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2021

  1. Add helper scripts.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    273142d View commit details
    Browse the repository at this point in the history

  2. Make record_retv parallel

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    52fd1c8 View commit details
    Browse the repository at this point in the history

  3. Add comment about parallelism in extract_testcases

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    5e7b310 View commit details
    Browse the repository at this point in the history

  4. Make check against_recorded_retv parallel

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    32848e0 View commit details
    Browse the repository at this point in the history

  5. Change working directory in parallel loop for record_retv

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    0b83398 View commit details
    Browse the repository at this point in the history

  6. Set WD in parallel loop for extract_testcases.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    c402938 View commit details
    Browse the repository at this point in the history

  7. Do not use next in parallel loop

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    c67d826 View commit details
    Browse the repository at this point in the history

  8. List callr as dep

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    4a72ed0 View commit details
    Browse the repository at this point in the history

  9. Complete README with manual instructions.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    98a92c9 View commit details
    Browse the repository at this point in the history

  10. Correctly save seed in record_retv

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    2c9c7bc View commit details
    Browse the repository at this point in the history

  11. Typo

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    e7ec0c6 View commit details
    Browse the repository at this point in the history

  12. Use parallel record_retv in instructions.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    645f500 View commit details
    Browse the repository at this point in the history

  13. Correctly save seed.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    da79d8a View commit details
    Browse the repository at this point in the history

  14. Add selection of generated tests with their retv (filtered to keep only 

    those that succeed with GNUR)
    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    f039551 View commit details
    Browse the repository at this point in the history

  15. Use same pattern for extension added to failed tests.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    943b5a4 View commit details
    Browse the repository at this point in the history

  16. Add comment about isolation in min_nb_iter

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    5799897 View commit details
    Browse the repository at this point in the history

  17. Update usage of min_nb_iter.R

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    f4780e3 View commit details
    Browse the repository at this point in the history

  18. Don't load doParallel in scripts, use it through namespace.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    f589d2e View commit details
    Browse the repository at this point in the history

  19. Move capture.output outside of the hot loop in min_inner_it computation.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    4dfdbc1 View commit details
    Browse the repository at this point in the history

  20. min_nb_iter: source in a user-created environment

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    a999691 View commit details
    Browse the repository at this point in the history

  21. Update n_inner_it command in README

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    4ea7194 View commit details
    Browse the repository at this point in the history

  22. Commit n_inner_it for the 2000 first testcases.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    bccf9e8 View commit details
    Browse the repository at this point in the history

  23. Update packages.txt

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    5e6cb03 View commit details
    Browse the repository at this point in the history

  24. Remove mistakenly added line in packages.txt

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    68ecea1 View commit details
    Browse the repository at this point in the history

  25. Keep only the extracted testcases from the 100 first packages.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    e6d0241 View commit details
    Browse the repository at this point in the history

  26. Document scripts in inner_it

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    89ac82d View commit details
    Browse the repository at this point in the history

  27. Genthat is needed when running the benchmarks.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    c99ff8d View commit details
    Browse the repository at this point in the history

  28. Use the right result variable in the debug message.

    @vogr
    vogr committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    fc0ee9f View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2021

  1. Quote benchmark name in rebench command 

    This prevent problems with benchmark named a<-b.R for instance.
    @vogr
    vogr committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    fcdec7e View commit details
    Browse the repository at this point in the history

  2. Normalize path of extfile and Rfile in genthat-CRAN harness

    @vogr
    vogr committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    387e911 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2021

  1. Move the generated, but currently unused, genthat-benchmark files out 

    the genthat-CRAN directory (else they would be detected by the
configuration scripts).
    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    6e9bfa5 View commit details
    Browse the repository at this point in the history

  2. Make the genthat files compatible with the new harness.

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    c2a88bb View commit details
    Browse the repository at this point in the history

  3. Make the genthat harness compatible with the new genthat files.

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    e0430e5 View commit details
    Browse the repository at this point in the history

  4. Remove the archived tests

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    deaae4f View commit details
    Browse the repository at this point in the history

  5. Keep only a few tests.

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    e3b7745 View commit details
    Browse the repository at this point in the history

  6. Record retv is now integrated in genthat.

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    af50f50 View commit details
    Browse the repository at this point in the history

  7. Use the PRL-PRG branch for the modified genthat, instead of the one 

    owned by vogr
    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    c3ddcbf View commit details
    Browse the repository at this point in the history

  8. Use correct retv name in logging in harness.

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    ff7fc06 View commit details
    Browse the repository at this point in the history

  9. Update the extfile so that they match the expected format.

    @vogr
    vogr committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    12573c1 View commit details
    Browse the repository at this point in the history