Benchmark improvements #598
Conversation
LeoGrin
force-pushed
the
benchmark_improvements
branch
from
90d77f8
to
2b60c2e
Compare
There was a problem hiding this comment.
Thanks for the PR :)
On the parallelism of runs, I originally set them to run sequentially so the time and memory monitoring would be accurate.
If there is a reliable way to measure those while running the instances in parallel, I'm all for it! Though I don't think the current implementation is. I changed the inner structure quite a bit in #593, it might be easier to integrate parallelism with these.
I like the random search idea, though we should also have a way to set the seed, for reproducibility.
| else: | ||
| parametrization = list(product(parametrize)) | ||
| if not (n_random_search is None): |
There was a problem hiding this comment.
Can be simplified:
| if not (n_random_search is None): | |
| if n_random_search is not None: |
| @@ -199,7 +219,7 @@ def exec_func(*args, **kwargs) -> Dict[str, List[float]]: | |||
| "there is therefore nothing to monitor for. ", | |||
| stacklevel=2, | |||
| ) | |||
| return df | |||
| return None | |||
There was a problem hiding this comment.
Same thing:
| return None | |
| return |
|
Can you:
|
| @@ -0,0 +1,95 @@ | |||
| import pytest | |||
There was a problem hiding this comment.
Please add a short module-level docstring describing what this does
| @@ -0,0 +1,95 @@ | |||
| import pytest | |||
| from benchmarks.utils import monitor, repr_func # replace with the actual import path | |||
There was a problem hiding this comment.
Out of curiosity: is this supposed to work without installation? I'm a bit surprised that it does.
There was a problem hiding this comment.
Yes, it does work, since there is an __init__.py file in benchmarks/ :)
| @@ -1,11 +1,13 @@ | |||
| import tracemalloc | |||
| import random | |||
There was a problem hiding this comment.
Can we have a small module-level docstring that describe what this file does?
@LilianBoulard this might be useful for the gapencoder benchmark
@jovan-stojanovic