Enable Pending Points #319

Scienfitz · 2024-07-21T20:04:56Z

This is an important step towards enabling asynchronous workflows, by allowing users to identify pending_experiments

Added:

.recommend from Recommender and Campaign gets a new argument pending_experiments
All pure recommenders now have the option allow_recommending_pending_experiments. If set to False, pending experiments will be excluded from the candidates (this only holds for exact matches to the searchspace and is only relevant in purely discrete search spaces)
For Bayesian recommenders, additionally, the info is passed down to the acquisition function as X_pending. This only works for mc-acqfs and an error is thrown in case of an incompatible acqf
A test verifies that a situation with subsequent .recommend calls and marking points as pending in the second call (keeping the randoms seed same for both calls) does not produce duplicated recommendations
A minimal userguide to make users aware and give an example of how to use it
Found that it is rather easy to make fixtures available to doctests and did that for the new guide, but did not check whether any other previous ~~~ code block could benefit from that
README entry mentioning the asynchronous capabilities

NOT added:

the Campaign does not do any further handling of pending experiments, eg like storing them and so on. Providing pending_experiments also invalidates cached recommendations

AdrianSosic

Hi @Scienfitz, thanks a lot for tackling the pending measurements issue. Here a first batch of comments, mostly regarding the user guide (thanks for writing this as well, btw). Will how have a look at the code

README.md

docs/userguide/async.md

AVHopp

There are no tests/examples of adding measurements for points that were declared pending, and I think this is absolutely necessary.
Other than that this looks really good :)

CHANGELOG.md

baybe/campaign.py

baybe/acquisition/base.py

docs/userguide/async.md

docs/userguide/userguide.md

tests/docs/test_docs.py

tests/test_pending_points.py

Co-authored-by: AdrianSosic <[email protected]>

AdrianSosic

Hi @Scienfitz, again, thanks a lot for the PR. Have made some suggestions but the big elephant in the room is the decision on how to handle the points in the first place. Have mentioned one idea in a thread.

docs/userguide/userguide.md

README.md

CHANGELOG.md

baybe/searchspace/discrete.py

baybe/recommenders/pure/base.py

baybe/recommenders/pure/bayesian/base.py

baybe/recommenders/pure/nonpredictive/base.py

CHANGELOG.md

Co-authored-by: AdrianSosic <[email protected]>

AVHopp

Re-reviewing this PR now solved my confusion regarding the adding of pending points. All of my comments are only minor or already raised by Adrian, hence approving.

docs/userguide/async.md

This reverts commit 36221c8.

# Conflicts: # CHANGELOG.md # baybe/acquisition/base.py # baybe/campaign.py # baybe/recommenders/naive.py # baybe/recommenders/pure/base.py # baybe/recommenders/pure/bayesian/base.py # baybe/recommenders/pure/nonpredictive/clustering.py # baybe/recommenders/pure/nonpredictive/sampling.py # baybe/searchspace/continuous.py # baybe/searchspace/discrete.py

AVHopp

User guide is much clearer now imo and I am happy with it.

docs/userguide/async.md

baybe/recommenders/pure/nonpredictive/base.py

baybe/recommenders/pure/base.py

baybe/recommenders/base.py

baybe/acquisition/base.py

Co-authored-by: AdrianSosic <[email protected]>

CHANGELOG.md

# Conflicts: # CHANGELOG.md # README.md # baybe/acquisition/base.py # baybe/exceptions.py # baybe/recommenders/pure/bayesian/botorch.py # tests/conftest.py

Scienfitz added the new feature New functionality label Jul 21, 2024

Scienfitz self-assigned this Jul 21, 2024

Scienfitz requested review from AdrianSosic and AVHopp as code owners July 21, 2024 20:04

Scienfitz force-pushed the feature/pending_points branch 2 times, most recently from 74101aa to ca866e1 Compare July 25, 2024 12:29

AdrianSosic requested changes Jul 26, 2024

View reviewed changes

AVHopp requested changes Jul 28, 2024

View reviewed changes

Scienfitz force-pushed the feature/pending_points branch from 563349d to ccd3055 Compare July 30, 2024 13:19

Scienfitz and others added 11 commits August 1, 2024 12:22

Add test

e0e6369

Enable nonpredictive recommenders

46ba000

Enable other recommenders

769e7bb

Ignore warning in test

770029e

Add test for incompatible acqfs

530917a

Add userguide for async

4308efa

Make campaign fixture available in doc tests

051f08a

Rework signatures

73471d8

Update README.md

c61e5a3

Co-authored-by: AdrianSosic <[email protected]>

Update CHANGELOG.md

13faee5

Update Userguide

3195053

Co-authored-by: AdrianSosic <[email protected]>

Scienfitz force-pushed the feature/pending_points branch from ccd3055 to 3195053 Compare August 1, 2024 10:23

AdrianSosic requested changes Aug 14, 2024

View reviewed changes

Improve text

f9c16b9

Co-authored-by: AdrianSosic <[email protected]>

Scienfitz force-pushed the feature/pending_points branch from 093be43 to f9c16b9 Compare August 28, 2024 15:19

Scienfitz added 2 commits August 28, 2024 17:24

Add defaults

f053911

Rework parameter name property

342037f

Scienfitz force-pushed the feature/pending_points branch from 2bfe15a to 56572f1 Compare August 29, 2024 06:37

AVHopp approved these changes Sep 2, 2024

View reviewed changes

docs/userguide/async.md Outdated Show resolved Hide resolved

docs/userguide/async.md Outdated Show resolved Hide resolved

docs/userguide/async.md Outdated Show resolved Hide resolved

Scienfitz added 2 commits September 3, 2024 16:09

Change nomenclature

48e758e

Add flag to enforce unique batches where relevant

36221c8

Scienfitz added 2 commits September 3, 2024 16:25

Revert "Add flag to enforce unique batches where relevant"

297ce78

This reverts commit 36221c8.

Make pending experiment candidate exclusion configurable

74cf068

Scienfitz force-pushed the feature/pending_points branch from 56572f1 to 74cf068 Compare September 3, 2024 17:28

Scienfitz added 2 commits September 3, 2024 20:07

Update user guide

2125dcc

Scienfitz force-pushed the feature/pending_points branch from ea12979 to 69c95b3 Compare September 3, 2024 19:31

AVHopp approved these changes Sep 4, 2024

View reviewed changes

AdrianSosic reviewed Sep 4, 2024

View reviewed changes

Refine User Guide

ad0daa8

Co-authored-by: AdrianSosic <[email protected]>

Scienfitz force-pushed the feature/pending_points branch from 8a39254 to ad0daa8 Compare September 4, 2024 10:44

Scienfitz added 2 commits September 4, 2024 12:51

Adjust docstrings

b32cffc

Fix warning

bcb68d3

AdrianSosic reviewed Sep 4, 2024

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

Expand tests

e8c9658

Scienfitz force-pushed the feature/pending_points branch from 3d8106f to e8c9658 Compare September 5, 2024 05:16

Scienfitz added 2 commits September 5, 2024 07:31

Mention new flag in user guide

6c92e29

Add new flag to conftest

8597da1

AdrianSosic approved these changes Sep 6, 2024

View reviewed changes

Scienfitz force-pushed the feature/pending_points branch from d04973b to 5490a3e Compare September 6, 2024 14:33

Merge branch 'main' into feature/pending_points2

9983981

# Conflicts: # CHANGELOG.md # README.md # baybe/acquisition/base.py # baybe/exceptions.py # baybe/recommenders/pure/bayesian/botorch.py # tests/conftest.py

Scienfitz force-pushed the feature/pending_points branch from 5490a3e to 9983981 Compare September 6, 2024 14:42

Scienfitz merged commit ac1fa4d into main Sep 6, 2024
11 checks passed

Scienfitz deleted the feature/pending_points branch September 6, 2024 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Pending Points #319

Enable Pending Points #319

Scienfitz commented Jul 21, 2024 •

edited by AdrianSosic

Loading

AdrianSosic left a comment

AVHopp left a comment

AdrianSosic left a comment

AVHopp left a comment

AVHopp left a comment

Enable Pending Points #319

Enable Pending Points #319

Conversation

Scienfitz commented Jul 21, 2024 • edited by AdrianSosic Loading

AdrianSosic left a comment

Choose a reason for hiding this comment

AVHopp left a comment

Choose a reason for hiding this comment

AdrianSosic left a comment

Choose a reason for hiding this comment

AVHopp left a comment

Choose a reason for hiding this comment

AVHopp left a comment

Choose a reason for hiding this comment

Scienfitz commented Jul 21, 2024 •

edited by AdrianSosic

Loading