attach trials

[Fa20]

<<def run_experiment(ax_client: AxClient, predefined_trials: list, num_trials: int = 7) -> None:
   
    for i, trial_params in enumerate(predefined_trials):
        ax_client.attach_trial(parameters=trial_params)
        result = evaluate_parameters(trial_params)
        ax_client.complete_trial(trial_index=i, raw_data=result)
        ax_client.save_to_json_file("experiment_state.json")


    for i in range(num_trials):
        print(f'Running trial {i + 1}')
        parameters, trial_index = ax_client.get_next_trial()
        result = evaluate_parameters(parameters)
        ax_client.complete_trial(trial_index=trial_index, raw_data=result)
        print(f"Checkpoint after {i + 1} trials")
        ax_client.save_to_json_file("experiment_state.json")






def plot_pareto_front(ax_client:AxClient):

    objectives = ax_client.experiment.optimization_config.objective.objectives
    objective11 =objectives[0].metric
    print('objective11',objective11)
    objective12 = objectives[1].metric


    frontier = compute_posterior_pareto_frontier(
        experiment=ax_client.experiment,
        data=ax_client.experiment.fetch_data(),
        primary_objective=objective12,
        secondary_objective=objective11,
        absolute_metrics=["objective_1", "objective_2"],
        num_points=4,
    )
    print('frontier',frontier)
    render(plot_pareto_frontier(frontier, CI_level=0.90))

    param_dicts = frontier.param_dicts
    means = frontier.means
    sems = frontier.sems

    params_df = pd.DataFrame(param_dicts)

    means_df = pd.DataFrame(means)
    sems_df = pd.DataFrame(sems)

    sems_df.columns = [f"{col}_se" for col in sems_df.columns]

    combined_df = pd.concat([params_df, means_df, sems_df], axis=1)

    print(ax_client.generation_strategy.trials_as_df)


if __name__ == '__main__':
   
    predefined_trials = [{'x1': 133.393258, 'x2': 28.187442, 'x3': 1.204922}]
    ax_client=setup_experiment()
    run_experiment(ax_client,predefined_trials)
    plot_pareto_front(ax_client)>>

I have tried to attach trial to my experiment but when ax_client.generation_strategy.trials_as_df I can not see the attached experiment and when I run the code again I got

<<  263 run_experiment(ax_client,predefined_trials)
    264 plot_pareto_front(ax_client)

Cell In[38], line 206, in run_experiment(ax_client, predefined_trials, num_trials)
    204     ax_client.attach_trial(parameters=trial_params)
    205     result = evaluate_parameters(trial_params)
--> 206     ax_client.complete_trial(trial_index=i, raw_data=result)
    207     ax_client.save_to_json_file("experiment_state.json")
    209 # Run additional trials

File ~\anaconda3\Lib\site-packages\ax\service\ax_client.py:748, in AxClient.complete_trial(self, trial_index, raw_data, metadata, sample_size)
    746 # Validate that trial can be completed.
    747 trial = self.get_trial(trial_index)
--> 748 trial._validate_can_attach_data()
    749 if not isinstance(trial_index, int):
    750     raise ValueError(f"Trial index must be an int, got: {trial_index}.")

File ~\anaconda3\Lib\site-packages\ax\core\base_trial.py:855, in BaseTrial._validate_can_attach_data(self)
    853 """Determines whether a trial is in a state that can be attached data."""
    854 if self.status.is_completed:
--> 855     raise UnsupportedError(
    856         f"Trial {self.index} has already been completed with data."
    857         "To add more data to it (for example, for a different metric), "
    858         "use `Trial.update_trial_data()` or "
    859         "BatchTrial.update_batch_trial_data()."
    860     )
    861 if self.status.is_abandoned or self.status.is_failed:
    862     raise UnsupportedError(
    863         f"Trial {self.index} has been marked {self.status.name}, so it "
    864         "no longer expects data."
    865     )

UnsupportedError: Trial 0 has already been completed with data.To add more data to it (for example, for a different metric), use `Trial.update_trial_data()` or BatchTrial.update_batch_trial_data().   >>  my question how can I fix this problem such that to attach the trail and  be able to load it as completed trial see it

[danielcohenlive]

Hi @Fa20, your code is missing definitions for setup_experiment() and evaluate_parameters(), so I had to guess a little. For me, it works. You may have gotten this error by running your code twice, in which case on

ax_client.complete_trial(trial_index=i, raw_data=result)

i is a list index, say 0. But if there already were 20 trials on the experiment, you're trying to complete trial 0 when you should be trying to complete trial 20. Trial 0 is probably already completed, so this error being raised to save you from silent failures/bad results. You could either fix this by making sure the experiment is clean at the beginning, or by changing

        ax_client.attach_trial(parameters=trial_params)
        result = evaluate_parameters(trial_params)
        ax_client.complete_trial(trial_index=i, raw_data=result)

to

        params, trial_index = ax_client.attach_trial(parameters=trial_params)
        result = evaluate_parameters(trial_params)
        ax_client.complete_trial(trial_index= trial_index, raw_data=result)

like in the loop below. Probably you just want to make sure you start with a clean experiment though.

If this isn't it, please fix your minimal repro and I'll be happy to look again. Your minimal repro should include imports and all, with any private information substituted, so that I can copy and paste it into my own notebook and click run without any modification and see the error (or general problem) you're having.

[Fa20]

@danielcohenlive thanks for your answer.No I do not want clean and start from the beginning the problem that I need to evaluate the objective functions by simulation which need to keep the Ax code running till to get the result from the simulation and I thought that I can avoid this by stop run the code till the simulation is finished and attach the parmas . Is there in Ax better way to solve this problem in case that I do not want to keep the code running till get the result from Simulation?

[danielcohenlive]

Is there in Ax better way to solve this problem in case that I do not want to keep the code running till get the result from Simulation?

@Fa20 could you clarify what you mean by not wanting to keep the code running... If you're talking about running a human in the loop experiment see https://ax.dev/tutorials/human_in_the_loop/human_in_the_loop.html. It's not written in AxClient though, so you'll have to cross reference with the service API tutorial.

[Fa20]

@danielcohenlive I mean when we run the Loop start the Generation step then evaluate the objective with this values of parameters. In my case I do not have this value of objective functions and I should calculate it Numerical which need some time and the Ax Code should wait till to get this value and then go to the next step .is there any way to that after the first Generation step that we can step excuted the code till the calculation of objective finished and then run it again but this time the step of evaluate the objective can be excuted

[Abrikosoff]

@danielcohenlive I mean when we run the Loop start the Generation step then evaluate the objective with this values of parameters. In my case I do not have this value of objective functions and I should calculate it Numerical which need some time and the Ax Code should wait till to get this value and then go to the next step .is there any way to that after the first Generation step that we can step excuted the code till the calculation of objective finished and then run it again but this time the step of evaluate the objective can be excuted

Hi @Fa20, again drive-by commenting, but maybe you can:

1. run the following (replace the num_trials by however much trials you need):

for i in range(num_trials):

    parameterization, trial_index = ax_client.get_next_trial()

    # extract parameters
    p1 = parameterization["Polymer1"]
    p2 = parameterization["Polymer2"]

    # Save suggestions in the dictionary
    experiment_suggestions[f"Iteration_{i}"] = {
        'Polymer1': p1,
        'Polymer2': p2,
   }

2. save your ax_client after finishing the loop (probably via ax_client.save_to_json_file; note this saves your ax_client instance, not the data)
3. do your simulations using data in the saved dictionary;
4. reload your saved ax_client and do ax_client.complete_trial(), with the correct trial indices.

Probably you can also set up a scheduler, but I'm not sure how that works in Service.

[Fa20]

@Abrikosoff step 4
Do I need to use for loop
To evaluate the objectives and then complete the trial

[Abrikosoff]

@Abrikosoff step 4 Do I need to use for loop To evaluate the objectives and then complete the trial

You probably would want to, although I think you can also partially complete the set of suggested trials. In that case if the remainder is larger than your max_parallelism number (I think!) you won't be able to generate new trials.