mlos_bench.storage.sql.experiment_data

An interface to access the benchmark experiment data stored in SQL DB using the ExperimentData interface.

Classes

ExperimentSqlData

SQL interface for accessing the stored experiment benchmark data.

Module Contents

class mlos_bench.storage.sql.experiment_data.ExperimentSqlData(*, engine: sqlalchemy.engine.Engine, schema: mlos_bench.storage.sql.schema.DbSchema, experiment_id: str, description: str, root_env_config: str, git_repo: str, git_commit: str)

Bases: mlos_bench.storage.base_experiment_data.ExperimentData

SQL interface for accessing the stored experiment benchmark data.

An experiment groups together a set of trials that are run with a given set of scripts and mlos_bench configuration files.

Parameters:

• engine (sqlalchemy.engine.Engine)

• schema (mlos_bench.storage.sql.schema.DbSchema)

• experiment_id (str)

• description (str)

• root_env_config (str)

• git_repo (str)

• git_commit (str)

property default_tunable_config_id: int | None

Retrieves the (tunable) config id for the default tunable values for this experiment.

Note: this is by default the first trial executed for this experiment. However, it is currently possible that the user changed the tunables config in between resumptions of an experiment.

Return type:

int

property objectives: Dict[str, Literal['min', 'max']]

Retrieve the experiment’s objectives data from the storage.

Returns:

objectives – A dictionary of the experiment’s objective names (optimization_targets) and their directions (e.g., min or max).

Return type:

Dict[str, Literal[“min”, “max”]]

property results_df: pandas.DataFrame

Retrieve all experimental results as a single DataFrame.

Returns:

results – A DataFrame with configurations and results from all trials of the experiment. Has columns [trial_id, tunable_config_id, tunable_config_trial_group_id, ts_start, ts_end, status] followed by tunable config parameters (prefixed with “config.”) and trial results (prefixed with “result.”). The latter can be NULLs if the trial was not successful.

Return type:

pandas.DataFrame

property trials: Dict[int, mlos_bench.storage.base_trial_data.TrialData]

Retrieve the experiment’s trials’ data from the storage.

Returns:

trials – A dictionary of the trials’ data, keyed by trial id.

Return type:

Dict[int, TrialData]

property tunable_config_trial_groups: Dict[int, mlos_bench.storage.base_tunable_config_trial_group_data.TunableConfigTrialGroupData]

Retrieve the Experiment’s (Tunable) Config Trial Group data from the storage.

Returns:

trials – A dictionary of the trials’ data, keyed by (tunable) by config id.

Return type:

Dict[int, TunableConfigTrialGroupData]

property tunable_configs: Dict[int, mlos_bench.storage.base_tunable_config_data.TunableConfigData]

Retrieve the experiment’s (tunable) configs’ data from the storage.

Returns:

trials – A dictionary of the configs’ data, keyed by (tunable) config id.

Return type:

Dict[int, TunableConfigData]