mlos_bench.storage.base_experiment_data

Base interface for accessing the stored benchmark experiment data.

An experiment is a collection of trials that are run with a given set of scripts and target system.

Each trial is associated with a configuration (e.g., set of tunable parameters), but multiple trials may use the same config (e.g., for repeat run variability analysis).

See also

mlos_bench.storage

The base storage module for mlos_bench, which includes some basic examples in the documentation.

ExperimentData.results_df

Retrieves a pandas DataFrame of the Experiment’s trials’ results data.

ExperimentData.trials

Retrieves a dictionary of the Experiment’s trials’ data.

ExperimentData.tunable_configs

Retrieves a dictionary of the Experiment’s sampled configs data.

ExperimentData.tunable_config_trial_groups

Retrieves a dictionary of the Experiment’s trials’ data, grouped by shared tunable config.

mlos_bench.storage.base_trial_data.TrialData

Base interface for accessing the stored benchmark trial data.

Classes

ExperimentData

Base interface for accessing the stored experiment benchmark data.

Module Contents

class mlos_bench.storage.base_experiment_data.ExperimentData(*, experiment_id: str, description: str, root_env_config: str, git_repo: str, git_commit: str)

Base interface for accessing the stored experiment benchmark data.

An experiment groups together a set of trials that are run with a given set of scripts and mlos_bench configuration files.

Parameters:

• experiment_id (str)

• description (str)

• root_env_config (str)

• git_repo (str)

• git_commit (str)

__repr__() → str

Return type:

str

CONFIG_COLUMN_PREFIX = 'config.'

Prefix given to columns in ExperimentData.results_df that contain trial config parameters.

For example, if the config parameter name is “param1”, the column name will be “config.param1”.

RESULT_COLUMN_PREFIX = 'result.'

Prefix given to columns in ExperimentData.results_df that contain trial results metrics.

For example, if the result metric is “time”, the column name will be “result.time”.

property default_tunable_config_id: int | None

Retrieves the (tunable) config id for the default tunable values for this experiment.

Note: this is by default the first trial executed for this experiment. However, it is currently possible that the user changed the tunables config in between resumptions of an experiment.

Return type:

int

property description: str

Description of the experiment.

Return type:

str

property experiment_id: str

ID of the experiment.

Return type:

str

property objectives: Dict[str, Literal['min', 'max']]

Abstractmethod:

Return type:

Dict[str, Literal[‘min’, ‘max’]]

Retrieve the experiment’s objectives data from the storage.

Returns:

objectives – A dictionary of the experiment’s objective names (optimization_targets) and their directions (e.g., min or max).

Return type:

Dict[str, Literal[“min”, “max”]]

property results_df: pandas.DataFrame

Abstractmethod:

Return type:

pandas.DataFrame

Retrieve all experimental results as a single DataFrame.

Returns:

results – A DataFrame with configurations and results from all trials of the experiment. Has columns [trial_id, tunable_config_id, tunable_config_trial_group_id, ts_start, ts_end, status] followed by tunable config parameters (prefixed with “config.”) and trial results (prefixed with “result.”). The latter can be NULLs if the trial was not successful.

Return type:

pandas.DataFrame

See also

ExperimentData.CONFIG_COLUMN_PREFIX, ExperimentData.RESULT_COLUMN_PREFIX

property root_env_config: Tuple[str, str, str]

Root environment configuration.

Returns:

(root_env_config, git_repo, git_commit) – A tuple of (root_env_config, git_repo, git_commit) for the root environment.

Return type:

Tuple[str, str, str]

property trials: Dict[int, mlos_bench.storage.base_trial_data.TrialData]

Abstractmethod:

Return type:

Dict[int, mlos_bench.storage.base_trial_data.TrialData]

Retrieve the experiment’s trials’ data from the storage.

Returns:

trials – A dictionary of the trials’ data, keyed by trial id.

Return type:

Dict[int, TrialData]

property tunable_config_trial_groups: Dict[int, mlos_bench.storage.base_tunable_config_trial_group_data.TunableConfigTrialGroupData]

Abstractmethod:

Return type:

Dict[int, mlos_bench.storage.base_tunable_config_trial_group_data.TunableConfigTrialGroupData]

Retrieve the Experiment’s (Tunable) Config Trial Group data from the storage.

Returns:

trials – A dictionary of the trials’ data, keyed by (tunable) by config id.

Return type:

Dict[int, TunableConfigTrialGroupData]

property tunable_configs: Dict[int, mlos_bench.storage.base_tunable_config_data.TunableConfigData]

Abstractmethod:

Return type:

Dict[int, mlos_bench.storage.base_tunable_config_data.TunableConfigData]

Retrieve the experiment’s (tunable) configs’ data from the storage.

Returns:

trials – A dictionary of the configs’ data, keyed by (tunable) config id.

Return type:

Dict[int, TunableConfigData]