mlos_bench.storage.sql.experiment

Storage.Experiment interface implementation for saving and restoring the benchmark experiment data using SQLAlchemy backend.

Classes

Experiment

Logic for retrieving and storing the results of a single experiment.

Module Contents

class mlos_bench.storage.sql.experiment.Experiment(*, engine: sqlalchemy.engine.Engine, schema: mlos_bench.storage.sql.schema.DbSchema, tunables: mlos_bench.tunables.tunable_groups.TunableGroups, experiment_id: str, trial_id: int, root_env_config: str, description: str, opt_targets: Dict[str, Literal['min', 'max']])

Bases: mlos_bench.storage.base_storage.Storage.Experiment

Logic for retrieving and storing the results of a single experiment.

Parameters:

• engine (sqlalchemy.engine.Engine)

• schema (mlos_bench.storage.sql.schema.DbSchema)

• tunables (mlos_bench.tunables.tunable_groups.TunableGroups)

• experiment_id (str)

• trial_id (int)

• root_env_config (str)

• description (str)

• opt_targets (Dict[str, Literal['min', 'max']])

load(last_trial_id: int = -1) → Tuple[List[int], List[dict], List[Dict[str, Any] | None], List[mlos_bench.environments.status.Status]]

Load (tunable values, benchmark scores, status) to warm-up the optimizer.

If last_trial_id is present, load only the data from the (completed) trials that were scheduled after the given trial ID. Otherwise, return data from ALL merged-in experiments and attempt to impute the missing tunable values.

Parameters:

last_trial_id (int) – (Optional) Trial ID to start from.

Returns:

(trial_ids, configs, scores, status) – Trial ids, Tunable values, benchmark scores, and status of the trials.

Return type:

([int], [dict], [Optional[dict]], [Status])

load_telemetry(trial_id: int) → List[Tuple[datetime.datetime, str, Any]]

Retrieve the telemetry data for a given trial.

Parameters:

trial_id (int) – Trial ID.

Returns:

metrics – Telemetry data.

Return type:

List[Tuple[datetime.datetime, str, Any]]

load_tunable_config(config_id: int) → Dict[str, Any]

Load tunable values for a given config ID.

Parameters:

config_id (int)

Return type:

Dict[str, Any]

merge(experiment_ids: List[str]) → None

Merge in the results of other (compatible) experiments trials. Used to help warm up the optimizer for this experiment.

Parameters:

experiment_ids (List[str]) – List of IDs of the experiments to merge in.

Return type:

None

pending_trials(timestamp: datetime.datetime, *, running: bool) → Iterator[mlos_bench.storage.base_storage.Storage.Trial]

Return an iterator over the pending trials that are scheduled to run on or before the specified timestamp.

Parameters:

• timestamp (datetime.datetime) – The time in UTC to check for scheduled trials.

• running (bool) – If True, include the trials that are already running. Otherwise, return only the scheduled trials.

Returns:

trials – An iterator over the scheduled (and maybe running) trials.

Return type:

Iterator[Storage.Trial]