mlos_bench.storage.sql.experiment

Storage.Experiment interface implementation for saving and restoring the benchmark experiment data using SQLAlchemy backend.

Classes

Experiment

Logic for retrieving and storing the results of a single experiment.

Module Contents

class mlos_bench.storage.sql.experiment.Experiment(*, engine: sqlalchemy.engine.Engine, schema: mlos_bench.storage.sql.schema.DbSchema, tunables: mlos_bench.tunables.tunable_groups.TunableGroups, experiment_id: str, trial_id: int, root_env_config: str, description: str, opt_targets: Dict[str, Literal['min', 'max']])[source]

Bases: mlos_bench.storage.base_storage.Storage.Experiment

Logic for retrieving and storing the results of a single experiment.

Parameters:

engine (sqlalchemy.engine.Engine)
schema (mlos_bench.storage.sql.schema.DbSchema)
tunables (mlos_bench.tunables.tunable_groups.TunableGroups)
experiment_id (str)
trial_id (int)
root_env_config (str)
description (str)
opt_targets (Dict[str, Literal['min', 'max']])

load(last_trial_id: int = -1) → Tuple[List[int], List[dict], List[Dict[str, Any] | None], List[mlos_bench.environments.status.Status]][source]

Load (tunable values, benchmark scores, status) to warm-up the optimizer.

If last_trial_id is present, load only the data from the (completed) trials that were scheduled after the given trial ID. Otherwise, return data from ALL merged-in experiments and attempt to impute the missing tunable values.

Parameters:: last_trial_id (int) – (Optional) Trial ID to start from.
Returns:: (trial_ids, configs, scores, status) – Trial ids, Tunable values, benchmark scores, and status of the trials.
Return type:: ([int], [dict], [Optional[dict]], [Status])

load_telemetry(trial_id: int) → List[Tuple[datetime.datetime, str, Any]][source]

Retrieve the telemetry data for a given trial.

Parameters:: trial_id (int) – Trial ID.
Returns:: metrics – Telemetry data.
Return type:: List[Tuple[datetime.datetime, str, Any]]

load_tunable_config(config_id: int) → Dict[str, Any][source]

Load tunable values for a given config ID.

Parameters:: config_id (int)
Return type:: Dict[str, Any]

merge(experiment_ids: List[str]) → None[source]

Merge in the results of other (compatible) experiments trials. Used to help warm up the optimizer for this experiment.

Parameters:: experiment_ids (List[str]) – List of IDs of the experiments to merge in.
Return type:: None

pending_trials(timestamp: datetime.datetime, *, running: bool) → Iterator[mlos_bench.storage.base_storage.Storage.Trial][source]

Return an iterator over the pending trials that are scheduled to run on or before the specified timestamp.

Parameters:

timestamp (datetime.datetime) – The time in UTC to check for scheduled trials.
running (bool) – If True, include the trials that are already running. Otherwise, return only the scheduled trials.

Returns:

trials – An iterator over the scheduled (and maybe running) trials.

Return type:

Iterator[Storage.Trial]