mlos_bench.storage.sql.storage

Saving and restoring the benchmark data in SQL database.

Classes

SqlStorage

An implementation of the Storage interface using SQLAlchemy

Module Contents

class mlos_bench.storage.sql.storage.SqlStorage(config: dict, global_config: dict | None = None, service: mlos_bench.services.base_service.Service | None = None)

Bases: mlos_bench.storage.base_storage.Storage

An implementation of the Storage interface using SQLAlchemy backend.

Create a new storage object.

Parameters:

• config (dict) – Free-format key/value pairs of configuration parameters.

• global_config (dict | None)

• service (mlos_bench.services.base_service.Service | None)

__exit__(exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: types.TracebackType | None) → Literal[False]

Close the engine connection when exiting the context.

Parameters:

• exc_type (type[BaseException] | None)

• exc_val (BaseException | None)

• exc_tb (types.TracebackType | None)

Return type:

Literal[False]

__getstate__() → dict

Return the state of the object for pickling.

Return type:

dict

__repr__() → str

Return type:

str

__setstate__(state: dict) → None

Restore the state of the object from pickling.

Parameters:

state (dict)

Return type:

None

dispose() → None

Closes the database connection pool.

Return type:

None

experiment(*, experiment_id: str, trial_id: int, root_env_config: str, description: str, tunables: mlos_bench.tunables.tunable_groups.TunableGroups, opt_targets: dict[str, Literal['min', 'max']]) → mlos_bench.storage.base_storage.Storage.Experiment

Create or reload an experiment in the Storage.

Notes

We need the opt_target parameter here to know what metric to retrieve when we load the data from previous trials. Later we will replace it with full metadata about the optimization direction, multiple objectives, etc.

Parameters:

• experiment_id (str) – Unique identifier of the experiment.

• trial_id (int) – Starting number of the trial.

• root_env_config (str) – A path to the root JSON configuration file of the benchmarking environment.

• description (str) – Human-readable description of the experiment.

• tunables (TunableGroups)

• opt_targets (dict[str, Literal["min", "max"]]) – Names of metrics we’re optimizing for and the optimization direction {min, max}.

Returns:

experiment – An object that allows to update the storage with the results of the experiment and related data.

Return type:

Storage.Experiment

get_experiment_by_id(experiment_id: str, tunables: mlos_bench.tunables.tunable_groups.TunableGroups, opt_targets: dict[str, Literal['min', 'max']]) → mlos_bench.storage.base_storage.Storage.Experiment | None

Gets an Experiment by its ID.

Parameters:

• experiment_id (str) – ID of the Experiment to retrieve.

• tunables (TunableGroups) – The tunables for the Experiment.

• opt_targets (dict[str, Literal["min", "max"]]) – The optimization targets for the Experiment’s Optimizer.

Returns:

experiment – The Experiment object, or None if it doesn’t exist.

Return type:

Storage.Experiment | None

Notes

Tunables are not stored in the database for the Experiment, only for the Trials, so currently they can change if the user (incorrectly) adjusts the configs on disk between resume runs. Since this method is generally meant to load th Experiment from the database for a child process to execute a Trial in the background we are generally safe to simply pass these values from the parent process rather than look them up in the database.

update_schema() → None

Update the database schema.

Return type:

None

property experiments: dict[str, mlos_bench.storage.base_experiment_data.ExperimentData]

Retrieve the experiments’ data from the storage.

Returns:

experiments – A dictionary of the experiments’ data, keyed by experiment id.

Return type:

dict[str, ExperimentData]