mlos_bench.storage

Interfaces to the storage backends for mlos_bench.

Storage backends (for instance sql) are used to store and retrieve the results of experiments and implement a persistent queue for schedulers.

The Storage class is the main interface and provides the ability to

  • Create or reload a new Experiment with one or more associated Trial instances which are used by the schedulers during mlos_bench run time to execute Trials.

    In MLOS terms, an Experiment is a group of Trials that share the same scripts and target system.

    A Trial is a single run of the target system with a specific Configuration (e.g., set of tunable parameter values). (Note: other systems may call this a sample)

  • Retrieve the TrialData results with the trials property on a ExperimentData instance via the Storage instance’s experiments property.

    These can be especially useful with mlos_viz for interactive exploration in a Jupyter Notebook interface, for instance.

The from_config() storage_factory function can be used to get a Storage instance from a STORAGE type json config.

Example

TODO: Add example usage.

See also

mlos_bench.storage.base_storage

Base interface for backends.

mlos_bench.storage.base_experiment_data

Base interface for ExperimentData.

mlos_bench.storage.base_trial_data

Base interface for TrialData.

Notes

Submodules