mlos_bench.environments.mock_env

Scheduler-side environment to mock the benchmark results.

Classes

MockEnv

Scheduler-side environment to mock the benchmark results.

Module Contents

class mlos_bench.environments.mock_env.MockEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: mlos_bench.tunables.tunable_groups.TunableGroups | None = None, service: mlos_bench.services.base_service.Service | None = None)[source]

Bases: mlos_bench.environments.base_environment.Environment

Scheduler-side environment to mock the benchmark results.

Create a new environment that produces mock benchmark data.

Parameters:

  • name (str) – Human-readable name of the environment.

  • config (dict) – Free-format dictionary that contains the benchmark environment configuration.

  • global_config (dict) – Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config. Optional arguments are mock_env_seed, mock_env_range, and mock_env_metrics. Set mock_env_seed to -1 for deterministic behavior, 0 for default randomness.

  • tunables (TunableGroups) – A collection of tunable parameters for all environments.

  • service (Service) – An optional service object. Not used by this class.

run() → tuple[mlos_bench.environments.status.Status, datetime.datetime, dict[str, mlos_bench.tunables.tunable_types.TunableValue] | None][source]

Produce mock benchmark data for one experiment.

Returns:

(status, timestamp, output) – 3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. The keys of the output dict are the names of the metrics specified in the config; by default it’s just one metric named “score”. All output metrics have the same value.

Return type:

(Status, datetime.datetime, dict)

status() → tuple[mlos_bench.environments.status.Status, datetime.datetime, list[tuple[datetime.datetime, str, Any]]][source]

Produce mock benchmark status telemetry for one experiment.

Returns:

(benchmark_status, timestamp, telemetry) – 3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

Return type:

(Status, datetime.datetime, list)