mlos_bench.environments.base_environment

A hierarchy of benchmark environments.

Classes

Environment

An abstract base of all benchmark environments.

Module Contents

class mlos_bench.environments.base_environment.Environment(*, name: str, config: dict, global_config: dict | None = None, tunables: mlos_bench.tunables.tunable_groups.TunableGroups | None = None, service: mlos_bench.services.base_service.Service | None = None)[source]

An abstract base of all benchmark environments.

Create a new environment with a given config.

Parameters:

  • name (str) – Human-readable name of the environment.

  • config (dict) – Free-format dictionary that contains the benchmark environment configuration. Each config must have at least the “tunable_params” and the “const_args” sections.

  • global_config (dict) – Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config.

  • tunables (TunableGroups) – A collection of groups of tunable parameters for all environments.

  • service (Service) – An optional service object (e.g., providing methods to deploy or reboot a VM/Host, etc.).

__enter__() → Environment[source]

Enter the environment’s benchmarking context.

Return type:

Environment

__exit__(ex_type: Type[BaseException] | None, ex_val: BaseException | None, ex_tb: types.TracebackType | None) → Literal[False][source]

Exit the context of the benchmarking environment.

Parameters:
Return type:

Literal[False]

__repr__() → str[source]
Return type:

str

__str__() → str[source]
Return type:

str

classmethod new(*, env_name: str, class_name: str, config: dict, global_config: dict | None = None, tunables: mlos_bench.tunables.tunable_groups.TunableGroups | None = None, service: mlos_bench.services.base_service.Service | None = None) → Environment[source]

Factory method for a new environment with a given config.

Parameters:

  • env_name (str) – Human-readable name of the environment.

  • class_name (str) – FQN of a Python class to instantiate, e.g., “mlos_bench.environments.remote.HostEnv”. Must be derived from the Environment class.

  • config (dict) – Free-format dictionary that contains the benchmark environment configuration. It will be passed as a constructor parameter of the class specified by name.

  • global_config (dict) – Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config.

  • tunables (TunableGroups) – A collection of groups of tunable parameters for all environments.

  • service (Service) – An optional service object (e.g., providing methods to deploy or reboot a VM/Host, etc.).

Returns:

env – An instance of the Environment class initialized with config.

Return type:

Environment

pprint(indent: int = 4, level: int = 0) → str[source]

Pretty-print the environment configuration. For composite environments, print all children environments as well.

Parameters:

  • indent (int) – Number of spaces to indent the output. Default is 4.

  • level (int) – Current level of indentation. Default is 0.

Returns:

pretty – Pretty-printed environment configuration. Default output is the same as __repr__.

Return type:

str

run() → Tuple[mlos_bench.environments.status.Status, datetime.datetime, Dict[str, mlos_bench.tunables.tunable.TunableValue] | None][source]

Execute the run script for this environment.

For instance, this may start a new experiment, download results, reconfigure the environment, etc. Details are configurable via the environment config.

Returns:

(status, timestamp, output) – 3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

Return type:

(Status, datetime.datetime, dict)

setup(tunables: mlos_bench.tunables.tunable_groups.TunableGroups, global_config: dict | None = None) → bool[source]

Set up a new benchmark environment, if necessary. This method must be idempotent, i.e., calling it several times in a row should be equivalent to a single call.

Parameters:

  • tunables (TunableGroups) – A collection of tunable parameters along with their values.

  • global_config (dict) – Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_success – True if operation is successful, false otherwise.

Return type:

bool

status() → Tuple[mlos_bench.environments.status.Status, datetime.datetime, List[Tuple[datetime.datetime, str, Any]]][source]

Check the status of the benchmark environment.

Returns:

(benchmark_status, timestamp, telemetry) – 3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

Return type:

(Status, datetime.datetime, list)

teardown() → None[source]

Tear down the benchmark environment.

This method must be idempotent, i.e., calling it several times in a row should be equivalent to a single call.

Return type:

None

config[source]
name[source]
property parameters: Dict[str, mlos_bench.tunables.tunable.TunableValue][source]

Key/value pairs of all environment parameters (i.e., const_args and tunable_params). Note that before .setup() is called, all tunables will be set to None.

Returns:

parameters – Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

Return type:

Dict[str, TunableValue]

property tunable_params: mlos_bench.tunables.tunable_groups.TunableGroups[source]

Get the configuration space of the given environment.

Returns:

tunables – A collection of covariant groups of tunable parameters.

Return type:

TunableGroups