mlos_bench.environments.composite_env

Composite benchmark environment.

Classes

CompositeEnv

Composite benchmark environment.

Module Contents

class mlos_bench.environments.composite_env.CompositeEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: mlos_bench.tunables.tunable_groups.TunableGroups | None = None, service: mlos_bench.services.base_service.Service | None = None)

Bases: mlos_bench.environments.base_environment.Environment

Composite benchmark environment.

Create a new environment with a given config.

Parameters:

• name (str) – Human-readable name of the environment.

• config (dict) – Free-format dictionary that contains the environment configuration. Must have a “children” section.

• global_config (dict) – Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config.

• tunables (TunableGroups) – A collection of groups of tunable parameters for all environments.

• service (Service) – An optional service object (e.g., providing methods to deploy or reboot a VM, etc.).

__enter__() → mlos_bench.environments.base_environment.Environment

Enter the environment’s benchmarking context.

Return type:

mlos_bench.environments.base_environment.Environment

__exit__(ex_type: Type[BaseException] | None, ex_val: BaseException | None, ex_tb: types.TracebackType | None) → Literal[False]

Exit the context of the benchmarking environment.

Parameters:

• ex_type (Optional[Type[BaseException]])

• ex_val (Optional[BaseException])

• ex_tb (Optional[types.TracebackType])

Return type:

Literal[False]

pprint(indent: int = 4, level: int = 0) → str

Pretty-print the environment and its children.

Parameters:

• indent (int) – Number of spaces to indent the output at each level. Default is 4.

• level (int) – Current level of indentation. Default is 0.

Returns:

pretty – Pretty-printed environment configuration.

Return type:

str

run() → Tuple[mlos_bench.environments.status.Status, datetime.datetime, Dict[str, mlos_bench.tunables.tunable.TunableValue] | None]

Submit a new experiment to the environment. Return the result of the last child environment if successful, or the status of the last failed environment otherwise.

Returns:

(status, timestamp, output) – 3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

Return type:

(Status, datetime.datetime, dict)

setup(tunables: mlos_bench.tunables.tunable_groups.TunableGroups, global_config: dict | None = None) → bool

Set up the children environments.

Parameters:

• tunables (TunableGroups) – A collection of tunable parameters along with their values.

• global_config (dict) – Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_success – True if all children setup() operations are successful, false otherwise.

Return type:

bool

status() → Tuple[mlos_bench.environments.status.Status, datetime.datetime, List[Tuple[datetime.datetime, str, Any]]]

Check the status of the benchmark environment.

Returns:

(benchmark_status, timestamp, telemetry) – 3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

Return type:

(Status, datetime.datetime, list)

teardown() → None

Tear down the children environments.

This method is idempotent, i.e., calling it several times is equivalent to a single call. The environments are being torn down in the reverse order.

Return type:

None

property children: List[mlos_bench.environments.base_environment.Environment]

Return the list of child environments.

Return type:

List[mlos_bench.environments.base_environment.Environment]