mlos_bench.environments
Tunable Environments for mlos_bench.
Overview
Environments are classes that represent an execution setting (i.e., environment) for running a benchmark or tuning process.
For instance, a LocalEnv represents a local execution environment, a
RemoteEnv represents a remote execution environment, a
VMEnv represents a virtual
machine, etc.
An Environment goes through a series of phases (e.g.,
setup(), run(),
teardown(), etc.) that can be used to prepare a VM, workload,
etc.; run a benchmark, script, etc.; and clean up afterwards.
Often, what these phases do (e.g., what commands to execute) will depend on the
specific Environment and the configs that Environment was loaded with.
This lets Environments be very flexible in what they can accomplish.
Environments can be stacked together with the CompositeEnv class to
represent complex setups (e.g., an appication running on a remote VM with a
benchmark running from a local machine).
See below for the set of Environments currently available in this package.
Note that additional ones can also be created by extending the base
Environment class and referencing them in the json configs using the class key.
Environment Tunables
Each environment can use
TunableGroups to specify the set of
configuration parameters that can be optimized or searched.
At each iteration of the optimization process, the optimizer will generate a set of
values for the Tunables that the
environment can use to configure itself.
At a python level, this happens by passing a
TunableGroups() object to the
tunable_groups parameter of the Environment constructor, but that
is typically handled by the
load_environment()
method of the
ConfigPersistenceService() invoked
by the mlos_bench command line tool’s mlos_bench.launcher.Launcher
class.
In the typical json user level configs, this is specified in the
include_tunables section of the Environment config to include the
TunableGroups definitions from other
json files when the Launcher processes the initial
set of config files.
The tunable_params setting in the config section of the Environment config
can also be used to limit which of the TunableGroups should be used for the
Environment.
Since json configs also support $variable
substitution in the values using the globals mechanism, this setting can used to
dynamically change the set of active TunableGroups for a given Experiment using only
globals, allowing for configs to be more modular and composable.
Environment Services
Environments can also reference services that provide the
necessary support to perform the actions that environment needs for each of its
phases depending upon where its being deployed (e.g., local machine, remote machine,
cloud provider VM, etc.)
Although this can be done in the Environment config directly with the
include_services key, it is often more useful to do it in the global or
cli config to allow for the same Environment to be
used in different settings (e.g., local machine, SSH accessible machine, Azure VM,
etc.) without having to change the Environment config.
Variable Propogation
TODO: Document how variable propogation works in the script environments using required_args, const_args, etc.
Examples
While this documentation is generated from the source code and is intended to be a
useful reference on the internal details, most users will be more interested in
generating json configs to be used with the mlos_bench command line tool.
For a simple working user oriented example please see the test_local_env_bench.jsonc file or other examples in the source tree linked below.
For more developer oriented examples please see the mlos_bench/tests/environments directory in the source tree.
Notes
See mlos_bench/environments/README.md for additional documentation in the source tree.
See mlos_bench/config/environments/README.md for additional config examples in the source tree.