Coverage for mlos_bench/mlos_bench/environments/__init__.py: 100%
8 statements
« prev ^ index » next coverage.py v7.8.0, created at 2025-04-01 00:52 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5"""
6Tunable Environments for mlos_bench.
8.. contents:: Table of Contents
9 :depth: 3
11Overview
12++++++++
14Environments are classes that represent an execution setting (i.e., environment) for
15running a benchmark or tuning process.
17For instance, a :py:class:`~.LocalEnv` represents a local execution environment, a
18:py:class:`~.RemoteEnv` represents a remote execution environment, a
19:py:class:`~mlos_bench.environments.remote.vm_env.VMEnv` represents a virtual
20machine, etc.
22An Environment goes through a series of *phases* (e.g.,
23:py:meth:`~.Environment.setup`, :py:meth:`~.Environment.run`,
24:py:meth:`~.Environment.teardown`, etc.) that can be used to prepare a VM, workload,
25etc.; run a benchmark, script, etc.; and clean up afterwards.
26Often, what these phases do (e.g., what commands to execute) will depend on the
27specific Environment and the configs that Environment was loaded with.
28This lets Environments be very flexible in what they can accomplish.
30Environments can be stacked together with the :py:class:`.CompositeEnv` class to
31represent complex setups (e.g., an application running on a remote VM with a
32benchmark running from a local machine).
34See below for the set of Environments currently available in this package.
36Note that additional ones can also be created by extending the base
37:py:class:`~.Environment` class and referencing them in the :py:mod:`json configs
38<mlos_bench.config>` using the ``class`` key.
40Environment Parameterization
41++++++++++++++++++++++++++++
43Each :py:class:`~.Environment` can have a set of parameters that define the
44environment's configuration. These parameters can be *constant* (i.e., immutable from one trial
45run to the next) or *tunable* (i.e., suggested by the optimizer or provided by the user). The
46following clauses in the environment configuration are used to declare these parameters:
48- ``tunable_params``:
49 A list of :py:mod:`tunable <mlos_bench.tunables>` parameters' (covariant) *groups*.
50 At each trial, the Environment will obtain the new values of these parameters
51 from the outside (e.g., from the :py:mod:`Optimizer <mlos_bench.optimizers>`).
53 Typically, this is set using variable expansion via the special
54 ``tunable_params_map`` key in the `globals config
55 <../config/index.html#globals-and-variable-substitution>`_.
57- ``const_args``:
58 A dictionary of *constant* parameters along with their values.
60- ``required_args``:
61 A list of *constant* parameters supplied to the environment externally
62 (i.e., from a parent environment, global config file, or command line).
64Again, tunable parameters change on every trial, while constant parameters stay fixed for the
65entire experiment.
67During the ``setup`` and ``run`` phases, MLOS will combine the constant and
68tunable parameters and their values into a single dictionary and pass it to the
69corresponding method.
71Values of constant parameters defined in the Environment config can be
72overridden with the values from the command line and/or external config files.
73That allows MLOS users to have reusable immutable environment configurations and
74move all experiment-specific or sensitive data outside of the version-controlled
75files. We discuss the `variable propagation <index.html#variable-propagation>`_ mechanism
76in the section below.
78Environment Tunables
79++++++++++++++++++++
81Each environment can use
82:py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` to specify the set of
83configuration parameters that can be optimized or searched.
84At each iteration of the optimization process, the optimizer will generate a set of
85values for the :py:class:`Tunables <mlos_bench.tunables.tunable.Tunable>` that the
86environment can use to configure itself.
88At a python level, this happens by passing a
89:py:meth:`~mlos_bench.tunables.tunable_groups.TunableGroups` object to the
90``tunable_groups`` parameter of the :py:class:`~.Environment` constructor, but that
91is typically handled by the
92:py:meth:`~mlos_bench.services.config_persistence.ConfigPersistenceService.load_environment`
93method of the
94:py:meth:`~mlos_bench.services.config_persistence.ConfigPersistenceService` invoked
95by the ``mlos_bench`` command line tool's :py:class:`mlos_bench.launcher.Launcher`
96class.
98In the typical json user level configs, this is specified in the
99``include_tunables`` section of the Environment config to include the
100:py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` definitions from other
101json files when the :py:class:`~mlos_bench.launcher.Launcher` processes the initial
102set of config files.
104The ``tunable_params`` setting in the ``config`` section of the Environment config can then be
105used to limit *which* of the ``TunableGroups`` should be used for the Environment.
107Tunable Parameters Map
108^^^^^^^^^^^^^^^^^^^^^^
110Although the full set of tunable parameters (and groups) of each Environment is always known in
111advance, in practice we often want to limit it to a smaller subset for a given experiment. This
112can be done by adding an extra level of indirection and specifying the ``tunable_params_map`` in
113the global config. ``tunable_params_map`` associates a variable name with a list of
114:py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` names, e.g.,
116 .. code-block:: json
118 // experiment-globals.mlos.jsonc
119 {
120 "tunable_params_map": {
121 "tunables_ref1": ["tunable_group1", "tunable_group2"],
122 "tunables_ref2": [] // Useful to disable all tunables.
123 }
124 }
126Later, in the Environment config, we can use these variable names to refer to the
127tunable groups we want to use for that Environment:
129 .. code-block:: json
131 // environment.mlos.jsonc
132 {
133 // ...
134 "config": {
135 "tunable_params": [
136 "$tunables_ref1", // Will be replaced with "tunable_group1", "tunable_group2"
137 "$tunables_ref2", // A no-op
138 "tunable_group3" // Can still refer to a group directly.
139 ],
140 // ... etc.
142Note: this references the `dummy-tunables.jsonc
143<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc>`_
144file for simplicity.
146Using such ``"$tunables_ref"`` variables in the Environment config allows us to dynamically
147change the set of active ``TunableGroups`` for a given Environment using the global config
148without modifying the Environment configuration files for each experiment, thus making them
149more modular and composable.
151Variable Propagation
152++++++++++++++++++++
154Parameters declared in the ``const_args`` or ``required_args`` sections of the Environment
155config can be overridden with values specified in the external config files or the command
156line. In fact, ``const_args`` or ``required_args`` sections can be viewed as placeholders
157for the parameters that are being pushed to the environment from the outside.
159The same parameter can be present in both ``const_args`` and ``required_args`` sections.
160``required_args`` is just a way to emphasize the importance of the parameter and create a
161placeholder for it when no default value can be specified the ``const_args`` section.
162If a ``required_args`` parameter is not present in the ``const_args`` section,
163and can't be resolved from the ``globals`` this allows MLOS to fail fast and
164return an error to the user indicating an incomplete config.
166Note that the parameter **must** appear in the child Environment ``const_args`` or
167``required_args`` section; if a parameter is not present in one of these
168placeholders of the Environment config, it will not be propagated. This allows MLOS
169users to have small immutable Environment configurations and combine and parameterize
170them with external (global) configs.
172Taking it to the next level outside of the Environment configs, the parameters
173can be defined in the external key-value JSON config files (usually referred to
174as `global config files
175<../config/index.html#globals-and-variable-substitution>`_ in MLOS lingo).
176See :py:mod:`mlos_bench.config` for more details.
178We can summarize the parameter propagation rules as follows:
1801. An environment will only get the parameters defined in its ``const_args`` or
181 ``required_args`` sections.
1822. Values of the parameters defined in the global config files will override the values of the
183 corresponding parameters in all environments.
1843. Values of the command line parameters take precedence over values defined in the global or
185 environment configs.
187Examples
188--------
189Here's a simple working example of a local environment config (written in Python
190instead of JSON for testing) to show how variable propagation works:
192Note: this references the `dummy-tunables.jsonc
193<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc>`_
194file for simplicity.
196>>> # globals.jsonc
197>>> globals_json = '''
198... {
199... "experiment_id": "test_experiment",
200...
201... "const_arg_from_globals_1": "Substituted from globals - 1",
202... "const_arg_from_globals_2": "Substituted from globals - 2",
203...
204... "const_arg_from_cli_1": "Will be overridden from CLI",
205...
206... // Define reference names to represent tunable groups in the Environment configs.
207... "tunable_params_map": {
208... "tunables_ref1": ["dummy_params_group1", "dummy_params_group2"],
209... "tunables_ref2": [], // Useful to disable all tunables for the Environment.
210... }
211... }
212... '''
214>>> # environment.jsonc
215>>> environment_json = '''
216... {
217... "class": "mlos_bench.environments.local.local_env.LocalEnv",
218... "name": "test_env1",
219... "include_tunables": [
220... "tunables/dummy-tunables.jsonc" // For simplicity, include all tunables available.
221... ],
222... "config": {
223... "tunable_params": [
224... "$tunables_ref1", // Includes "dummy_params_group1", "dummy_params_group2"
225... "$tunables_ref2", // A no-op
226... "dummy_params_group3" // Can still refer to a group directly.
227... ],
228... "const_args": {
229... // Environment-specific non-tunable constant parameters:
230... "const_arg_1": "Default value of const_arg_1",
231... "const_arg_from_globals_1": "To be replaced from global config",
232... "const_arg_from_cli_1": "To be replaced from CLI"
233... },
234... "required_args": [
235... // These parameters always come from elsewhere:
236... "const_arg_from_globals_2",
237... "const_arg_from_cli_2",
238... // We already define these parameters in "const_args" section above;
239... // mentioning them here is optional, but can be used for clarity:
240... "const_arg_from_globals_1",
241... "const_arg_from_cli_1"
242... ],
243... "run": [
244... "echo Hello world"
245... ]
246... }
247... }
248... '''
250Now that we have our environment and global configurations, we can instantiate the
251:py:class:`~.Environment` and inspect it. In this example we will simulate the command line execution to demonstrate how CLI parameters propagate to the environment.
253>>> # Load the globals and environment configs defined above via the Launcher as
254>>> # if we were calling `mlos_bench` directly on the CLI.
255>>> from mlos_bench.launcher import Launcher
256>>> argv = [
257... "--environment", environment_json,
258... "--globals", globals_json,
259... # Override some values via CLI directly:
260... "--const_arg_from_cli_1", "Substituted from CLI - 1",
261... "--const_arg_from_cli_2", "Substituted from CLI - 2",
262... ]
263>>> launcher = Launcher("sample_launcher", argv=argv)
264>>> env = launcher.root_environment
265>>> env.name
266'test_env1'
268``env`` is an instance of :py:class:`~.Environment` class that we can use to setup, run, and tear
269down the environment. It also has a set of properties and methods that we can use to access the
270object's parameters. This way we can check the actual runtime configuration of the environment.
272First, let's check the tunable parameters:
274>>> assert env.tunable_params.get_param_values() == {
275... "dummy_param": "dummy",
276... "dummy_param_int": 0,
277... "dummy_param_float": 0.5,
278... "dummy_param3": 0.0
279... }
281We can see the tunables from ``dummy_params_group1`` and ``dummy_params_group2`` groups specified
282via ``$tunables_ref1``, as well as the tunables from ``dummy_params_group3`` that we specified
283directly in the Environment config. All tunables are initialized to their default values.
285Now let's see how the variable propagation works.
287>>> env.const_args["const_arg_1"]
288'Default value of const_arg_1'
290``const_arg_1`` has the value we have assigned in the ``"const_args"`` section of the
291Environment config. No surprises here.
293>>> env.const_args["const_arg_from_globals_1"]
294'Substituted from globals - 1'
295>>> env.const_args["const_arg_from_globals_2"]
296'Substituted from globals - 2'
298``const_arg_from_globals_1`` and ``const_arg_from_globals_2`` were declared in the Environment's
299``const_args`` and ``required_args`` sections, respectively. Their values were overridden by the
300values from the global config.
302>>> env.const_args["const_arg_from_cli_1"]
303'Substituted from CLI - 1'
304>>> env.const_args["const_arg_from_cli_2"]
305'Substituted from CLI - 2'
307Likewise, ``const_arg_from_cli_1`` and ``const_arg_from_cli_2`` got their values from the
308command line. Note that for ``const_arg_from_cli_1`` the value from the command line takes
309precedence over the values specified in the Environment's ``const_args`` section **and** the one
310in the global config.
312Now let's set up the environment and see how the constant and tunable parameters get combined.
313We'll also assign some non-default values to the tunables, as the optimizer would do on each
314trial.
316>>> env.tunable_params["dummy_param_int"] = 99
317>>> env.tunable_params["dummy_param3"] = 0.999
318>>> with env:
319... assert env.setup(env.tunable_params)
320... assert env.parameters == {
321... "const_arg_1": "Default value of const_arg_1",
322... "const_arg_from_globals_1": "Substituted from globals - 1",
323... "const_arg_from_globals_2": "Substituted from globals - 2",
324... "const_arg_from_cli_1": "Substituted from CLI - 1",
325... "const_arg_from_cli_2": "Substituted from CLI - 2",
326... "trial_id": 1,
327... "trial_runner_id": 1,
328... "experiment_id": "test_experiment",
329... "dummy_param": "dummy",
330... "dummy_param_int": 99,
331... "dummy_param_float": 0.5,
332... "dummy_param3": 0.999
333... }
335These are the values visible to the implementations of the :py:meth:`~.Environment.setup`,
336:py:meth:`~.Environment.run`, and :py:meth:`~.Environment.teardown` methods. We can see both
337the constant and tunable parameters combined into a single dictionary
338:py:attr:`~.Environment.parameters` with proper values assigned to each of them on each iteration.
339When implementing a new :py:class:`~.Environment`-derived class, developers can rely on the
340:py:attr:`~.Environment.parameters` data in their versions of :py:meth:`~.Environment.setup` and
341other methods. For example, :py:class:`~mlos_bench.environments.remote.vm_env.VMEnv` would then
342pass the :py:attr:`~.Environment.parameters` into an ARM template when provisioning a new VM,
343and :py:class:`~mlos_bench.environments.local.local_env.LocalEnv` can dump them into a JSON file
344specified in the ``dump_params_file`` config property, or/and cherry-pick some of these values
345and make them shell variables with the ``shell_env_params``.
347A few `Well Known Parameters <../config/index.html#well-known-variables>`_
348parameters like ``trial_id`` and ``trial_runner_id`` are added by the
349:py:mod:`Scheduler <mlos_bench.schedulers>` and used for trials parallelization
350and storage of the results. It is sometimes useful to add them, for example, to
351the paths used by the Environment, as in, e.g.,
352``"/storage/$experiment_id/$trial_id/data/"``, to prevent conflicts when running
353multiple Experiments and Trials in parallel.
355We will discuss passing the parameters to external scripts and using them in referencing files
356and directories in local and shared storage in the documentation of the concrete
357:py:class:`~.Environment` implementations, especially
358:py:class:`~mlos_bench.environments.script_env.ScriptEnv` and
359:py:class:`~mlos_bench.environments.local.local_env.LocalEnv`.
361Environment Services
362++++++++++++++++++++
364Environments can also reference :py:mod:`~mlos_bench.services` that provide the
365necessary support to perform the actions that environment needs for each of its
366phases depending upon where its being deployed (e.g., local machine, remote machine,
367cloud provider VM, etc.)
369Although this can be done in the Environment config directly with the
370``include_services`` key, it is often more useful to do it in the global or
371:py:mod:`cli config <mlos_bench.config>` to allow for the same Environment to be
372used in different settings (e.g., local machine, SSH accessible machine, Azure VM,
373etc.) without having to change the Environment config.
375Variable propagation rules described in the previous section for the environment
376configs also apply to the :py:mod:`Service <mlos_bench.services>`
377configurations.
379That is, every parameter defined in the Service config can be overridden by a
380corresponding parameter from the global config or the command line.
382All global configs, command line parameters, Environment ``const_args`` and
383``required_args`` sections, and Service config parameters thus form one flat
384name space of parameters. This imposes a certain risk of name clashes, but also
385simplifies the configuration process and allows users to keep all
386experiment-specific data in a few human-readable files.
388We will discuss the examples of such global and local configuration parameters in the
389documentation of the concrete :py:mod:`~mlos_bench.services` and
390:py:mod:`~mlos_bench.environments`.
392Examples
393--------
394While this documentation is generated from the source code and is intended to be a
395useful reference on the internal details, most users will be more interested in
396generating json configs to be used with the ``mlos_bench`` command line tool.
398For a simple working user oriented example please see the `test_local_env_bench.jsonc
399<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/config/environments/local/test_local_env.jsonc>`_
400file or other examples in the source tree linked below.
402For more developer oriented examples please see the `mlos_bench/tests/environments
403<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/>`_
404directory in the source tree.
406Notes
407-----
408- See `mlos_bench/environments/README.md
409 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/environments/>`_
410 for additional documentation in the source tree.
411- See `mlos_bench/config/environments/README.md
412 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/environments/>`_
413 for additional config examples in the source tree.
415See Also
416--------
417:py:mod:`mlos_bench.config` :
418 Overview of the configuration system.
419:py:mod:`mlos_bench.services` :
420 Overview of the Services available to the Environments and their configurations.
421:py:mod:`mlos_bench.tunables` :
422 Overview of the Tunables available to the Environments and their configurations.
423""" # pylint: disable=line-too-long # noqa: E501
425from mlos_bench.environments.base_environment import Environment
426from mlos_bench.environments.composite_env import CompositeEnv
427from mlos_bench.environments.local.local_env import LocalEnv
428from mlos_bench.environments.local.local_fileshare_env import LocalFileShareEnv
429from mlos_bench.environments.mock_env import MockEnv
430from mlos_bench.environments.remote.remote_env import RemoteEnv
431from mlos_bench.environments.status import Status
433__all__ = [
434 "Status",
435 "Environment",
436 "MockEnv",
437 "RemoteEnv",
438 "LocalEnv",
439 "LocalFileShareEnv",
440 "CompositeEnv",
441]