Coverage for mlos_bench/mlos_bench/environments/__init__.py: 100%

1# 

2# Copyright (c) Microsoft Corporation. 

3# Licensed under the MIT License. 

4# 

5""" 

6Tunable Environments for mlos_bench. 

7 

8.. contents:: Table of Contents 

9 :depth: 3 

10 

11Overview 

12++++++++ 

13 

14Environments are classes that represent an execution setting (i.e., environment) for 

15running a benchmark or tuning process. 

16 

17For instance, a :py:class:`~.LocalEnv` represents a local execution environment, a 

18:py:class:`~.RemoteEnv` represents a remote execution environment, a 

19:py:class:`~mlos_bench.environments.remote.vm_env.VMEnv` represents a virtual 

20machine, etc. 

21 

22An Environment goes through a series of *phases* (e.g., 

23:py:meth:`~.Environment.setup`, :py:meth:`~.Environment.run`, 

24:py:meth:`~.Environment.teardown`, etc.) that can be used to prepare a VM, workload, 

25etc.; run a benchmark, script, etc.; and clean up afterwards. 

26Often, what these phases do (e.g., what commands to execute) will depend on the 

27specific Environment and the configs that Environment was loaded with. 

28This lets Environments be very flexible in what they can accomplish. 

29 

30Environments can be stacked together with the :py:class:`.CompositeEnv` class to 

31represent complex setups (e.g., an appication running on a remote VM with a 

32benchmark running from a local machine). 

33 

34See below for the set of Environments currently available in this package. 

35 

36Note that additional ones can also be created by extending the base 

37:py:class:`.Environment` class and referencing them in the :py:mod:`json configs 

38<mlos_bench.config>` using the ``class`` key. 

39 

40Environment Tunables 

41++++++++++++++++++++ 

42 

43Each environment can use 

44:py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` to specify the set of 

45configuration parameters that can be optimized or searched. 

46At each iteration of the optimization process, the optimizer will generate a set of 

47values for the :py:class:`Tunables <mlos_bench.tunables.tunable.Tunable>` that the 

48environment can use to configure itself. 

49 

50At a python level, this happens by passing a 

51:py:meth:`~mlos_bench.tunables.tunable_groups.TunableGroups` object to the 

52``tunable_groups`` parameter of the :py:class:`~.Environment` constructor, but that 

53is typically handled by the 

54:py:meth:`~mlos_bench.services.config_persistence.ConfigPersistenceService.load_environment` 

55method of the 

56:py:meth:`~mlos_bench.services.config_persistence.ConfigPersistenceService` invoked 

57by the ``mlos_bench`` command line tool's :py:class:`mlos_bench.launcher.Launcher` 

58class. 

59 

60In the typical json user level configs, this is specified in the 

61``include_tunables`` section of the Environment config to include the 

62:py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` definitions from other 

63json files when the :py:class:`~mlos_bench.launcher.Launcher` processes the initial 

64set of config files. 

65 

66The ``tunable_params`` setting in the ``config`` section of the Environment config 

67can also be used to limit *which* of the ``TunableGroups`` should be used for the 

68Environment. 

69 

70Since :py:mod:`json configs <mlos_bench.config>` also support ``$variable`` 

71substitution in the values using the `globals` mechanism, this setting can used to 

72dynamically change the set of active TunableGroups for a given Experiment using only 

73`globals`, allowing for configs to be more modular and composable. 

74 

75Environment Services 

76++++++++++++++++++++ 

77 

78Environments can also reference :py:mod:`~mlos_bench.services` that provide the 

79necessary support to perform the actions that environment needs for each of its 

80phases depending upon where its being deployed (e.g., local machine, remote machine, 

81cloud provider VM, etc.) 

82 

83Although this can be done in the Environment config directly with the 

84``include_services`` key, it is often more useful to do it in the global or 

85:py:mod:`cli config <mlos_bench.config>` to allow for the same Environment to be 

86used in different settings (e.g., local machine, SSH accessible machine, Azure VM, 

87etc.) without having to change the Environment config. 

88 

89Variable Propagation 

90++++++++++++++++++++ 

91TODO: Document how variable propagation works in the script environments using 

92required_args, const_args, etc. 

93 

94Examples 

95-------- 

96While this documentation is generated from the source code and is intended to be a 

97useful reference on the internal details, most users will be more interested in 

98generating json configs to be used with the ``mlos_bench`` command line tool. 

99 

100For a simple working user oriented example please see the `test_local_env_bench.jsonc 

101<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/config/environments/local/test_local_env.jsonc>`_ 

102file or other examples in the source tree linked below. 

103 

104For more developer oriented examples please see the `mlos_bench/tests/environments 

105<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/>`_ 

106directory in the source tree. 

107 

108Notes 

109----- 

110- See `mlos_bench/environments/README.md 

111 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/environments/>`_ 

112 for additional documentation in the source tree. 

113- See `mlos_bench/config/environments/README.md 

114 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/environments/>`_ 

115 for additional config examples in the source tree. 

116 

117See Also 

118-------- 

119:py:mod:`mlos_bench.config` : Overview of the configuration system. 

120:py:mod:`mlos_bench.services` : Overview of the Services available to the 

121 Environments and their configurations. 

122:py:mod:`mlos_bench.tunables` : Overview of the Tunables available to the 

123 Environments and their configurations. 

124""" 

125 

126from mlos_bench.environments.base_environment import Environment 

127from mlos_bench.environments.composite_env import CompositeEnv 

128from mlos_bench.environments.local.local_env import LocalEnv 

129from mlos_bench.environments.local.local_fileshare_env import LocalFileShareEnv 

130from mlos_bench.environments.mock_env import MockEnv 

131from mlos_bench.environments.remote.remote_env import RemoteEnv 

132from mlos_bench.environments.status import Status 

133 

134__all__ = [ 

135 "Status", 

136 "Environment", 

137 "MockEnv", 

138 "RemoteEnv", 

139 "LocalEnv", 

140 "LocalFileShareEnv", 

141 "CompositeEnv", 

142]