Coverage for mlos_bench/mlos_bench/tunables/__init__.py: 100%
3 statements
« prev ^ index » next coverage.py v7.6.9, created at 2024-12-20 00:44 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5"""
6Tunables classes for Environments in mlos_bench.
8.. contents:: Table of Contents
9 :depth: 3
11Overview
12^^^^^^^^
14mlos_bench uses the concept of "tunables" to define the configuration space for an
15:py:class:`~mlos_bench.environments.base_environment.Environment`.
17An :py:class:`~mlos_bench.optimizers.base_optimizer.Optimizer` can then use these
18tunables to explore the configuration space in order to improve some target
19objective metrics (e.g., reduce tail latency, reduce cost, improve throughput,
20etc.).
22They are similar to the concept of "hyperparameters" in machine learning, but are
23used to configure the system being tested.
25Classes
26^^^^^^^
28Tunable
29+++++++
31The :py:class:`~mlos_bench.tunables.tunable.Tunable` class is used to define a
32single tunable parameter.
33A ``Tunable`` can be a ``categorical`` or numeric (``int`` or ``float``) and always has
34at least a domain (``range`` or set of ``values``) and default.
35Each type can also have a number of additional properties that can optionally be set
36to help control the sampling of the tunable.
38For instance:
40- Numeric tunables can have a ``distribution`` property to specify the sampling
41 distribution. ``log`` sampling can also be enabled for numeric tunables.
42- Categorical tunables can have a ``values_weights`` property to specify biased
43 sampling of the values
44- ``special`` values can be marked to indicate that they need more explicit testing
45 This can be useful for values that indicate "automatic" or "disabled" behavior.
47The full set of supported properties can be found in the `JSON schema for tunable
48parameters
49<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/config/schemas/tunables/tunable-params-schema.json>`_
50and seen in some of the `test examples in the source tree
51<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/schemas/tunable-params/test-cases/good/>`_.
53CovariantGroup
54++++++++++++++
56The :py:class:`~mlos_bench.tunables.covariant_group.CovariantTunableGroup` class is
57used to define a group of related tunable parameters that are all configured
58together with the same ``cost`` (e.g., is a more expensive operation required to
59reconfigure the system like redeployed vs. restarted vs. reloaded).
60Optimizers can use this information to explore the configuration space more efficiently.
62TunableGroups
63+++++++++++++
65The :py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` class is used to
66define an entire set of tunable parameters (e.g., combined set of covariant groups).
68Usage
69^^^^^
71Most user interactions with tunables will be through JSON configuration files.
73Since tunables are associated with an Environment, their configs are typically
74colocated with the environment configs (e.g., ``env-name-tunables.jsonc``) and
75loaded with the Environment using the ``include_tunables`` property in the
76Environment config.
78Then individual covariant groups can be enabled via the ``tunable_params`` and
79``tunable_params_map`` properties, possibly via ``globals`` variable expansion.
81See the :py:mod:`mlos_bench.config` and :py:mod:`mlos_bench.environments` module
82documentation for more information.
84In benchmarking-only mode (e.g., without an ``Optimizer`` specified), ``mlos_bench``
85can still run with a particular set of ``--tunable-values`` (e.g., a simple
86key-value file declaring a set of values to assign to the set of configured tunable
87parameters) in order to manually explore a configuration space.
89See the :py:mod:`mlos_bench.run` module documentation for more information.
91During an Environment's ``setup`` and ``run`` phases the tunables can be exported to
92a JSON file using the ``dump_params_file`` property of the Environment config for
93the user scripts to use when configuring the target system.
94The ``meta`` property of the tunable config can be used to add additional
95information for this step (e.g., a unit suffix to append to the value).
97See the :py:mod:`mlos_bench.environments` module documentation for more information.
99Examples
100--------
101Here's a short (incomplete) example of some of the TunableGroups JSON configuration
102options, expressed in Python (for testing purposes).
103However, most of the time you will be loading these from a JSON config file stored
104along with the associated Environment config.
106For more tunable parameters examples refer to the `JSON schema
107<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/config/schemas/tunables/tunable-params-schema.json>`_
108or some of the `test examples in the source tree
109<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/schemas/tunable-params/test-cases/good/>`_.
111There are also examples of `tunable values in the source tree
112<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/schemas/tunable-values/test-cases/good/>`_.
114>>> # Load tunables from JSON string.
115>>> import json5
116>>> from mlos_bench.services.config_persistence import ConfigPersistenceService
117>>> service = ConfigPersistenceService()
118>>> json_config = '''
119... {
120... "group_1": {
121... "cost": 1,
122... "params": {
123... "colors": {
124... "type": "categorical",
125... // Values for the categorical tunable.
126... "values": ["red", "blue", "green"],
127... // Weights for each value in the categorical distribution.
128... "values_weights": [0.1, 0.2, 0.7],
129... // Default value.
130... "default": "green",
131... },
132... "int_param": {
133... "type": "int",
134... "range": [-1, 10],
135... "default": 5,
136... // Mark some values as "special", that need more explicit testing.
137... // e.g., maybe these indicate "automatic" or "disabled" behavior for
138... // the system being tested instead of an explicit size
139... "special": [-1, 0],
140... // Optionally specify a sampling distribution.
141... "distribution": {
142... "type": "uniform" // alternatively, "beta" or "normal"
143... },
144... // Free form key-value pairs that can be used with the
145... // tunable upon sampling for composing configs.
146... "meta": {
147... "suffix": "MB"
148... }
149... },
150... "float_param": {
151... "type": "float",
152... "range": [1, 10000],
153... "default": 1,
154... // Quantize the range into 100 bins
155... "quantization_bins": 100,
156... // enable log sampling of the bins
157... "log": true
158... }
159... }
160... }
161... }
162... '''
163>>> tunables = service.load_tunables(jsons=[json_config])
164>>> # Retrieve the current value for the tunable groups.
165>>> tunables.get_param_values()
166{'colors': 'green', 'int_param': 5, 'float_param': 1.0}
167>>> # Or an individual parameter:
168>>> tunables["colors"]
169'green'
170>>> # Assign new values to the tunable groups.
171>>> tunable_values = json5.loads('''
172... {
173... // can be partially specified
174... "colors": "red"
175... }
176... ''')
177>>> _ = tunables.assign(tunable_values)
178>>> tunables.get_param_values()
179{'colors': 'red', 'int_param': 5, 'float_param': 1.0}
180>>> # Check if the tunables have been updated.
181>>> # mlos_bench uses this to reinvoke the setup() phase of the
182>>> # associated Environment to reconfigure the system.
183>>> tunables.is_updated()
184True
185>>> # Reset the tunables to their default values.
186>>> # As a special case, an empty json object will reset all tunables to the defaults.
187>>> tunable_values = json5.loads('''
188... {}
189... ''')
190>>> _ = tunables.assign(tunable_values)
191>>> tunables.is_defaults()
192True
193>>> tunables.get_param_values()
194{'colors': 'green', 'int_param': 5, 'float_param': 1.0}
196Notes
197-----
198Internally, :py:class:`.TunableGroups` are converted to
199:external:py:class:`ConfigSpace.ConfigurationSpace` objects for use with
200:py:mod:`mlos_core`.
201See the "Spaces" section in the :py:mod:`mlos_core` module documentation for more
202information.
204See Also
205--------
206:py:mod:`mlos_bench.config` : Overview of the configuration system.
207:py:mod:`mlos_bench.environments` : Overview of Environments and their configurations.
208:py:mod:`mlos_core.optimizers` : Overview of mlos_core optimizers.
209:py:mod:`mlos_core.spaces` : Overview of the mlos_core configuration space system.
210:py:meth:`.TunableGroups.assign` : Notes on special cases for assigning tunable values.
211"""
213from mlos_bench.tunables.tunable import Tunable, TunableValue
214from mlos_bench.tunables.tunable_groups import TunableGroups
216__all__ = [
217 "Tunable",
218 "TunableValue",
219 "TunableGroups",
220]