Coverage for mlos_bench/mlos_bench/tunables/__init__.py: 100%
4 statements
« prev ^ index » next coverage.py v7.8.0, created at 2025-04-01 00:52 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5"""
6Tunables classes for Environments in mlos_bench.
8.. contents:: Table of Contents
9 :depth: 3
11Overview
12^^^^^^^^
14mlos_bench uses the concept of "tunables" to define the configuration space for an
15:py:class:`~mlos_bench.environments.base_environment.Environment`.
17An :py:class:`~mlos_bench.optimizers.base_optimizer.Optimizer` can then use these
18tunables to explore the configuration space in order to improve some target
19objective metrics (e.g., reduce tail latency, reduce cost, improve throughput,
20etc.).
22They are similar to the concept of "hyperparameters" in machine learning, but are
23used to configure the system being tested.
25Classes
26^^^^^^^
28Tunable
29+++++++
31The :py:class:`~mlos_bench.tunables.tunable.Tunable` class is used to define a
32single tunable parameter.
33A ``Tunable`` has a :py:attr:`~.Tunable.type` and can be a ``categorical`` or
34numeric (``int`` or ``float``) and always has at least a domain
35(:py:attr:`~.Tunable.range` or set of :py:attr:`~.Tunable.values`) and a
36:py:attr:`~.Tunable.default`.
37Each type can also have a number of additional properties that can optionally be set
38to help control the sampling of the tunable.
40For instance:
42- Numeric tunables can have a :py:attr:`~.Tunable.distribution` property to specify the sampling
43 distribution. :py:attr:`log <.Tunable.is_log>` sampling can also be enabled for
44 numeric tunables.
45- Categorical tunables can have a :py:attr:`values_weights <.Tunable.weights>`
46 property to specify biased sampling of the values
47- :py:attr:`~.Tunable.special` values can be marked to indicate that they need more
48 explicit testing. This can be useful for values that indicate "automatic" or
49 "disabled" behavior.
51The :py:class:`~mlos_bench.tunables.tunable.Tunable` class attributes documentation
52and :py:class:`~mlos_bench.tunables.tunable_types.TunableDict` class documentation
53provides some more information and example on the available properties.
55The full set of supported properties is specified in the `JSON schema for tunable
56parameters
57<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/config/schemas/tunables/tunable-params-schema.json>`_
58and can be seen in some of the `test examples in the source tree
59<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/schemas/tunable-params/test-cases/good/>`_.
61CovariantTunableGroup
62+++++++++++++++++++++
64The :py:class:`~mlos_bench.tunables.covariant_group.CovariantTunableGroup` class is
65used to define a group of related tunable parameters that are all configured
66together with the same ``cost`` (e.g., is a more expensive operation required to
67reconfigure the system like redeployed vs. restarted vs. reloaded).
68Optimizers can use this information to explore the configuration space more efficiently.
70TunableGroups
71+++++++++++++
73The :py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` class is used to
74define an entire set of tunable parameters (e.g., combined set of covariant groups).
76Limitations
77-----------
78Currently we lack a config language for expressing constraints between tunables
80(e.g., ``a < b`` or ``a + b < c``)
82This is supported in the underlying :py:mod:`mlos_core` library, but is not yet
83exposed in the ``mlos_bench`` config API.
85Usage
86^^^^^
88Most user interactions with tunables will be through JSON configuration files.
90Since tunables are associated with an Environment, their configs are typically
91colocated with the environment configs (e.g., ``env-name-tunables.jsonc``) and
92loaded with the Environment using the ``include_tunables`` property in the
93Environment config.
95Then individual covariant groups can be enabled via the ``tunable_params`` and
96``tunable_params_map`` properties, possibly via ``globals`` `Variable Expansion
97<../config/index.html#globals-and-variable-substitution>`_.
99See the :py:mod:`mlos_bench.config` and :py:mod:`mlos_bench.environments` module
100documentation for more information.
102In benchmarking-only mode (e.g., without an ``Optimizer`` specified),
103`mlos_bench <../../../mlos_bench.run.usage.html>`_ can still run with a
104particular set of ``--tunable-values`` (e.g., a simple key-value file declaring
105a set of values to assign to the set of configured tunable parameters) in order
106to manually explore a configuration space.
108See the :py:class:`~mlos_bench.optimizers.one_shot_optimizer.OneShotOptimizer`
109and :py:mod:`mlos_bench.run` module documentation and the for more information.
111During an Environment's
112:py:meth:`~mlos_bench.environments.base_environment.Environment.setup` and
113:py:meth:`~mlos_bench.environments.base_environment.Environment.run` phases the
114tunables can be exported to a JSON file using the ``dump_params_file`` property
115of the Environment config for the user scripts to use when configuring the
116target system.
117The :py:attr:`~.Tunable.meta` property of the tunable config can be used to add
118additional information for this step (e.g., a unit suffix to append to the
119value).
121See the :py:mod:`mlos_bench.environments` module documentation for more information.
123Examples
124--------
125Here's a short (incomplete) example of some of the :py:class:`.TunableGroups`
126JSON configuration options, expressed in Python (for testing purposes).
127However, most of the time you will be loading these from a JSON config file
128stored along with the associated
129:py:class:`~mlos_bench.environments.base_environment.Environment` config.
131For more tunable parameters examples refer to the `JSON schema
132<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/config/schemas/tunables/tunable-params-schema.json>`_
133or some of the `test examples in the source tree
134<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/schemas/tunable-params/test-cases/good/>`_.
136There are also examples of `tunable values in the source tree
137<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/schemas/tunable-values/test-cases/good/>`_.
139>>> # Load tunables from JSON string.
140>>> import json5
141>>> from mlos_bench.services.config_persistence import ConfigPersistenceService
142>>> service = ConfigPersistenceService()
143>>> json_config = '''
144... // Use json5 (or jsonc) syntax to allow comments and other more flexible syntax.
145... {
146... "group_1": {
147... "cost": 1,
148... "params": {
149... "colors": {
150... "type": "categorical",
151... // Values for the categorical tunable.
152... "values": ["red", "blue", "green"],
153... // Weights for each value in the categorical distribution.
154... "values_weights": [0.1, 0.2, 0.7],
155... // Default value.
156... "default": "green",
157... },
158... "int_param": {
159... "type": "int",
160... "range": [1, 10],
161... "default": 5,
162... // Mark some values as "special", that need more explicit testing.
163... // e.g., maybe these indicate "automatic" or "disabled" behavior for
164... // the system being tested instead of an explicit size
165... "special": [-1, 0],
166... // Optionally specify a sampling distribution
167... // to influence which values to prioritize.
168... "distribution": {
169... "type": "uniform" // alternatively, "beta" or "normal"
170... },
171... // Free form key-value pairs that can be used with the
172... // tunable upon sampling for composing configs.
173... // These can be retrieved later to help generate
174... // config files from the sampled tunables.
175... "meta": {
176... "suffix": "MB"
177... }
178... },
179... "float_param": {
180... "type": "float",
181... "range": [1, 10000],
182... "default": 1,
183... // Quantize the range into 100 bins
184... "quantization_bins": 100,
185... // enable log sampling of the bins
186... "log": true
187... }
188... }
189... }
190... }
191... '''
192>>> tunables = service.load_tunables(jsons=[json_config])
193>>> # Retrieve the current value for the tunable groups.
194>>> tunables.get_param_values()
195{'colors': 'green', 'int_param': 5, 'float_param': 1.0}
196>>> # Or an individual parameter:
197>>> tunables["colors"]
198'green'
199>>> # Assign new values to the tunable groups.
200>>> tunable_values = json5.loads('''
201... {
202... // can be partially specified
203... "colors": "red"
204... }
205... ''')
206>>> _ = tunables.assign(tunable_values)
207>>> tunables.get_param_values()
208{'colors': 'red', 'int_param': 5, 'float_param': 1.0}
209>>> # Check if the tunables have been updated.
210>>> # mlos_bench uses this to reinvoke the setup() phase of the
211>>> # associated Environment to reconfigure the system.
212>>> tunables.is_updated()
213True
214>>> # Reset the tunables to their default values.
215>>> # As a special case, an empty json object will reset all tunables to the defaults.
216>>> tunable_values = json5.loads('''
217... {}
218... ''')
219>>> _ = tunables.assign(tunable_values)
220>>> tunables.is_defaults()
221True
222>>> tunables.get_param_values()
223{'colors': 'green', 'int_param': 5, 'float_param': 1.0}
225Notes
226-----
227Internally, :py:class:`.TunableGroups` are converted to
228:external:py:class:`ConfigSpace.ConfigurationSpace` objects for use with
229:py:mod:`mlos_core` using the :py:mod:`mlos_bench.optimizers.convert_configspace`.
230See the "Spaces" section in the :py:mod:`mlos_core` module documentation for more
231information.
233In order to handle sampling of :py:attr:`.Tunable.special` values, the ``!``
234character is prohibited from being used in the name of a :py:class:`.Tunable`.
236See Also
237--------
238:py:mod:`mlos_bench.config` : Overview of the configuration system.
239:py:mod:`mlos_bench.environments` : Overview of Environments and their configurations.
240:py:mod:`mlos_core.optimizers` : Overview of mlos_core optimizers.
241:py:mod:`mlos_core.spaces` : Overview of the mlos_core configuration space system.
242:py:meth:`.TunableGroups.assign` : Notes on special cases for assigning tunable values.
243"""
245from mlos_bench.tunables.tunable import Tunable
246from mlos_bench.tunables.tunable_groups import TunableGroups
247from mlos_bench.tunables.tunable_types import TunableValue
249__all__ = [
250 "Tunable",
251 "TunableValue",
252 "TunableGroups",
253]