Coverage for mlos_bench/mlos_bench/config/__init__.py: 100%
0 statements
« prev ^ index » next coverage.py v7.6.9, created at 2024-12-20 00:44 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5"""
6A module for and documentation about the structure and management of json configs, their
7schemas and validation for various components of MLOS.
9.. contents:: Table of Contents
10 :depth: 3
12Overview
13++++++++
15MLOS is a framework for doing benchmarking and autotuning for systems.
16The bulk of the code to do that is written in python. As such, all of the code
17classes documented here take python objects in their construction.
19However, most users of MLOS will interact with the system via the ``mlos_bench`` CLI
20and its json config files and their own scripts for MLOS to invoke. This module
21attempts to document some of those high level interactions.
23General JSON Config Structure
24+++++++++++++++++++++++++++++
26We use `json5 <https://pypi.org/project/json5/>`_ to parse the json files, since it
27allows for inline C style comments (e.g., ``//``, ``/* */``), trailing commas, etc.,
28so it is slightly more user friendly than strict json.
30By convention files use the ``*.mlos.json`` or ``*.mlos.jsonc`` extension to
31indicate that they are an ``mlos_bench`` config file.
33This allows tools that support `JSON Schema Store
34<https://www.schemastore.org/json/>`_ (e.g., `VSCode
35<https://code.visualstudio.com/>`_ with an `extension
36<https://marketplace.visualstudio.com/items?itemName=remcohaszing.schemastore>`_) to
37provide helpful autocomplete and validation of the json configs while editing.
39Organization
40^^^^^^^^^^^^
42Ultimately, each experiment is slightly different so it can take some time to get
43the automation right.
45Therefore the configs are intended to be modular and reusable to reduce the time to
46do that for the next set.
47Hence, they are usually split into several files and directory structures.
49We attempt to provide some examples and reusable templates in the core ``mlos_bench``
50package, but users are encouraged to create their own configs as needed, or to
51`submit PRs or Issues <https://github.com/microsoft/MLOS/CONTRIBUTING.md>`_ to add
52additional ones.
54References to some examples are provided below.
56Additional details about the organization of the files and directories are as follows:
58- ``cli/``:
59 Contains the cli configs that control the overall setup for a set of Experiments.
61- ``environments/``:
62 Contains the configs for :py:mod:`~mlos_bench.environments`, and their
63 associated scripts (if relevant, e.g., for
64 :py:class:`~mlos_bench.environments.remote.remote_env.RemoteEnv` or
65 :py:class:`~mlos_bench.environments.script_env.ScriptEnv`) and
66 :py:mod:`~mlos_bench.tunables`.
68 There is usually one *root* environment that chains the others together to build
69 a full experiment (e.g., via
70 :py:class:`~mlos_bench.environments.composite_env.CompositeEnv` and the
71 ``include_children`` field).
72 The *root* environment is the one referenced in the CLI config ``environment``
73 field.
75 Note that each separate Environment config is really more of a template that
76 allows for variable expansion so that the same environment can be used in
77 multiple experiments with different configurations (see below).
79 Similarly, Environments declare a need for a particular Service, but not which
80 implementation of it.
81 This allows for easy swapping of Services (e.g., a different cloud vendor) using
82 a different ``services`` config in the CLI config.
84 Grouping the scripts and tunables together with the environment allows for
85 easier reuse, readability, and debugging.
87 Note that tunables are also separated into "groups" each of which can be enabled
88 for tuning or not, again controllable via ``globals`` variable expansion.
90- ``experiments/``:
91 Contains some ``globals`` (variables) that help expand a set of other config
92 templates out into a full set of configs.
93 Since each experiment may only slightly differ from a previous one, this allows
94 a greater reuse across individual experiments.
96- ``optimizers/``:
97 Contains the configs for :py:mod:`mlos_bench.optimizers`.
98 The optimizer is referenced in the CLI config's ``optimizer`` field.
99 This config controls which optimizer to use and any custom settings for it.
101- ``services/``:
102 Contains the configs for :py:mod:`mlos_bench.services`.
104 In general services can simply be referenced in the CLI config's ``services``
105 field, though sometimes additional settings are required, possibly provided by
106 an additional ``globals`` config in the CLI config.
108- ``storage/``:
109 Contains the configs for :py:mod:`mlos_bench.storage`.
111 The storage config is referenced in the CLI config's ``storage`` field and
112 controls how data is stored and retrieved for the experiments and trials.
114See below for additional details about each configuration type.
116CLI Configs
117^^^^^^^^^^^
119:py:attr:`~.mlos_bench.config.schemas.config_schemas.ConfigSchema.CLI` style configs
120are typically used to start the ``mlos_bench`` CLI using the ``--config`` argument
121and a restricted key-value dict form where each key corresponds to a CLI argument.
123For instance:
125.. code-block:: json
127 // cli-config.mlos.json
128 {
129 "experiment": "path/to/base/experiment-config.mlos.json",
130 "services": [
131 "path/to/some/service-config.mlos.json",
132 ],
133 "globals": "path/to/basic-globals-config.mlos.json",
134 }
136.. code-block:: json
138 // basic-globals-config.mlos.json
139 {
140 "location": "westus",
141 "vm_size": "Standard_D2s_v5",
142 }
144Typically CLI configs will reference some other configs, especially the base
145Environment and Services configs, but some ``globals`` may be left to be specified
146on the command line.
148For instance:
150.. code-block:: shell
152 mlos_bench --config path/to/cli-config.mlos.json --globals experiment-config.mlos.json
154where ``experiment-config.mlos.json`` might look something like this:
156.. code-block:: json
158 // experiment-config.mlos.json (also a set of globals)
159 {
160 "experiment_id": "my_experiment",
161 "some_var": "some_value",
162 }
164This allows some of the ``globals`` to be specified on the CLI to alter the behavior
165of a set of Experiments without having to adjust many of the other config files
166themselves.
168See below for examples.
170Notes
171-----
172- See `mlos_bench CLI usage <../../../mlos_bench.run.usage.html>`_ for more details on the
173 CLI arguments.
174- See `mlos_bench/config/cli
175 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/cli>`_
176 and `mlos_bench/tests/config/cli
177 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/cli>`_
178 for some examples of CLI configs.
180Globals and Variable Substitution
181+++++++++++++++++++++++++++++++++
183:py:attr:`Globals <mlos_bench.config.schemas.config_schemas.ConfigSchema.GLOBALS>`
184are basically just key-value variables that can be used in other configs using
185``$variable`` substitution via the
186:py:meth:`~mlos_bench.dict_templater.DictTemplater.expand_vars` method.
188For instance:
190.. code-block:: json
192 // globals-config.mlos.json
193 {
194 "experiment_id": "my_experiment",
195 "some_var": "some_value",
196 // environment variable expansion also works here
197 "current_dir": "$PWD",
198 "some_expanded_var": "$some_var: $experiment_id",
199 "location": "eastus",
200 }
202There are additional details about variable propagation in the
203:py:mod:`mlos_bench.environments` module.
205Well Known Variables
206^^^^^^^^^^^^^^^^^^^^
208Here is a list of some well known variables that are provided or required by the
209system and may be used in the config files:
211- ``$experiment_id``: A unique identifier for the experiment.
212 Typically provided in globals.
213- ``$trial_id``: A unique identifier for the trial currently being executed.
214 This can be useful in the configs for :py:mod:`mlos_bench.environments` for
215 instance (e.g., when writing scripts).
216- TODO: Document more variables here.
218Tunable Configs
219^^^^^^^^^^^^^^^
221There are two forms of tunable configs:
223- "TunableParams" style configs
225 Which are used to define the set of
226 :py:mod:`~mlos_bench.tunables.tunable_groups.TunableGroups` (i.e., tunable
227 parameters).
229 .. code-block:: json
231 // some-env-tunables.json
232 {
233 // a group of tunables that are tuned together
234 "covariant_group_name": [
235 {
236 "name": "tunable_name",
237 "type": "int",
238 "range": [0, 100],
239 "default": 50,
240 },
241 // more tunables
242 ],
243 // another group of tunables
244 // both can be enabled at the same time
245 "another_group_name": [
246 {
247 "name": "another_tunable_name",
248 "type": "categorical",
249 "values": ["red", "yellow", "green"],
250 "default": "green"
251 },
252 // more tunables
253 ],
254 }
256 Since TunableParams are associated with an :py:mod:`~mlos_bench.environments`,
257 they are typically kept in the same directory as that Environment's config and
258 named something like ``env-tunables.json``.
260- "TunableValues" style configs which are used to specify the values for an
261 instantiation of a set of tunables params.
263 These are essentially just a dict of the tunable names and their values.
264 For instance:
266 .. code-block:: json
268 // tunable-values.mlos.json
269 {
270 "tunable_name": 25,
271 "another_tunable_name": "red",
272 }
274 These can be used with the
275 :py:class:`~mlos_bench.optimizers.one_shot_optimizer.OneShotOptimizer`
276 :py:class:`~mlos_bench.optimizers.manual_optimizer.ManualOptimizer` to run a
277 benchmark with a particular config or set of configs.
279For more information on tunable configs, see the :py:mod:`mlos_bench.tunables`
280module.
282Class Configs
283^^^^^^^^^^^^^
285Class style configs include most anything else and roughly take this form:
287.. code-block:: json
289 // class configs (environments, services, etc.)
290 {
291 // some mlos class name to load
292 "class": "mlos_bench.type.ClassName",
293 "config": {
294 // class specific config
295 "key": "value",
296 "key2": "$some_var", // variable substitution is allowed here too
297 }
298 }
300Where ``type`` is one of the core classes in the system:
302- :py:mod:`~mlos_bench.environments`
303- :py:mod:`~mlos_bench.optimizers`
304- :py:mod:`~mlos_bench.services`
305- :py:mod:`~mlos_bench.schedulers`
306- :py:mod:`~mlos_bench.storage`
308Each of which have their own submodules and classes that dictate the allowed and
309expected structure of the ``config`` section.
311In certain cases (e.g., script command execution) the variable substitution rules
312take on slightly different behavior
313See various documentation in :py:mod:`mlos_bench.environments` for more details.
315Config Processing
316+++++++++++++++++
318Config files are processed by the :py:class:`~mlos_bench.launcher.Launcher` and
319:py:class:`~mlos_bench.services.config_persistence.ConfigPersistenceService` classes
320at startup time by the ``mlos_bench`` CLI.
322The typical entrypoint is a CLI config which references other configs, especially
323the base Environment config, Services, Optimizer, and Storage.
325See `mlos_bench CLI usage <../../../mlos_bench.run.usage.html>`__ for more details
326on those arguments.
328Schema Definitions
329++++++++++++++++++
331For further details on the schema definitions and validation, see the
332:py:class:`~mlos_bench.config.schemas.config_schemas.ConfigSchema` class
333documentation, which also contains links to the actual schema definitions in the
334source tree (see below).
336Debugging
337+++++++++
339Most of the time issues in running an Experiment involve issues with the json
340configs and/or user scripts that are being run by the framework.
342It can help to run ``mlos_bench`` with ``--log-level DEBUG`` to see more detailed
343output about the steps it is taking.
344Alternatively, it can help to add additional debug logging to the user scripts
345themselves to see what about the unique automation process is failing.
347Notes
348-----
349See `mlos_bench/config/README.md
350<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_ and
351`mlos_bench/tests/config/README.md
352<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/>`_
353for additional documentation and examples in the source tree.
354"""