Coverage for mlos_bench/mlos_bench/config/__init__.py: 100%
0 statements
« prev ^ index » next coverage.py v7.6.7, created at 2024-11-22 01:18 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5"""
6A module for and documentation about the structure and mangement of json configs, their
7schemas and validation for various components of MLOS.
9.. contents:: Table of Contents
10 :depth: 3
12Overview
13++++++++
15MLOS is a framework for doing benchmarking and autotuning for systems.
16The bulk of the code to do that is written in python. As such, all of the code
17classes documented here take python objects in their construction.
19However, most users of MLOS will interact with the system via the ``mlos_bench`` CLI
20and its json config files and their own scripts for MLOS to invoke. This module
21attempts to document some of those high level interactions.
23General JSON Config Structure
24+++++++++++++++++++++++++++++
26We use `json5 <https://pypi.org/project/json5/>`_ to parse the json files, since it
27allows for inline C style comments (e.g., ``//``, ``/* */``), trailing commas, etc.,
28so it is slightly more user friendly than strict json.
30By convention files use the ``*.mlos.json`` or ``*.mlos.jsonc`` extension to
31indicate that they are an ``mlos_bench`` config file.
33This allows tools that support `JSON Schema Store
34<https://www.schemastore.org/json/>`_ (e.g., `VSCode
35<https://code.visualstudio.com/>`_ with an `extension
36<https://marketplace.visualstudio.com/items?itemName=remcohaszing.schemastore>`_) to
37provide helpful autocomplete and validation of the json configs while editing.
39CLI Configs
40^^^^^^^^^^^
42:py:attr:`~.mlos_bench.config.schemas.config_schemas.ConfigSchema.CLI` style configs
43are typically used to start the ``mlos_bench`` CLI using the ``--config`` argument
44and a restricted key-value dict form where each key corresponds to a CLI argument.
46For instance:
48.. code-block:: json
50 // cli-config.mlos.json
51 {
52 "experiment": "path/to/base/experiment-config.mlos.json",
53 "services": [
54 "path/to/some/service-config.mlos.json",
55 ],
56 "globals": "path/to/basic-globals-config.mlos.json",
57 }
59.. code-block:: json
61 // basic-globals-config.mlos.json
62 {
63 "location": "westus",
64 "vm_size": "Standard_D2s_v5",
65 }
67Typically CLI configs will reference some other configs, especially the base
68Environment and Services configs, but some ``globals`` may be left to be specified
69on the command line.
71For instance:
73.. code-block:: shell
75 mlos_bench --config path/to/cli-config.mlos.json --globals experiment-config.mlos.json
77where ``experiment-config.mlos.json`` might look something like this:
79.. code-block:: json
81 // experiment-config.mlos.json (also a set of globals)
82 {
83 "experiment_id": "my_experiment",
84 "some_var": "some_value",
85 }
87This allows some of the ``globals`` to be specified on the CLI to alter the behavior
88of a set of Experiments without having to adjust many of the other config files
89themselves.
91See below for examples.
93Notes
94-----
95- See `mlos_bench CLI usage </mlos_bench.run.usage.html>`_ for more details on the
96 CLI arguments.
97- See `mlos_bench/config/cli
98 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/cli>`_
99 and `mlos_bench/tests/config/cli
100 <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/cli>`_
101 for some examples of CLI configs.
103Globals and Variable Substitution
104+++++++++++++++++++++++++++++++++
106:py:attr:`Globals <mlos_bench.config.schemas.config_schemas.ConfigSchema.GLOBALS>`
107are basically just key-value variables that can be used in other configs using
108``$variable`` substituion via the
109:py:meth:`~mlos_bench.dict_templater.DictTemplater.expand_vars` method.
111For instance:
113.. code-block:: json
115 // globals-config.mlos.json
116 {
117 "experiment_id": "my_experiment",
118 "some_var": "some_value",
119 // environment variable expansion also works here
120 "current_dir": "$PWD",
121 "some_expanded_var": "$some_var: $experiment_id",
122 "location": "eastus",
123 }
125There are additional details about variable propogation in the
126:py:mod:`mlos_bench.environments` module.
128Well Known Variables
129^^^^^^^^^^^^^^^^^^^^
131Here is a list of some well known variables that are provided or required by the
132system and may be used in the config files:
134- ``$experiment_id``: A unique identifier for the experiment.
135 Typically provided in globals.
136- ``$trial_id``: A unique identifier for the trial currently being executed.
137 This can be useful in the configs for :py:mod:`mlos_bench.environments` for
138 instance (e.g., when writing scripts).
139- TODO: Document more variables here.
141Tunable Configs
142^^^^^^^^^^^^^^^
144There are two forms of tunable configs:
146- "TunableParams" style configs
148 Which are used to define the set of
149 :py:mod:`~mlos_bench.tunables.tunable_groups.TunableGroups` (i.e., tunable
150 parameters).
152 .. code-block:: json
154 // some-env-tunables.json
155 {
156 // a group of tunables that are tuned together
157 "covariant_group_name": [
158 {
159 "name": "tunable_name",
160 "type": "int",
161 "range": [0, 100],
162 "default": 50,
163 },
164 // more tunables
165 ],
166 // another group of tunables
167 // both can be enabled at the same time
168 "another_group_name": [
169 {
170 "name": "another_tunable_name",
171 "type": "categorical",
172 "values": ["red", "yellow", "green"],
173 "default": "green"
174 },
175 // more tunables
176 ],
177 }
179 Since TunableParams are associated with Environments, they are typically kept
180 in the same directory as that environment and named something like
181 ``env-tunables.json``.
183- "TunableValues" style configs which are used to specify the values for an
184 instantiation of a set of tunables params.
186 These are essentially just a dict of the tunable names and their values.
187 For instance:
189 .. code-block:: json
191 // tunable-values.mlos.json
192 {
193 "tunable_name": 25,
194 "another_tunable_name": "red",
195 }
197 These can be used with the
198 :py:class:`~mlos_bench.optimizers.one_shot_optimizer.OneShotOptimizer`
199 :py:class:`~mlos_bench.optimizers.manual_optimizer.ManualOptimizer` to run a
200 benchmark with a particular config or set of configs.
202Class Configs
203^^^^^^^^^^^^^
205Class style configs include most anything else and roughly take this form:
207.. code-block:: json
209 // class configs (environments, services, etc.)
210 {
211 // some mlos class name to load
212 "class": "mlos_bench.type.ClassName",
213 "config": {
214 // class specific config
215 "key": "value",
216 "key2": "$some_var", // variable substitution is allowed here too
217 }
218 }
220Where ``type`` is one of the core classes in the system:
222- :py:mod:`~mlos_bench.environments`
223- :py:mod:`~mlos_bench.optimizers`
224- :py:mod:`~mlos_bench.services`
225- :py:mod:`~mlos_bench.schedulers`
226- :py:mod:`~mlos_bench.storage`
228Each of which have their own submodules and classes that dictate the allowed and
229expected structure of the ``config`` section.
231In certain cases (e.g., script command execution) the variable substitution rules
232take on slightly different behavior
233See various documentation in :py:mod:`mlos_bench.environments` for more details.
235Config Processing
236+++++++++++++++++
238Config files are processed by the :py:class:`~mlos_bench.launcher.Launcher` and
239:py:class:`~mlos_bench.services.config_persistence.ConfigPersistenceService` classes
240at startup time by the ``mlos_bench`` CLI.
242The typical entrypoint is a CLI config which references other configs, especially
243the base Environment config, Services, Optimizer, and Storage.
245See `mlos_bench CLI usage </mlos_bench.run.usage.html>`_ for more details on those
246arguments.
248Schema Definitions
249++++++++++++++++++
251For further details on the schema definitions and validation, see the
252:py:class:`~mlos_bench.config.schemas.config_schemas.ConfigSchema` class
253documentation, which also contains links to the actual schema definitions in the
254source tree (see below).
256Notes
257-----
258See `mlos_bench/config/README.md
259<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_ and
260`mlos_bench/tests/config/README.md
261<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/tests/config/>`_
262for additional documentation and examples in the source tree.
263"""