Coverage for mlos_bench/mlos_bench/__init__.py: 100%
2 statements
« prev ^ index » next coverage.py v7.8.0, created at 2025-04-01 00:52 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5r"""
6mlos_bench is a framework to help automate benchmarking and OS/application parameter
7autotuning and the data management of the results.
9.. contents:: Table of Contents
10 :depth: 3
12Overview
13++++++++
15``mlos_bench`` can be installed from `pypi <https://pypi.org/project/mlos-bench>`_
16via ``pip install mlos-bench`` and executed using the ``mlos_bench`` `command
17<../../mlos_bench.run.usage.html>`_ using a collection of `json` `configs
18<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_
19which are managed and documented in the :py:mod:`~mlos_bench.config` module and
20elsewhere in this package documentation.
22It is intended to be used with :py:mod:`mlos_core` via
23:py:class:`~mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer` to help
24navigate complex parameter spaces more efficiently, though other
25:py:mod:`~mlos_bench.optimizers` are also available to help customize the search
26process easily by simply swapping out the
27:py:class:`~mlos_bench.optimizers.base_optimizer.Optimizer` class in the associated
28json configs. For instance,
29:py:class:`~mlos_bench.optimizers.grid_search_optimizer.GridSearchOptimizer` can be
30used to perform a grid search over the parameter space instead, or no tuning at all
31and ``mlos_bench`` can be used as only a repeatable benchmarking tool.
33Goals
34^^^^^
36The overall goal of the MLOS project is to enable *reproducible* and *trackable*
37benchmarking and *efficient* autotuning for systems software.
39In this, automation of the benchmarking process is a key component that
40``mlos_bench`` seeks to enable.
42Interaction
43^^^^^^^^^^^
45Users are expected to provide JSON :py:mod:`mlos_bench.config` s that instruct the
46framework how to automate their benchmark, autotuning, or other Experiment.
48This may involve several steps such as
501. deploying a VM
512. installing some software
523. loading a dataset
534. running a benchmark
545. collecting and storing the results
556. repeating for statistical and noise measures (we call each iteration a ``Trial``)
567. (optionally) repeating with a different configuration (e.g., for autotuning purposes)
578. analyzing the results
59Since many of these phases are common across different benchmarks, the framework is
60intended to be modular and composable to allow reuse and faster development of new
61benchmarking environments or autotuning experiments.
63Where possible, the framework will provide common configs for reuse (e.g., deploying
64a VM on Azure, or run `benchbase <https://github.com/cmu-db/benchbase>`_ against
65a database system) to allow users to focus on the specifics of their
66experiments.
68Where none are currently available, one can create them external to MLOS, however
69users are also encouraged to `submit PRs or Issues
70<https://github.com/microsoft/MLOS/CONTRIBUTING.md>`_ to add new classes or config
71and script snippets for others to use as well.
73For more details on the configuration files, please see the documentation in the
74:py:mod:`mlos_bench.config` module.
76Classes Overview
77++++++++++++++++
79The other core classes in this package are:
81- :py:mod:`~mlos_bench.environments` which provide abstractions for representing an
82 execution environment.
84 These are generally the target of the optimization process and are used to
85 evaluate the performance of a given configuration, though can also be used to
86 simply run a single benchmark. They can be used, for instance, to provision a
87 :py:mod:`VM <mlos_bench.environments.remote.vm_env>`, run benchmarks or execute
88 any other arbitrary code on a :py:mod:`remote machine
89 <mlos_bench.environments.remote.remote_env>`, and many other things.
91- Environments are often associated with :py:mod:`~mlos_bench.tunables` which
92 provide a language for specifying the set of configuration parameters that can be
93 optimized or searched over with the :py:mod:`~mlos_bench.optimizers`.
95- :py:mod:`~mlos_bench.services` provide the necessary abstractions to run interact
96 with the :py:mod:`~mlos_bench.environments` in different settings.
98 For instance, the
99 :py:class:`~mlos_bench.services.remote.azure.azure_vm_services.AzureVMService` can
100 be used to run commands on Azure VMs for a remote
101 :py:mod:`~mlos_bench.environments.remote.vm_env.VMEnv`.
103 Alternatively, one could swap out that service for
104 :py:class:`~mlos_bench.services.remote.ssh.ssh_host_service.SshHostService` in
105 order to target a different VM without having to change the
106 :py:class:`~mlos_bench.environments.base_environment.Environment` configuration at
107 all since they both implement the same
108 :py:class:`~mlos_bench.services.types.remote_exec_type.SupportsRemoteExec`
109 :py:mod:`Services type<mlos_bench.services.types>` interfaces.
111 This is particularly useful when running the same benchmark on different
112 ecosystems and makes the configs more modular and composable.
114- :py:mod:`~mlos_bench.storage` which provides abstractions for storing and
115 retrieving data from the experiments.
117 For instance, nearly any :py:mod:`SQL <mlos_bench.storage.sql>` backend that
118 `sqlalchemy <https://www.sqlalchemy.org>`_ supports can be used.
120The data management and automation portions of experiment data is a key component of
121MLOS as it provides a unified way to manage experiment data across different
122Environments, enabling more reusable visualization and analysis by mapping benchmark
123metrics into common semantic types (e.g., via `OpenTelemetry
124<https://opentelemetry.io>`_).
126Without this most experiments are effectively siloed and require custom, and more
127critically, non-reusable scripts to setup and later parse results and are hence
128harder to scale to many users.
130With these features as a part of the MLOS ecosystem, benchmarking can become a
131*service* that any developer, admin, researcher, etc. can use and adapt.
133See below for more information on the classes in this package.
135Notes
136-----
137Note that while the docstrings in this package are generated from the source
138code and hence sometimes more focused on the implementation details, we do try
139to provide some (testable) examples here, but most user interactions with the
140package will be through the `json configs
141<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_.
142Even so it may be useful to look at the documentation here (perhaps especially
143starting with :py:mod:`mlos_bench.config`) and, if necessary, the `source code
144<https://github.com/microsoft/MLOS>`_ to understand how those are interpreted.
146Examples
147--------
148Here is an example that shows how to run a simple benchmark using the command line.
150The entry point for these configs can be found `here
151<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc>`_.
153>>> from subprocess import run
154>>> # Note: we show the command wrapped in python here for testing purposes.
155>>> # Alternatively replace test-cli-local-env-bench.jsonc with
156>>> # test-cli-local-env-opt.jsonc for one that does an optimization loop.
157>>> cmd = "mlos_bench \
158... --config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc \
159... --globals experiment_test_local.jsonc \
160... --tunable_values tunable-values/tunable-values-local.jsonc"
161>>> print(f"Here's the shell command you'd actually run:\n# {cmd}")
162Here's the shell command you'd actually run:
163# mlos_bench --config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc --globals experiment_test_local.jsonc --tunable_values tunable-values/tunable-values-local.jsonc
164>>> # Now we run the command and check the output.
165>>> result = run(cmd, shell=True, capture_output=True, text=True, check=True)
166>>> assert result.returncode == 0
167>>> lines = result.stderr.splitlines()
168>>> first_line = lines[0]
169>>> last_line = lines[-1]
170>>> expected = "INFO Launch: mlos_bench"
171>>> assert first_line.endswith(expected)
172>>> expected = "INFO Final score: {'score': 123.4, 'total_time': 123.4, 'throughput': 1234567.0}"
173>>> assert last_line.endswith(expected)
175Notes
176-----
177- `mlos_bench/README.md <https://github.com/microsoft/MLOS/tree/main/mlos_bench/>`_
178 for additional documentation and examples in the source tree.
180- `mlos_bench/DEVNOTES.md <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/DEVNOTES.md>`_
181 for additional developer notes in the source tree.
183- There is also a working example of using ``mlos_bench`` in a *separate config
184 repo* (the more expected case for most users) in the `sqlite-autotuning
185 <https://github.com/Microsoft-CISL/sqlite-autotuning>`_ repo.
186""" # pylint: disable=line-too-long # noqa: E501
187from mlos_bench.version import VERSION
189__version__ = VERSION