Coverage for mlos_bench/mlos_bench/__init__.py: 100%
2 statements
« prev ^ index » next coverage.py v7.6.9, created at 2024-12-20 00:44 +0000
1#
2# Copyright (c) Microsoft Corporation.
3# Licensed under the MIT License.
4#
5r"""
6mlos_bench is a framework to help automate benchmarking and OS/application parameter
7autotuning and the data management of the results.
9.. contents:: Table of Contents
10 :depth: 3
12Overview
13++++++++
15``mlos_bench`` can be installed from `pypi <https://pypi.org/project/mlos-bench>`_
16via ``pip install mlos-bench`` and executed using the ``mlos_bench`` `command
17<../../mlos_bench.run.usage.html>`_ using a collection of `json` `configs
18<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_.
20It is intended to be used with :py:mod:`mlos_core` via
21:py:class:`~mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer` to help
22navigate complex parameter spaces more efficiently, though other
23:py:mod:`~mlos_bench.optimizers` are also available to help customize the search
24process easily by simply swapping out the
25:py:class:`~mlos_bench.optimizers.base_optimizer.Optimizer` class in the associated
26json configs. For instance,
27:py:class:`~mlos_bench.optimizers.grid_search_optimizer.GridSearchOptimizer` can be
28used to perform a grid search over the parameter space instead, or no tuning at all
29and ``mlos_bench`` can be used as only a repeatable benchmarking tool.
31Goals
32^^^^^
34The overall goal of the MLOS project is to enable *reproducible* and *trackable*
35benchmarking and *efficient* autotuning for systems software.
37In this, automation of the benchmarking process is a key component that
38``mlos_bench`` seeks to enable.
40Interaction
41^^^^^^^^^^^
43Users are expected to provide JSON :py:mod:`mlos_bench.config` s that instruct the
44framework how to automate their benchmark, autotuning, or other Experiment.
46This may involve several steps such as
481. deploying a VM
492. installing some software
503. loading a dataset
514. running a benchmark
525. collecting and storing the results
536. repeating for statistical and noise measures (we call each iteration a ``Trial``)
547. (optionally) repeating with a different configuration (e.g., for autotuning purposes)
558. analyzing the results
57Since many of these phases are common across different benchmarks, the framework is
58intended to be modular and composable to allow reuse and faster development of new
59benchmarking environments or autotuning experiments.
61Where possible, the framework will provide common configs for reuse (e.g., deploying
62a VM on Azure, or run benchbase against a database system) to allow users to focus
63on the specifics of their experiments.
65Where none are currently available, one can create them external to MLOS, however
66users are also encouraged to `submit PRs or Issues
67<https://github.com/microsoft/MLOS/CONTRIBUTING.md>`_ to add new classes or config
68and script snippets for others to use as well.
70For more details on the configuration files, please see the documentation in the
71:py:mod:`mlos_bench.config` module.
73Classes Overview
74++++++++++++++++
76The other core classes in this package are:
78- :py:mod:`~mlos_bench.environments` which provide abstractions for representing an
79 execution environment.
81 These are generally the target of the optimization process and are used to
82 evaluate the performance of a given configuration, though can also be used to
83 simply run a single benchmark. They can be used, for instance, to provision a
84 :py:mod:`VM <mlos_bench.environments.remote.vm_env>`, run benchmarks or execute
85 any other arbitrary code on a :py:mod:`remote machine
86 <mlos_bench.environments.remote.remote_env>`, and many other things.
88- Environments are often associated with :py:mod:`~mlos_bench.tunables` which
89 provide a language for specifying the set of configuration parameters that can be
90 optimized or searched over with the :py:mod:`~mlos_bench.optimizers`.
92- :py:mod:`~mlos_bench.services` provide the necessary abstractions to run interact
93 with the :py:mod:`~mlos_bench.environments` in different settings.
95 For instance, the
96 :py:class:`~mlos_bench.services.remote.azure.azure_vm_services.AzureVMService` can
97 be used to run commands on Azure VMs for a remote
98 :py:mod:`~mlos_bench.environments.remote.vm_env.VMEnv`.
100 Alternatively, one could swap out that service for
101 :py:class:`~mlos_bench.services.remote.ssh.ssh_host_service.SshHostService` in
102 order to target a different VM without having to change the
103 :py:class:`~mlos_bench.environments.base_environment.Environment` configuration at
104 all since they both implement the same
105 :py:class:`~mlos_bench.services.types.remote_exec_type.SupportsRemoteExec`
106 :py:mod:`Services type<mlos_bench.services.types>` interfaces.
108 This is particularly useful when running the same benchmark on different
109 ecosystems and makes the configs more modular and composable.
111- :py:mod:`~mlos_bench.storage` which provides abstractions for storing and
112 retrieving data from the experiments.
114 For instance, nearly any :py:mod:`SQL <mlos_bench.storage.sql>` backend that
115 `sqlalchemy <https://www.sqlalchemy.org>`_ supports can be used.
117The data management and automation portions of experiment data is a key component of
118MLOS as it provides a unified way to manage experiment data across different
119Environments, enabling more reusable visualization and analysis by mapping benchmark
120metrics into common semantic types (e.g., via `OpenTelemetry
121<https://opentelemetry.io>`_).
123Without this most experiments are effectively siloed and require custom, and more
124critically, non-reusable scripts to setup and later parse results and are hence
125harder to scale to many users.
127With these features as a part of the MLOS ecosystem, benchmarking can become a
128*service* that any developer, admin, research, etc. can use and adapt.
130See below for more information on the classes in this package.
132Notes
133-----
134Note that while the docstrings in this package are generated from the source code
135and hence sometimes more focused on the implementation details, most user
136interactions with the package will be through the `json configs
137<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_. Even
138so it may be useful to look at the source code to understand how those are
139interpreted.
141Examples
142--------
143Here is an example that shows how to run a simple benchmark using the command line.
145The entry point for these configs can be found `here
146<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc>`_.
148>>> from subprocess import run
149>>> # Note: we show the command wrapped in python here for testing purposes.
150>>> # Alternatively replace test-cli-local-env-bench.jsonc with
151>>> # test-cli-local-env-opt.jsonc for one that does an optimization loop.
152>>> cmd = "mlos_bench \
153... --config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc \
154... --globals experiment_test_local.jsonc \
155... --tunable_values tunable-values/tunable-values-local.jsonc"
156>>> print(f"Here's the shell command you'd actually run:\n# {cmd}")
157Here's the shell command you'd actually run:
158# mlos_bench --config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc --globals experiment_test_local.jsonc --tunable_values tunable-values/tunable-values-local.jsonc
159>>> # Now we run the command and check the output.
160>>> result = run(cmd, shell=True, capture_output=True, text=True, check=True)
161>>> assert result.returncode == 0
162>>> lines = result.stderr.splitlines()
163>>> first_line = lines[0]
164>>> last_line = lines[-1]
165>>> expected = "INFO Launch: mlos_bench"
166>>> assert first_line.endswith(expected)
167>>> expected = "INFO Final score: {'score': 123.4, 'total_time': 123.4, 'throughput': 1234567.0}"
168>>> assert last_line.endswith(expected)
170Notes
171-----
172- `mlos_bench/README.md <https://github.com/microsoft/MLOS/tree/main/mlos_bench/>`_
173 for additional documentation and examples in the source tree.
175- `mlos_bench/DEVNOTES.md <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/DEVNOTES.md>`_
176 for additional developer notes in the source tree.
178- There is also a working example of using ``mlos_bench`` in a *separate config
179 repo* (the more expected case for most users) in the `sqlite-autotuning
180 <https://github.com/Microsoft-CISL/sqlite-autotuning>`_ repo.
181""" # pylint: disable=line-too-long # noqa: E501
182from mlos_bench.version import VERSION
184__version__ = VERSION