Coverage for mlos_bench/mlos_bench/__init__.py: 100%

1# 

2# Copyright (c) Microsoft Corporation. 

3# Licensed under the MIT License. 

4# 

5r""" 

6mlos_bench is a framework to help automate benchmarking and OS/application parameter 

7autotuning and the data management of the results. 

8 

9It can be installed from `pypi <https://pypi.org/project/mlos-bench>`_ via ``pip 

10install mlos-bench`` and executed using the ``mlos_bench`` `command 

11<../../mlos_bench.run.usage.html>`_ using a collection of `json` `configs 

12<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_. 

13 

14It is intended to be used with :py:mod:`mlos_core` via 

15:py:class:`~mlos_bench.optimizers.mlos_core_optimizer.MlosCoreOptimizer` to help 

16navigate complex parameter spaces more effeciently, though other 

17:py:mod:`~mlos_bench.optimizers` are also available to help customize the search 

18process easily by simply swapping out the 

19:py:class:`~mlos_bench.optimizers.base_optimizer.Optimizer` class in the associated 

20json configs. For instance, 

21:py:class:`~mlos_bench.optimizers.grid_search_optimizer.GridSearchOptimizer` can be 

22used to perform a grid search over the parameter space instead. 

23 

24The other core classes in this package are: 

25 

26- :py:mod:`~mlos_bench.environments` which provide abstractions for representing an 

27 execution environment. 

28 

29 These are generally the target of the optimization process and are used to 

30 evaluate the performance of a given configuration, though can also be used to 

31 simply run a single benchmark. They can be used, for instance, to provision a 

32 :py:mod:`VM <mlos_bench.environments.remote.vm_env>`, run benchmarks or execute 

33 any other arbitrary code on a :py:mod:`remote machine 

34 <mlos_bench.environments.remote.remote_env>`, and many other things. 

35 

36- Environments are often associated with :py:mod:`~mlos_bench.tunables` which 

37 provide a language for specifying the set of configuration parameters that can be 

38 optimized or searched over with the :py:mod:`~mlos_bench.optimizers`. 

39 

40- :py:mod:`~mlos_bench.services` provide the necessary abstractions to run interact 

41 with the :py:mod:`~mlos_bench.environments` in different settings. 

42 

43 For instance, the 

44 :py:class:`~mlos_bench.services.remote.azure.azure_vm_services.AzureVMService` can 

45 be used to run commands on Azure VMs for a remote 

46 :py:mod:`~mlos_bench.environments.remote.vm_env.VMEnv`. 

47 

48 Alternatively, one could swap out that service for 

49 :py:class:`~mlos_bench.services.remote.ssh.ssh_host_service.SshHostService` in 

50 order to target a different VM without having to change the 

51 :py:class:`~mlos_bench.environments.base_environment.Environment` configuration at 

52 all since they both implement the same 

53 :py:class:`~mlos_bench.services.types.remote_exec_type.SupportsRemoteExec` 

54 :py:mod:`Services type<mlos_bench.services.types>` interfaces. 

55 

56 This is particularly useful when running the same benchmark on different 

57 ecosystems and makes the configs more modular and composable. 

58 

59- :py:mod:`~mlos_bench.storage` which provides abstractions for storing and 

60 retrieving data from the experiments. 

61 

62 For instance, nearly any :py:mod:`SQL <mlos_bench.storage.sql>` backend that 

63 `sqlalchemy <https://www.sqlalchemy.org>`_ supports can be used. 

64 

65The data management and automation portions of experiment data is a key component of 

66MLOS as it provides a unified way to manage experiment data across different 

67Environments, enabling more reusable visualization and analysis by mapping benchmark 

68metrics into common semantic types (e.g., via `OpenTelemetry 

69<https://opentelemetry.io>`_). 

70 

71Without this most experiments are effectively siloed and require custom, and more 

72critically, non-reusable scripts to setup and later parse results and are hence 

73harder to scale to many users. 

74 

75With these features as a part of the MLOS ecosystem, benchmarking can become a 

76*service* that any developer, admin, research, etc. can use and adapt. 

77 

78See below for more information on the classes in this package. 

79 

80Notes 

81----- 

82Note that while the docstrings in this package are generated from the source code 

83and hence sometimes more focused on the implementation details, most user 

84interactions with the package will be through the `json configs 

85<https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/>`_. Even 

86so it may be useful to look at the source code to understand how those are 

87interpretted. 

88 

89Examples 

90-------- 

91Here is an example that shows how to run a simple benchmark using the command line. 

92 

93The entry point for these configs can be found `here 

94<https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc>`_. 

95 

96>>> from subprocess import run 

97>>> # Note: we show the command wrapped in python here for testing purposes. 

98>>> # Alternatively replace test-cli-local-env-bench.jsonc with 

99>>> # test-cli-local-env-opt.jsonc for one that does an optimization loop. 

100>>> cmd = "mlos_bench \ 

101... --config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc \ 

102... --globals experiment_test_local.jsonc \ 

103... --tunable_values tunable-values/tunable-values-local.jsonc" 

104>>> print(f"Here's the shell command you'd actually run:\n# {cmd}") 

105Here's the shell command you'd actually run: 

106# mlos_bench --config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc --globals experiment_test_local.jsonc --tunable_values tunable-values/tunable-values-local.jsonc 

107>>> # Now we run the command and check the output. 

108>>> result = run(cmd, shell=True, capture_output=True, text=True, check=True) 

109>>> assert result.returncode == 0 

110>>> lines = result.stderr.splitlines() 

111>>> first_line = lines[0] 

112>>> last_line = lines[-1] 

113>>> expected = "INFO Launch: mlos_bench" 

114>>> assert first_line.endswith(expected) 

115>>> expected = "INFO Final score: {'score': 123.4, 'total_time': 123.4, 'throughput': 1234567.0}" 

116>>> assert last_line.endswith(expected) 

117 

118Notes 

119----- 

120- `mlos_bench/README.md <https://github.com/microsoft/MLOS/tree/main/mlos_bench/>`_ 

121 for additional documentation and examples in the source tree. 

122 

123- `mlos_bench/DEVNOTES.md <https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/DEVNOTES.md>`_ 

124 for additional developer notes in the source tree. 

125 

126- There is also a working example of using ``mlos_bench`` in a *separate config 

127 repo* (the more expected case for most users) in the `sqlite-autotuning 

128 <https://github.com/Microsoft-CISL/sqlite-autotuning>`_ repo. 

129""" # pylint: disable=line-too-long # noqa: E501 

130from mlos_bench.version import VERSION 

131 

132__version__ = VERSION