mlos_bench.tunables 

Tunables classes for Environments in mlos_bench.

Overview 

mlos_bench uses the concept of “tunables” to define the configuration space for an Environment.

An Optimizer can then use these tunables to explore the configuration space in order to improve some target objective metrics (e.g., reduce tail latency, reduce cost, improve throughput, etc.).

They are similar to the concept of “hyperparameters” in machine learning, but are used to configure the system being tested.

The Tunable class is used to define a single tunable parameter. A Tunable has a type and can be a categorical or numeric (int or float) and always has at least a domain (range or set of values) and a default. Each type can also have a number of additional properties that can optionally be set to help control the sampling of the tunable.

For instance:

Numeric tunables can have a distribution property to specify the sampling distribution. log sampling can also be enabled for numeric tunables.
Categorical tunables can have a values_weights property to specify biased sampling of the values
special values can be marked to indicate that they need more explicit testing. This can be useful for values that indicate “automatic” or “disabled” behavior.

The Tunable class attributes documentation and TunableDict class documentation provides some more information and example on the available properties.

The full set of supported properties is specified in the JSON schema for tunable parameters and can be seen in some of the test examples in the source tree.

CovariantTunableGroup 

The CovariantTunableGroup class is used to define a group of related tunable parameters that are all configured together with the same cost (e.g., is a more expensive operation required to reconfigure the system like redeployed vs. restarted vs. reloaded). Optimizers can use this information to explore the configuration space more efficiently.

TunableGroups 

The TunableGroups class is used to define an entire set of tunable parameters (e.g., combined set of covariant groups).

Limitations

Currently we lack a config language for expressing constraints between tunables

(e.g., a < b or a + b < c)

This is supported in the underlying mlos_core library, but is not yet exposed in the mlos_bench config API.

Usage 

Most user interactions with tunables will be through JSON configuration files.

Since tunables are associated with an Environment, their configs are typically colocated with the environment configs (e.g., env-name-tunables.jsonc) and loaded with the Environment using the include_tunables property in the Environment config.

Then individual covariant groups can be enabled via the tunable_params and tunable_params_map properties, possibly via globals Variable Expansion.

See the mlos_bench.config and mlos_bench.environments module documentation for more information.

In benchmarking-only mode (e.g., without an Optimizer specified), mlos_bench can still run with a particular set of --tunable-values (e.g., a simple key-value file declaring a set of values to assign to the set of configured tunable parameters) in order to manually explore a configuration space.

See the OneShotOptimizer and mlos_bench.run module documentation and the for more information.

During an Environment’s setup() and run() phases the tunables can be exported to a JSON file using the dump_params_file property of the Environment config for the user scripts to use when configuring the target system. The meta property of the tunable config can be used to add additional information for this step (e.g., a unit suffix to append to the value).

See the mlos_bench.environments module documentation for more information.

Examples

Here’s a short (incomplete) example of some of the TunableGroups JSON configuration options, expressed in Python (for testing purposes). However, most of the time you will be loading these from a JSON config file stored along with the associated Environment config.

For more tunable parameters examples refer to the JSON schema or some of the test examples in the source tree.

There are also examples of tunable values in the source tree.

>>> # Load tunables from JSON string.
>>> import json5
>>> from mlos_bench.services.config_persistence import ConfigPersistenceService
>>> service = ConfigPersistenceService()
>>> json_config = '''
... // Use json5 (or jsonc) syntax to allow comments and other more flexible syntax.
... {
...   "group_1": {
...     "cost": 1,
...     "params": {
...       "colors": {
...         "type": "categorical",
...         // Values for the categorical tunable.
...         "values": ["red", "blue", "green"],
...         // Weights for each value in the categorical distribution.
...         "values_weights": [0.1, 0.2, 0.7],
...         // Default value.
...         "default": "green",
...       },
...       "int_param": {
...         "type": "int",
...         "range": [1, 10],
...         "default": 5,
...         // Mark some values as "special", that need more explicit testing.
...         // e.g., maybe these indicate "automatic" or "disabled" behavior for
...         // the system being tested instead of an explicit size
...         "special": [-1, 0],
...         // Optionally specify a sampling distribution
...         // to influence which values to prioritize.
...         "distribution": {
...             "type": "uniform" // alternatively, "beta" or "normal"
...         },
...         // Free form key-value pairs that can be used with the
...         // tunable upon sampling for composing configs.
...         // These can be retrieved later to help generate
...         // config files from the sampled tunables.
...         "meta": {
...           "suffix": "MB"
...         }
...       },
...       "float_param": {
...         "type": "float",
...         "range": [1, 10000],
...         "default": 1,
...         // Quantize the range into 100 bins
...         "quantization_bins": 100,
...         // enable log sampling of the bins
...         "log": true
...       }
...     }
...   }
... }
... '''
>>> tunables = service.load_tunables(jsons=[json_config])
>>> # Retrieve the current value for the tunable groups.
>>> tunables.get_param_values()
{'colors': 'green', 'int_param': 5, 'float_param': 1.0}
>>> # Or an individual parameter:
>>> tunables["colors"]
'green'
>>> # Assign new values to the tunable groups.
>>> tunable_values = json5.loads('''
... {
...   // can be partially specified
...   "colors": "red"
... }
... ''')
>>> _ = tunables.assign(tunable_values)
>>> tunables.get_param_values()
{'colors': 'red', 'int_param': 5, 'float_param': 1.0}
>>> # Check if the tunables have been updated.
>>> # mlos_bench uses this to reinvoke the setup() phase of the
>>> # associated Environment to reconfigure the system.
>>> tunables.is_updated()
True
>>> # Reset the tunables to their default values.
>>> # As a special case, an empty json object will reset all tunables to the defaults.
>>> tunable_values = json5.loads('''
... {}
... ''')
>>> _ = tunables.assign(tunable_values)
>>> tunables.is_defaults()
True
>>> tunables.get_param_values()
{'colors': 'green', 'int_param': 5, 'float_param': 1.0}

Notes

Internally, TunableGroups are converted to ConfigSpace.ConfigurationSpace objects for use with mlos_core using the mlos_bench.optimizers.convert_configspace. See the “Spaces” section in the mlos_core module documentation for more information.

In order to handle sampling of Tunable.special values, the ! character is prohibited from being used in the name of a Tunable.

mlos_bench.tunables 

Overview 

Classes 

Tunable 

CovariantTunableGroup 

TunableGroups 

Limitations

Usage 

Submodules 