mlos_bench.tunables 

Tunables classes for Environments in mlos_bench.

Overview 

mlos_bench uses the concept of “tunables” to define the configuration space for an Environment.

An Optimizer can then use these tunables to explore the configuration space in order to improve some target objective metrics (e.g., reduce tail latency, reduce cost, improve throughput, etc.).

They are similar to the concept of “hyperparameters” in machine learning, but are used to configure the system being tested.

The Tunable class is used to define a single tunable parameter. A Tunable can be a categorical or numeric (int or float) and always has at least a domain (range or set of values) and default. Each type can also have a number of additional properties that can optionally be set to help control the sampling of the tunable.

For instance:

Numeric tunables can have a distribution property to specify the sampling distribution. log sampling can also be enabled for numeric tunables.
Categorical tunables can have a values_weights property to specify biased sampling of the values
special values can be marked to indicate that they need more explicit testing This can be useful for values that indicate “automatic” or “disabled” behavior.

The full set of supported properties can be found in the JSON schema for tunable parameters and seen in some of the test examples in the source tree.

CovariantGroup 

The CovariantTunableGroup class is used to define a group of related tunable parameters that are all configured together with the same cost (e.g., is a more expensive operation required to reconfigure the system like redeployed vs. restarted vs. reloaded). Optimizers can use this information to explore the configuration space more efficiently.

TunableGroups 

The TunableGroups class is used to define an entire set of tunable parameters (e.g., combined set of covariant groups).

Usage 

Most user interactions with tunables will be through JSON configuration files.

Since tunables are associated with an Environment, their configs are typically colocated with the environment configs (e.g., env-name-tunables.jsonc) and loaded with the Environment using the include_tunables property in the Environment config.

Then individual covariant groups can be enabled via the tunable_params and tunable_params_map properties, possibly via globals variable expansion.

See the mlos_bench.config and mlos_bench.environments module documentation for more information.

In benchmarking-only mode (e.g., without an Optimizer specified), mlos_bench can still run with a particular set of --tunable-values (e.g., a simple key-value file declaring a set of values to assign to the set of configured tunable parameters) in order to manually explore a configuration space.

See the mlos_bench.run module documentation for more information.

During an Environment’s setup and run phases the tunables can be exported to a JSON file using the dump_params_file property of the Environment config for the user scripts to use when configuring the target system. The meta property of the tunable config can be used to add additional information for this step (e.g., a unit suffix to append to the value).

See the mlos_bench.environments module documentation for more information.

Examples

Here’s a short (incomplete) example of some of the TunableGroups JSON configuration options, expressed in Python (for testing purposes). However, most of the time you will be loading these from a JSON config file stored along with the associated Environment config.

For more tunable parameters examples refer to the JSON schema or some of the test examples in the source tree.

There are also examples of tunable values in the source tree.

>>> # Load tunables from JSON string.
>>> import json5
>>> from mlos_bench.services.config_persistence import ConfigPersistenceService
>>> service = ConfigPersistenceService()
>>> json_config = '''
... {
...   "group_1": {
...     "cost": 1,
...     "params": {
...       "colors": {
...         "type": "categorical",
...         // Values for the categorical tunable.
...         "values": ["red", "blue", "green"],
...         // Weights for each value in the categorical distribution.
...         "values_weights": [0.1, 0.2, 0.7],
...         // Default value.
...         "default": "green",
...       },
...       "int_param": {
...         "type": "int",
...         "range": [-1, 10],
...         "default": 5,
...         // Mark some values as "special", that need more explicit testing.
...         // e.g., maybe these indicate "automatic" or "disabled" behavior for
...         // the system being tested instead of an explicit size
...         "special": [-1, 0],
...         // Optionally specify a sampling distribution.
...         "distribution": {
...             "type": "uniform" // alternatively, "beta" or "normal"
...         },
...         // Free form key-value pairs that can be used with the
...         // tunable upon sampling for composing configs.
...         "meta": {
...           "suffix": "MB"
...         }
...       },
...       "float_param": {
...         "type": "float",
...         "range": [1, 10000],
...         "default": 1,
...         // Quantize the range into 100 bins
...         "quantization_bins": 100,
...         // enable log sampling of the bins
...         "log": true
...       }
...     }
...   }
... }
... '''
>>> tunables = service.load_tunables(jsons=[json_config])
>>> # Retrieve the current value for the tunable groups.
>>> tunables.get_param_values()
{'colors': 'green', 'int_param': 5, 'float_param': 1.0}
>>> # Or an individual parameter:
>>> tunables["colors"]
'green'
>>> # Assign new values to the tunable groups.
>>> tunable_values = json5.loads('''
... {
...   // can be partially specified
...   "colors": "red"
... }
... ''')
>>> _ = tunables.assign(tunable_values)
>>> tunables.get_param_values()
{'colors': 'red', 'int_param': 5, 'float_param': 1.0}
>>> # Check if the tunables have been updated.
>>> # mlos_bench uses this to reinvoke the setup() phase of the
>>> # associated Environment to reconfigure the system.
>>> tunables.is_updated()
True
>>> # Reset the tunables to their default values.
>>> # As a special case, an empty json object will reset all tunables to the defaults.
>>> tunable_values = json5.loads('''
... {}
... ''')
>>> _ = tunables.assign(tunable_values)
>>> tunables.is_defaults()
True
>>> tunables.get_param_values()
{'colors': 'green', 'int_param': 5, 'float_param': 1.0}

Notes

Internally, TunableGroups are converted to ConfigSpace.ConfigurationSpace objects for use with mlos_core. See the “Spaces” section in the mlos_core module documentation for more information.

mlos_bench.tunables 

Overview 

Classes 

Tunable 

CovariantGroup 

TunableGroups 

Usage 

Submodules 