mlos_bench.tunables.tunable

Definitions for Tunable parameters.

Tunable parameters are one of the core building blocks of the mlos_bench framework. Together with TunableGroups, they provide a description of a configuration parameter space for a benchmark or an autotuning optimization task.

Some details about the configuration of an individual Tunable parameter are available in the Examples docstrings below.

However, Tunables are generally provided as a part of a TunableGroups config specified in a JSON config file.

See also

mlos_bench.tunables

For more information on Tunable parameters and their configuration.

Classes

Tunable

A Tunable parameter definition and its current value.

Module Contents

class mlos_bench.tunables.tunable.Tunable(name: str, config: dict)

A Tunable parameter definition and its current value.

Create an instance of a new Tunable parameter.

Parameters:

• name (str) – Human-readable identifier of the Tunable parameter. NOTE: ! characters are currently disallowed in Tunable names in order handle “special” values sampling logic. See: mlos_bench.optimizers.convert_configspace for details.

• config (dict) – Python dict that represents a Tunable (e.g., deserialized from JSON) NOTE: Must be convertible to a TunableDict.

See also

mlos_bench.tunables

For more information on Tunable parameters and their configuration.

__eq__(other: object) → bool

Check if two Tunable objects are equal.

Parameters:

other (Tunable) – A tunable object to compare to.

Returns:

is_equal – True if the Tunables correspond to the same parameter and have the same value and type. NOTE: ranges and special values are not currently considered in the comparison.

Return type:

bool

__lt__(other: object) → bool

Compare the two Tunable objects.

We mostly need this to create a canonical list of Tunable objects when hashing a TunableGroups.

Parameters:

other (Tunable) – A tunable object to compare to.

Returns:

is_less – True if the current Tunable is less then the other one, False otherwise.

Return type:

bool

__repr__() → str

Produce a human-readable version of the Tunable (mostly for logging).

Returns:

string – A human-readable version of the Tunable.

Return type:

str

copy() → Tunable

Deep copy of the Tunable object.

Returns:

tunable – A new Tunable object that is a deep copy of the original one.

Return type:

Tunable

static from_json(name: str, json_str: str) → Tunable

Create a Tunable object from a JSON string.

Parameters:

• name (str) – Human-readable identifier of the Tunable parameter.

• json_str (str) – JSON string that represents a Tunable.

Returns:

tunable – A new Tunable object created from the JSON string.

Return type:

Tunable

Notes

This is mostly for testing purposes. Generally Tunables will be created as a part of loading TunableGroups.

See also

ConfigPersistenceService.load_tunables

in_range(value: int | float | str | None) → bool

Check if the value is within the range of the Tunable.

Do NOT check for special values. Return False if the Tunable or value is categorical or None.

Parameters:

value (int | float | str | None)

Return type:

bool

is_default() → bool

Checks whether the currently assigned value of the Tunable is at its default.

Return type:

bool

is_valid(value: mlos_bench.tunables.tunable_types.TunableValue) → bool

Check if the value can be assigned to the Tunable.

Parameters:

value (int | float | str) – Value to validate.

Returns:

is_valid – True if the value is valid, False otherwise.

Return type:

bool

update(value: mlos_bench.tunables.tunable_types.TunableValue) → bool

Assign the value to the Tunable. Return True if it is a new value, False otherwise.

Parameters:

value (int | float | str) – Value to assign.

Returns:

is_updated – True if the new value is different from the previous one, False otherwise.

Return type:

bool

property cardinality: int | None

Gets the cardinality of elements in this Tunable, or else None (e.g., when the Tunable is continuous float and not quantized).

If the Tunable has quantization set, this returns the number of quantization bins.

Returns:

cardinality – Either the number of points in the Tunable or else None.

Return type:

int

Examples

>>> json_config = '''
... {
...    "type": "categorical",
...    "default": "red",
...    "values": ["red", "blue", "green"],
... }
... '''
>>> categorical_tunable = Tunable.from_json("categorical_tunable", json_config)
>>> categorical_tunable.cardinality
3

>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10000],
... }
... '''
>>> basic_tunable = Tunable.from_json("basic_tunable", json_config)
>>> basic_tunable.cardinality
10001

>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10000],
...    // Enable quantization.
...    "quantization_bins": 10,
... }
... '''
>>> quantized_tunable = Tunable.from_json("quantized_tunable", json_config)
>>> quantized_tunable.cardinality
10

>>> json_config = '''
... {
...    "type": "float",
...    "default": 50.0,
...    "range": [0, 100],
... }
... '''
>>> float_tunable = Tunable.from_json("float_tunable", json_config)
>>> assert float_tunable.cardinality is None

property categories: list[str | None]

Get the list of all possible values of a categorical Tunable. Return None if the Tunable is not categorical.

Returns:

values – List of all possible values of a categorical Tunable.

Return type:

list[str]

See also

Tunable.values

For more examples on getting the categorical values of a Tunable.

property category: str | None

Get the current value of the Tunable as a string.

Return type:

str | None

property default: mlos_bench.tunables.tunable_types.TunableValue

Get the default value of the Tunable.

Return type:

mlos_bench.tunables.tunable_types.TunableValue

property description: str | None

Get the description of the Tunable.

Return type:

str | None

property distribution: mlos_bench.tunables.tunable_types.DistributionName | None

Get the name of the distribution if specified.

Returns:

distribution – Name of the distribution or None.

Return type:

str | None

See also

distribution_params

For more examples on configuring a Tunable with a distribution.

Examples

>>> # Example values of the DistributionName
>>> from mlos_bench.tunables.tunable_types import DistributionName
>>> DistributionName
typing.Literal['uniform', 'normal', 'beta']

property distribution_params: dict[str, float]

Get the parameters of the distribution, if specified.

Returns:

distribution_params – Parameters of the distribution or None.

Return type:

dict[str, float]

Examples

>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10],
...    // No distribution specified.
... }
... '''
>>> base_config = json.loads(json_config)
>>> basic_tunable = Tunable("basic_tunable", base_config)
>>> assert basic_tunable.distribution is None
>>> basic_tunable.distribution_params
{}

>>> # Example of a uniform distribution (the default if not specified)
>>> config_with_dist = base_config | {
...    "distribution": {
...        "type": "uniform"
...    }
... }
>>> uniform_tunable = Tunable("uniform_tunable", config_with_dist)
>>> uniform_tunable.distribution
'uniform'
>>> uniform_tunable.distribution_params
{}

>>> # Example of a normal distribution params
>>> config_with_dist = base_config | {
...    "distribution": {
...        "type": "normal",
...        "params": {
...            "mu": 0.0,
...            "sigma": 1.0,
...        }
...    }
... }
>>> normal_tunable = Tunable("normal_tunable", config_with_dist)
>>> normal_tunable.distribution
'normal'
>>> normal_tunable.distribution_params
{'mu': 0.0, 'sigma': 1.0}

>>> # Example of a beta distribution params
>>> config_with_dist = base_config | {
...    "distribution": {
...        "type": "beta",
...        "params": {
...            "alpha": 1.0,
...            "beta": 1.0,
...        }
...    }
... }
>>> beta_tunable = Tunable("beta_tunable", config_with_dist)
>>> beta_tunable.distribution
'beta'
>>> beta_tunable.distribution_params
{'alpha': 1.0, 'beta': 1.0}

property dtype: mlos_bench.tunables.tunable_types.TunableValueType

Get the actual Python data type of the Tunable.

This is useful for bulk conversions of the input data.

Returns:

dtype – Data type of the Tunable - one of: {int, float, str}

Return type:

type

Examples

>>> # Example values of the TunableValueType
>>> from mlos_bench.tunables.tunable_types import TunableValueType
>>> TunableValueType
type[int] | type[float] | type[str]

>>> # Example values of the TUNABLE_DTYPE
>>> from mlos_bench.tunables.tunable_types import TUNABLE_DTYPE
>>> TUNABLE_DTYPE
{'int': <class 'int'>, 'float': <class 'float'>, 'categorical': <class 'str'>}

property is_categorical: bool

Check if the Tunable is categorical.

Returns:

is_categorical – True if the Tunable is categorical, False otherwise.

Return type:

bool

property is_log: bool | None

Check if numeric Tunable is log scale.

Returns:

log – True if numeric Tunable is log scale, False if linear.

Return type:

bool

Examples

>>> # Example values of the log scale
>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10000],
...    // Enable log sampling.
...    "log": true,
... }
... '''
>>> tunable = Tunable.from_json("log_tunable", json_config)
>>> tunable.is_log
True

property is_numerical: bool

Check if the Tunable is an integer or float.

Returns:

is_int – True if the Tunable is an integer or float, False otherwise.

Return type:

bool

property is_special: bool

Check if the current value of the Tunable is special.

Returns:

is_special – True if the current value of the Tunable is special, False otherwise.

Return type:

bool

property meta: dict[str, Any]

Get the Tunable’s metadata.

This is a free-form dictionary that can be used to store any additional information about the Tunable (e.g., the unit information) which can be useful when using the dump_params_file and dump_meta_file properties of the environments config to generate a configuration file for the target system.

Examples

>>> json_config = '''
... {
...    "type": "int",
...    "range": [0, 10],
...    "default": 1,
...    "meta": {
...        "unit": "seconds",
...    },
...    "description": "Time to wait before timing out a request.",
... }
... '''
>>> tunable = Tunable.from_json("timer_tunable", json_config)
>>> tunable.meta
{'unit': 'seconds'}

Return type:

dict[str, Any]

property name: str

Get the name / string ID of the Tunable.

Return type:

str

property numerical_value: int | float

Get the current value of the Tunable as a number.

Return type:

int | float

property quantization_bins: int | None

Get the number of quantization bins, if specified.

Returns:

quantization_bins – Number of quantization bins, or None.

Return type:

int | None

Examples

>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10000],
...    // Enable quantization.
...    "quantization_bins": 11,
... }
... '''
>>> quantized_tunable = Tunable.from_json("quantized_tunable", json_config)
>>> quantized_tunable.quantization_bins
11
>>> list(quantized_tunable.quantized_values)
[0, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000]

>>> json_config = '''
... {
...    "type": "float",
...    "default": 0,
...    "range": [0, 1],
...    // Enable quantization.
...    "quantization_bins": 5,
... }
... '''
>>> quantized_tunable = Tunable.from_json("quantized_tunable", json_config)
>>> quantized_tunable.quantization_bins
5
>>> list(quantized_tunable.quantized_values)
[0.0, 0.25, 0.5, 0.75, 1.0]

property quantized_values: collections.abc.Iterable[int] | collections.abc.Iterable[float] | None

Get a sequence of quantized values for this Tunable.

Returns:

If the Tunable is quantizable, returns a sequence of those elements, else None (e.g., for unquantized float type Tunables).

Return type:

Iterable[int] | Iterable[float] | None

See also

quantization_bins

For more examples on configuring a Tunable with quantization.

property range: tuple[int, int] | tuple[float, float]

Get the range of the Tunable if it is numerical, None otherwise.

Returns:

range – A 2-tuple of numbers that represents the range of the Tunable. Numbers can be int or float, depending on the type of the Tunable.

Return type:

tuple[int, int] | tuple[float, float]

Examples

>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10000],
... }
... '''
>>> int_tunable = Tunable.from_json("int_tunable", json_config)
>>> int_tunable.range
(0, 10000)

>>> json_config = '''
... {
...    "type": "float",
...    "default": 0.0,
...    "range": [0.0, 100.0],
... }
... '''
>>> float_tunable = Tunable.from_json("float_tunable", json_config)
>>> float_tunable.range
(0.0, 100.0)

property range_weight: float | None

Get weight of the range of the numeric Tunable. Return None if there are no weights or a Tunable is categorical.

Returns:

weight – Weight of the range or None.

Return type:

float

See also

Tunable.weights

For example of range_weight configuration.

property span: int | float

Gets the span of the range.

Note: this does not take quantization into account.

Returns:

(max - min) for numerical Tunables.

Return type:

int | float

property special: list[int] | list[float]

Get the special values of the Tunable. Return an empty list if there are none.

Special values are used to mark some values as “special” that need more explicit testing. For example, these might indicate “automatic” or “disabled” behavior for the system being tested instead of an explicit size and hence need more explicit sampling.

Notes

Only numerical Tunable parameters can have special values.

Returns:

special – A list of special values of the Tunable. Can be empty.

Return type:

[int] | [float]

Examples

>>> # Example values of the special values
>>> json_config = '''
... {
...    "type": "int",
...    "default": 50,
...    "range": [1, 100],
...    // These are special and sampled
...    // Note that the types don't need to match or be in the range.
...    "special": [
...      -1,     // e.g., auto
...       0,     // e.g., disabled
...       true,  // e.g., enabled
...       null,  // e.g., unspecified
...    ],
... }
... '''
>>> tunable = Tunable.from_json("tunable_with_special", json_config)
>>> # JSON values are converted to Python types
>>> tunable.special
[-1, 0, True, None]

property type: mlos_bench.tunables.tunable_types.TunableValueTypeName

Get the string name of the data type of the Tunable.

Returns:

type – String representation of the data type of the Tunable.

Return type:

TunableValueTypeName

Examples

>>> # Example values of the TunableValueTypeName
>>> from mlos_bench.tunables.tunable_types import TunableValueTypeName
>>> TunableValueTypeName
typing.Literal['int', 'float', 'categorical']

Examples

>>> json_config = '''
... {
...    "type": "categorical",
...    "default": "red",
...    "values": ["red", "blue", "green"],
... }
... '''
>>> categorical_tunable = Tunable.from_json("categorical_tunable", json_config)
>>> categorical_tunable.type
'categorical'

>>> json_config = '''
... {
...    "type": "int",
...    "default": 0,
...    "range": [0, 10000],
... }
... '''
>>> int_tunable = Tunable.from_json("int_tunable", json_config)
>>> int_tunable.type
'int'

>>> json_config = '''
... {
...    "type": "float",
...    "default": 0.0,
...    "range": [0.0, 10000.0],
... }
... '''
>>> float_tunable = Tunable.from_json("float_tunable", json_config)
>>> float_tunable.type
'float'

property value: mlos_bench.tunables.tunable_types.TunableValue

Get the current value of the Tunable.

Return type:

mlos_bench.tunables.tunable_types.TunableValue

property values: collections.abc.Iterable[str | None] | collections.abc.Iterable[int] | collections.abc.Iterable[float] | None

Gets the categories or quantized_values for this Tunable.

Returns:

Categories or quantized values.

Return type:

Iterable[str | None] | Iterable[int] | Iterable[float] | None

Examples

>>> # Example values of the Tunable categories
>>> json_config = '''
... {
...    "type": "categorical",
...    "values": ["red", "blue", "green"],
...    "default": "red",
... }
... '''
>>> categorical_tunable = Tunable.from_json("categorical_tunable", json_config)
>>> list(categorical_tunable.values)
['red', 'blue', 'green']
>>> assert categorical_tunable.values == categorical_tunable.categories

>>> # Example values of the Tunable int
>>> json_config = '''
... {
...    "type": "int",
...    "range": [0, 5],
...    "default": 1,
... }
... '''
>>> int_tunable = Tunable.from_json("int_tunable", json_config)
>>> list(int_tunable.values)
[0, 1, 2, 3, 4, 5]

>>> # Example values of the quantized Tunable float
>>> json_config = '''
... {
...    "type": "float",
...    "range": [0, 1],
...    "default": 0.5,
...    "quantization_bins": 3,
... }
... '''
>>> float_tunable = Tunable.from_json("float_tunable", json_config)
>>> list(float_tunable.values)
[0.0, 0.5, 1.0]

property weights: list[float] | None

Get the weights of the categories or special values of the Tunable. Return None if there are none.

Returns:

weights – A list of weights or None.

Return type:

[float]

Examples

>>> json_config = '''
... {
...    "type": "categorical",
...    "default": "red",
...    "values": ["red", "blue", "green"],
...    "values_weights": [0.1, 0.2, 0.7],
... }
... '''
>>> categorical_tunable = Tunable.from_json("categorical_tunable", json_config)
>>> categorical_tunable.weights
[0.1, 0.2, 0.7]
>>> dict(zip(categorical_tunable.values, categorical_tunable.weights))
{'red': 0.1, 'blue': 0.2, 'green': 0.7}

>>> json_config = '''
... {
...    "type": "float",
...    "default": 50.0,
...    "range": [1, 100],
...    "special": [-1, 0],
...    "special_weights": [0.1, 0.2],
...    "range_weight": 0.7,
... }
... '''
>>> float_tunable = Tunable.from_json("float_tunable", json_config)
>>> float_tunable.weights
[0.1, 0.2]
>>> dict(zip(float_tunable.special, float_tunable.weights))
{-1: 0.1, 0: 0.2}