mlos_bench.optimizers.grid_search_optimizer

Grid search Optimizer for mlos_bench.

Grid search is a simple optimizer that exhaustively searches the configuration space.

To do this it generates a grid of configurations to try, and then suggests them one by one.

Therefore, the number of configurations to try is the product of the cardinality of each of the tunables. (i.e., non quantized tunables are not supported).

Examples

Load tunables from a JSON string. Note: normally these would be automatically loaded from the Environment’s include_tunables config parameter.

>>> import json5 as json
>>> from mlos_bench.environments.status import Status
>>> from mlos_bench.services.config_persistence import ConfigPersistenceService
>>> service = ConfigPersistenceService()
>>> json_config = '''
... {
...   "group_1": {
...     "cost": 1,
...     "params": {
...       "colors": {
...         "type": "categorical",
...         "values": ["red", "blue", "green"],
...         "default": "green",
...       },
...       "int_param": {
...         "type": "int",
...         "range": [1, 3],
...         "default": 2,
...       },
...       "float_param": {
...         "type": "float",
...         "range": [0, 1],
...         "default": 0.5,
...         // Quantize the range into 3 bins
...         "quantization_bins": 3,
...       }
...     }
...   }
... }
... '''
>>> tunables = service.load_tunables(jsons=[json_config])
>>> # Check the defaults:
>>> tunables.get_param_values()
{'colors': 'green', 'int_param': 2, 'float_param': 0.5}

Now create a GridSearchOptimizer from a JSON config string.

>>> optimizer_json_config = '''
... {
...   "class": "mlos_bench.optimizers.grid_search_optimizer.GridSearchOptimizer",
...   "description": "GridSearchOptimizer",
...     "config": {
...         "max_suggestions": 100,
...         "optimization_targets": {"score": "max"},
...         "start_with_defaults": true
...     }
... }
... '''
>>> config = json.loads(optimizer_json_config)
>>> grid_search_optimizer = service.build_optimizer(
...   tunables=tunables,
...   service=service,
...   config=config,
... )
>>> # Should have 3 values for each of the 3 tunables
>>> len(list(grid_search_optimizer.pending_configs))
27
>>> next(grid_search_optimizer.pending_configs)
{'colors': 'red', 'float_param': 0, 'int_param': 1}

Here are some examples of suggesting and registering configurations.

>>> suggested_config_1 = grid_search_optimizer.suggest()
>>> # Default should be suggested first, per json config.
>>> suggested_config_1.get_param_values()
{'colors': 'green', 'int_param': 2, 'float_param': 0.5}
>>> # Get another suggestion.
>>> # Note that multiple suggestions can be pending prior to
>>> # registering their scores, supporting parallel trial execution.
>>> suggested_config_2 = grid_search_optimizer.suggest()
>>> suggested_config_2.get_param_values()
{'colors': 'red', 'int_param': 1, 'float_param': 0.0}
>>> # Register some scores.
>>> # Note: Maximization problems track negative scores to produce a minimization problem.
>>> grid_search_optimizer.register(suggested_config_1, Status.SUCCEEDED, {"score": 42})
{'score': -42.0}
>>> grid_search_optimizer.register(suggested_config_2, Status.SUCCEEDED, {"score": 7})
{'score': -7.0}
>>> (best_score, best_config) = grid_search_optimizer.get_best_observation()
>>> best_score
{'score': 42.0}
>>> assert best_config == suggested_config_1

Classes

GridSearchOptimizer

Grid search optimizer.

Module Contents

class mlos_bench.optimizers.grid_search_optimizer.GridSearchOptimizer(tunables: mlos_bench.tunables.tunable_groups.TunableGroups, config: dict, global_config: dict | None = None, service: mlos_bench.services.base_service.Service | None = None)

Bases: mlos_bench.optimizers.track_best_optimizer.TrackBestOptimizer

Grid search optimizer.

See above for more details.

Create a new optimizer for the given configuration space defined by the tunables.

Parameters:

• tunables (TunableGroups) – The tunables to optimize.

• config (dict) – Free-format key/value pairs of configuration parameters to pass to the optimizer.

• global_config (dict | None)

• service (Service | None)

bulk_register(configs: collections.abc.Sequence[dict], scores: collections.abc.Sequence[dict[str, mlos_bench.tunables.tunable_types.TunableValue] | None], status: collections.abc.Sequence[mlos_bench.environments.status.Status] | None = None) → bool

Pre-load the optimizer with the bulk data from previous experiments.

Parameters:

• configs (Sequence[dict]) – Records of tunable values from other experiments.

• scores (Sequence[Optional[dict[str, TunableValue]]]) – Benchmark results from experiments that correspond to configs.

• status (Optional[Sequence[Status]]) – Status of the experiments that correspond to configs.

Returns:

is_not_empty – True if there is data to register, false otherwise.

Return type:

bool

not_converged() → bool

Return True if not converged, False otherwise.

Base implementation just checks the iteration count.

Return type:

bool

register(tunables: mlos_bench.tunables.tunable_groups.TunableGroups, status: mlos_bench.environments.status.Status, score: dict[str, mlos_bench.tunables.tunable_types.TunableValue] | None = None) → dict[str, float] | None

Register the observation for the given configuration.

Parameters:

• tunables (TunableGroups) – The configuration that has been benchmarked. Usually it’s the same config that the .suggest() method returned.

• status (Status) – Final status of the experiment (e.g., SUCCEEDED or FAILED).

• score (Optional[dict[str, TunableValue]]) – A dict with the final benchmark results. None if the experiment was not successful.

Returns:

value – Benchmark scores extracted (and possibly transformed) from the dataframe that’s being MINIMIZED.

Return type:

Optional[dict[str, float]]

suggest() → mlos_bench.tunables.tunable_groups.TunableGroups

Generate the next grid search suggestion.

Return type:

mlos_bench.tunables.tunable_groups.TunableGroups

MAX_CONFIGS = 10000

Maximum number of configurations to enumerate.

property pending_configs: collections.abc.Iterable[dict[str, mlos_bench.tunables.tunable_types.TunableValue]]

Gets the set of pending configs in this grid search optimizer.

Return type:

Iterable[dict[str, TunableValue]]

property suggested_configs: collections.abc.Iterable[dict[str, mlos_bench.tunables.tunable_types.TunableValue]]

Gets the set of configs that have been suggested but not yet registered.

Return type:

Iterable[dict[str, TunableValue]]