Benchmark#

NATS-Bench#

class archai.discrete_search.evaluators.benchmark.natsbench_tss.NatsbenchMetric(search_space: NatsbenchTssSearchSpace, metric_name: str, epochs: int | None = None, raise_not_found: bool | None = True, more_info_kwargs: Dict[str, Any] | None = None, cost_info_kwargs: Dict[str, Any] | None = None)[source]#

Evaluate a model using a metric from the NATS-Bench API.

evaluate(model: ArchaiModel, budget: float | None = None) → float | None[source]#

Evaluate an ArchaiModel instance, optionally using a budget value.

Parameters:

  • arch – Model to be evaluated.

  • dataset – A dataset provider object.

  • budget – A budget multiplier value, used by search algorithms like SuccessiveHalving to specify how much compute should be spent in this evaluation. In order to use this type of search algorithm, the implementation of evaluate() must use the passed budget value accordingly.

Returns:

Evaluation result.