mlos_bench CLI usage

Here is the current --help output for the mlos_bench CLI script:

See the mlos_bench.config module documentation for more information on configuration files.

usage: mlos_bench [-h] [--config CONFIG] [--log_file LOG_FILE]
                  [--log_level LOG_LEVEL]
                  [--config_path CONFIG_PATH [CONFIG_PATH ...]]
                  [--service SERVICE [SERVICE ...]]
                  [--environment ENVIRONMENT] [--optimizer OPTIMIZER]
                  [--trial_config_repeat_count TRIAL_CONFIG_REPEAT_COUNT]
                  [--scheduler SCHEDULER] [--storage STORAGE] [--random_init]
                  [--random_seed RANDOM_SEED]
                  [--tunable_values TUNABLE_VALUES [TUNABLE_VALUES ...]]
                  [--globals GLOBALS [GLOBALS ...]] [--no_teardown]
                  [--experiment_id EXPERIMENT_ID]

mlos_bench : Systems autotuning and benchmarking tool

options:
  -h, --help            show this help message and exit
  --config CONFIG       Main JSON5 configuration file. Its keys are the same
                        as the command line options and can be overridden by
                        the latter. See the `mlos_bench/config/` tree at
                        https://github.com/microsoft/MLOS/ for additional
                        config examples for this and other arguments.
  --log_file, --log-file LOG_FILE
                        Path to the log file. Use stdout if omitted.
  --log_level, --log-level LOG_LEVEL
                        Logging level. Default is INFO. Set to DEBUG for
                        debug, WARNING for warnings only.
  --config_path, --config-path, --config-paths, --config_paths CONFIG_PATH [CONFIG_PATH ...]
                        One or more locations of JSON config files.
  --service, --services SERVICE [SERVICE ...]
                        Path to JSON file with the configuration of the
                        service(s) for environment(s) to use.
  --environment ENVIRONMENT
                        Path to JSON file with the configuration of the
                        benchmarking environment(s).
  --optimizer OPTIMIZER
                        Path to the optimizer configuration file. If omitted,
                        run a single trial with default (or specified in
                        --tunable_values).
  --trial_config_repeat_count, --trial-config-repeat-count TRIAL_CONFIG_REPEAT_COUNT
                        Number of times to repeat each config. Default is 1
                        trial per config, though more may be advised.
  --scheduler SCHEDULER
                        Path to the scheduler configuration file. By default,
                        use a single worker synchronous scheduler.
  --storage STORAGE     Path to the storage configuration file. If omitted,
                        use the ephemeral in-memory SQL storage.
  --random_init, --random-init
                        Initialize tunables with random values. (Before
                        applying --tunable_values).
  --random_seed, --random-seed RANDOM_SEED
                        Seed to use with --random_init
  --tunable_values, --tunable-values TUNABLE_VALUES [TUNABLE_VALUES ...]
                        Path to one or more JSON files that contain values of
                        the tunable parameters. This can be used for a single
                        trial (when no --optimizer is specified) or as default
                        values for the first run in optimization.
  --globals GLOBALS [GLOBALS ...]
                        Path to one or more JSON files that contain additional
                        [private] parameters of the benchmarking environment.
  --no_teardown, --no-teardown
                        Disable teardown of the environment after the
                        benchmark.
  --experiment_id, --experiment-id EXPERIMENT_ID
                        Experiment ID to use for the benchmark. If omitted,
                        the value from the --cli config or --globals is used.
                        This is used to store and reload trial results from
                        the storage. NOTE: It is **important** to change this
                        value when incompatible changes are made to config
                        files, scripts, versions, etc. This is left as a
                        manual operation as detection of what is
                        "incompatible" is not easily automatable across
                        systems.

Additional --key=value pairs can be specified to augment or override values
listed in --globals. Other required_args values can also be pulled from shell
environment variables. For additional details, please see the website or the
README.md files in the source tree:
<https://github.com/microsoft/MLOS/tree/main/mlos_bench/>