Cloud Execution

Azure

A key benefit and design motivation of OpticSim is being able to execute many simulations/optimizations at once. This is best enabled through the use of cloud computing services such as Azure.

As part of the base package, we provide support for cloud execution using an Azure Machine Learning workspace.

To use this functionally you'll first need to set up an AML workspace with a compute cluster. Then you'll need to provide a few bits of information to OpticSim:

Subscription ID, of the form XXXXXXXX-XXX-XXX-XXXX-XXXXXXXXXXXX
Resource group name
Workspace name
Compute cluster name

This information can be cached either to a specific file, or globally:

OpticSim.Cloud.cache_run_config — Function

cache_run_config(subscription_id::String, resource_group::String, workspace_name::String, compute_name::String[, path::String])

Writes the AML config information to a file at path. If path isn't set then the config will be used globally for that OpticSim install.

source

OpticSim.Cloud.get_cached_run_config — Function

get_cached_run_config([path::String])

Reads the AML config information from a file at path. If not specified then the global config will be read.

source

You should also include an .amlignore file in the root of your project. This is similar to a .gitignore file and should include any files which should not be uploaded to AML as part of your source snapshot, for examples test/.

Note

Manifest.toml must be listed in your .amlignore file.

If an .amlignore doesn't already exist then one will be created on the first submission of a run to AML.

Once everything is configured, you can submit a run:

OpticSim.Cloud.submit_run_to_AML — Function

submit_run_to_AML(run_name::String, path_to_script::String, script_args::Vector{String} = nothing, sampled_args:Dict{String,Vector{String}} = nothing, config_path::String; hyperdrive_concurrent_runs::Int = 10)
submit_run_to_AML(run_name::String, path_to_script::String, subscription_id::String, resource_group::String, workspace_name::String, compute_name::String, script_args::Vector{String} = nothing, sampled_args::Dict{String, Vector{String}} = nothing; hyperdrive_concurrent_runs::Int = 10)

Submit a run to AML, path_to_script is relative to your local package root (i.e. location of Project.toml). script_args are a series of arguments to your script as strings. sampled_args is a dictionary where keys are argument names and values are lists of values (as strings) that that argument will take. config_path is a path to a config file as written by cache_run_config, if not specified the global config is used. Alternatively this information can be provided directly using the second method above. hyperdrive_concurrent_runs is the maximum number of concurrent runs that will execute on AML (limited by your compute cluster size).

source

To retrieve outputs from your run simply write files to the outputs/ directory and the files will automatically appear as part of the AML run.

Examples

using OpticSim.Cloud

cache_run_config([subscription_id], [resource_group_name], [workspace_name], [compute_name], [path_to_config])

submit_run_to_AML("example-run", [path_to_script], ["--arg1", "1", "--arg2", "2"], nothing, [path_to_config])

submit_run_to_AML("example-hyperdrive-run", [path_to_script], ["--arg1", "1"], Dict("--arg2" => ["1", "2", "3"]), [path_to_config])

Other Cloud Services

Currently no other services are supported, though it should be reasonably straightforward to add similar functionality to that for AML.