1. Re-install horovod using our prepared script
We build on top of torchvision, horovod and pytorch_lightning, so we need to reinstall horovod by building on specific versions of those packages. Please download our horovod installation script and upload it to databricks dbfs.
Add the path of this script to
Init Scripts section when configuring the spark cluster.
Restarting the cluster will automatically install horovod v0.25.0 with pytorch_lightning v1.5.0 and torchvision v0.12.0.
2. Install SynapseML Deep Learning Component
You could install the single synapseml-deep-learning wheel package to get the full functionality of deep vision classification. Run the following command:
pip install https://mmlspark.blob.core.windows.net/pip/$SYNAPSEML_SCALA_VERSION/synapseml_deep_learning-$SYNAPSEML_PYTHON_VERSION-py2.py3-none-any.whl
An alternative is installing the SynapseML jar package in library management section, by adding:
If you install the jar package, you need to follow the first two cell of this sample to make horovod recognizing our module.
3. Try our sample notebook
You could follow the rest of this sample and have a try on your own dataset.
Supported models (
backbone parameter for
DeepVisionClassifer) should be string format of torchvision supported models;
You could also check by running
backbone in torchvision.models.__dict__.