SynapseML

Coming from MMLSpark? We have been renamed to SynapseML!

Cognitive Services
Deep Learning
Responsible AI
LightGBM
OpenCV

from synapse.ml.cognitive import *

sentiment_df = (TextSentiment()
    .setTextCol("text")
    .setLocation("eastus")
    .setSubscriptionKey(key)
    .setOutputCol("sentiment")
    .setErrorCol("error")
    .setLanguageCol("language")
    .transform(input_df))

from synapse.ml.onnx import *

model_prediction_df = (ONNXModel()
    .setModelPayload(model_payload_ml)
    .setDeviceType("CPU")
    .setFeedDict({"input": "features"})
    .setFetchDict({"probability": "probabilities", "prediction": "label"})
    .setMiniBatchSize(64)
    .transform(input_df))

from synapse.ml.explainers import *

interpretation_df = (TabularSHAP()
    .setInputCols(features)
    .setOutputCol("shapValues")
    .setTargetCol("probability")
    .setTargetClasses([1])
    .setNumSamples(5000)
    .setModel(model)
    .transform(input_df))

from synapse.ml.lightgbm import *

quantile_df = (LightGBMRegressor()
    .setApplication('quantile')
    .setAlpha(0.3)
    .setLearningRate(0.3)
    .setNumIterations(100)
    .setNumLeaves(31)
    .fit(train_df)
    .transform(test_df))

from synapse.ml.opencv import *

image_df = (ImageTransformer()
    .setInputCol("images")
    .setOutputCol("transformed_images")
    .resize(224, True)
    .centerCrop(224, 224)
    .normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], color_scale_factor = 1/255)
    .transform(input_df))

Simple

Quickly create, train, and use distributed machine learning tools in only a few lines of code.

Scalable

Scale ML workloads to hundreds of machines on your Apache Spark cluster.

Multilingual

Use SynapseML from any Spark compatible language including Python, Scala, R, Java, .NET and C#.

Open

SynapseML is Open Source and can be installed on supported Spark 3.5 and Spark 4 infrastructure, including your local machine, Databricks, Synapse Analytics, and others.

Installation

SynapseML's Python package supplies language wrappers; Spark must also load the JVM artifact matching its Scala binary version. Open source and cloud native.

Code line	Spark	Scala	Python baseline	Release tag	Maven coordinate
`master`	3.5.x	2.12	3.11	`v1.1.3`	`com.microsoft.azure:synapseml_2.12:1.1.3`
`spark4.0`	4.0.1+ (<4.1)	2.13	3.12	`v1.1.3-spark4.0`	`com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0`
`spark4.1`	4.1.x	2.13	3.13	`v1.1.3-spark4.1`	`com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1`

All released Python variants use synapseml==1.1.3. To try the latest successful master build instead of the release, use the copy-ready snapshot command in the installation guide.

Current Synapse pools use Spark 3.5. Add the following to the first cell of a notebook:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

SynapseML is preinstalled on Fabric. Before overriding it, check the runtime's Spark and Scala versions. This example selects the published Spark 4.1 / Scala 2.13 artifact:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.13,org.scalactic:scalactic_2.13,org.scalatest:scalatest_2.13,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

SynapseML can be conveniently installed on existing Spark clusters via the --packages option:

# Spark 4.1
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"

# Spark 4.0
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"

# Spark 3.5
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" --packages "com.microsoft.azure:synapseml_2.12:1.1.3"

This can be used in other Spark contexts too. For example, you can use SynapseML in AZTK by adding it to the .aztk/spark-defaults.conf file.

To install SynapseML on the Databricks cloud, create a new library from Maven coordinates in your workspace.

Choose the coordinate matching the cluster runtime:

Spark 4.1 / Scala 2.13:

com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1

Spark 4.0 / Scala 2.13:

com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0

Spark 3.5 / Scala 2.12:

com.microsoft.azure:synapseml_2.12:1.1.3

Use the following resolver:

https://mmlspark.blob.core.windows.net/maven

Ensure this library is attached to your target cluster(s).

Restart the cluster after attaching the library so the JVM artifact is available before importing synapse.ml.

You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:

https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv1.1.3.dbc

The easiest way to evaluate SynapseML is via our pre-built Docker container. To do so, run the following command:

docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release

Navigate to http://localhost:8888 in your web browser to run the sample notebooks. See the documentation for more on Docker use.

To read the EULA for using the docker image, run

docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula

Choose exactly one Python/PySpark runtime variant matching the selected JVM artifact.

Spark 4.1 / Python 3.13:

python -m pip install "synapseml==1.1.3" "pyspark>=4.1,<4.2"

Spark 4.0 / Python 3.12:

python -m pip install "synapseml==1.1.3" "pyspark>=4.0.1,<4.1"

Spark 3.5 / Python 3.11:

python -m pip install "synapseml==1.1.3" "pyspark>=3.5,<3.6"

from pyspark.sql import SparkSession

# Spark 4.1; use "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0" for Spark 4.0 or
# "com.microsoft.azure:synapseml_2.12:1.1.3" for Spark 3.5.
coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"
spark = (
    SparkSession.builder.appName("MyApp")
    .config("spark.jars.packages", coordinate)
    .config("spark.jars.repositories", "https://mmlspark.blob.core.windows.net/maven")
    .getOrCreate()
)
import synapse.ml

If you are building a Spark application in Scala, add the following lines to your build.sbt:

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"

// Spark 4.1; use "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0" for Spark 4.0.
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.1"

// Spark 3.5:
// libraryDependencies +=
//   "com.microsoft.azure" % "synapseml_2.12" % "1.1.3"