Skip to main content
Version: 0.10.2

SynapseML

SynapseML is an ecosystem of tools aimed towards expanding the distributed computing framework Apache Spark in several new directions. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK), LightGBM and OpenCV. These tools enable powerful and highly-scalable predictive and analytical models for a variety of datasources.

SynapseML also brings new networking capabilities to the Spark Ecosystem. With the HTTP on Spark project, users can embed any web service into their SparkML models. In this vein, SynapseML provides easy to use SparkML transformers for a wide variety of Microsoft Cognitive Services. For production grade deployment, the Spark Serving project enables high throughput, sub-millisecond latency web services, backed by your Spark cluster.

SynapseML requires Scala 2.12, Spark 3.0+, and Python 3.6+. See the API documentation for Scala and for PySpark.

Get Started

Examples

CognitiveServices - OverviewClassification - Adult CensusCognitiveServices - OverviewGeospatialServices - OverviewConditionalKNN - Exploring Art Across CulturesCyberML - Anomalous Access DetectionDataBalanceAnalysis - Adult Census IncomeInterpretability - Image ExplainersONNX - Inference on SparkLightGBM - OverviewVowpal Wabbit - Overview

Explore our Features

Image alt text

The Cognitive Services on Spark

Leverage the Microsoft Cognitive Services at unprecedented scales in your existing SparkML pipelines.
Image alt text

Stress Free Serving

Spark is well known for it's ability to switch between batch and streaming workloads by modifying a single line. We push this concept even further and enable distributed web services with the same API as batch and streaming workloads.
Image alt text

Lightning Fast Gradient Boosting

SynapseML adds GPU enabled gradient boosted machines from the popular framework LightGBM. Users can mix and match frameworks in a single distributed environment and API.
Image alt text

Fast and Sparse Text Analytics

Vowpal Wabbit on Spark enables new classes of workloads in scalable and performant text analytics
Image alt text

Distributed Microservices

SynapseML provides powerful and idiomatic tools to communicate with any HTTP endpoint service using Spark. Users can now use Spark as a elastic micro-service orchestrator.
Image alt text

Large Scale Model Interpretability

Understand any image classifier with a distributed implementation of Local Interpretable Model Agnostic Explanations (LIME).
Image alt text

Scalable Deep Learning

SynapseML integrates the distributed computing framework Apache Spark with the flexible deep learning framework CNTK. Enabling deep learning at unprecedented scales.
Image alt text

Broad Language Support

SynapseML's API spans Scala, Python, Java, R, .NET and C# so you can integrate with any ecosystem.

Papers