What the Hack Hackathon

Updated by Laura Edell, Sr. Data Scientist | Microsoft MSUS CTO CSU Organization
Date: 1/31/2019

Got Data? An Introduction to Using Spark on Azure Databricks

Use the challenges in this repo to get started using Spark in Azure Databricks.

0a. Start by following the Setup Guide to prepare your Azure environment.

0b. Download the all the Student files in the Files area in the Teams site for your hack.

After you successfully complete both steps listed as 0a and 0b, please complete the challenges in the following in order:

  1. Challenge 1 - Getting Started with Spark In this challenge, you’ll learn how to provision a Spark cluster in an Azure Databricks workspace, followed by interacting with the data using Python or Scala.

  2. Challenge 2 - Running a Spark Job. In this challenge, you’ll learn how to configure a Spark job for silent execution allowing you to schedule your batch processing workloads.

  3. Challenge 3 - Using Structured Streaming. In this challenge, you’ll learn how to use Spark to process stream(s) of real-time data using IoT sensor data.

  4. Challenge 4 - Introduction to Machine Learning. In this challenge, you’ll be introduced to using Spark to train & evaluate a classification model.

In the Advanced Databricks workshop, you will learn more about MMLSpark and how to build several types of Supervised and Unsupervised Machine Learning models for different business use cases https://github.com/annedroid/Ready2019_AA_AI319