Spark Bicep Deployment Guide

This guide explains how to deploy the Spark infrastructure using Azure PowerShell.

Prerequisites

  1. Azure PowerShell Module - Install the Az module: 
    1

    Install-Module -Name Az -Repository PSGallery -Force -AllowClobber
  2. Bicep CLI - Azure PowerShell uses Bicep automatically, but ensure you have the latest version: 
    1

    az bicep upgrade
  3. Azure Subscription - You need an Azure subscription with appropriate permissions to create resources.

Authentication

Azure Commercial 

1
2
3
4
5

# Connect to Azure Commercial
Connect-AzAccount

# Select your subscription
Set-AzContext -SubscriptionId "<your-subscription-id>"

Azure Government 

1
2
3
4
5

# Connect to Azure Government
Connect-AzAccount -Environment AzureUSGovernment

# Select your subscription
Set-AzContext -SubscriptionId "<your-subscription-id>"

Deployment Templates

Template Description
automationaccount-spark.bicep Azure Automation Account with runbooks, modules, and variables
function-app-spark.bicep Azure Function App for Spark
sql.bicep Azure SQL Database

Deploying the Automation Account

If multiple Automation accounts are needed just run the script with different names

Using Parameter Files

Azure Commercial Deployment 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# Set variables
$resourceGroupName = "rg-spark-dev"
$location = "eastus2"

# Create resource group if it doesn't exist
New-AzResourceGroup -Name $resourceGroupName -Location $location -Force

# Get the certificate base64 value
$certBase64 = Get-Content -Path "path/to/certificate.pfx" -AsByteStream | 
    ForEach-Object { [Convert]::ToBase64String($_) }

$certBase64Secure = ConvertTo-SecureString -String $certBase64 -AsPlainText -Force

# Deploy using parameter file
New-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\automationaccount-spark.bicep" `
    -TemplateParameterFile ".\automationaccount.bicepparam" `
    -sendMailCertificateBase64Value $certBase64Secure `
    -Verbose

Azure Government Deployment 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Connect to Azure Government first
Connect-AzAccount -Environment AzureUSGovernment

# Set variables
$resourceGroupName = "rg-spark-prod"
$location = "usgovvirginia"

# Create resource group if it doesn't exist
New-AzResourceGroup -Name $resourceGroupName -Location $location -Force

# Get the certificate base64 value
$certBase64 = Get-Content -Path "path/to/certificate.pfx" -AsByteStream | 
    ForEach-Object { [Convert]::ToBase64String($_) }

$certBase64Secure = ConvertTo-SecureString -String $certBase64 -AsPlainText -Force

# Deploy using parameter file
New-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\automationaccount-spark.bicep" `
    -TemplateParameterFile ".\automationaccount.bicepparam" `
    -sendMailCertificateBase64Value $certBase64Secure `
    -Verbose

Example using Inline Parameters if needed 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$resourceGroupName = "rg-spark-dev"
$location = "eastus2"

# Create resource group
New-AzResourceGroup -Name $resourceGroupName -Location $location -Force

# Prepare secure string for certificate
$certBase64Secure = ConvertTo-SecureString -String $certBase64 -AsPlainText -Force

# Deploy with inline parameters
New-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\automationaccount-spark.bicep" `
    -automationAccountName "aa-spark-automation0" `
    -location $location `
    -azureEnvironment "AzureCloud" `
    -userAssignedManagedIdentityResourceId "/subscriptions/<sub-id>/resourcegroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/uami-spark-spoactions" `
    -sendMailCertificateBase64Value $certBase64Secure `
    -sendMailCertificateThumbprint "<your-thumbprint>" `
    -Verbose

Deploying the Function App 

1
2
3
4
5
6
7
8

$resourceGroupName = "rg-spark-dev"

# Azure Commercial
New-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\function-app-spark.bicep" `
    -TemplateParameterFile ".\function-app-spark.bicepparam" `
    -Verbose

Deploying SQL Database 

1
2
3
4
5
6
7
8

$resourceGroupName = "rg-spark-dev"

# Azure Commercial
New-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\sql.bicep" `
    -TemplateParameterFile ".\sql.bicepparam" `
    -Verbose

What-If Deployment (Preview Changes)

Before deploying, you can preview the changes that will be made:

1
2
3
4
5
6

New-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\automationaccount-spark.bicep" `
    -TemplateParameterFile ".\automationaccount.bicepparam" `
    -sendMailCertificateBase64Value $certBase64Secure `
    -WhatIf

Validating Templates

Validate the template without deploying:

1
2
3
4
5

Test-AzResourceGroupDeployment `
    -ResourceGroupName $resourceGroupName `
    -TemplateFile ".\automationaccount-spark.bicep" `
    -TemplateParameterFile ".\automationaccount.bicepparam" `
    -sendMailCertificateBase64Value $certBase64Secure

Complete Deployment Script

Here’s a complete script to deploy all Spark infrastructure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

#Requires -Modules Az

param(
    [Parameter(Mandatory)]
    [string]$SubscriptionId,
    
    [Parameter(Mandatory)]
    [string]$ResourceGroupName,
    
    [Parameter(Mandatory)]
    [ValidateSet("AzureCloud", "AzureUSGovernment")]
    [string]$Environment,
    
    [Parameter(Mandatory)]
    [string]$CertificatePath
)

# Set error action
$ErrorActionPreference = "Stop"

# Connect to Azure
if ($Environment -eq "AzureUSGovernment") {
    Connect-AzAccount -Environment AzureUSGovernment
    $location = "usgovvirginia"
    $automationParamFile = ".\automationaccount.bicepparam"
    $functionParamFile = ".\function-app-spark.bicepparam"
    $sqlParamFile = ".\sql.bicepparam"
} else {
    Connect-AzAccount
    $location = "eastus2"
    $automationParamFile = ".\automationaccount.bicepparam"
    $functionParamFile = ".\function-app-spark.bicepparam"
    $sqlParamFile = ".\sql.bicepparam"
}

# Set subscription context
Set-AzContext -SubscriptionId $SubscriptionId


# Read certificate
Write-Host "Reading certificate..." -ForegroundColor Cyan
$certBytes = [System.IO.File]::ReadAllBytes($CertificatePath)
$certBase64 = [Convert]::ToBase64String($certBytes)
$certBase64Secure = ConvertTo-SecureString -String $certBase64 -AsPlainText -Force

# Deploy Automation Account
Write-Host "Deploying Automation Account..." -ForegroundColor Cyan
$automationResult = New-AzResourceGroupDeployment `
    -ResourceGroupName $ResourceGroupName `
    -TemplateFile ".\automationaccount.bicep" `
    -TemplateParameterFile $automationParamFile `
    -sendMailCertificateBase64Value $certBase64Secure `
    -Verbose

Write-Host "Automation Account deployed: $($automationResult.Outputs.automationAccountName.Value)" -ForegroundColor Green

# Deploy Function App
Write-Host "Deploying Function App..." -ForegroundColor Cyan
$functionResult = New-AzResourceGroupDeployment `
    -ResourceGroupName $ResourceGroupName `
    -TemplateFile ".\function-app-spark.bicep" `
    -TemplateParameterFile $functionParamFile `
    -Verbose

Write-Host "Function App deployed successfully" -ForegroundColor Green

# Deploy SQL Database
Write-Host "Deploying SQL Database..." -ForegroundColor Cyan
$sqlResult = New-AzResourceGroupDeployment `
    -ResourceGroupName $ResourceGroupName `
    -TemplateFile ".\sql.bicep" `
    -TemplateParameterFile $sqlParamFile `
    -Verbose

Write-Host "SQL Database deployed successfully" -ForegroundColor Green

Write-Host "`nDeployment Complete!" -ForegroundColor Green
Write-Host "===================" -ForegroundColor Green

Save this as Deploy-SparkInfrastructure.ps1 and run it:

1
2
3
4
5

.\Deploy-SparkInfrastructure.ps1 `
    -SubscriptionId "your-subscription-id" `
    -ResourceGroupName "rg-spark-dev" `
    -Environment "AzureCloud" `
    -CertificatePath "C:\certs\sendmail.pfx"

Troubleshooting

Common Issues

  1. Authentication Errors 
    1
2
3

    # Clear cached credentials and re-authenticate
Disconnect-AzAccount
Connect-AzAccount
  2. Bicep Compilation Errors 
    1
2

    # Build the Bicep file to see detailed errors
az bicep build --file .\automationaccount.bicep
  3. Resource Provider Not Registered 
    1
2
3
4

    # Register required resource providers
Register-AzResourceProvider -ProviderNamespace Microsoft.Automation
Register-AzResourceProvider -ProviderNamespace Microsoft.Web
Register-AzResourceProvider -ProviderNamespace Microsoft.Sql
  4. Deployment Debugging 
    1
2
3
4

    # Get detailed deployment logs
Get-AzResourceGroupDeploymentOperation `
    -ResourceGroupName $resourceGroupName `
    -DeploymentName "<deployment-name>"

Parameter File Reference

Automation Account Parameters

Parameter Description Required
automationAccountName Name of the Automation Account Yes
location Azure region for deployment No (defaults to resource group location)
azureEnvironment AzureCloud or AzureUSGovernment No (defaults to AzureCloud)
userAssignedManagedIdentityResourceId Resource ID of the UAMI Yes
sendMailCertificateBase64Value Base64 encoded certificate Yes
sendMailCertificateThumbprint Certificate thumbprint Yes
deployRunbooks Deploy runbook definitions No (defaults to true)
deployPowerShell72Modules Deploy PS 7.2 modules No (defaults to true)
deployPowerShell51Modules Deploy PS 5.1 modules No (defaults to true)
deployRuntimeEnvironments Deploy runtime environments No (defaults to true)
skuName Basic or Free No (defaults to Basic)
tags Resource tags No

Additional Resources

Deploying Function App Code

After deploying the Function App infrastructure using function-app-spark.bicep, you need to deploy the actual function code from /assets/12-function-apps.

The simplest approach is to use the provided deployment script:

1
2
3
4
5
6
7
8
9
10
11
12
13

# Navigate to the function apps folder
cd ../assets/12-function-apps

# Deploy to Azure Commercial
.\Deploy-FunctionApp.ps1 `
    -FunctionAppName "fx-spark-prod" `
    -ResourceGroupName "rg-spark-prod"

# Deploy to Azure Government
.\Deploy-FunctionApp.ps1 `
    -FunctionAppName "fx-spark-gov" `
    -ResourceGroupName "rg-spark-gov" `
    -AzureEnvironment "AzureUSGovernment"

Function App Structure

The deployment creates the following HTTP-triggered functions:

Function Description
fx-spark-getallsites Retrieves all sites from inventory
fx-spark-getsitecounts Gets site count statistics
fx-spark-getsitesbyaction Filters sites by action type
fx-spark-getsitesbykeyword Searches sites by keyword
fx-spark-getsparksites Gets Spark-managed sites
fx-spark-updatesparksites Updates Spark site configurations