Exercise 7: Event-Driven Autoscaling with KEDA

Kubernetes Event-driven Autoscaling (KEDA) allows you to scale your Kubernetes workloads based on the number of events that need to be processed. Unlike the standard Horizontal Pod Autoscaler (HPA) which scales based on CPU and memory metrics, KEDA can scale based on external metrics such as queue length, event hub message count, and more.

Task 1: Install KEDA using the AKS Add-on

AKS offers KEDA as a managed add-on, making it simple to deploy and integrate with your cluster.

  1. Enable the KEDA add-on for your AKS cluster:

    ​
    az aks update --enable-keda --name $AKS_NAME --resource-group $RESOURCE_GROUP 
    az aks update --enable-keda --name $AKS_NAME --resource-group $RESOURCE_GROUP

  2. Verify that KEDA has been installed correctly:

    ​
    kubectl get po -n kube-system | select-string 'keda'
    kubectl get po -n kube-system | grep 'keda'

    You should see several KEDA pods running, such as the KEDA Operator, KEDA Metrics Server, and KEDA Admission Webhooks.

  3. Create a dedicated namespace for our KEDA demo:

    ​
    kubectl create namespace keda-demo
    kubectl create namespace keda-demo

Task 2: Create an Azure Service Bus Queue

KEDA can scale based on various event sources. For this exercise, we’ll use an Azure Service Bus queue.

  1. Create an Azure Service Bus namespace:

    ​
    $SB_NAMESPACE = "kedasb$((New-Guid).ToString().Substring(0,8))"
az servicebus namespace create `
  --name $SB_NAMESPACE `
  --resource-group $RESOURCE_GROUP `
  --location $LOCATION `
  --sku Standard
    SB_NAMESPACE="kedasb$(openssl rand -hex 4)"
az servicebus namespace create \
  --name $SB_NAMESPACE \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --sku Standard

  2. Create a queue in the Service Bus namespace:

    ​
    az servicebus queue create `
  --name kedaqueue `
  --namespace-name $SB_NAMESPACE `
  --resource-group $RESOURCE_GROUP
    az servicebus queue create \
  --name kedaqueue \
  --namespace-name $SB_NAMESPACE \
  --resource-group $RESOURCE_GROUP

  3. Get the connection string for the Service Bus namespace:

    ​
    $SB_CONNECTION_STRING = az servicebus namespace authorization-rule keys list `
  --name RootManageSharedAccessKey `
  --namespace-name $SB_NAMESPACE `
  --resource-group $RESOURCE_GROUP `
  --query primaryConnectionString `
  -o tsv
    SB_CONNECTION_STRING=$(az servicebus namespace authorization-rule keys list \
  --name RootManageSharedAccessKey \
  --namespace-name $SB_NAMESPACE \
  --resource-group $RESOURCE_GROUP \
  --query primaryConnectionString \
  -o tsv)

  4. Create a Kubernetes secret with the connection string:

    ​
    kubectl create secret generic servicebus-connection `
  --from-literal=connection-string=$SB_CONNECTION_STRING `
  --namespace keda-demo
    kubectl create secret generic servicebus-connection \
  --from-literal=connection-string="$SB_CONNECTION_STRING" \
  --namespace keda-demo

  5. Grant your user account the Azure Service Bus Data Owner role on the Service Bus namespace:

    ​
    az role assignment create `
  --assignee $(az ad signed-in-user show --query userPrincipalName -o tsv) `
  --role "Azure Service Bus Data Owner" `
  --scope $(az servicebus namespace show --name $SB_NAMESPACE --resource-group $RESOURCE_GROUP --query id -o tsv)
    az role assignment create \
  --assignee "$(az ad signed-in-user show --query userPrincipalName -o tsv)" \
  --role "Azure Service Bus Data Owner" \
  --scope "$(az servicebus namespace show --name $SB_NAMESPACE --resource-group $RESOURCE_GROUP --query id -o tsv)"

Task 3: Deploy a Sample Application and ScaledObject

Now, let’s deploy the official KEDA sample application that will process messages from the Service Bus queue and scale based on queue length.

  1. Create a deployment for the KEDA sample application in the keda-demo namespace:

    ​
       $deploymentYaml = @"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-processor
  namespace: keda-demo
  labels:
    app: order-processor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: order-processor
  template:
    metadata:
      labels:
        app: order-processor
    spec:
      containers:
        - name: order-processor
          image: k8sonazureworkshoppublic.azurecr.io/ghcr.io/kedacore/sample-dotnet-worker-servicebus-queue:latest
          env:
            - name: KEDA_SERVICEBUS_AUTH_MODE
              value: ConnectionString
            - name: KEDA_SERVICEBUS_QUEUE_CONNECTIONSTRING
              valueFrom:
                secretKeyRef:
                  name: servicebus-connection
                  key: connection-string
            - name: KEDA_SERVICEBUS_QUEUE_NAME
              value: kedaqueue
"@

   $deploymentYaml | kubectl apply -f -
   ```
   
       cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-processor
  namespace: keda-demo
  labels:
    app: order-processor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: order-processor
  template:
    metadata:
      labels:
        app: order-processor
    spec:
      containers:
        - name: order-processor
          image: k8sonazureworkshoppublic.azurecr.io/ghcr.io/kedacore/sample-dotnet-worker-servicebus-queue:latest
          env:
            - name: KEDA_SERVICEBUS_AUTH_MODE
              value: ConnectionString
            - name: KEDA_SERVICEBUS_QUEUE_CONNECTIONSTRING
              valueFrom:
                secretKeyRef:
                  name: servicebus-connection
                  key: connection-string
            - name: KEDA_SERVICEBUS_QUEUE_NAME
              value: kedaqueue

EOF

  2. Create a ScaledObject to enable KEDA autoscaling:

    ​
       $scaledObjectYaml = @"
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-servicebus
  namespace: keda-demo
spec:
  secretTargetRef:
  - parameter: connection
    name: servicebus-connection
    key: connection-string
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor-scaler
  namespace: keda-demo
spec:
  scaleTargetRef:
    name: order-processor
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 5
  cooldownPeriod: 30
  triggers:
  - type: azure-servicebus
    metadata:
      queueName: kedaqueue
      messageCount: '5'
    authenticationRef:
      name: trigger-auth-servicebus
"@

   $scaledObjectYaml | kubectl apply -f -
   ```
       cat <<EOF | kubectl apply -f -
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-servicebus
  namespace: keda-demo
spec:
  secretTargetRef:
  - parameter: connection
    name: servicebus-connection
    key: connection-string
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor-scaler
  namespace: keda-demo
spec:
  scaleTargetRef:
    name: order-processor
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 5
  cooldownPeriod: 30
  triggers:
  - type: azure-servicebus
    metadata:
      queueName: kedaqueue
      messageCount: '5'
    authenticationRef:
      name: trigger-auth-servicebus
EOF

  3. Check that the ScaledObject has been created correctly:

    ​
    kubectl get scaledobjects -n keda-demo
kubectl get triggerauthentications -n keda-demo
    kubectl get scaledobjects -n keda-demo
kubectl get triggerauthentications -n keda-demo

Task 4: Test KEDA Autoscaling

Now, let’s test KEDA by sending messages to the Service Bus queue and observing how KEDA scales our application.

  1. First, check the current number of pods:

    ​
    kubectl get pods -l app=order-processor -n keda-demo
    kubectl get pods -l app=order-processor -n keda-demo

    You should see one pod running.

  2. Send multiple messages to the Service Bus queue using Azure CLI:

    ​
    $requestUri="https://$SB_NAMESPACE.servicebus.windows.net/kedaqueue/messages"
$authToken = az account get-access-token --resource "https://servicebus.azure.net/" --query accessToken -o tsv
   
for ($i = 1; $i -le 200; $i++) {
   $messageBody = @{
      Id = "order-$i"
      Amount = Get-Random -Minimum 1 -Maximum 1000
      ArticleNumber = "Item-$i"
      Customer = @{
         FirstName = "Test"
         LastName = "User-$i"
      }
   } | ConvertTo-Json -Compress
      
   Invoke-RestMethod -Uri $requestUri -Method Post -Headers @{
      "Authorization" = "Bearer $authToken"
      "Content-Type" = "application/json"
   } -Body $messageBody
      
   Write-Host "Sent message $i to Service Bus queue"
}
    requestUri="https://$SB_NAMESPACE.servicebus.windows.net/kedaqueue/messages"
authToken=$(az account get-access-token --resource "https://servicebus.azure.net/" --query accessToken -o tsv)

for i in {1..200}; do
   messageBody=$(cat <<EOF
{
   "Id": "order-$i",
   "Amount": $((RANDOM % 1000 + 1)),
   "ArticleNumber": "Item-$i",
   "Customer": {
      "FirstName": "Test",
      "LastName": "User-$i"
   }
}
EOF
   )
   
   curl -X POST "$requestUri" \
   -H "Authorization: Bearer $authToken" \
   -H "Content-Type: application/json" \
   -d "$messageBody"
   
   echo "Sent message $i to Service Bus queue"
done

  3. Watch the pods scale up as KEDA detects the messages in the queue:

    ​
    kubectl get pods -l app=order-processor -n keda-demo -w
    kubectl get pods -l app=order-processor -n keda-demo -w

    You should see new pods being created to handle the messages. KEDA scales the deployment based on the number of messages in the queue.

  4. In another terminal, you can also watch the HPA that KEDA created:

    ​
    kubectl get hpa -n keda-demo -w
    kubectl get hpa -n keda-demo -w

    KEDA works by creating an HPA object with custom metrics.

  5. You can check the logs of one of the processor pods to see the messages being processed:

    ​
    kubectl logs -f $(kubectl get pods -l app=order-processor -n keda-demo -o name | Select-Object -First 1) -n keda-demo
    kubectl logs -f $(kubectl get pods -l app=order-processor -n keda-demo -o name | head -1) -n keda-demo

After a few minutes, as the messages are processed, KEDA will scale down the deployment back to the minimum number of replicas.

Task 5: Clean Up Resources

  1. Delete the ScaledObject, TriggerAuthentication, and the deployment:

    ​
    kubectl delete scaledobject order-processor-scaler -n keda-demo
kubectl delete triggerauthentication trigger-auth-servicebus -n keda-demo
kubectl delete deployment order-processor -n keda-demo
    kubectl delete scaledobject order-processor-scaler -n keda-demo
kubectl delete triggerauthentication trigger-auth-servicebus -n keda-demo
kubectl delete deployment order-processor -n keda-demo

  2. Delete the Service Bus queue and namespace:

    ​
    az servicebus queue delete `
  --name kedaqueue `
  --namespace-name $SB_NAMESPACE `
  --resource-group $RESOURCE_GROUP

az servicebus namespace delete `
  --name $SB_NAMESPACE `
  --resource-group $RESOURCE_GROUP
    az servicebus queue delete \
  --name kedaqueue \
  --namespace-name $SB_NAMESPACE \
  --resource-group $RESOURCE_GROUP

az servicebus namespace delete \
  --name $SB_NAMESPACE \
  --resource-group $RESOURCE_GROUP

  3. Delete the namespace:

    ​
    kubectl delete namespace keda-demo
    kubectl delete namespace keda-demo