Exercise 5: Secret Management with Azure Key Vault

The Azure Key Vault Provider for Secrets Store CSI Driver allows you to:

  • Mount secrets from Azure Key Vault as volumes in your pods
  • Automatically rotate secrets
  • Keep sensitive data out of your manifests and container images

Task 1 - Set Up Azure Key Vault

The steps below assume the environment variables you setup in Lab 1 are still available. If not, please follow the steps to recreate them.

  1. Create a Key Vault:

    ​
    # Create Key Vault
$KEYVAULT_NAME = "keyvault-$($INITIALS)"
az keyvault create --name $KEYVAULT_NAME `
                   --resource-group $RESOURCE_GROUP `
                   --location $LOCATION

Write-Host "Created Key Vault: $KEYVAULT_NAME"
    # Create Key Vault
KEYVAULT_NAME="keyvault-$INITIALS"
az keyvault create --name $KEYVAULT_NAME \
                   --resource-group $RESOURCE_GROUP \
                   --location $LOCATION

echo "Created Key Vault: $KEYVAULT_NAME"

  2. Assign yourself the “Key Vault Secrets Officer” role to ensure you have permission to manage secrets:

    ​
    # Get your user principal
$USER_ID = (az ad signed-in-user show --query id -o tsv)

# Get Key Vault resource ID
$KEYVAULT_ID = (az keyvault show --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP --query id -o tsv)

# Assign Key Vault Secrets Officer role
az role assignment create `
    --assignee $USER_ID `
    --role "Key Vault Secrets Officer" `
    --scope $KEYVAULT_ID

Write-Host "Assigned Key Vault Secrets Officer role to your account"
    # Get your user principal
USER_ID=$(az ad signed-in-user show --query id -o tsv)

# Get Key Vault resource ID
KEYVAULT_ID=$(az keyvault show --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP --query id -o tsv)

# Assign Key Vault Secrets Officer role
az role assignment create \
    --assignee $USER_ID \
    --role "Key Vault Secrets Officer" \
    --scope $KEYVAULT_ID

echo "Assigned Key Vault Secrets Officer role to your account"

  3. Add a secret to the Key Vault:

    ​
    az keyvault secret set --vault-name $KEYVAULT_NAME `
                       --name "app-secret" `
                       --value "This is a secret value from Key Vault"
    az keyvault secret set --vault-name $KEYVAULT_NAME \
                       --name "app-secret" \
                       --value "This is a secret value from Key Vault"

Task 2 - Configure AKS Access to Key Vault

  1. Check if the Key Vault Secrets Provider add-on is enabled and get its identity:

    ​
    # Get AKS name
$AKS_NAME = (az aks list --resource-group $RESOURCE_GROUP --query '[0].name' -o tsv)

# Check the Key Vault add-on status and get identity details
$ADDON_INFO = (az aks show --resource-group $RESOURCE_GROUP --name $AKS_NAME --query 'addonProfiles.azureKeyvaultSecretsProvider' -o json | ConvertFrom-Json)

# Check if add-on is enabled
if ($ADDON_INFO.enabled -ne $true) {
    Write-Host "Enabling Azure Key Vault Secrets Provider add-on..."
    az aks enable-addons --addons azure-keyvault-secrets-provider --name $AKS_NAME --resource-group $RESOURCE_GROUP
    # Refresh add-on info after enabling
    $ADDON_INFO = (az aks show --resource-group $RESOURCE_GROUP --name $AKS_NAME --query 'addonProfiles.azureKeyvaultSecretsProvider' -o json | ConvertFrom-Json)
}

# Get the managed identity client ID
$IDENTITY_CLIENT_ID = $ADDON_INFO.identity.clientId

# Get the managed identity principal ID (object ID)
$IDENTITY_PRINCIPAL_ID = $ADDON_INFO.identity.objectId

Write-Host "Key Vault Add-on is enabled: $($ADDON_INFO.enabled)"
Write-Host "Identity Client ID: $IDENTITY_CLIENT_ID"
Write-Host "Identity Principal ID: $IDENTITY_PRINCIPAL_ID"
    # Get AKS name
AKS_NAME=$(az aks list --resource-group "$RESOURCE_GROUP" --query '[0].name' -o tsv)

# Check if the Key Vault add-on is enabled
ENABLED=$(az aks show --resource-group "$RESOURCE_GROUP" --name "$AKS_NAME" \
  --query 'addonProfiles.azureKeyvaultSecretsProvider.enabled' -o tsv)

if [[ "$ENABLED" != "true" ]]; then
  echo "Enabling Azure Key Vault Secrets Provider add-on..."
  az aks enable-addons --addons azure-keyvault-secrets-provider --name "$AKS_NAME" --resource-group "$RESOURCE_GROUP"
  # Refresh enabled state after enabling
  ENABLED=$(az aks show --resource-group "$RESOURCE_GROUP" --name "$AKS_NAME" \
    --query 'addonProfiles.azureKeyvaultSecretsProvider.enabled' -o tsv)
fi

# Get the managed identity client ID and principal (object) ID
IDENTITY_CLIENT_ID=$(az aks show --resource-group "$RESOURCE_GROUP" --name "$AKS_NAME" \
  --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.clientId' -o tsv)
IDENTITY_PRINCIPAL_ID=$(az aks show --resource-group "$RESOURCE_GROUP" --name "$AKS_NAME" \
  --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.objectId' -o tsv)

echo "Key Vault Add-on is enabled: $ENABLED"
echo "Identity Client ID: $IDENTITY_CLIENT_ID"
echo "Identity Principal ID: $IDENTITY_PRINCIPAL_ID"
    Tip

    You should see a message saying Key Vault Add-on is enabled: true followed by a GUID for each of the Identity Client ID and Identity Principal ID.

  2. Assign the appropriate role to the managed identity to access Key Vault:

    ​
    $TENANT_ID = (az account show --query tenantId -o tsv)

# Use RBAC for Key Vault access (more modern approach)
az role assignment create `
    --assignee $IDENTITY_PRINCIPAL_ID `
    --role "Key Vault Secrets User" `
    --scope $KEYVAULT_ID

Write-Host "Tenant ID: $TENANT_ID"
    TENANT_ID=$(az account show --query tenantId -o tsv)

# Use RBAC for Key Vault access (more modern approach)
az role assignment create \
    --assignee $IDENTITY_PRINCIPAL_ID \
    --role "Key Vault Secrets User" \
    --scope $KEYVAULT_ID

echo "Tenant ID: $TENANT_ID"

Task 3 - Install the CSI Driver

Since we already have the Azure Key Vault Secrets Provider add-on enabled, we don’t need to install the CSI driver manually. The add-on provides all the necessary components.

  1. Verify the CSI driver installation:

    ​
    kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'
    kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'

    You should see pods for both the CSI driver and the Azure provider running.

Task 4 - Create SecretProviderClass and Pod

  1. Create a SecretProviderClass with your Key Vault information:

    ​
    $secretProviderClass = @"
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "$IDENTITY_CLIENT_ID"
    keyvaultName: "$KEYVAULT_NAME"
    objects: |
      array:
        - |
          objectName: app-secret
          objectType: secret
          objectVersion: ""
    tenantId: "$TENANT_ID"
"@

$secretProviderClass | kubectl apply -f -
    cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "$IDENTITY_CLIENT_ID"
    keyvaultName: "$KEYVAULT_NAME"
    objects: |
      array:
        - |
          objectName: app-secret
          objectType: secret
          objectVersion: ""
    tenantId: "$TENANT_ID"
EOF

  2. Create a pod that uses the SecretProviderClass:

    ​
    $secretPod = @"
apiVersion: v1
kind: Pod
metadata:
  name: nginx-secrets-store
spec:
  containers:
    - name: nginx
      image: k8sonazureworkshoppublic.azurecr.io/nginx
      volumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname"
"@

$secretPod | kubectl apply -f -
    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-secrets-store
spec:
  containers:
    - name: nginx
      image: k8sonazureworkshoppublic.azurecr.io/nginx
      volumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname"
EOF

  3. Wait for the pod to be ready:

    ​
    kubectl wait --for=condition=Ready pod/nginx-secrets-store --timeout=60s
    kubectl wait --for=condition=Ready pod/nginx-secrets-store --timeout=60s

Task 5 - Verify Secret Access

  1. Check if the secret is mounted:

    ​
    kubectl exec -it nginx-secrets-store -- ls -la /mnt/secrets-store/
    kubectl exec -it nginx-secrets-store -- ls -la /mnt/secrets-store/

    You should see the app-secret file.

  2. View the contents of the secret:

    ​
    kubectl exec -it nginx-secrets-store -- cat /mnt/secrets-store/app-secret
    kubectl exec -it nginx-secrets-store -- cat /mnt/secrets-store/app-secret

    This should display the secret value you stored in Key Vault.

Task 6 - Create Kubernetes Secret from Key Vault

Rather than just mounting the secret, you can also create a Kubernetes Secret from the Key Vault secret. This allows you to enable native integration with Kubernetes features that require Secret objects, such as environment variables, init containers, and Helm charts.

  1. Update the SecretProviderClass to create a Kubernetes Secret:

    ​
    $secretProviderClassSync = @"
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-sync
spec:
  provider: azure
  secretObjects:
  - secretName: keyvault-secret
    type: Opaque
    data: 
    - objectName: app-secret
      key: app-secret
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "$IDENTITY_CLIENT_ID"
    keyvaultName: "$KEYVAULT_NAME"
    objects: |
      array:
        - |
          objectName: app-secret
          objectType: secret
          objectVersion: ""
    tenantId: "$TENANT_ID"
"@

$secretProviderClassSync | kubectl apply -f -
    cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-sync
spec:
  provider: azure
  secretObjects:
  - secretName: keyvault-secret
    type: Opaque
    data: 
    - objectName: app-secret
      key: app-secret
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "$IDENTITY_CLIENT_ID"
    keyvaultName: "$KEYVAULT_NAME"
    objects: |
      array:
        - |
          objectName: app-secret
          objectType: secret
          objectVersion: ""
    tenantId: "$TENANT_ID"
EOF

  2. The secret won’t actually exist until a pod consumes it. Create a pod that uses this SecretProviderClass:

    ​
    $secretPodSync = @"
apiVersion: v1
kind: Pod
metadata:
  name: nginx-secrets-sync
spec:
  containers:
    - name: nginx
      image: k8sonazureworkshoppublic.azurecr.io/nginx
      volumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
      env:
      - name: SECRET_FROM_KEYVAULT
        valueFrom:
          secretKeyRef:
            name: keyvault-secret
            key: app-secret
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-sync"
"@

$secretPodSync | kubectl apply -f -
    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-secrets-sync
spec:
  containers:
    - name: nginx
      image: k8sonazureworkshoppublic.azurecr.io/nginx
      volumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
      env:
      - name: SECRET_FROM_KEYVAULT
        valueFrom:
          secretKeyRef:
            name: keyvault-secret
            key: app-secret
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-sync"
EOF

  3. Wait for the pod to be ready:

    ​
    kubectl wait --for=condition=Ready pod/nginx-secrets-sync --timeout=60s
    kubectl wait --for=condition=Ready pod/nginx-secrets-sync --timeout=60s

  4. Verify the Kubernetes Secret was created:

    ​
    kubectl get secret keyvault-secret
    kubectl get secret keyvault-secret

  5. Check that the environment variable is set in the pod:

    ​
    kubectl exec -it nginx-secrets-sync -- sh -c 'echo $SECRET_FROM_KEYVAULT'
    kubectl exec -it nginx-secrets-sync -- sh -c 'echo $SECRET_FROM_KEYVAULT'

    You should see the secret value from Key Vault.

Task 7 - Clean Up

  1. Delete the pod and SecretProviderClass:

    ​
    kubectl delete pod nginx-secrets-store
kubectl delete pod nginx-secrets-sync
kubectl delete secretproviderclass azure-kvname
kubectl delete secretproviderclass azure-kvname-sync
    kubectl delete pod nginx-secrets-store
kubectl delete pod nginx-secrets-sync
kubectl delete secretproviderclass azure-kvname
kubectl delete secretproviderclass azure-kvname-sync

  2. Delete the key vault and its contents:

    ​
    az keyvault delete --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP
    az keyvault delete --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP