Azure Red Hat OpenShift Workshop

Azure Red Hat OpenShift is a fully managed Red Hat OpenShift service in Azure that is jointly engineered and supported by Microsoft and Red Hat. In this lab, you’ll go through a set of tasks that will help you understand some of the concepts of deploying and securing container based applications on top of Azure Red Hat OpenShift.

You can use this guide as an OpenShift tutorial and as study material to help you get started to learn OpenShift.

Some of the things you’ll be going through:

• Creating a project on the Azure Red Hat OpenShift Web Console
• Deploying a MongoDB container that uses Azure Disks for persistent storage
• Deploying a Node JS API and frontend app from Git Hub using Source-To-Image (S2I)
• Exposing the web application frontend using Routes
• Creating a network policy to control communication between the different tiers in the application

You’ll be doing the majority of the labs using the OpenShift CLI, but you can also accomplish them using the Azure Red Hat OpenShift web console.

Prerequisites

Azure subscription and Azure Red Hat OpenShift environment

Tools

Azure Cloud Shell

You can use the Azure Cloud Shell accessible at https://shell.azure.com once you login with an Azure subscription.

OpenShift CLI (oc)

You’ll need to download the latest OpenShift CLI (oc) client tools for OpenShift 4. You can follow the steps below on the Azure Cloud Shell.

cd ~
curl https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz > openshift-client-linux.tar.gz

mkdir openshift

tar -zxvf openshift-client-linux.tar.gz -C openshift

echo 'export PATH=$PATH:~/openshift' >> ~/.bashrc && source ~/.bashrc

GitHub Account

You’ll need a personal GitHub account. You can sign up for free here.

Create cluster

We will now create our own cluster.

If you choose to install and use the CLI locally, this tutorial requires that you are running the Azure CLI version 2.6.0 or later. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.

Azure Red Hat OpenShift requires a minimum of 40 cores to create and run an OpenShift cluster. The default Azure resource quota for a new Azure subscription does not meet this requirement. To request an increase in your resource limit, see Standard quota: Increase limits by VM series.

Verify your permissions

To create an Azure Red Hat OpenShift cluster, verify the following permissions on your Azure subscription, Azure Active Directory user, or service principal:

Register the resource providers

Next, you need to register the following resource providers in your subscription.

Create a virtual network containing two empty subnets

Next, you will create a virtual network containing two empty subnets.

Get a Red Hat pull secret

A Red Hat pull secret enables your cluster to access Red Hat container registries along with additional content. This step is required to be able to pull Red Hat images.

Obtain your pull secret by navigating to https://cloud.redhat.com/openshift/install/azure/aro-provisioned and clicking Download pull secret. You will need to log in to your Red Hat account or create a new Red Hat account with your business email and accept the terms and conditions.

Note You can upload that file to Azure Cloud Shell by dragging and dropping the file into the window.

Create the cluster

Run the following command to create a cluster. When running the az aro create command, you can reference your pull secret using the –pull-secret @pull-secret.txt parameter. Execute az aro create from the directory where you stored your pull-secret.txt file. Otherwise, replace @pull-secret.txt with @<path-to-my-pull-secret-file>.

az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--pull-secret @pull-secret.txt
# --domain foo.example.com # [OPTIONAL] custom domain

Get kubeadmin credentials

az aro list-credentials \
  --name $CLUSTER \
  --resource-group $RESOURCEGROUP

{
  "kubeadminPassword": "<generated password>",
  "kubeadminUsername": "kubeadmin"
}

Get web console address

az aro list -o table

Connect to the cluster

This section assumes you have been given cluster credentials by your lab administrator, or that you have created a cluster in the previous section. You should have;

• Web Console URL
• A username (eg: kubeadmin, student01)
• A password

Login to the web console

Open the Web Console URL in your web browser, and login with the credentials you have. After logging in, you should be able to see the Azure Red Hat OpenShift Web Console.

Choose an option to use the oc command

Lab 1 - Go Microservices

Now that you have your environment provisioned and the prerequisites fulfilled, it is time to start working on the labs.

Application Overview

You will be deploying a ratings application on Azure Red Hat OpenShift.

The application consists of 3 components:

Once you’re done, you’ll have an experience similar to the below.

Create Project

Create a project

oc new-project workshop

Resources

• ARO Documentation - Getting started with the CLI
• ARO Documentation - Projects

Deploy MongoDB

Create mongoDB from Docker hub

oc new-app bitnami/mongodb \
  -e MONGODB_USERNAME=ratingsuser \
  -e MONGODB_PASSWORD=ratingspassword \
  -e MONGODB_DATABASE=ratingsdb \
  -e MONGODB_ROOT_USER=root \
  -e MONGODB_ROOT_PASSWORD=ratingspassword

Verify if the mongoDB pod was created successfully

oc get all

Retrieve mongoDB service hostname

oc get svc mongodb

Deploy Ratings API

The rating-api is a NodeJS application that connects to mongoDB to retrieve and rate items. Below are some of the details that you’ll need to deploy this.

• rating-api on GitHub: https://github.com/MicrosoftDocs/mslearn-aks-workshop-ratings-api
• The container exposes port 8080
• MongoDB connection is configured using an environment variable called MONGODB_URI

Fork the application to your own GitHub repository

To be able to setup CI/CD webhooks, you’ll need to fork the application into your personal GitHub repository.

Fork

Use the OpenShift CLI to deploy the rating-api

Note You’re going to be using source-to-image (S2I) as a build strategy.

oc new-app https://github.com/<your GitHub username>/mslearn-aks-workshop-ratings-api --strategy=source --name=rating-api

Configure the required environment variables

oc set env deploy/rating-api MONGODB_URI=mongodb://ratingsuser:ratingspassword@mongodb.workshop.svc.cluster.local:27017/ratingsdb

Verify that the service is running

Retrieve rating-api service hostname

oc get svc rating-api

Setup GitHub webhook

To trigger S2I builds when you push code into your GitHib repo, you’ll need to setup the GitHub webhook.

oc get bc/rating-api -o=jsonpath='{.spec.triggers..github.secret}'

oc describe bc/rating-api

Resources

• ARO Documentation - Triggering builds

Deploy Ratings frontend

The rating-web is a NodeJS application that connects to the rating-api. Below are some of the details that you’ll need to deploy this.

• rating-web on GitHub: https://github.com/MicrosoftDocs/mslearn-aks-workshop-ratings-web
• The rating-web frontend Dockerfile needs modification to the image to run on ARO 4.x versions
• The container exposes port 8080
• The web app connects to the API over the internal cluster DNS, using a proxy through an environment variable named API

Fork the application to your own GitHub repository

To be able to setup CI/CD webhooks, you’ll need to fork the application into your personal GitHub repository.

Fork

Modify Dockerfile in your repository

Note The Dockerfile needs to be modified to use the correct node.js image to build correctly.

Note The git commands shown below have been run in a local shell pre-authenticated against the GitHub repository

git clone https://github.com/<your GitHub username>/mslearn-aks-workshop-ratings-web.git
cd mslearn-aks-workshop-ratings-web

wget https://raw.githubusercontent.com/sajitsasi/rating-web/master/Dockerfile -O ./Dockerfile
wget https://raw.githubusercontent.com/sajitsasi/rating-web/master/src/components/Footer.vue -O ./src/components/Footer.vue

git status
git add .
git commit -m "Modified Dockerfile and Footer.vue"
git push

Use the OpenShift CLI to deploy the rating-web

Note You’re going to be using source-to-image (S2I) as a build strategy.

oc new-app https://github.com/<your GitHub username>/mslearn-aks-workshop-ratings-web --strategy=docker --name=rating-web

Configure the required environment variables

oc set env deploy rating-web API=http://rating-api:8080

Expose the rating-web service using a Route

oc expose svc/rating-web

oc get route rating-web

Try the service

Setup GitHub webhook

To trigger S2I builds when you push code into your GitHib repo, you’ll need to setup the GitHub webhook.

oc get bc/rating-web -o=jsonpath='{.spec.triggers..github.secret}'

oc describe bc/rating-web

Make a change to the website app and see the rolling update

Resources

• ARO Documentation - Triggering builds

Create Network Policy

Now that you have the application working, it is time to apply some security hardening. You’ll use network policies to restrict communication to the rating-api.

Switch to the Cluster Console

Create network policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-from-web
  namespace: workshop
spec:
  podSelector:
    matchLabels:
      app: rating-api
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: rating-web

Resources

• ARO Documentation - Managing Networking with Network Policy

Lab 2 - ARO Internals

Application Overview

Resources

• The source code for this app is available here: https://github.com/openshift-cs/ostoy
• OSToy front-end container image: https://quay.io/repository/ostoylab/ostoy-frontend?tab=tags
• OSToy microservice container image: https://quay.io/repository/ostoylab/ostoy-microservice?tab=tags
• Deployment Definition YAMLs:
  • ostoy-frontend-deployment.yaml
  • ostoy-microservice-deployment.yaml

Note In order to simplify the deployment of the app (which you will do next) we have included all the objects needed in the above YAMLs as “all-in-one” YAMLs. In reality though, an enterprise would most likely want to have a different yaml file for each Kubernetes object.

About OSToy

OSToy is a simple Node.js application that we will deploy to Azure Red Hat OpenShift. It is used to help us explore the functionality of Kubernetes. This application has a user interface which you can:

• write messages to the log (stdout / stderr)
• intentionally crash the application to view self-healing
• toggle a liveness probe and monitor OpenShift behavior
• read config maps, secrets, and env variables
• if connected to shared storage, read and write files
• check network connectivity, intra-cluster DNS, and intra-communication with an included microservice
• increase the load to view automatic scaling of the pods to handle the load (via the Horizontal Pod Autoscaler)
• connect to an Azure Blob Storage container to read and write objects (if Blob Storage already present)

OSToy Application Diagram

Familiarization with the Application UI

1. Shows the pod name that served your browser the page.
2. Home: The main page of the application where you can perform some of the functions listed which we will explore.
3. Persistent Storage: Allows us to write data to the persistent volume bound to this application.
4. Config Maps: Shows the contents of configmaps available to the application and the key:value pairs.
5. Secrets: Shows the contents of secrets available to the application and the key:value pairs.
6. ENV Variables: Shows the environment variables available to the application.
7. Auto Scaling: Explore the Horizontal Pod Autoscaler to see how increased loads are handled.
8. Networking: Tools to illustrate networking within the application.
9. ASO - Blob Storage: Integrate with Azure Blob Storage to read and write objects to a container.
10. About: Shows some more information about the application.

Learn more about the application

To learn more, click on the “About” menu item on the left once we deploy the app.

Application Deployment

Retrieve login command

If not logged in via the CLI, click on the dropdown arrow next to your name in the top-right and select Copy Login Command.

$ oc login --token=sha256~qWBXdQ_X_4wWZor0XZO00ZZXXXXXXXXXXXX --server=https://api.abcs1234.westus.aroapp.io:6443
Logged into "https://api.abcd1234.westus.aroapp.io:6443" as "kube:admin" using the token provided.

You have access to 67 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".

Create new project

Create a new project called “OSToy” in your cluster.

$ oc new-project ostoy
Now using project "ostoy" on server "https://api.abcd1234.westus2.aroapp.io:6443".
[...]

View the YAML deployment manifest

View the Kubernetes deployment manifest. If you wish you can download them from the following locations to your Azure Cloud Shell, to your local machine, or just use the direct link in the next steps.

Deploy backend microservice

The microservice serves internal web requests and returns a JSON object containing the current hostname and a randomly generated color string.

$ oc apply -f https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/ostoy-microservice-deployment.yaml
deployment.apps/ostoy-microservice created
service/ostoy-microservice-svc created

Deploy the front-end service

This deployment contains the node.js frontend for our application along with a few other Kubernetes objects.

$ oc apply -f https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/ostoy-frontend-deployment.yaml
persistentvolumeclaim/ostoy-pvc created
deployment.apps/ostoy-frontend created
service/ostoy-frontend-svc created
route.route.openshift.io/ostoy-route created
configmap/ostoy-configmap-env created
secret/ostoy-secret-env created
configmap/ostoy-configmap-files created
secret/ostoy-secret created

Get route

Get the route so that we can access the application via:

oc get route

NAME           HOST/PORT                                                      PATH      SERVICES              PORT      TERMINATION   WILDCARD
ostoy-route   ostoy-route-ostoy.apps.abcd1234.westus2.aroapp.io             ostoy-frontend-svc   <all>                   None

Logging and Metrics

Assuming you can access the application via the Route provided and are still logged into the CLI (please go back to part 2 if you need to do any of those) we’ll start to use this application. As stated earlier, this application will allow you to “push the buttons” of OpenShift and see how it works. We will do this to test the logs.

Click on the Home menu item and then click in the message box for “Log Message (stdout)” and write any message you want to output to the stdout stream. You can try “All is well!”. Then click “Send Message”.

Click in the message box for “Log Message (stderr)” and write any message you want to output to the stderr stream. You can try “Oh no! Error!”. Then click “Send Message”.

View logs directly from the pod

$ oc get pods -o name
pod/ostoy-frontend-679cb85695-5cn7x
pod/ostoy-microservice-86b4c6f559-p594d

$ oc logs ostoy-frontend-679cb85695-5cn7x
[...]
ostoy-frontend-679cb85695-5cn7x: server starting on port 8080
Redirecting to /home
stdout: All is well!
stderr: Oh no! Error!

View metrics and logs by integrating with Azure Arc

Exploring Health Checks

In this section we will intentionally crash our pods and also make a pod non-responsive to the liveness probes and see how Kubernetes behaves. We will first intentionally crash our pod and see that Kubernetes will self-heal by immediately spinning it back up. Then we will trigger the health check by stopping the response on the /health endpoint in our app. After three consecutive failures, Kubernetes should kill the pod and then recreate it.

Persistent Storage

In this section we will execute a simple example of using persistent storage by creating a file that will be stored on a persistent volume in our cluster and then confirm that it will “persist” across pod failures and recreation.

$ oc get pods
NAME                                  READY     STATUS    RESTARTS   AGE
ostoy-frontend-5fc8d486dc-wsw24       1/1       Running   0          18m
ostoy-microservice-6cf764974f-hx4qm   1/1       Running   0          18m

$ oc rsh ostoy-frontend-5fc8d486dc-wsw24

/ $ cd /var/demo_files/

/var/demo_files $ ls
lost+found   test-pv.txt

/var/demo_files $ cat test-pv.txt
Azure Red Hat OpenShift is the greatest thing since sliced bread!

Configuration

In this section we’ll take a look at how OSToy can be configured using ConfigMaps, Secrets, and Environment Variables. This section won’t go into details explaining each (the links above are for that), but will show you how they are exposed to the application.

Configuration using ConfigMaps

ConfigMaps allow you to decouple configuration artifacts from container image content to keep containerized applications portable.

kind: ConfigMap
apiVersion: v1
metadata:
  name: ostoy-configmap-files
data:
  config.json:  '{ "default": "123" }'

Configuration using Secrets

Kubernetes Secret objects allow you to store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it, verbatim, into a pod definition or a container image.

apiVersion: v1
kind: Secret
metadata:
  name: ostoy-secret
data:
  secret.txt: VVNFUk5BTUU9bXlfdXNlcgpQQVNTV09SRD1AT3RCbCVYQXAhIzYzMlk1RndDQE1UUWsKU01UUD1sb2NhbGhvc3QKU01UUF9QT1JUPTI1
type: Opaque

Configuration using Environment Variables

Using environment variables is an easy way to change application behavior without requiring code changes. It allows different deployments of the same application to potentially behave differently based on the environment variables, and OpenShift makes it simple to set, view, and update environment variables for Pods/Deployments.

  env:
  - name: ENV_TOY_CONFIGMAP
    valueFrom:
      configMapKeyRef:
        name: ostoy-configmap-env
        key: ENV_TOY_CONFIGMAP
  - name: ENV_TOY_SECRET
    valueFrom:
      secretKeyRef:
        name: ostoy-secret-env
        key: ENV_TOY_SECRET
  - name: MICROSERVICE_NAME
    value: OSTOY_MICROSERVICE_SVC
  - name: NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace

Networking and Scaling

In this section we’ll see how OSToy uses intra-cluster networking to separate functions by using microservices and visualize the scaling of pods.

Let’s review how this application is set up…

As can be seen in the image above we have defined at least 2 separate pods, each with its own service. One is the frontend web application (with a service and a publicly accessible route) and the other is the backend microservice with a service object created so that the frontend pod can communicate with the microservice (across the pods if more than one). Therefore this microservice is not accessible from outside this cluster (or from other namespaces/projects, if configured, due to OpenShifts’ network policy, ovs-networkpolicy). The sole purpose of this microservice is to serve internal web requests and return a JSON object containing the current hostname and a randomly generated color string. This color string is used to display a box with that color displayed in the tile titled “Intra-cluster Communication”.

Networking

Click on Networking in the left menu. Review the networking configuration.

apiVersion: v1
kind: Service
metadata:
  name: ostoy-microservice-svc
  labels:
    app: ostoy-microservice
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: ostoy-microservice

Scaling

OpenShift allows one to scale up/down the number of pods for each part of an application as needed. This can be accomplished via changing our replicaset/deployment definition (declarative), by the command line (imperative), or via the web console (imperative). In our deployment definition (part of our ostoy-frontend-deployment.yaml) we stated that we only want one pod for our microservice to start with. This means that the Kubernetes Replication Controller will always strive to keep one pod alive. We can also define pod autoscaling using the Horizontal Pod Autoscaler (HPA) based on load to expand past what we defined. We will do this in a later section of this lab.

$ oc get pods
NAME                                   READY     STATUS    RESTARTS   AGE
ostoy-frontend-679cb85695-5cn7x       1/1       Running   0          1h
ostoy-microservice-86b4c6f559-p594d   1/1       Running   0          1h

spec:
    selector:
      matchLabels:
        app: ostoy-microservice
    replicas: 3

Pod Autoscaling

In this section we will explore how the Horizontal Pod Autoscaler (HPA) can be used and works within Kubernetes/OpenShift.

As defined in the documentation:

[…] you can use a horizontal pod autoscaler (HPA) to specify how OpenShift Container Platform should automatically increase or decrease the scale of a replication controller or deployment configuration, based on metrics collected from the pods that belong to that replication controller or deployment configuration.

In more simple words, “if there is a lot of work, make more pods”.

We will create a HPA and then use OSToy to generate CPU intensive workloads. We will then observe how the HPA will scale up the number of pods in order to handle the increased workloads.

oc autoscale deployment/ostoy-microservice --cpu-percent=80 --min=1 --max=10

oc get pods --field-selector=status.phase=Running | grep microservice

$ oc get pods --field-selector=status.phase=Running
NAME                                READY   STATUS    RESTARTS       AGE
ostoy-frontend-64c8668694-7pq6k     1/1     Running   3 (105m ago)   23h
ostoy-microservice-cf8bfb4c-bkwx8   1/1     Running   0              104s
ostoy-microservice-cf8bfb4c-j24f9   1/1     Running   0              23h
ostoy-microservice-cf8bfb4c-xls2t   1/1     Running   0              104s

Managing Worker Nodes

There may be times when you need to change aspects of your worker nodes. Things like scaling, changing the type, adding labels or taints to name a few. Most of these things are done through the use of machine sets. A machine is a unit that describes the host for a node and a machine set is a group of machines. Think of a machine set as a “template” for the kinds of machines that make up the worker nodes of your cluster. Similar to how a replicaset is to pods. A machine set allows users to manage many machines as a single entity though it is contained to a specific availability zone. If you’d like to learn more see Overview of machine management

Scaling worker nodes

$ oc get machinesets -n openshift-machine-api
NAME                           DESIRED   CURRENT   READY   AVAILABLE   AGE
ok0620-rq5tl-worker-westus21   1         1         1       1           72m
ok0620-rq5tl-worker-westus22   1         1         1       1           72m
ok0620-rq5tl-worker-westus23   1         1         1       1           72m

$ oc get machine -n openshift-machine-api
NAME                                 PHASE     TYPE              REGION    ZONE   AGE
ok0620-rq5tl-master-0                Running   Standard_D8s_v3   westus2   1      73m
ok0620-rq5tl-master-1                Running   Standard_D8s_v3   westus2   2      73m
ok0620-rq5tl-master-2                Running   Standard_D8s_v3   westus2   3      73m
ok0620-rq5tl-worker-westus21-n6lcs   Running   Standard_D4s_v3   westus2   1      73m
ok0620-rq5tl-worker-westus22-ggcmv   Running   Standard_D4s_v3   westus2   2      73m
ok0620-rq5tl-worker-westus23-hzggb   Running   Standard_D4s_v3   westus2   3      73m

$ oc scale --replicas=2 machineset ok0620-rq5tl-worker-westus23 -n openshift-machine-api
machineset.machine.openshift.io/ok0620-rq5tl-worker-westus23 scaled

$ oc get machinesets -n openshift-machine-api
NAME                           DESIRED   CURRENT   READY   AVAILABLE   AGE
ok0620-rq5tl-worker-westus21   1         1         1       1           73m
ok0620-rq5tl-worker-westus22   1         1         1       1           73m
ok0620-rq5tl-worker-westus23   2         2         1       1           73m

$ oc get machine -n openshift-machine-api
NAME                                 PHASE         TYPE              REGION    ZONE   AGE
ok0620-rq5tl-master-0                Running       Standard_D8s_v3   westus2   1      74m
ok0620-rq5tl-master-1                Running       Standard_D8s_v3   westus2   2      74m
ok0620-rq5tl-master-2                Running       Standard_D8s_v3   westus2   3      74m
ok0620-rq5tl-worker-westus21-n6lcs   Running       Standard_D4s_v3   westus2   1      74m
ok0620-rq5tl-worker-westus22-ggcmv   Running       Standard_D4s_v3   westus2   2      74m
ok0620-rq5tl-worker-westus23-5fhm5   Provisioned   Standard_D4s_v3   westus2   3      54s
ok0620-rq5tl-worker-westus23-hzggb   Running       Standard_D4s_v3   westus2   3      74m

az aro show \
   --name <CLUSTER-NAME> \
   --resource-group <RESOURCEGROUP> \
   --query "consoleProfile.url" -o tsv

Cluster Autoscaling

apiVersion: "autoscaling.openshift.io/v1beta1"
kind: "MachineAutoscaler"
metadata:
  name: "ok0620-rq5tl-worker-westus21-autoscaler"
  namespace: "openshift-machine-api"
spec:
  minReplicas: 1
  maxReplicas: 7
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: ok0620-rq5tl-worker-westus21

$ oc create -f machine-autoscaler.yaml
machineautoscaler.autoscaling.openshift.io/ok0620-rq5tl-worker-westus21-mautoscaler created

$ oc get machineautoscaler -n openshift-machine-api
NAME                           REF KIND     REF NAME                      MIN   MAX   AGE
ok0620-rq5tl-worker-westus21   MachineSet   ok0620-rq5tl-worker-westus2   1     7     40s

$ oc create -f https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/cluster-autoscaler.yaml
clusterautoscaler.autoscaling.openshift.io/default created

$ oc get pods
NAME                     READY   STATUS              RESTARTS   AGE
work-queue-28n9m-29qgj   1/1     Running             0          53s
work-queue-28n9m-2c9rm   0/1     Pending             0          53s
work-queue-28n9m-57vnc   0/1     Pending             0          53s
work-queue-28n9m-5gz7t   0/1     Pending             0          53s
work-queue-28n9m-5h4jv   0/1     Pending             0          53s
work-queue-28n9m-6jz7v   0/1     Pending             0          53s
work-queue-28n9m-6ptgh   0/1     Pending             0          53s
work-queue-28n9m-78rr9   1/1     Running             0          53s
work-queue-28n9m-898wn   0/1     ContainerCreating   0          53s
work-queue-28n9m-8wpbt   0/1     Pending             0          53s
work-queue-28n9m-9nm78   1/1     Running             0          53s
work-queue-28n9m-9ntxc   1/1     Running             0          53s
[...]

$ oc get machinesets -n openshift-machine-api
NAME                           DESIRED   CURRENT   READY   AVAILABLE   AGE
ok0620-rq5tl-worker-westus21   5         5         1       1           7h17m
ok0620-rq5tl-worker-westus22   1         1         1       1           7h17m
ok0620-rq5tl-worker-westus23   1         1         1       1           7h17m

$ oc get machines -n openshift-machine-api
NAME                                 PHASE         TYPE              REGION    ZONE   AGE
ok0620-rq5tl-master-0                Running       Standard_D8s_v3   westus2   1      7h18m
ok0620-rq5tl-master-1                Running       Standard_D8s_v3   westus2   2      7h18m
ok0620-rq5tl-master-2                Running       Standard_D8s_v3   westus2   3      7h18m
ok0620-rq5tl-worker-westus21-7hqgz   Provisioned   Standard_D4s_v3   westus2   1      72s
ok0620-rq5tl-worker-westus21-7j22r   Provisioned   Standard_D4s_v3   westus2   1      73s
ok0620-rq5tl-worker-westus21-7n7nf   Provisioned   Standard_D4s_v3   westus2   1      72s
ok0620-rq5tl-worker-westus21-8m94b   Provisioned   Standard_D4s_v3   westus2   1      73s
ok0620-rq5tl-worker-westus21-qnlfl   Running       Standard_D4s_v3   westus2   1      13m
ok0620-rq5tl-worker-westus22-9dtk5   Running       Standard_D4s_v3   westus2   2      22m
ok0620-rq5tl-worker-westus23-hzggb   Running       Standard_D4s_v3   westus2   3      7h15m

$ oc get machinesets -n openshift-machine-api
NAME                           DESIRED   CURRENT   READY   AVAILABLE   AGE
ok0620-rq5tl-worker-westus21   5         5         5       5           7h23m
ok0620-rq5tl-worker-westus22   1         1         1       1           7h23m
ok0620-rq5tl-worker-westus23   1         1         1       1           7h23m

$ oc get machinesets -n openshift-machine-api
NAME                           DESIRED   CURRENT   READY   AVAILABLE   AGE
ok0620-rq5tl-worker-westus21   4         4         4       4           7h27m
ok0620-rq5tl-worker-westus22   1         1         1       1           7h27m
ok0620-rq5tl-worker-westus23   1         1         1       1           7h27m


$ oc get machines -n openshift-machine-api
NAME                                 PHASE      TYPE              REGION    ZONE   AGE
ok0620-rq5tl-master-0                Running    Standard_D8s_v3   westus2   1      7h28m
ok0620-rq5tl-master-1                Running    Standard_D8s_v3   westus2   2      7h28m
ok0620-rq5tl-master-2                Running    Standard_D8s_v3   westus2   3      7h28m
ok0620-rq5tl-worker-westus21-7hqgz   Running    Standard_D4s_v3   westus2   1      10m
ok0620-rq5tl-worker-westus21-7j22r   Running    Standard_D4s_v3   westus2   1      10m
ok0620-rq5tl-worker-westus21-8m94b   Deleting   Standard_D4s_v3   westus2   1      10m
ok0620-rq5tl-worker-westus21-qnlfl   Running    Standard_D4s_v3   westus2   1      22m
ok0620-rq5tl-worker-westus22-9dtk5   Running    Standard_D4s_v3   westus2   2      32m
ok0620-rq5tl-worker-westus23-hzggb   Running    Standard_D4s_v3   westus2   3      7h24m

Adding node labels

To add a node label it is recommended to set the label in the machine set. While you can directly add a label the node, this is not recommended since nodes could be overwritten and then the label would disappear. Once the machine set is modified to contain the desired label any new machines created from that set would have the newly added labels. This means that existing machines (nodes) will not get the label. Therefore, to make sure all nodes have the label, you should scale the machine set down to zero and then scale the machine set back up.

Using the web console

Select “MachineSets” from the left menu. You will see the list of machinesets.

We’ll select the first one “ok0620-rq5tl-worker-westus21”

Click on the second tab “YAML”

Click into the YAML and under spec.template.spec.metadata add “labels:” then under that add a key:value pair for the label you want. In our example we can add a label “tier: frontend”. Click Save.

The already existing machine won’t get this label but any new machines will. So to ensure that all machines get the label, we will scale down this machine set to zero, then once completed we will scale it back up as we did earlier.

Click on the node that was just created.

You can see that the label is now there.

Azure Service Operator - Blob Store

Integrating with Azure services

So far, our OSToy application has functioned independently without relying on any external services. While this may be nice for a workshop environment, it’s not exactly representative of real-world applications. Many applications require external services like databases, object stores, or messaging services.

In this section, we will learn how to integrate our OSToy application with other Azure services, specifically Azure Blob Storage and Key Vault. By the end of this section, our application will be able to securely create and read objects from Blob Storage.

To achieve this, we will use the Azure Service Operator (ASO) to create the necessary services for our application directly from Kubernetes. We will also utilize Key Vault to securely store the connection secret required for accessing the Blob Storage container. We will create a Kubernetes secret to retrieve this secret from Key Vault, enabling our application to access the Blob Storage container using the secret.

To demonstrate this integration, we will use OSToy to create a basic text file and save it in Blob Storage. Finally, we will confirm that the file was successfully added and can be read from Blob Storage.

Azure Service Operator (ASO)

The Azure Service Operator (ASO) allows you to create and use Azure services directly from Kubernetes. You can deploy your applications, including any required Azure services directly within the Kubernetes framework using a familiar structure to declaratively define and create Azure services like Storage Blob or CosmosDB databases.

Key Vault

Azure Key Vault is a cloud-based service provided by Microsoft Azure that allows you to securely store and manage cryptographic keys, secrets, and certificates used by your applications and services.

Why should you use Key Vault to store secrets?

Using a secret store like Azure Key Vault allows you to take advantage of a number of benefits.

1. Scalability - Using a secret store service is already designed to scale to handle a large number of secrets over placing them directly in the cluster.
2. Centralization - You are able to keep all your organizations secrets in one location.
3. Security - Features like access control, monitoring, encryption and audit are already baked in.
4. Rotation - Decoupling the secret from your cluster makes it much easier to rotate secrets since you only have to update it in Key Vault and the Kubernetes secret in the cluster will reference that external secret store. This also allows for separation of duties as someone else can manage these resources.

Section overview

To provide a clearer understanding of the process, the procedure we will be following consists of three primary parts.

1. Install the Azure Service Operator - This allows you to create/delete Azure services (in our case, Blob Storage) through the use of a Kubernetes Custom Resource. Install the controller which will also create the required namespace and the service account and then create the required resources.
2. Setup Key Vault - Perform required prerequisites (ex: install CSI drivers), create a Key Vault instance, add the connection string.
3. Application access - Configuring the application to access the stored connection string in Key Vault and thus enable the application to access the Blob Storage location.

Below is an updated application diagram of what this will look like after completing this section.

Access the cluster

1. Login to the cluster using the oc CLI if you are not already logged in.

Setup

Define helper variables

1. Set helper environment variables to facilitate execution of the commands in this section. Replace <REGION> with the Azure region you are deploying into (ex: eastus or westus2).

    export AZURE_SUBSCRIPTION_ID=$(az account show --query "id" --output tsv)
    export AZ_TENANT_ID=$(az account show -o tsv --query tenantId)
    export MY_UUID=$(uuidgen | cut -d - -f 2 | tr '[:upper:]' '[:lower:]')
    export PROJECT_NAME=ostoy-${MY_UUID}
    export KEYVAULT_NAME=secret-store-${MY_UUID}
    export REGION=<REGION>
   

Create a service principal

If you don’t already have a Service Principal to use then we need to create one. It is recommended to create one with Contributor level permissions for this workshop so that the Azure Service Operator can create resources and that access can be granted to Key Vault.

1. Create a service principal for use in the lab and store the client secret in an environment variable.

    export SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac -n aro-lab-sp-${MY_UUID} --role contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID --query 'password' -o tsv)"
   

2. Get the service principal Client Id.

    export SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name aro-lab-sp-${MY_UUID} --query '[0].appId' -o tsv)"
   

3. Install Helm if you don’t already have it. You can also check the Official Helm site for other install options.

    curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
   

Install the Azure Service Operator

1. Set up ASO

   1. We first need to install Cert Manager. Run the following.

       oc apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
      

   2. Confirm that the cert-manager pods have started successfully before continuing.

      oc get pods -n cert-manager
      

      You will see a response like:

      NAME                                       READY   STATUS    RESTARTS   AGE
      cert-manager-677874db78-t6wgn              1/1     Running   0          1m
      cert-manager-cainjector-6c5bf7b759-l722b   1/1     Running   0          1m
      cert-manager-webhook-5685fdbc4b-rlbhz      1/1     Running   0          1m
      

   3. We then need to add the latest Helm chart for the ASO.

       helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
      

   4. Update the Helm repository.

       helm repo update
      

   5. Install the ASO.

       helm upgrade --install --devel aso2 aso2/azure-service-operator \
       --create-namespace \
       --namespace=azureserviceoperator-system \
       --set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
       --set azureTenantID=$AZ_TENANT_ID \
       --set azureClientID=$SERVICE_PRINCIPAL_CLIENT_ID \
       --set azureClientSecret=$SERVICE_PRINCIPAL_CLIENT_SECRET
      

   6. Ensure that the pods are running successfully. This could take about 2 minutes.

      oc get pods -n azureserviceoperator-system
      

      You will see a response like:

      NAME                                                      READY   STATUS    RESTARTS   AGE
      azureserviceoperator-controller-manager-5b4bfc59df-lfpqf   2/2     Running   0          24s
      

Create Storage Accounts and containers using the ASO

Now we need to create a Storage Account for our Blob Storage, to use with OSToy. We could create this using the CLI or the Azure Portal, but wouldn’t it be nice if we could do so using standard Kubernetes objects? We could have defined the all these resources in once place (like in the deployment manifest), but for the purpose of gaining experience we will create each resource separately below.

1. Create a new OpenShift project for our OSToy app (even if you already have one from earlier).

   oc new-project $PROJECT_NAME
   

2. Create a resource group.

    cat << EOF | oc apply -f -
    apiVersion: resources.azure.com/v1api20200601
    kind: ResourceGroup
    metadata:
      name: ${PROJECT_NAME}-rg
      namespace: $PROJECT_NAME
    spec:
      location: $REGION
    EOF
   

3. Confirm that the Resource Group was actually created. You will see the name returned. It may take a minute or two to appear.

    az group list --query '[].name' --output tsv | grep ${MY_UUID}
   

4. Create a Storage Account.

    cat << EOF | oc apply -f -
    apiVersion: storage.azure.com/v1api20210401
    kind: StorageAccount
    metadata:
      name: ostoystorage${MY_UUID}
      namespace: $PROJECT_NAME
    spec:
      location: $REGION
      kind: BlobStorage
      sku:
        name: Standard_LRS
      owner:
        name: ${PROJECT_NAME}-rg
      accessTier: Hot
    EOF
   

5. Confirm that it was created. It may take a minute or two to appear.

    az storage account list --query '[].name' --output tsv | grep ${MY_UUID}
   

6. Create a Blob service.

    cat << EOF | oc apply -f -
    apiVersion: storage.azure.com/v1api20210401
    kind: StorageAccountsBlobService
    metadata:
      name: ostoystorage${MY_UUID}service
      namespace: $PROJECT_NAME
    spec:
      owner:
        name: ostoystorage${MY_UUID}
    EOF
   

7. Create a container.

    cat << EOF | oc apply -f -
    apiVersion: storage.azure.com/v1api20210401
    kind: StorageAccountsBlobServicesContainer
    metadata:
      name: ${PROJECT_NAME}-container
      namespace: $PROJECT_NAME
    spec:
      owner:
        name: ostoystorage${MY_UUID}service
    EOF
   

8. Confirm that the container was created. It make take a minute or two to appear.

    az storage container list --auth-mode login --account-name ostoystorage${MY_UUID} --query '[].name' -o tsv
   

9. Obtain the connection string of the Storage Account for use in the next section. The connection string contains all the information required to connect to the storage account. This should be guarded and securely stored. The --name parameter is the name of the Storage Account we created using the ASO.

    export CONNECTION_STRING=$(az storage account show-connection-string --name ostoystorage${MY_UUID} --resource-group ${PROJECT_NAME}-rg -o tsv)
   

The storage account is now set up for use with our application.

Install Kubernetes Secret Store CSI

In this part we will create a Key Vault location to store the connection string to our Storage account. Our application will use this to connect to the Blob Storage container we created, enabling it to display the contents, create new files, as well as display the contents of the files. We will mount this as a secret in a secure volume mount within our application. Our application will then read that to access the Blob storage.

1. To simplify the process for the workshop, a script is provided that will do the prerequisite work in order to use Key Vault stored secrets. If you are curious, please feel free to read the script, otherwise just run it. This should take about 1-2 minutes to complete.

    curl https://raw.githubusercontent.com/microsoft/aroworkshop/master/resources/setup-csi.sh | bash
   

   Or, if you’d rather not live on the edge, feel free to download it first.

   Note: Instead, you could connect your cluster to Azure ARC and use the KeyVault extension

2. Create an Azure Key Vault in the resource group we created using the ASO above.

    az keyvault create -n $KEYVAULT_NAME --resource-group ${PROJECT_NAME}-rg --location $REGION
   

3. Store the connection string as a secret in Key Vault.

    az keyvault secret set --vault-name $KEYVAULT_NAME --name connectionsecret --value $CONNECTION_STRING
   

4. Set an Access Policy for the Service Principal. This allows the Service Principal to get secrets from the Key Vault instance.

    az keyvault set-policy -n $KEYVAULT_NAME --secret-permissions get --spn $SERVICE_PRINCIPAL_CLIENT_ID
   

5. Create a secret for Kubernetes to use to access the Key Vault. When this command is executed, the Service Principal’s credentials are stored in the secrets-store-creds Secret object, where it can be used by the Secret Store CSI driver to authenticate with Azure Key Vault and retrieve secrets when needed.

    oc create secret generic secrets-store-creds \
    -n $PROJECT_NAME \
    --from-literal clientid=$SERVICE_PRINCIPAL_CLIENT_ID \
    --from-literal clientsecret=$SERVICE_PRINCIPAL_CLIENT_SECRET
   

6. Create a label for the secret. By default, the secret store provider has filtered watch enabled on secrets. You can allow it to find the secret in the default configuration by adding this label to the secret.

    oc -n $PROJECT_NAME label secret secrets-store-creds secrets-store.csi.k8s.io/used=true
   

7. Create the Secret Provider Class to give access to this secret. To learn more about the fields in this class see Secret Provider Class object.

    cat <<EOF | oc apply -f -
    apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: azure-kvname
      namespace: $PROJECT_NAME
    spec:
      provider: azure
      parameters:
        usePodIdentity: "false"
        useVMManagedIdentity: "false"
        userAssignedIdentityID: ""
        keyvaultName: "${KEYVAULT_NAME}"
        objects: |
          array:
            - |
              objectName: connectionsecret
              objectType: secret
              objectVersion: ""
        tenantId: "${AZ_TENANT_ID}"
    EOF
   

Create a custom Security Context Constraint (SCC)

SCCs are outside the scope of this workshop. Though, in short, OpenShift SCCs are a mechanism for controlling the actions and resources that a pod or container can access in an OpenShift cluster. SCCs can be used to enforce security policies at the pod or container level, which helps to improve the overall security of an OpenShift cluster. For more details please see Managing security context constraints.

1. Create a new SCC that will allow our OSToy app to use the Secrets Store Provider CSI driver. The SCC that is used by default, restricted, does not allow it. So in this custom SCC we are explicitly allowing access to CSI. If you are curious feel free to view the file first, the last line in specific.

    oc apply -f https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/ostoyscc.yaml
   

2. Create a Service Account for the application.

    oc create sa ostoy-sa -n $PROJECT_NAME
   

3. Grant permissions to the Service Account using the custom SCC we just created.

    oc adm policy add-scc-to-user ostoyscc system:serviceaccount:${PROJECT_NAME}:ostoy-sa
   

Deploy the OSToy application

1. Deploy the application. First deploy the microservice.

    oc apply -n $PROJECT_NAME -f https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/ostoy-microservice-deployment.yaml
   

2. Run the following to deploy the frontend. This will automatically remove the comment symbols for the new lines that we need in order to use the secret.

    curl https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/ostoy-frontend-deployment.yaml | sed 's/#//g' | oc apply -n $PROJECT_NAME -f -
   

See the storage contents through OSToy

After about a minute we can use our app to see the contents of our Blob storage container.

1. Get the route for the newly deployed application.

    oc get route ostoy-route -n $PROJECT_NAME -o jsonpath='{.spec.host}{"\n"}'
   

2. Open a new browser tab and enter the route from above. Ensure that it is using http:// and not https://.
3. A new menu item will appear. Click on “ASO - Blob Storage” in the left menu in OSToy.

4. You will see a page that lists the contents of the bucket, which at this point should be empty.

   

5. Move on to the next step to add some files.

Create files in your Azure Blob Storage Container

For this step we will use OStoy to create a file and upload it to the Blob Storage Container. While Blob Storage can accept any kind of file, for this workshop we’ll use text files so that the contents can easily be rendered in the browser.

1. Click on “ASO - Blob Storage” in the left menu in OSToy.
2. Scroll down to the section underneath the “Existing files” section, titled “Upload a text file to Blob Storage”.
3. Enter a file name for your file.
4. Enter some content for your file.

5. Click “Create file”.

   

6. Scroll up to the top section for existing files and you should see your file that you just created there.

7. Click on the file name to view the file.

   

8. Now to confirm that this is not just some smoke and mirrors, let’s confirm directly via the CLI. Run the following to list the contents of our bucket.

    az storage blob list --account-name ostoystorage${MY_UUID} --connection-string $CONNECTION_STRING -c ${PROJECT_NAME}-container --query "[].name" -o tsv
   

   We should see our file(s) returned.

Concepts

Contributors

The following people have contributed to this workshop, thanks!

@chzbrgr71

@haroldwongms

@jschluchter

@jamesread

@nichochen

@0kashi

@sabbour

@wgordon17

@lgmorand