Automating Arena's CI/CD pipelines with Azure App Service and GitHub
In this joint development effort, Microsoft teamed up with Arena’s CTO, Rodrigo Reis, to streamline Arena’s deployment process to staging and ultimately production by using Microsoft Azure App Service.
In our solution, we used Azure App Service, web deployment slots, and continuous integration and deployment with GitHub to remove manual steps requiring developer intervention and to automate and accelerate deployment to staging and production.
Key technologies used
- Rodrigo Reis – Co-founder and CTO, Arena
- Martin Schray – Senior Technical Evangelist, Microsoft
- Ryan Joy – Senior Technical Evangelist, Microsoft
Arena is a digital media and entertainment startup based in Los Angeles. The Arena platform allows fans to interact with one another during live events. Working with sports like UFC (Ultimate Fighting Championship), Arena gives fans a place to connect that’s not as cluttered as Twitter or Facebook, and that focuses just on the live event.
Arena boasts some impressive audience numbers, saying they have more than 700,000 monthly active users and had over 100,000 uniques for a single UFC event. The startup is building widgets to allow websites and blogs to host Arena chats directly on their sites. In addition, Arena hopes to be at 10 million monthly active users in the next 12 months.
Arena’s CEO Paulo Martins and CTO Rodrigo Reis hanging out in Chicago’s premier startup space 1871
Arena wanted to move from their current AWS Elastic Beanstalk environment to an approach that incorporates automated deployment using continuous integration and continuous deployment (CI/CD). Arena’s initial build process required building their application locally, zipping the build, transferring the build files using FTP to their cloud provider, unzipping the application, and running scripts to start the application.
This meant that the Arena developers were spending their valuable development cycles doing low-value, repetitive, manual tasks to prepare their application for deployment and deploying it. This also meant developers were spending time waiting for files to be zipped, file transfers to complete, and files to be extracted from .zip files.
During our conversations with the Arena team, we explored their current challenges and opportunities in moving code from development to staging and finally, production. To explore the current process and what the ideal state could be, we built a value stream map (detailed in the conclusion section of this report) of the current process of deploying (e.g., moving) code from development to staging to production. The value stream map showed time-consuming manual processes that could be optimized and automated.
As a team, we agreed that automating and optimizing Arena’s deployment process from development to staging to production was the area that would provide the most immediate benefit to Arena by freeing developers from those repetitive, manual tasks. This automation and optimization would benefit Arena developers by saving around 10 minutes for each deployment from development to staging, which happens several times a day.
Solution, steps, and delivery
Arena did not want to manage server infrastructure so they chose to use Azure App Service, which is Azure’s platform as a service (PaaS) offering. Through the Azure portal we created Arena’s Azure App Service and configured options for auto-scale, always on, logging and environment variables for the app. Auto-scale was important to Arena because their site faces widely varying levels of demand with large peaks during popular sports events. Arena chose to scale on CPU% utilization. When the CPU utilization hits its threshold, Azure App Service automatically adds additional virtual machines and configures the load balancer to leverage these new VMs. After a configurable cool-down period when CPU% utilization drops below configured levels, these new virtual machines are taken offline after removing them from the load balancer.
Arena wanted their environment to allow them to deploy directly from GitHub to their staging environment for more complete testing before making the decision to deploy to production. Using the Azure portal, we used the Azure App Service we had created for Arena in the previous step and added a web deployment slot for staging. The same virtual machines managed by Azure host both the production and any additional staging slots. After we had created the staging web deployment slot, we used the Azure portal for a staging slot to set up Azure continuous deployment (CD). We selected GitHub as the deployment source, and the Arena team provided their GitHub credentials and selected the repo to deploy from.
Using the Service Control Manager to generate customer deployment scripts
If, for example, your website was http://arena.azurewebsite.net, you could reach the Service Control Manager by navigating to http://arena.scm.azurewebsite.net. The SCM site generates the template files and zips them and allows you to download them. The zipped file includes .deploy (that tells Kudu the name of the custom deploy file) and deploy.cmd, which is the script for Kudu’s deployment. We made two modifications to deploy.cmd:
- Having it check for the existence of the site\deployments\tools\PostDeploymentActions directory and creating one if it did not exist, and copying the newly created postdeploy.cmd file into that directory.
- By default, Kudu’s deployment script only deploys production node packages (due to -production argument to npm). However, because Arena wanted to deploy production and dev packages via npm to enable their post-build processes (such as minification, hashmarking, and so on), we removed the -production setting.
Arena’s post-build activities such as minification and hashmarking are invoked from postdeploy.cmd via npm, so postdeploy.cmd simply invokes an npm command. All of these post-build activities are based on Kudu’s post-deployment support. Kudu’s post-deployment actions will automatically execute any scripts (or executable) in the site\deployments\tools\PostDeploymentActions folder.
Arena used the Azure portal to transfer their custom domains to Azure so their customers would see the Arena URL, not the azurewebsites.net URL.
Rodrigo reviewing the Azure Portal for the Arena Azure App Service
Using Azure App Service and its continuous deployment capabilities allowed Arena to be more adaptable and flexible and to accelerate deployment from development to staging and ultimately to production. Azure App Service allowed Arena to speed their development and more quickly add innovative new features and capabilities to their products.
The centerpiece of Arena’s use of Azure is Azure’s platform as a service (PaaS) offering. While there are several components to Azure’s PaaS offering, which is called Azure App Service (Logic Apps, API Apps, Web Apps, Mobile Apps), our work was focused on Web Apps.
Arena deployment architecture
After creating Arena’s Web Apps service, we set up auto-scale, configured logging, set up the staging web deployment slot, configured continuous deployment from GitHub, set up post-installation build action, and transferred Arena’s custom domain. Each of these items is explained in greater detail in the Steps section.
Arena demo video
We leveraged Azure PaaS architecture for this effort. Specifically, we used the Web Apps feature of Azure App Service, which supports Node.js application deployment. We further leveraged Kudu deployment to allow post-deployment build steps such as minificiation, webpack, and so on. Arena’s application architecture is a Single Page Application (SPA) built using Node.js and React.
- The Arena team wanted to have some post-deployment activities, such as minification and hashmarking, occur after deployment had completed. As we began exploring how to build custom approaches for post-deployment actions, we found that Web Apps has built-in support for post-deployment actions.
- Web Apps has the ability to run custom post-deployment scripts after deployment completes. Any script or executable in the site\deployments\tools\PostDeploymentActions directory will be executed after deployment completes. Because Arena wanted to execute post-deployment actions such as minification, it was very easy to set them up using this approach.
- The Service Control Manager web application, which is part of Kudu, provides a menu selection that automatically generates the two files (deploy.cmd and .deploy) needed for custom deployments and downloads them in a .zip file.
The template file, deploy.cmd, generated from the Service Control Manager, can be intimidating until you realize you can probably just have it add a few instructions to copy your scripts and executables into site\deployments\tools\PostDeploymentAction (creating it if it doesn’t exist). The scripts or executables in the site\deployments\tools\PostDeploymentAction directory will automatically be executed when the deployment is done. The following snippet of code checks for the existence of the PostDeploymentActions and creates it if it does not exist, and then copies the PostDeploy.cmd file into the PostDeploymentActions directory.
<pre><code> :: 4. Post deployment actions :: copy the post deployment script into place so its executed automatically by Kudu echo "copy the post deployment script into place" :: set the directory to where the site has been deployed pushd "%DEPLOYMENT_TARGET%" :: Test for the existence of the PostDeploymentActions directory (create if it doesn't exist) IF NOT EXIST %DEPLOYMENT_TARGET%\..\deployments\tools\PostDeploymentActions ( echo "create PostDeploymentActions" call :ExecuteCmd mkdir %DEPLOYMENT_TARGET%\..\deployments\tools\PostDeploymentActions :: If there is an error goto the error handler IF !ERRORLEVEL! NEQ 0 goto error ) ELSE ( echo "PostDeploymentActions already exists" ) :: Copy the PostDeploy.cmd script into the PostDeployment call :ExecuteCmd copy %DEPLOYMENT_SOURCE%\PostDeploy.cmd %DEPLOYMENT_TARGET%\..\deployments\tools\PostDeploymentActions\ :: If there is an error goto the error handler IF !ERRORLEVEL! NEQ 0 goto error :: Restore the directory popd </pre></code>
- By default, the deployment for Node.js as part of Azure continuous deployment only installs the production npm packages due to the use of the -production switch passed to npm. Because Arena wanted to execute post-deployment steps such as minification, we also needed the -dev npm packages installed. We modified deploy.cmd to remove the -production switch when initiating npm on package.json.
- It only took adding three files to the Arena source tree to customize the deployment (deploy.cmd and .deploy) and set up post-deployment actions. After you get the idea that any script or executable in site\deployments\tools\PostDeploymentActions is automatically executed, it is really easy to set up post-deployment actions.
You could simply use npm from your post-deployment script to execute post-deployment actions. The following example changes the directory to where the site was deployed and then executes npm commands to perform post-deployment actions.
<pre><code> :: 1. SET :: ---------------------- :: Post Deploy Script :: Version: 1.0.0 :: Execute the post deployment steps :: ---------------------- echo "**********************" echo "Post Deployment Script" echo "**********************" :: Change to the directory where the site was deployed pushd "%DEPLOYMENT_TARGET%" :: Execute post deploy actions via NPM leveraging package.json npm run build:js && npm run build:embed && npm run build:vendors && npm run build:icons :: If there is an error goto the error handler IF !ERRORLEVEL! NEQ 0 goto error :: Restore the directory popd </pre></code>
- It is a best practice to deploy to staging and, after appropriate testing, use virtual IP (VIP) swap on web deployment slots to “promote” staging to production. This makes moving from staging to production very easy.
- Kudu has variables that hold important information such as source and target directories.
- Kudu can easily fire webhooks when a build is complete.
- Azure App Service supports Node.js as a first-class citizen supporting npm and package.json.
- Azure App Service allows easy autoscaling of apps. In Web Apps, auto-scale automatically will add new instances and remove instances based on how you configured it. This is a great capability as it takes care of adding and removing this instance from the load balancer.
- Azure App Service will run custom post-deployment scripts after deployment completes, allowing you to easily automate post-deployment actions.
- Developers often want post-deployment activities such as minification to occur once the deployment is complete, and Azure App Service supports post-deployment activities.
- Using Azure deployment slots, you can deploy to a non-production site for testing and quickly move to production using VIP swap.
Looking at the value stream map, we removed a number of manual steps from the current state process.
Arena current state diagram
By implementing the future state model using the Azure App Service, the Arena team was able to deploy in a much more repeatable, expedient, and automated fashion. The continuous integration to an Azure deployment slot for staging removed many manual steps for the Arena developers and shaved several minutes off of each and every deployment.
Arena future state diagram
The Arena team is an incredibly agile startup and they are quite happy that their development and deployment process now mirrors their agility. Using Azure App Service allowed Arena to focus on their application, not their infrastructure. Azure App Service provided capabilities such as continuous deployment that helped save developer time, attention, and energy and allowed for automated building and deployment for Arena’s site.
We found that changes that removed friction for developers by automating time-consuming manual deployment steps increased developer productivity and morale. Developers were able to accelerate their ability to build, test, and deploy new features. Time saved by automating deployment freed more time for developers to develop, accelerating Arena’s ability to bring innovations to their product.
If you consider DevOps to be an evolving practice and do not insist on perfection at the start, you can quickly begin to add DevOps practices into your environment (and likely gain valuable feedback and insights as part of the implementation process). DevOps involves people, process, and technology and should be an ever-evolving process. The natural evolution of your product or service and team will likely necessitate changes to your DevOps practices. The Arena team decided to start down on the path on the DevOps process by focusing on continuous integration and continuous deployment. While there were many potential places to start, the Arena team realized from the value stream mapping process that CI/CD were the biggest opportunity areas for their dev team.
Creating a value stream map of the current state and future (that is, the desired state) provides a powerful approach and vehicle for capturing the ways things are now and the end state you are driving toward. Done correctly, the value stream map can provide measures for success as well.
Developers often want post-deployment steps such as minification and hashmarking performed after their site is deployed from version control rather than performing these steps and checking the resulting files into version control. Web Apps support for post-deployment actions, which is detailed in the “Architecture,” “Key learnings,” and “Resources” sections, provides an easy and built-in way to accomplish post-deployment actions.
How the learnings and insights can be applied elsewhere
There are often opportunities to further optimize and automate processes around build, development, and deployment. Pursuing opportunities that remove friction in build, development, and deployment by automating time-consuming manual steps increases developer productivity and, potentially, morale. Although it does not seem that pursuing changes in something such as deployment can accelerate feature development, anything that frees developer time and increases productivity can accelerate feature development. Investing time to review DevOps practices such as paying down “technical debt” can pay substantial rewards through increased efficiency.
Value stream mapping can be a powerful mechanism for exploring opportunities to improve current processes. Why? By creating a value stream map of the current state and future (desired state), you have a chance to start a conversation and create measures for success. In your value stream map for the current state, make sure to get information on how long the process steps take so you can measure improvements.
- Channel 9 DevOps Fundamentals Video Series - DevOps Fundamentals
- Channel 9 DevOps Fundamentals Video Series - Continuous Integration
- Channel 9 DevOps Fundamentals Video Series - Continuous Deployment
- Phoenix Project DevOps Book
- Phoenix Project DevOps Blog - The Three Ways: The Principles Underpinning DevOps
- What is Kudu?
- Publishing a web site from source control
- Kudu Post Deployment Actions
- Customizing deployments with Kudu
- The Azure Runtime Environment
- Setting Continuous Deployment
- Map an existing custom DNS name to Azure Web Apps
- Scale up an app in Azure
- Set up staging environments for web apps in Azure App Service