Unit4, a large, independent software vendor, reached out to Microsoft DX Netherlands, wanting to accelerate their Azure cloud adoption and find support for their DevOps efforts. We teamed up and agreed to jointly hack Unit4’s Business World On! solution and its components.

This report describes how Unit4, Chef, and Microsoft collectively applied DevOps practices at an enterprise level and what lessons we learned in the process. It explains which DevOps practices we implemented in this particular hack, including:

  • Infrastructure as Code (IaC)
  • Continuous Integration (CI)
  • Continuous Deployment (CD)
  • Release Management (RM)
  • Configuration Management (CM)

Core team

The team for this engagement consisted of the following participants from Unit4, joined by IT professionals from Chef and Microsoft.

Unit4:

  • Øivind Rype – Software Architect
  • Arve Svendsen – Build Master
  • Bård Eik-Hvidsten – Development Manager
  • Arun Deep – System Architect
  • Einar Onsum – Delivery/Deployment Engineer
  • Magnus Tengmo – Operations Engineer
  • Conny Zell – Operations Engineer
  • Åsmund Riiser Kristiansen – Product Services Engineer
  • Mirosław Coma – Operations Engineer
  • Erik Marcussen – Technical Platform Lead (Dev)
  • Jon Braathen – Solution Architect
  • Mariusz Konarski – Operations Engineer
  • Ryan Gloeckler – Cloud Architect
  • Pasi Tuukkanen – Operations Lead
  • Svanhild Odden – Product Services Lead
  • Lars Fredrik Hatlehol – Product Development Director
  • Claus Jepsen – Chief Architect
  • Per Engh – Product Development Director

Chef:

  • Yvo van Doorn – EMEA Technical Services Manager
  • Joe Gardiner – Solutions Architect

Microsoft:

  • Astrid Hackenberg – Senior Software Developer
  • Greg Oliver – Senior Software Developer
  • Stuart Leeks – Senior Software Developer
  • Marco van der Kooij – Partner Development Evangelist
  • Rasmus Hald – Principal Technical Evangelist
  • Aleksandar Dordevic – Principal Technical Evangelist

Customer profile

Unit4 is a leading provider of enterprise applications used by thousands of organizations worldwide. Its solutions are in use among sectors providing professional services, education, public services, non-profit, real estate, wholesale, and financial services. It has approximately 4,000 employees.

The hackfest partnership began with a meeting of the Unit4 and Microsoft teams at Microsoft in The Netherlands. The Unit4 team was responsible for cloud adoption and transformation of their Dev and Ops teams, and innovating around creation and delivery of software to their clients. We discussed their current way of working and how to reinvent it toward more agile teams and ways of operating. During this meeting, we narrowed the options to executable/doable steps that would be inline and helpful to their DevOps journey.

We decided to deliver a value stream mapping (VSM) workshop followed by a DevOps hackfest. We would deliver this engagement in two steps: first, a one-day VSM workshop in mid-March 2016; second, a three-day hackfest in May 2016.

A few months after our initial meeting, the joint team convened at Unit4’s office in Oslo, Norway, for the VSM workshop and to capture how Unit4 conceives ideas, makes user stories, develops new features, tests new code, and ultimately moves deployables via different environments.

But before delivering the VSM workshop, we focused on the high-level architecture of the solution. We needed everyone to be in alignment on how the Business World On! solution works and what the major building blocks are.


Figure 1: High-level conceptual architecture

High-level architecture


Business World On! is an Enterprise Resource Planning (ERP) solution, one of many Unit4 has in its large portfolio of solutions. From a user perspective, Business World On! had spanned from a desktop-rich endpoint to mobile and browser usage of the application. Business logic is separated within a module and there is a central data store. These two components are the heart of the system, while other components are in service at the core of the system. Client access and reporting layers are separated in adequate individual components.

We simplified and deconstructed the app structure into these fundamental building blocks:

  • Endpoint access layer with pluggable modules
  • DB layer
  • Background worker roles / batch jobs

It is important to note that at the time of the hackfest, Unit4 was primarily leveraging its production solution via an internal private cloud based on virtual machines. Their company goal was to use the Microsoft cloud and optimize code and deployment to support it—they wanted to be fully “Microsoft cloud ready,” both from a development and operations perspective. Unit4 already had started this cloud journey and we were pleased to be able to see the current state, work on accelerating the transformation, and finally create tangible and measurable elements together with them.

Problem statement

To identify bottlenecks and areas in need of improvement, the team crafted a full value stream, from idea creation to delivery in production. This value stream map exercise lasted a full day, and it was worth the effort. For the first time, some members of the team saw the overall process of creating and delivering value with the Business World On! application.


Figure 2: Mapping the full value stream of the application

Full value stream


The team was able to visualize the value and the workflow, see how it intersects departments, and identify areas with wasteful activities. This led to interesting open discussions, exchanges of ideas, and various countermeasure proposals.

The following figure shows the results of our value stream mapping workshop.


Figure 3: Results of the value stream mapping workshop

VSM results


We used the most recent release to production of the Business World On! application for the VSM exercise.

Based on the VSM results, we identified the areas with the most opportunity within development and release. The types of problems were very common and frequently seen in large development teams: large batch size and relatively low levels of automation while releasing code to staging and production.

Their development process took place in a three-week sprint and, depending on the complexity of the feature, could take many sprints before it moved on to the next step of the value flow. An additional discovery was that multiple features were developed at the same time, creating a “siloed” mode of working for the teams. The team discussed how to tackle this challenge and what is affected by it. We quickly concluded that small batch is the key and that the pipeline needs to be adjusted to support it.

From a release standpoint, we saw that lead time of deployment to different environments sometimes took considerable time and processes were more manual than desired. The team proposed automated Infrastructure as Code (IaC) as one of the countermeasures. In addition, after more open and creative discussion, the team proposed implementing Release Management practices along with IaC in order to fully respect and comply with Unit4’s company policies. This was significant because it allows Unit4 to choose our partner Chef and its capabilities to deliver IaC.

Following is a summary of the identified countermeasures and recommendations agreed on by the team after the VSM workshop:

  • Implement IaC and Configuration Management
    • Automate deployment to different environments, both staging and production, and use IaC and Release Management practices (Team Foundation Server, Visual Studio Team Services, and/or other tools of preference).
  • Implement Release Management (RM)
    • Fully automate deployment and testing to the staging environment.
  • Fully implement Continuous Integration and Continuous Deployment (CI/CD)
  • Use Azure Platform as a Service (PaaS) capabilities in the app development/release
    • Web tier of the app solution (historically virtual machines) can and should move fully to Azure Web Apps or, in the future, to Azure Service Fabric (work in progress).
    • SQL tier of the app solution (historically virtual machines) can and should move fully to Azure SQL Database.
  • Innovate development process and people skill development
    • Leverage innovation and accelerate future feature development by planning and scheduling development/use of new skills/practices/tools as standard and planned work.
    • Create smaller deployment batches, continue to drive development toward continuous innovation.
    • Invest in efforts to lower technical debt.


Figure 4: Participants in the value stream map workshop

Participants


The VSM workshop was key to defining a focus for the upcoming DevOps hackfest, both its execution and solution aims. Unit4 decided that the most valuable achievement for them would be to prototype the velocity of pushing new features, test development for and with the Microsoft cloud, and test deployment to multiple cloud environments. In addition, since Unit4 offers the Business World On! solution within non-Azure environments as driven by compliance needs, it was crucial to cover Infrastructure as a Service (virtual machines) and PaaS delivery.

Everyone on the team accepted these tasks and began preparing for the hackfest that would occur one month later.

Solution, steps, and delivery

In the interest of Unit4’s cloud development goals and to test the identified countermeasures, the team decided to hack Identity Services, a production service component that provides identity-related services to various Unit4 solutions including Business World On!. (The entire Business World On! suite itself would be too broad to be hacked in three days.) Unit4 suggested using Identity Services because it is a component of many of their solutions, it is security-related, and it would be a proper proof of concept (PoC) for them. In addition, this was a great opportunity to experiment with small batch size and velocity of pushing changes to production environments.

The initial measurable starting points for the Identity Services app were the following high-water measures experienced in some parts of the solution:

  • Deployment lead time ~5 days (to provision and configure new environment)
  • Developing and delivering hotfix, total lead time ~3 weeks
  • A few scripts that needed to be run manually

We as a team were determined to dramatically improve these numbers.

In brief, the Identity Services component is a claim-based component that runs as a web service and relies on a database and digital certificate to deliver its function. The following image shows the high-level blocks of the app.


Figure 5: The high-level blocks of the application

High-level blocks


Before getting into technical achievements and nuggets, here is a look at the plan for how the overall process and architecture should look and what key milestones should be delivered as a result of the hack.


Figure 6: The solution envisioned by the team

The solution


We had a single code source and the output of that source needed to be deployed in two different environments—IaaS and PaaS. That led to the planning of the appropriate architecture that will provide the needed functionality, availability, and security that this type of solution requires. We brainstormed as a team and created an architecture that would enable us to start the hack. Here, side by side, is what IaaS versus PaaS architecture looks like:


Figure 7: IaaS architecture versus PaaS architecture

IaaS vs PaaS


Now let’s take a look at the most interesting technical achievements delivered at this DevOps hackfest that supported planned architecture and set goals.

It is difficult to decide which one or two nuggets are the best to share, but here are my choices. I’ll aggregate to cover the whole pipeline, in major blocks, and to show how some of it was done at PaaS versus IaaS to achieve the same functionality.

Infrastructure as Code

We all struggle with the correct way to handle and use secrets, certificates, and so forth. During the hackfest we discovered a way to push and hand over the needed certificate to Azure Web Apps in an automated and secure way. In addition, we enabled scenarios in which the deployment team just refers to the secrets, in the delivery phase, while the security team, for example, maintains and manages Azure Key Vault.

Assuming you have successfully put your secrets in Azure Key Vault, here is how to reference those secrets in the Azure Resource Manager (ARM) template. In our case, we wanted cert, with private and public key, to be automatically available to the environment during provisioning of infrastructure.

In the following code, you will see snippets of the ARM template and a corresponding parameter file that enables you to make a reference to the Azure Key Vault secrets.

Snippet code from the ARM template file:

  	{
      "apiVersion": "2015-08-01",
      "name": "[concat(parameters('webSiteName'), '-cert')]",
      "type": "Microsoft.Web/certificates",
      "tags": {
        "displayName": "cert"
      },
      "location": "[resourceGroup().location]",
      "properties": {
        "pfxBlob": "[parameters('cert')]",
        "password": "[parameters('certpassword')]"
      }
    },

    {
      "apiVersion": "2015-08-01",
      "name": "[parameters('webSiteName')]",
      "type": "Microsoft.Web/sites",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[concat('Microsoft.Web/serverFarms/', parameters('hostingPlanName'))]",
        "[concat('Microsoft.Storage/storageAccounts/', variables('logstorageName'))]"
      ],
      "tags": {
        "[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "empty",
        "displayName": "Website"
      },
      "properties": {
        "name": "[parameters('webSiteName')]",
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]"
      },
      "resources": [
        {
          "apiVersion": "2015-08-01",
          "type": "config",
          "name": "appsettings",
          "dependsOn": [
            "[concat('Microsoft.Web/Sites/', parameters('webSiteName'))]",
            "[concat('Microsoft.Cache/Redis/', parameters('RedisName'))]",
            "[concat('Microsoft.Web/certificates/', parameters('webSiteName'), '-cert')]"
          ],
          "properties": {
            "cache:type": "redis",
            "cache:default-duration": "60",
            "cache:redis-configuration": "[concat(reference(concat('Microsoft.Cache/Redis/', parameters('RedisName'))).hostName, ':6380,password=', listKeys(resourceId('Microsoft.Cache/Redis', parameters('RedisName')), '2015-08-01').primaryKey, ',ssl=True,abortConnect=False')]",
            "admin:api:authentication:basic:username": "[parameters('idserver-username')]",
            "admin:api:authentication:basic:password": "[parameters('idserver-password')]",
            "admin:api:authentication:type": "basic",
            "SigningCertificateThumbprint": "[reference(concat('Microsoft.Web/certificates/', parameters('webSiteName'), '-cert')).thumbprint]",
            "WEBSITE_LOAD_CERTIFICATES": "[reference(concat('Microsoft.Web/certificates/', parameters('webSiteName'), '-cert')).thumbprint]"
          }
        }
      ],

In the snippet code above, you can find the [concat(parameters('webSiteName'), '-cert')] resource that is “pooling” the certificate from Azure Key Vault. The Signing Certificate and WEBSITE_LOAD_CERTIFICATES configuration under the WebSite resource is configuring Azure Web Apps to make use of the supplied certificate. As you see in the code, we are referencing the pfxBlob and password properties of the resource cert to a parameter file. Here is a look at what is important to be set there.

The corresponding ARM parameters file:

    "cert": {
      "reference": {
        "keyVault": { "id": "/subscriptions/<ID>/resourceGroups/hacksecrets/providers/Microsoft.KeyVault/vaults/hackvault" },
        "secretName": "hackcert"
      }
    },
    "certpassword": {
      "reference": {
        "keyVault": { "id": "/subscriptions/<ID> /resourceGroups/hacksecrets/providers/Microsoft.KeyVault/vaults/hackvault" },
        "secretName": "hackcertpwd",
        "secretVersion": "zxcx5dabcb6d4e44a334649c3132vcxz"
      }
    }

In the parameter file, you are defining a “location” in Azure Key Vault from where, during the provisioning, ARM will pick up the needed certificate and password.

Here is the result on the Azure portal and how it looks when successfully deployed:


Figure 8: The result on the Azure portal and how it looks when successfully deployed

Result on portal


Configuring web server with Chef

Now that we have shown how to pass and configure an application securely with the needed certificate in a PaaS setup, let’s see how the team achieved it in an IaaS environment.

I’ll briefly explain how Chef did the overall configuration. We had all virtual machines automatically configured with a Chef extension—it was done with the appropriate ARM template during the initial phase of infrastructure provisioning. After that, all configuration management was done by Chef.

As noted earlier, let’s now see how in the IaaS world you can get the same scale of configuration with Chef, with regard to passing certificate and application configuration to the infrastructure environment.

Here is a recipe for installing a certificate on all front-end virtual machines—our PoC had two front-end nodes.


Figure 9: A recipe for installing a certificate on all front-end virtual machines

Recipe for installing certificate


Corresponding code from the ssl.rb file:

	# Cookbook Name:: IdentityServices
	# Recipe:: ssl
	#
	# Copyright (c) 2016 The Authors, All Rights Reserved.

	# Create directory for storing certificate
	directory 'C:\UNIT4\#Certificates' do
  	  recursive true
    	action :create
	end

	# Copy the certificate onto the node
	cookbook_file 'C:\UNIT4\#Certificates\cert.pfx' do
  	  source 'cert.pfx'
    	action :create
  	  not_if { ::File.exist?('C:\UNIT4\#Certificates\complete.txt') }
	end

	# Install certificate
	windows_certificate 'C:\UNIT4\#Certificates\cert.pfx' do
  	  pfx_password '###########'
  	    action :create
  	  not_if { ::File.exist?('C:\UNIT4\#Certificates\complete.txt') }
	end

	# create completion file
	file 'C:\UNIT4\#Certificates\complete.txt' do
    	action :create_if_missing
    	notifies :create, 'windows_certificate_binding[*.u4demo.com]', :immediately
	end

	# Bind the cert to 443 in personal store
	windows_certificate_binding '*.u4demo.com' do
    	name_kind :subject
  	  store_name 'MY'
  	  port 443
    	action :nothing
	end

	# Tidy up
	file 'C:\UNIT4\#Certificates\cert.pfx' do
    	action :delete
	end

For more information about SSL Chef recipes, see the Additional resources section.

As you can see in the previous screenshot/file, IaC is set up to check on whether a certificate exists, and if not, to force its installment, binding to the appropriate port, and finally doing a cleanup so we don’t leave a PFX file on the file system. After all nodes have the appropriate certificate, the next step is to configure the application, Identity Service, to use it. This is done by pushing to Internet Information Services (IIS) web.conf configuration to all virtual machines and front-end nodes.

The team created a template, webconfig.erb, and was referring to it in one of the cookbooks—take a look.


Figure 10: Cookbook refers to webconfig.erb

webconfig.erb


For links to more Chef documentation, see the Additional resources section.

Let’s zoom in to the specific place at webconfig.erb where the application is getting the needed setting.

Because we previously installed cert on virtual machines, it is now only a matter of “telling” the app to start to use it.


Figure 11: Closer look at webconfig.erb

Closer look


As the code below shows, we are referencing configuration of the app to look and use a previously imported certificate. This way we ensured that application, Identity Service, has all the prerequisites (DB connection string, signing certificate, and so forth) to run smoothly on “n” number of front-end virtual machines.

Here is a snippet of the code used to address signing certificate and connection strings inside the webconfig.erb file:

	<!--GENERAL SETUP-->
    	<add key="identityserver:providers:saml2:entity-id" value="https://<%= node['hostname'] %>.u4demo.com//IdentityService/identity"/>

    	<!--SECURE SETUP-->
    	<add key="security-policy:hsts:max-age" value="30"/>
    	<!--Signing certificates-->
    	<!-- Allowed values: Localmachine,CurrentUser-->
    	<add key="SigningCertificateStoreLocation" value="Localmachine"/>
    	<!-- Allowed values: Personal, TrustedRootCAs, OtherUsers,TrustedPeople    -->
    	<add key="SigningCertificateStoreName" value="PERSONAL"/>
    	<add key="SigningCertificateThumbprint" value="<%= node['identity']['certificate']['thumbprint'] %>"/>  

	<…other standard configuration data for ASP.NET Config…>

	<connectionStrings>
  	  <add name="U4IDSConfig" providerName="System.Data.SqlClient" connectionString="Server=<%= node['identity']['database']['hostname'] %>;Initial Catalog=<%= node['identity']['database']['dbname'] %>;User Id=<%= node['identity']['database']['user'] %>;Password=<%= node['identity']['database']['password'] %>"/>
	</connectionStrings>

Build definitions

Here is the unfiltered build definition, which shows how we progressed. For example, the first milestone was to have CI/CD without RM, so that is why some items are disabled. As we managed to get to the milestone with RM, we kept the build steps and disabled them so that we could share those learnings with others.

Here is how our PaaS Build definition looked.


Figure 12: How the PaaS Build definition looked

PaaS build definition


It is more or less a standard build for the web app. One interesting point at this stage is publishing “Infra_Resources.” This step takes IaC scripts and templates from source control and publishes them to the Release Management component of our overall pipeline.

Release definitions

We created two release/delivery pipelines to reflect both worlds in which the application needed to live (IaaS and PaaS): U4IDA Global Cloud and U4IDA Local Cloud. Unit4 Global Cloud = Azure; Local Cloud = non-Azure datacenter running Azure Stack or other private cloud stack.

Both release definitions on Visual Studio Team Services (VSTS):


Figure 13: Both release definitions on Visual Studio Team Services (VSTS)

Release definitions on VSTS


Now let’s explore Global Cloud and see how the team used technical capabilities in order to comply with Unit4 policies and delivery plan. Here are the details of the U4IDA Global Cloud Release Management definition, high level.


Figure 14: High-level details of the U4IDA Global Cloud Release Management definition

U4IDA


The Global Cloud definition has two stages (environments) of application lifecycle: staging and production. Each stage has its own steps/tasks that are executed so the application, its configuration and infrastructure are deployed correctly and in order. After a successful build, Release Management automatically picks up the build artifacts and starts to carry out four steps related to the staging environment. If we break down those steps, we are achieving the following:

  • Creating/updating of infrastructure
  • Pushing new code to the staging environment

When new code is successfully pushed to staging, that code must be approved before it can move to the production environment. According to Unit4 policies, the product owner and head of operations must provide approval to go live to production, so we included that in our Release Management criteria. Here is how we configured it.


Figure 15: Configuration of approval for release to production

Release configuration


After Release Management gets both approvals, it proceeds with carrying out the tasks that are configured. The successful result of Release Management looks like this:


Figure 16: A successful release to production

Successful release


Conclusion

It was wonderful to see and contribute to Unit4’s pursuit of more agile ways of developing and delivering software. In addition to the passion shown by the Unit4 technical team, it was clear that Unit4’s executive committee cared about the DevOps transformation and the business benefits that can be achieved. The high point of the engagement was the opportunity to demonstrate our achievements to Unit4’s product board, which consists mostly of Unit4 executive committee members.

These are some facts that we shared with the product board:

  • Lead time for provisioning a new environment (or customer) was reduced from ~5 working days to a maximum of 30 minutes (depending on whether it is IaaS or PaaS).
    • IaaS environment – initial provisioning (infra & app) up to 30 min; pushing new code via pipeline in ~15 min
    • PaaS environment – initial provisioning (infra & app) up to 16 min; pushing new code via pipeline in ~7 min
    • Since the solution is using deployment slots, end users experience no downtime during the actual deploy from staging to production.
  • Lead time for pushing a new feature and/or bug fix was reduced from ~2 weeks to ~15 minutes in an IaaS environment or ~7 minutes in a PaaS environment.

Other highlights:

  • Demonstrated full automated delivery pipeline with IaC versus mostly manual deployment.
  • Worked as one team—Dev, Ops, and Architects!
  • Made 72 changes during the three-day hackfest and successfully pushed them via the pipeline.


Figure 17: The team made 72 changes during the hackfest

Changes during hackfest


A key takeaway is this: Enterprises are successfully adopting and applying DevOps. Unit4 is doing it and will continue to do so.

This is how we ended our team presentation to the product board.


Figure 18: Final summary to product board

Final summary


All of this was possible because of teamwork and great people. We successfully delivered all must-have milestones, and most nice-to-have milestones. The credit goes to each involved individual who made this happen—thank you, team!

Final note: In the time needed to finalize this document and ready it for publishing, it is worth mentioning that the Unit4 People Platform team has now implemented full end-to-end DevOps processes based on the experiences from the hackfest. All People Platform source control is now in VSTS, and the team uses Build and Release Management for CI and CD for most of their services, including but not limited to Identity Services that was the topic of the hackfest.

Further, the platform team handles all secrets, certificates, and passwords using Key Vault and deployment mechanisms of ARM. No secrets are shared by means of email, stored in source code or any other unsecure mechanism.

Additional resources

For more information about Resource Manager and Key Vault:

A good starting point for more information on Chef, Cookbooks/Recipes, and Microsoft Azure: