Hark is an IoT startup that required an automated build-and-release pipeline to accelerate the delivery of its IoT monitoring solution. Microsoft joined Hark to design the required build-and-release pipelines and to implement the pipeline in Visual Studio Team Services.

DevOps practices implemented:

  • Infrastructure as code
  • Continuous integration
  • Continuous delivery
  • Automated testing
  • Application monitoring

The core hack team included:

Customer profile

Hark, a technology company based in the United Kingdom, is building an interconnected cloud-based sensor platform that allows users to monitor, store, and gain insight into their environmental data in real time. Alongside their own hardware, they allow any industry-standard sensor to connect to their platform with an almost plug and play nature for rapid deployments.

Some of the Hark team have a background in building enterprise software with Microsoft technologies. However, a great deal of the team’s experience is with technologies such as Go, MongoDB, Apache Cassandra, Apache Kafka, TypeScript, and ReactJS. To accelerate the development process and allow the team to focus on the product rather than maintaining server-based services, they made the decision to use Azure native services wherever possible.

Problem statement

Hark requires a robust continuous delivery pipeline that will reduce operational overhead, ensure minimal downtime, and reduce the time taken for code to move from development to production. Hark is aiming to have its MVP in production use in October and a robust delivery pipeline is key to achieving this.

Current situation

The product is currently in alpha stages of development with a working prototype that is demonstrated to customers. The delivery pipeline currently exists of developers committing to GitHub and then carrying out manual deployments to a test environment. Hark is a small team with limited time and is looking to automate as much of the software delivery process as possible.

Architecture overview

The current architecture consists of a number of Azure services. This includes the Web Apps feature of Azure App Service that hosts both customer-facing websites and APIs, with a data layer consisting of Azure Table storage and Azure SQL databases. The architecture was sketched out and then Visio was used for clarification:

architecture

Architecture components

  • An SPA accessed by users. This uses Node.js and is hosted within a virtual machine on DigitalOcean.
  • Two applications providing APIs are currently hosted within a single web app, although once the code base is separated, they will be hosted separately.
  • A console application running inside an Azure virtual machine that processes data.
  • Two event hubs and an Azure Stream Analytics job that perform data calculations.
  • A number of storage locations, including Azure Storage accounts and Azure SQL.

Designing build-and-release pipelines

The first step in such an engagement would usually be to carry out a value stream mapping exercise to identify areas for process improvement. Because Hark is an early stage startup not yet in production, the software build-and-release process was ad-hoc and not defined. It soon became evident that trying to map nonexisting processes was not going to be a productive use of time.

As an alternative, we began to discuss and map out build-and-release pipelines for each part of the application architecture.

Front end

SenseNet

Database

Monitoring

Outside of the build-and-release pipeline, it is important to consider monitoring. We spent a short time discussing how the application would be monitored once deployed.

At the end of the day we discussed what areas Hark felt would be most beneficial to be the focus of a hack.

We decided to arrange a hack to focus on the following:

  • Creating infrastructure using Azure Resource Manager templates.
  • Enabling automated testing of the APIs.
  • Creating a build-and-release pipeline that implements continuous integration and continuous delivery (CI/CD).
  • Integrating Application Insights alerts into Slack.

The hack

Hack group



Azure Resource Manager templates

Azure Resource Manager templates allow solution infrastructure to be defined as code. This ensures consistency each time the application is deployed.

Mapping Resource Manager template to application architecture

Since the hack, the application architecture had been modified and clarified. The Payload API and Website API have been split and the processes running as a console application will be migrated from a virtual machine to Web Jobs. Using a web job as opposed to a virtual machine will reduce administration overhead (because there is no server overhead) and facilitate autoscaling.

By reviewing the build-and-release pipelines created earlier and taking into consideration the architecture changes, the decision was made to build separate Resource Manager templates for the following aspects of infrastructure:

  • SenseNet
  • TopHat
  • Data Layer

Creating the Resource Manager Templates

Because Hark had already deployed the application using the Azure portal rather than creating the templates from scratch, they were exported using the “Automation script” feature of a Resource Group:

Automation script

Once the template was exported, a number of errors were presented:

Automation script error bar


Automation script errors


These errors are because a number of resources are currently not supported for export; these included:

  • Azure Stream Analytics
  • Web App config, such as application settings
  • Web Site Extensions

A number of other issues with the template would need to be resolved, including:

  • The application had been deployed into a single resource group. We required multiple resource groups.
  • The template had a large number of parameters that will never be used.
  • Selected resource properties that were exported show resource status/current configuration and cannot be set using a Resource Manager template.

Data processing template

This template contains:

  • Web app with two web jobs
  • Two event hubs
  • Azure Stream Analytics
  • Storage account used by the event processing web jobs

Azure Stream Analytics

Unfortunately, there is no JSON schema available as of yet that defines how to create a Stream Analytics resource using Azure Resource Manager. However, the REST API for Stream Analytics has been documented on MSDN: https://msdn.microsoft.com/en-us/library/azure/dn834994.aspx. Using the REST API, some examples found online as part of the IoT Suite (https://github.com/Azure/azure-iot-remote-monitoring/tree/master/Common/Deployment), and a fair amount of trial and error, the Stream Analytics resource was successfully created.

{
      "apiVersion": "2015-06-01",
      "type": "Microsoft.StreamAnalytics/streamingjobs",
      "name": "[variables('stream_analytics_job_name')]",
      "location": "[resourceGroup().location]",
      "dependsOn": [
       ],
      "properties": {
        "sku": {
          "name": "standard"
        },
        "outputStartMode": "JobStartTime",
        "outputStartTime": null,
        "EventsOutOfOrderMaxDelayInSeconds": 10,
        "EventsOutOfOrderPolicy": "adjust",
        "Inputs": [
          {
            "Name": "[variables('sa_input_name')]",
            "Properties": {
              "DataSource": {
                "Properties": {
                  "Name": "[variables('sa_input_name')]",
                  "EventHubName": "[variables('eventhubs_name1')]",
                  "consumerGroupName": "[variables('consumergroups_name1')]",
                  "ServiceBusNamespace": "[variables('namespace_name')]",
                  "SharedAccessPolicyKey": "[listkeys(variables('namespace_auth_rule_id'), variables('namespace_api_version')).primaryKey]",
                  "SharedAccessPolicyName": "[variables('auth_rule_name')]",
                  "PartitionKey": "PartitionId"
                },
                "Type": "Microsoft.ServiceBus/EventHub"
              },
              "Serialization": {
                "Properties": {
                  "Encoding": "UTF8",
                  "Format": "Array"
                },
                "Type": "Json"
              },
              "Type": "Stream"
            }
          }
        ],
        "Outputs": [
          {
            "Name": "ConvertedMetricAverage",
            "Properties": {
              "DataSource": {
                "Properties": {
                  "EventHubName": "[variables('eventhubs_name2')]",
                  "consumerGroupName": "[variables('consumergroups_name2')]",

                  "ServiceBusNamespace": "[variables('namespace_name')]",
                  "SharedAccessPolicyKey": "[listkeys(variables('namespace_auth_rule_id'), variables('namespace_api_version')).primaryKey]",
                  "SharedAccessPolicyName": "[variables('auth_rule_name')]",
                  "PartitionKey": "PartitionId"
                },
                "Type": "Microsoft.ServiceBus/EventHub"
              },
              "Serialization": {
                "Properties": {
                  "Encoding": "UTF8",
                  "Format": "Array"
                },
                "Type": "Json"
              }
            }
          }
        ],
        "Transformation": {
          "Name": "[variables('sa_transformation_name')]",
          "Properties": {
            "Query": "SELECT\r\n  System.Timestamp as TimeValue,\r\n  SensorId,\r\n  AVG(Value) as ValueAvg,\r\n  COUNT(*) as ValueAvgCount,\r\n  SUM(Value) as ValueAvgSum,\r\n  MIN(Value) as ValueMin,\r\n  MAX(Value)  ValueMax,\r\n  '5MINUTE' as 'AverageType'\r\nINTO\r\n    ConvertedMetricAverages\r\nFROM\r\n    ConvertedMetrics1\r\nTIMESTAMP BY CreatedUtc\r\nGROUP BY SensorId, TUMBLINGWINDOW(minute, 5)\r\n",
            "StreamingUnits": 1
          }
        }
      }
    }


Web app configuration

Configuration for both web apps needed to be provided within the Resource Manager template. Some configuration items will change in each environment, and others will remain consistent, one example being Azure Application Insights. There is an instance of Application Insights for each web app in each environment.

The following JSON extract shows how the APPINSIGHTS_INSTRUMENTATIONKEY application setting is set based on an APPINSIGHTS_INSTRUMENTATIONKEY parameter passed in when deployed:

 "parameters": {
      "APPINSIGHTS_INSTRUMENTATIONKEY": {
      "type": "string"
    },
	...
 }


Setting the value in the appsettings resource:

{
            "apiVersion": "2015-08-01",
            "name": "appsettings",
            "type": "config",
            "dependsOn": [
              "[resourceId('Microsoft.Web/Sites', variables('TopHatSiteName'))]"
            ],
            "properties": {
                "APPINSIGHTS_INSTRUMENTATIONKEY": "[parameters('APPINSIGHTS_INSTRUMENTATIONKEY')]",
         ...
             }
},


Regions and release environment

Some Azure resources, such as storage accounts, require a unique name across the entirety of Azure to ensure redundancy across the application and hence Resource Manager templates will be deployed in multiple regions. This means that each time the template is deployed, parameters must be included to ensure uniqueness. We decided to pass in a parameter called EnvironmentSuffix such as dev, test, production; and a region suffix that will be set according to the region being deployed into, such as ne for North Europe, we for West Europe, and so on. Unique variables could have been generated, but using a naming convention such as this ensures resources can be easily identified.

 "parameters": {

    "EnvironmentSuffix": {
      "type": "string"
    },

    "RegionSuffix": {
      "type": "string"
    }
    ...
 }


  "EventHubStorageAccountName": "[concat('myappeh',parameters('EnvironmentSuffix'), parameters('RegionSuffix'))]",


The compete Resource Manager template can be found here: https://github.com/marrobi/ARMTemplates/tree/master/HARK. Please note that this was created during a hack, has been sanitized of certain information to enable publication, and will need to be worked on further before reuse.

Data layer

The template for the data layer proved to be much more straightforward. Once the exported template was cleaned up, not a lot of work was required other than introducing the Environment and Region suffixes as above. We also ensured the required firewall rules were applied to the SQL server to allow remote management:

       {
          "type": "firewallrules",
          "kind": "v12.0",
          "name": "ClientIPAddress1",
          "apiVersion": "2014-04-01-preview",
          "location": "[resourceGroup().location]",
          "properties": {
            "startIpAddress": "123.123.123.123",
            "endIpAddress": "123.123.123.123"
          },
          "resources": [ ],
          "dependsOn": [
            "[resourceId('Microsoft.Sql/servers', variables('sqlserver_name'))]"
          ]
        },


The completed Resource Manager template can be found here: https://github.com/marrobi/ARMTemplates/tree/master/HARK. Again, please note that this was created during a hack, has been sanitized of certain information to enable publication, and will need to be worked on further before reuse.

Traffic Manager

Because the application will be deployed across regions, Azure Traffic Manager is required to distribute requests. This was created separately to the other Resource Manager templates because it is only required once per environment.


Pipeline design

Initial implementation of continuous integration was aimed at getting the SLN file to compile with a standard “Visual Studio Build” task. The solution was an ASP.Net Core WebAPI project that used .NET Framework 4.5.2 libraries.

.NET Core build issues


Needed dotnet restore

Lock file error


The build warning informed us that we should run dotnet restore, so we added a standard command task to call restore before the build task.

Restore not working

2016-09-08T08:07:42.5478829Z ##[section]Starting: Run dotnet
2016-09-08T08:07:42.6418840Z ##[command]dotnet restore
2016-09-08T08:07:44.2200223Z 
2016-09-08T08:07:44.2220204Z Welcome to .NET Core!
2016-09-08T08:07:44.2220204Z ---------------------
2016-09-08T08:07:44.2220204Z Learn more about .NET Core @ https://aka.ms/dotnet-docs. Use dotnet --help to see available commands or go to https://aka.ms/dotnet-cli-docs.
2016-09-08T08:07:44.2220204Z Telemetry
2016-09-08T08:07:44.2220204Z --------------
2016-09-08T08:07:44.2220204Z The .NET Core tools collect usage data in order to improve your experience. The data is anonymous and does not include commandline arguments. The data is collected by Microsoft and shared with the community.
2016-09-08T08:07:44.2220204Z You can opt out of telemetry by setting a DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1 using your favorite shell.
2016-09-08T08:07:44.2220204Z You can read more about .NET Core tools telemetry @ https://aka.ms/dotnet-cli-telemetry.
2016-09-08T08:07:44.2220204Z Configuring...
2016-09-08T08:07:44.2220204Z -------------------
2016-09-08T08:07:44.2220204Z A command is running to initially populate your local package cache, to improve restore speed and enable offline access. This command will take up to a minute to complete and will only happen once.
2016-09-08T08:07:49.0924123Z Decompressing 0%Decompressing 1%Decompressing 2%Decompressing 3%Decompressing 4%Decompressing 5%Decompressing 99%Expanding 100% 20920 ms
2016-09-08T08:08:33.2825951Z log  : Restoring packages for C:\a\1\s\src\SenseNet.Api\project.json...
2016-09-08T08:08:41.4638227Z error: Unable to resolve 'SenseNet.Azure' for '.NETFramework,Version=v4.5.2'.
2016-09-08T08:08:41.4648230Z error: Unable to resolve 'SenseNet.Common' for '.NETFramework,Version=v4.5.2'.
2016-09-08T08:08:41.4648230Z error: Unable to resolve 'SenseNet.Messaging.Azure' for '.NETFramework,Version=v4.5.2'.
2016-09-08T08:08:41.4648230Z error: Unable to resolve 'SenseNet.Services' for '.NETFramework,Version=v4.5.2'.
2016-09-08T08:08:41.4648230Z error: Unable to resolve 'SenseNet.Storage' for '.NETFramework,Version=v4.5.2'.
2016-09-08T08:08:41.4648230Z error: Unable to resolve 'SenseNet.Storage.Azure' for '.NETFramework,Version=v4.5.2'.
2016-09-08T08:08:41.4648230Z error: Unable to resolve 'SenseNet.Storage.SqlServer' for '.NETFramework,Version=v4.5.2'.


As you can see, there is a problem that dotnet restore does not work with mixed core/framework projects. See https://github.com/dotnet/cli/issues/3199 and https://github.com/aspnet/Tooling/issues/741. We sought assistance, and Jelle Druyts—a Microsoft Premier Field Engineer—pointed out that the dotnet executable is actually a wrapper around NuGet. He suggested we try the latest version of NuGet directly from the NuGet website instead of dotnet restore. We could have downloaded it and checked in the EXE, but because it’s a temporary fix, we downloaded it on the fly—in this case, the RC1 version (https://dist.nuget.org/win-x86-commandline/v3.5.0-rc1/NuGet.exe).

Then you can use the standard NuGet restore task but pointed explicitly at the NuGet.exe file that was just downloaded. To do this, set the “Advanced” “Path to NuGet.exe variable” to: $(build.stagingdirectory)\NuGet.exe

After we had our solution building correctly, we needed to make sure we copied all of the required outputs of the various projects to the $(Build.ArtifactsStagingDirectory) folder so we could publish them easily. Unfortunately, because this was .NET Core, it was not straightforward. Here are the steps that we put in place to enable this:

  1. We had to tell dotnet that we are publishing to IIS:

    Publish API

  2. We had to create a fake parameters.xml file and copy it to our ArtifactStagingDirectory so that we can use the ‘Azure RM Web App Deployment’ Task that requires it. It doesn’t matter what is in there, but in our case, we had:

     <parameters> <parameter name='IIS Web Application Name' defaultValue='MyApp' tags='IisApp' /> </parameters>
    
  3. We had to zip our .NET Core application because it does not create a web deploy package automatically. A key thing to note is to ensure that the ‘Prefix root folder name to archive paths’ is unchecked because we do not want to preserve the file paths in the zip:


The build definition with all the required steps was as follows:

Build definition

Release

We had further discussions about the method for deployment of resource groups at each stage of the pipeline.

Hack release


Because the application is still in the early stages of deployment, it was decided to just have two environments: Development and Production. Hark intend to add a separate Testing environment at a later point in time. The following image shows the release pipeline for the deployment of the ARM Templates and associated WebApps.

Testing

The platform includes two APIs, one that receives data from the IoT gateway—the Payload API—and a second that is accessed by the web front end. A solution for testing these APIs was required that met the following criteria:

  • Allowed calls to be made to the payload API and the result to be verified by making a call to the web API.
  • Could be used to run tests within a release pipeline in Visual Studio Team Services; that is, supports command-line execution of tests.

After identifying a number of tools that could potentially help, Postman (https://www.getpostman.com) was selected because it is actively maintained and test collections can be executed at the command line using an OSS command-line test runner for Postman called Newman (https://github.com/postmanlabs/newman).



The test took the form of calls to the API and then verification that the correct responses were returned. These tests were saved as a collection and could be run within the UI and also exported as a JSON file.



The tool also allows environmental variables to be pulled out into a postman_environment.json file so that we can use the same tests with different URIs, which is perfect for our release definition.



Once we were happy with our Postman Collection, we checked in the JSON files and ensured that they were published as an artifact of the build so that we could use them in the release pipeline.

To use the Postman tests in the release pipeline, we used Newman. We added an npm install newman task to the pipeline and from a command-line task called Newman to execute the tests. We also redirected the output to be a JUnit test result:

This enabled us to use the ‘Publish test results’ task to get the results displayed as part of the release.

Application Insights

When it came to integrating Application Insights alerts into Slack, we soon discovered that Hark had carried this out themselves prior to the hack. For reference, they had used the Azure quickstart template from https://github.com/Azure/azure-quickstart-templates/tree/master/201-alert-to-slack-with-logic-app.

Further work will need to be done to integrate this into the new instances of Application Insights in the release pipeline.

Learnings

Other than the many technical learnings documented here, we came away with a couple more general insights:

  • There are often multiple ways to reach a desired goal, especially when designing release pipelines.
  • Value stream mapping is an extremely powerful way of identifying and removing areas of waste in the software development process.

However, because Hark is a startup with few defined processes, it became apparent that it was not appropriate to carry out a VSM. This is something we should remember when engaging with organizations with products at a similar stage of development. It would be interesting to carry out a VSM with Hark in 6 to 12 months once processes have matured.

Conclusion

Despite needing to overcome a number of technical hurdles, we made great progress during the hackfest. It has also been great to learn that since the hackfest, Hark has continued to add to the work carried out to create a comprehensive build-and-release pipeline. The following quotation illustrates what was achieved:

“During the hack, we deployed and tested a robust CI pipeline with specific Azure native services. This included services that we were unsure about how to continuously deploy, such as Stream Analytics and Event Hubs. Moving all our systems into VSTS (including our source control away from GitHub) was one of our main achievements along with deployment of geo-redundant deployment of application and infrastructure.

“The hack greatly accelerated the deployment of these pipelines and we learned a vast amount about the Microsoft tooling surrounding VSTS and Azure. The work will enable us to be in a production several times faster than our predicted roadmap and enable us to bring our first customers online in the coming weeks.”

— Jordan Appleson, CEO, Hark

Resources