DevOps practices help Foize manage the success of its 24Coms solution
In this DevOps hackfest, Microsoft teamed up with Foize to hack a component of a popular Foize solution, 24Coms. This article describes the process and results, and includes the following DevOps elements:
- Infrastructure as code
- Continuous integration/continuous delivery
- Staging and production environments
Foize is an emerging startup based in the Netherlands and operating globally. It creates software solutions for consumer and enterprise customers, with a focus on digital social communication, geofencing, phone as a sensor, and location-based innovations.
A Foize solution—24Coms—provides new possibilities for modern communications by adding a geotracking dimension to conventional forms of information exchange and alerting. Foize has seen growing numbers of customers for this solution.
The success of 24Coms, however, soon became a challenge. The application was growing so rapidly that the current infrastructure was unable to meet demands. With this in mind, Foize partnered with our Microsoft team for a DevOps hackfest. The purpose: to hack the 24Coms solution and apply DevOps practices to automate and migrate fully to the cloud.
The hack team:
- Leo Winder – Owner/Ops, Foize
- Michael Vlaar – C# and Ops Developer, Foize
- Leon Keijzer – iOS Developer, Foize
- Ronald Koster – Android and DBA Developer, Foize
- Valery Jacobs – Technical Evangelist, Azure, Dev, Microsoft
- Srdjan Bozovic – Technical Evangelist, Mobile/Azure, Dev, Microsoft
- Aleksandar Dordevic – Technical Evangelist, DevOps, Microsoft
The preparation for the DevOps hackfest involved assessing the current state of their software development lifecycle, looking for areas of improvement, and focusing on the actual hack. This consisted of three phases: forming, storming, and performing. During these phases, we narrowed down which areas to improve and what to deliver, and then proceeded to go and DO it!
The challenges (or areas of improvement) the Foize team has been facing can be summed up as follows:
- Scaling to support rapid growth of net new customers.
- Automating operations, development, and recovery.
- Crafting/accepting DevOps skills needed to provide accelerated migration to a full cloud solution.
In our initial “forming” session, the hackfest team discussed the overall application architecture, technologies used, current production, operations, and business cases that 24Coms addresses. We captured the initial improvement areas, such as level of automation, deployment techniques, and vNext for the solution, taking into consideration fast improvements and capabilities that the modern cloud offers.
This meeting set the focus on the GPS component of the solution, which was running on Azure because few worker roles have been developed with C#. It also was using Azure Service Bus as a main building block of this service.
Our first drill—or “dive-in”—was to perform value stream mapping (VSM), in which we captured the overall process, from development, to staging and testing, all the way to production usage. By using VSM, we identified current development and release processes in use by Foize and areas in need of improvement; for example, we found some manual deployment processes and more optimal uses of build processes and delivery to production/staging/development environments.
Take a look at what we captured and created during the day-long VSM session (Figure 1).
Figure 1. Value stream map of 24Coms solution
Foize has a hybrid solution based on hosted and Azure services and combines these worlds (hosted and cloud) when providing service to customers. The major building block of their solution is clearly visible in the value stream map:
- CI/CD/CM: Atlassian Bamboo (build servers at AWS)
- Hosted environment at local ISP (production and staging environments)
- Microsoft Azure and Visual Studio Team Services. (Their choice of vNext and interim mode of the 24Coms solution)
- Users and devices of the solution
As expected, the solution has its good, bad, and ugly sides (as does any solution). Because of the growth in customers, Foize had made a strategic decision to move from this current, working model toward a full cloud solution. They started this journey on their own, did an awesome job, and then partnered with Microsoft to improve the overall model.
The VSM provided us with focus areas and a challenge. We had to:
- Improve current DevOps processes.
- Create a “blueprint” or template for future services migration toward Azure.
- Keep the production running while we experimented.
In DevOps terminology, we decided to focus on and hack the following:
- Infrastructure as code (IaC)
- Continuous integration/continuous delivery (CI/CD)
- Staging and production environments
The team was excited to begin the next phase (Figure 2).
Figure 2. Ronald Koster (left), Leo Winder (center), and Michael Vlaar (right) of the Foize team, eager to start the DevOps hack
Solution, steps, and delivery
In the value stream mapping process, we captured the focus and hacking areas. At our next meeting, we crafted a plan and listed the steps toward a solution.
Figure 3. Meeting to finalize a plan
We focused on addressing manual deployment to production GPS service, manual recovery procedures, and improvements in continuous integration and delivery. We agreed first to do IaC, prepare the Visual Studio Team Services environment, focus on CI/CD, and lastly deal with staging and production environments.
Our “master” plan, shown in Figure 4, is simple but effective.
Figure 4. Overall architecture and action items for the hack
Drilling down on the action items, here is what we aimed to achieve:
- Use Visual Studio Team Services as a source repository and CI/CD/RM component.
- Create IaC that can re-create and “enforce” identical infrastructure environment setup and configuration.
The first part, using Visual Studio Team Services, was relatively easy because it involved the whole team and our collective experience with the platform. As a result, making changes to source code and enablement of CI was smooth.
But in parallel, we had some challenges with using IaC to create Service Bus, Queue storage, and Topic. With Azure, you can use an old management module, Service Manager, or the new Azure Resource Manager. Azure is currently in interim mode, and in this case the new model didn’t support resources we were trying to create. So the first tasks for the IaC team were to make it run from their machine and then make it work with Team Services. We had problems with this, but I’ll share how we solved them.
First, take a look at a failure (Figure 5).
Figure 5. A look at a failed effort
After a few failed attempts, we learned that in order to create Service Bus and other related components, we needed to use PowerShell and reference code to hook to Microsoft.ServiceBus.dll. It wasn’t pretty, but it did the job. It is easy to do this when you manually run script from your laptop, but if you want to automate and use agents, it is a challenge.
Our challenge was how to point the Team Services build agent to find Microsoft.ServiceBus.dll and then use it while running task/PS script in Team Services.
Here is how we solved it:
Add Microsoft.ServiceBus.dll to source repo.
Update the code, especially variables that will point out to the location of DLL ($srcDir).
Run the build again at Visual Studio Team Services and finally move forward.
Once we solved IaC, we accelerated rapidly on our hack because all team members could focus their efforts on the CI/CD processes.
The next step in our journey was to deliver CI/CD to staging, and we were able to do so by using Azure Cloud Service Deployment. We completed that final task and were ready to advance …
… But there was another obstacle: Our build and deploy was failing again. Azure Cloud Service Deployment was not getting the right context regarding CurrentStorageAccountName. So we went to GitHub, found a PS script for the task, examined it, and found a workaround. It was not the smoothest one, but we wanted to push forward and get to our goal. What did we do? Simple—we added a reference to PS script.
Here are the steps for fixing this:
- Go to the GitHub repository: https://github.com/Microsoft/vso-agent-tasks/tree/master/Tasks/AzureCloudPowerShellDeployment
- Download Publish-AzureCloudDeployment.ps1 and add to your source repo.
Add the following line at row 222:
- Save and commit.
- Add it as a build step instead of Azure Cloud Service Deployment.
- Run the build again.
- And … SUCCESS—FINALLY! (We were in need of some success to keep us motivated, so this came at the right moment.)
So far we had achieved the following: We were able to collaborate, check in our code, deliver CI, and use IaC to ensure Service Bus, Topic, and Queue.
The next step was to create worker roles and an environment in which to “drop” the code. Things were finally getting smoother. We were able to create worker roles with a few lines of PS code. Here is a look:
After completing this step, we were ready, excited, and nervous to see what would happen … and take a look: After about 8 minutes, on day 3 of our hack, we successfully assembled the solution. Success looked like this:
Our three-day DevOps hackfest was intense, demanding, and ultimately productive. It helped everyone involved to change, evolve team spirit, and most importantly, have a better understanding of failure. Because failure is an essential component of the improvement process, it will occur as we constantly try to excel in our work, our life … everything.
We did achieve what was agreed on and set as a target. Now the Foize team has an enriched “DevOps spirit.” They have learned new practices and polished some practices well known to them, and ultimately created automated recovery. If we were to choose one achievement that is easy to measure, it would be Mean Time to Repair (MTTR). Initially, Foize had a manual process of recovery, and by their own account it could take from 2 to 4 hours. Now it is done in about 15 minutes.
Of course, we had our fun with CI, and then with CD—but it was known to Foize already, they were using it, and so we polished it a bit.
The hack had nonfunctional benefits as well. It is always good to take time to understand, talk, exchange, learn, share, and have fun with peers from other businesses. It provides us all with a better understanding of what is out there, how our IT world is evolving, and what others are thinking. We got all of this, both technical and non-technical, and we can’t wait to do it again.
We are grateful to the Foize team for many reasons:
- They had a great idea, they were persistent, and after a while their idea became a product.
- As their solution began to get more attention, they recognized the potential bottleneck and were open-minded about solving it.
- They hosted us, as their extended team, for three days.
- Their spirit is awesome.
As with any work, it is far from done. This is just one step of many that Foize will be making. As we brought our hack to an end, we took a little time to imagine what the current solution could look like in the near future. Immediately a few ideas came to mind: Azure Service Fabric, Azure Container Service, and more. We decided to “go wild” for a bit and the result was a brief/draft future architecture (Figure 11). It is not detailed, but it represents a way forward.
Figure 11. Our brainstorm of next steps for the solution
Finally, it’s worth emphasizing that DevOps is a team game that shouldn’t be done alone. It was the team effort that made the hackfest work.