Lowering lead time and improving visibility with ExakTime
Microsoft worked with ExakTime to create a value stream map of its software delivery processes and to make improvements with DevOps practices. The following DevOps practices were implemented:
- Continuous integration
- Continuous deployment
- Release management
Key technologies used:
The core hack team:
- Erik Renken – Senior Software Architect, ExakTime
- Adam Freehling – Software Engineer, ExakTime
- David Tesar – Senior Technical Evangelist, Microsoft
Note: A larger team was present for the value stream map exercise, but these were the core participants in the implementation hackfest.
ExakTime uses innovative time-tracking solutions to eliminate the need for handwritten, paper time cards at job sites. The focus of the hackfest was on ExakTime’s core customer front-end services and back-end APIs, which run its time-tracking software. The time-tracking software all run as Azure app services.
ExakTime used different tools for work item tracking (Targetprocess), source code repository (Plastic SCM), and central build, test, and deployment (Jenkins). Although Jenkins was set up to do continuous integration and deployment to specific environments, the releases between environments were done manually and new builds were generated from the same version of source code control between each environment. At any given time, it was challenging to determine which environment code resided on and when that code might be ready to move to the next environment to go to production.
Furthermore, quite a bit of effort was required to pull together information across these separate tools to have visibility into which work items were related to a release, what commits into source code control were happening against those work items, and what builds contained what work items. There was no hard-set rule on when a release happened and it would take around a month or more to get even a single feature into production.
The ExakTime product has five different front-end services (Documents, Identity, Provisioning, Reports, and Web) with back-end APIs and they all run on top of Azure App Service. They had one Visual Studio SLN file that built all of these and more and one source code repository for this solution. Builds usually took an hour or longer, which slowed development and delayed feedback from automated tests in Jenkins. Also, the Jenkins server was a single virtual machine that had to be maintained with OS-level and task patches, required scheduled reboots, and occasionally was inoperative, interfering with the overall development life cycle.
Solutions, steps, and delivery
Value stream mapping helped ExakTime determine what its real lead time was and find where the biggest improvements could be made to its overall process of delivery, from ideation to production. The team decided to focus on the following key areas for improvement:
- Better visibility into work items, code commits, and deployment pipeline.
- Move toward true continuous integration.
- Implement release management.
Better visibility into work items, code commits, and deployment pipeline
The ExakTime team decided to migrate their work-item tracking system (Targetprocess) and source code control (Plastic SCM) to Visual Studio Team Services using the Team Services agile process template and Git repositories. They did this migration on their own outside of a Microsoft-assisted hackfest. This made it much easier to see the connection between the work items and the code being committed.
To continue to have visibility through the entire deployment pipeline to production, ExakTime needed to evaluate the tools available to move those corresponding work items and code commits into builds, and builds through the necessary environments into production. ExakTime didn’t want to spend time maintaining their Jenkins implementation, so they really needed a service as opposed to any other tool that they would also have to maintain. They liked the capabilities of Team Services build and release management and the visibility it could provide for their work items, code commits, builds, and deployments. Team Services was a natural fit.
Microsoft helped replace Jenkins with Team Services. ExakTime could use Team Services to complete all of the same functionality as with Jenkins, including continuous integration and automated tests, but now they had the added bonus of being able to easily see their work items, code commits, builds, AND tests that failed—all from a central location.
The continuous deployment functionality of Jenkins that was handled by a PowerShell script deploying to Azure App Service was replaced with Visual Studio Team Services release management and the Azure websites deployment task as covered below.
Move toward true continuous integration
ExakTime already had the “loose” definition of continuous integration enabled—namely, that when they checked in code to their branch, Jenkins would automatically start unit tests and create a build if those tests passed. This is useful, although the biggest delays were in getting the code through a number of branches and build definitions leading up to production.
The user story branches get checked in by the dev anywhere from 3 hours to 2 days. There might be anywhere from 2-10 user stories per feature and the feature branch doesn’t get manually merged into the dev branch until the entire user story is complete (6 hours to 20 days’ time). Then finally “loose CI” kicks in once it hits the dev branch. After this, the merge from the dev branch into the QA branch is a manual one taking anywhere from 1-2 days and another “loose CI” kicks off. The QA team does manual tests when it hits the QA environment.
This branching structure caused delays in getting feedback from the QA team of up to three weeks or more from the time the code was first written, causing a “waste” of task switching and taking developers longer to fix their code. Furthermore, merge conflicts were common, wasting much time simply trying to get branches into a healthy state. So many people were working on code subbranches that they didn’t even attempt to merge for many days if not weeks or months later.
To mitigate these challenges, a move toward true continuous integration was needed. This meant the ultimate goal was to have people committing to “trunk” or the main branch daily and that branch was always in a shippable state as verified by automated tests and the code running in production or production-like environments. Moving to this structure is a big culture shift it and takes time, so we proposed the following to help transition to this state:
Be able to build, test, and release the app services independently. This significantly reduced build times to around 10 minutes from 45-60 minutes. It enables faster feedback and opens up the possibility of deploying independently and breaking apart the code base into separate Git repositories.
Shrink the branch structure and environments. Removing the dev environment and user story branches was a quick way to reduce merge conflicts and lead time.
Run the new Git merge task nightly. In the transition to one branch and separate Git repositories for each service, Microsoft created a Visual Studio Team Services “Git merge” task that tests to see if any merge conflicts will happen if those branches merge individually or all together into their higher-level branch.
ExakTime runs this nightly. It has helped to find and resolve merge conflicts the next day versus having much more pain in trying to merge after days, weeks, or months across numerous developer merge conflicts. In the future, ExakTime might run this task even before allowing the code to be checked in to the subbranch.
Set up a deployment-and-release pipeline based on services. Developers usually worked on just one service at a time. These services have the ability to deliver end-user value in itself and could be deployed to customers independently. For these reasons, it makes sense to break with the “ship all services at once” mentality and deploy them more independently.
Implement release management
This new branch and service pair release structure made a smooth transition into their own continuous deployment and release definition pipelines. Each service pair has a build definition that builds both services and upon a successful build, continuous deployment triggers a new release for that build into the QA environment.
As you can see, a build only happens once and then this build artifact is automatically moved through the different environments. Because ExakTime is using Visual Studio Team Services for work-item tracking, source code repository, builds, tests, deployments, and release, they now have full end-to-end visibility!
ExakTime has reduced its lead time to a week from more than one month, reduced its build times, and is getting feedback much quicker, which saves time and produces higher quality code. Everyone at the company has an amazing new view in Visual Studio Team Services of the end-to-end lifecycle of the services, and this has motivated the team and saved them time—they no longer spend time maintaining a Jenkins virtual machine because Team Services handles everything.
Other teams should work toward smaller batch sizes for a release and strive to aim for “true continuous integration.” Consider the question, “What is the smallest new feature that could be developed that has user value?” and set that to be a release.
In the quest for true continuous integration, teams might consider using the Git merge Team Services task to minimize or eliminate the pain of merge conflicts.
- Git merge Visual Studio Team Services task in marketplace (coming soon)
- Git source code branch structure discussion on Channel 9
- Visual Studio Team Services release management course on Channel 9
- DevOps Practices