Microsoft and SUMITEM conducted a joint hackfest to introduce Mobile DevOps practices into SUMITEM’s Dev and Ops teams for their Maintenance Master Pro software.

Technologies used

Core team

  • SUMITEM
    • Masahiro Kato – Developer
    • Takuya Nakajima – Developer
    • Tetsuya Yamane – Developer
    • Yamamoto Nobuo – Developer
    • Yasuyuki Jinnouchi – Developer
    • Youichi Nakamura – Developer Lead
    • Meguchi Marubayashi – Developer
    • Miyu Takata – Developer
    • Daisuke Muraishi – Developer
    • Eiichi Yukawa – Developer
    • Hironari Hatta – Developer
  • Microsoft Japan
    • Akira Inoue – Senior Technical Evangelist
    • Junichi Anno – Principal Technical Evangelist


Hackfest members


Customer profile

Sumitomo Cement Systems Development Co., Ltd. (SUMITEM), is a Japanese company established in 1988, based on Sumitomo Osaka Cement’s Information Systems Division. It is developing its own IT-related service business as well as an IT environment for its supporting parent company. SUMITEM has received a very high evaluation from customers on the development and sale of their cement, specifically for the management of quality, sales, and shipping.

Their most recent product release is Maintenance Master Pro. The number of customers using it is rapidly increasing, so SUMITEM would like to speed up improvements and get better troubleshooting support in place.

Problem statement

This case study is, of course, related to Mobile DevOps. However, it is also the story of a growing young development team. Young and highly skilled development team members can overcome difficulties with their passion and oversight. They trust each other’s skills, and they themselves are full of confidence. As much as possible, they tend to eliminate work that seems to them to be inefficient, and instead devise seemingly more efficient development methods. “Inefficient work” may include tasks such as “review” and “sharing results,” and to me that’s a red flag. Therefore, in addition to introducing Mobile DevOps, we decided to introduce DevOps practices as a method of sharing evidence at this hackfest.

SUMITEM is currently using the following technologies:

SUMITEM’s current service can be accessed from a PC browser or from a mobile application for iOS and Android developed by Xamarin. When accessing it from the browser, it uses the Web Apps feature of Azure App Service. When accessing it from the Xamarin application, it uses the Web API (also deployed on Azure App Service).

Two kinds of databases are prepared in the backend. One is a database common to customers, where customer information, SQL strings, and the like are stored. In the other database, customer-specific information is stored separately.

They use C# with Visual Studio 2015. Their source code is stored and version-controlled in Team Foundation Server. However, they don’t use any technologies related to automation.

System overview


The issues heard from SUMITEM before the hackfest are as follows:

  • Unit tests are done manually by each developer, but there is no evidence from the tests.
  • There are no load tests for web apps.
  • There are no UI tests for Xamarin apps (iOS/Android).
  • Failure information that is generated on the mobile device cannot be fed back to the product effectively.
  • The database’s schema is manually changed.

Solution and steps

Value stream mapping

To further clarify the problem, we created a value stream mapping (VSM) before starting the hackfest.

VSM


Following are the results:

  1. Requests, requirement definition, and design

    • Lead time = 10 days; process time = 3 days
    • Requests for implementation of new features may come from the sales team or development teams.
    • The feasibility of the request and the person in charge of implementation are decided at a regular meeting every week.
    • A developer in charge of implementation redefines the request as a requirement and creates a design document. However, there are many cases where the request is misleading, and it can take up to 10 days to complete the requirement definition due to having to wait for responses from customers and sales representatives.
    • There is no review of the design document.
  2. Coding

    • Lead time = 20 days; process time = 16 days
    • Developers in charge of implementation develop code according to the design document they created.
    • During development, the developer creates a test specification in his head, but never documents it.
  3. Unit tests

    • Lead time = 6 days; process time = 4.5 days
    • As mentioned earlier, unit tests are done by the developers according to the test specifications in their head.
    • After the tests are complete and the developers are satisfied, they share the deliverables with the team.
    • Because the test is done manually and there is no documented evidence, the validity of the test is secured by trust.
  4. Integrated tests/load tests

    • Lead time = 5.5 days; process time = 3.5 days
    • From this step, work is handed to operations (Ops). The Ops team deploys the modules shared by each developer in the staging area and conducts integration tests. Integration testing is basically focused on operations. Test scenarios exist and are documented.
    • If a new function is added or a correction is applied, the same test is done manually each time.
    • If problems occur in the integration test and need to be corrected, the same test is performed again manually.
    • Integrated testing of mobile applications developed with Xamarin are also done manually in this step.
  5. Release

    • Lead time = 3 hours; process time = 2.5 hours
    • Release and existing customer system upgrades are also done manually by Ops.
    • The most time-consuming and labor-intensive work involves upgrading the database. Ops uses Visual Studio to create a data-tier application package (DACPAC) and applies it manually to each customer’s database when a schema change of the database occurs (basically adding columns) due to enhancement. According to Ops team members, this is the work that is the most nerve-racking.

The total lead time was 42 days. The total process time was 27 days.

To our surprise, SUMITEM developers don’t have a culture of sharing evidence. For example, each design document is prepared by each person on the team, and the designer starts coding without any meeting for review.

The same goes for tests; developers create their own test case specifications and tests while coding. Test results are not shared, and there are no steps to review them. In other words, each individual is highly skilled and their development ability is fully trusted. It can be said that this is an ideal process in a sense; however, this is a recipe for potential problems. At the moment, developers share only the finished program, and there is no way to check or confirm the process involved in making it. In the future, this will emerge as a prominent problem when the scale of development and the number of customers increases.

The current manual testing process can also be improved. Today’s unit test requires six days. With automated testing, that can be shortened to a few hours. In addition, if they succeed in introducing test-driven development (TDD), they can further shorten the testing time.

Therefore, we made the following suggestions for the hackfest:

  • Mobile DevOps
  • Basic DevOps practices
    • Continuous integration/continuous delivery (CI/CD) using Visual Studio Team Services
    • Kanban collaboration
    • Test-driven development (TDD) and automated testing
    • Test scenario recorder of Visual Studio 2015 and automated testing
  • SQL Database DevOps
    • Automated deployment of DACPAC

With these practices, SUMITEM developers should be able to easily share the test scenario, its results, and the process up to releasing the project.

Technical delivery

Implementing test-driven development and a Microsoft test project

At the beginning of the hackfest, we introduced test-driven development (TDD) and had a mini hands-on session. Following are the SUMITEM developers’ impressions after the hands-on.

“We are purely surprised that we can proceed with coding as we write the test code. Until now, we were vaguely thinking about the specifications of the test in the head during coding. As a result, there was something we forgot even though we tried to recall it later. However, it is amazing to be able to create tests at the same time as coding, using TDD. However, it seems that “thinking” at the time of coding seems to be a little different from what it was before. In this hackfest, TDD is one of the themes. If we concentrate only on TDD, other practices may be negligible. We will tackle TDD seriously after the hackfest.”

We used the Bowling Game Kata using C# to demonstrate test-driven development (posted on SlideShare). Using this game, the developers can learn test-driven development based on program development to calculate bowling scores. Bowling is a game everyone knows, and the rules are well-known. The most difficult thing for beginners in test-driven development is to make future predictions and test code creation at the same time. The skill of “predicting the future of the code” is gradually acquired and polished. This time I wanted to focus on writing test code, and save the “predicting” part for later.

Through this mini hands-on, the SUMITEM developers gained an understanding that test code and function code could be developed at the same time. In addition, their challenge that there was no evidence from tests was resolved at this time.

Migrating the TFS environment to Visual Studio Team Services

Until now, unit tests were conducted on individual PCs and the results were not shared. Now, unit tests can be fully automated by using Visual Studio Team Services and can be shared by team members.

This was the first time that the developers used Team Services, so they had to create a new project repository. For a version control system, they selected Team Foundation Version Control (TFVC). The repository in the following diagram is a private repo that SUMITEM managed.

Project dashboard


We suggested that they use a Git repository in the future. Git is a distributed version control system; you can find out more at Choosing the right version control for your project.

Each developer had a copy of the source repository on their machine. Developers could commit each set of changes on their dev machine and perform version control operations, such as history and compare, without a network connection.

After they created the repo, they migrated the source code by referencing the Team Foundation Server to Visual Studio Team Services step-by-step migration guide.

Now, they have a collaboration and sharing platform in the cloud.

Kanban


Originally the SUMITEM developers tended to dislike “waiting for approval” and “cost of collaboration.” It is certainly a virtue that individuals perform their duties with responsibility. However, if they can share the value of their work, that virtue is further refined. The advantage of Visual Studio Team Services is that it allows you to share facts and phenomena without increasing the cost of collaboration. The result of the work done by developers can always be shared in the central Team Services repo. Using Team Services provides a great growth opportunity for the team.

“As leaves develop photosynthesis, it will be a source of growth for all team members, as nutrients come back to the trunks.” —Junichi Anno, Principal Technical Evangelist, Microsoft Japan


Creating the CI/CD pipeline

Their next step was creating a pipeline for continuous integration/continuous delivery (CI/CD). Before creating the CI/CD pipeline, we decided on a new system architecture as shown in following diagram.

New System Diagram


Before the hackfest, SUMITEM had only two environments: one for developers and the other for customers. After manual integrated testing on the developer environment, they would release on the customer environment.

Going forward, after developing in the development environment and committing it, the source code is automatically synchronized with Team Services and built, and unit testing is done. When the unit test is completed, it moves to a similar staging environment such as the customer environment where integration testing is done, and if there is no problem, a swap is done and it is released to the production environment. The following diagram shows the new pipeline.

New Build/Release pipeline


Based on the changes in these processes, SUMITEM will be able to offer higher quality service to customers. In addition, in case of any trouble, it will be possible to immediately return to the old environment by using the swap function.

Creating the Xamarin (iOS) application pipeline

The Xamarin application uses the following pipeline. When code is committed in Visual Studio, the build process is started and automatically synchronizes with Team Services. An automated test is performed on Xamarin Test Cloud after completion of the build. When the test finishes successfully, it is sent to the beta tester rather than released, and HockeyApp is used at this time. They receive feedback from the beta tester, brush up again, carry out the build, run an automatic test, beta test again, and when it is confirmed that there is no problem in terms of quality, it is released on Azure Blob storage. At this time, releasing work to the application store (Apple and Google) is done manually.

Xamarin App Build/Release pipeline


As we mentioned, SUMITEM developers are manually testing the Xamarin application. When there is a change on the server side, even if Xamarin has not made any changes, they do all the integration tests again manually. For this reason, they have to spend a large amount of testing time to fix a few small issues. A main objective of this hackfest was to automate the process as much as possible to reduce lead time.

The build pipeline they created is as follows. For reference, I used Build your Xamarin app.

  1. Set up a custom build host on the Mac.
  2. Get source.
  3. Restore Xamarin components.
  4. Build Xamarin.iOS solution.
  5. Define automated testing step for Xamarin Test Cloud.
  6. Deploy to HockeyApp.
  7. Copy to artifact folder.

1. Set up a custom build host on the Mac

I will introduce in detail the application for iOS here. For iOS, some preparation is required before creating the build pipeline. A build of an iOS application needs to be done on the Mac operating system (macOS), not in the Team Services environment in the cloud. Therefore, it is necessary to set up the build agent on a local MacBook and register it in Team Services as a custom build agent.

Step 1 Set up a custom build host on the Mac


For the setup procedures for additional hosts, we referred to the following site: Deploy an agent on OSX.

NOTE: It is important to note that the build agent cannot be started simultaneously unless you add more queues. For example, even if you add a build agent on the macOS, you cannot operate it in parallel with the default queue. To make build time more efficient, it is necessary to purchase a new queue.


2. Get source

In this step, the source code is acquired. Because they are using Team Foundation Version Control this time, we must pay attention to the Workspace mappings setting. The repository option is ignored here. In Map, specify the location where the resources necessary for build are stored. By default, the project’s root folder is mapped.

If the folder contains a lot of unnecessary files, all of them are downloaded to the workspace, so there is a possibility of using too much time with one build. Therefore, Cloak is used to designate unnecessary folders. The path specified for Cloak is not necessary for build, so it cannot be downloaded.

Step 2 Get source


3. Restore Xamarin components

Next, specify the project file to be built and restore it.

Step 3 Restore Xamarin components


In the Email and Password fields, specify the ID and password you are using for your Xamarin account. It is better to register the password as a variable on the Variables tab in consideration of safety, instead of entering the password as raw. In this case, we specified the password with the variable name xamarin-yj-password.

Variables tab


4. Build Xamarin.iOS solution

Next, build the restored project.

Step 4 Build Xamarin.iOS solution


The important setting here is Signing & Provisioning. To build an application for iOS, you need two files, a .p12 file and a provisioning profile file. These files need to be prepared in advance.

The relationship between these files is a bit complicated, so it is shown in the following figure.

Relation among p12 file, provisioning profile, and Team Services


  • .p12. First, create a CSR file with a Keychain Access application on macOS. Upload it to the Apple Developer Center and download the generated certificate file. You can register the certificate file with the Keychain Access application and generate the .p12 file.

  • provisioning profile. In the Apple Developer Center, issue an App ID, register a device for development, and create a provisioning profile by using the previously created certificate file.

  • build agent. Next, define the custom agent host on the Mac. To build an iOS application, it is necessary to designate a custom agent created on the Mac. You can specify the agent on the Options tab.

    Define the custom agent host on macOS


5. Define automated testing step for Xamarin Test Cloud

Testing starts automatically when build is completed. Automated testing is done on the Xamarin Test Cloud, so you need to get a Xamarin Test Cloud account beforehand. You must set the API key on this screen. The API key is shown at Teams & Apps on the Xamarin Account page; add it to the Team API Key field.

Step 5 Define automated testing step for Xamarin Test Cloud


6. Deploy to HockeyApp

When the automatic test is completed, deliver the .ipa file to HockeyApp and enable the beta test.

Step 6 Deploy to HockeyApp


When an application is registered in HockeyApp, beta test members are notified by email so they can download it to their own device.

Overview of app registered in HockeyApp


Crash reports generated by beta tests are automatically reported to HockeyApp.

Crash report in HockeyApp


It is also possible to easily obtain feedback from the beta tester from the application.

Feedback from beta tester


HockeyApp can work with an external bug tracker. It also includes Visual Studio, which allows Kanban to efficiently share crash reports and feedback as backlogs within the team.

Working together external Bug Tracker


7. Copy to artifact folder

Finally, artifacts are duplicated in the drop area for release.

Step 7 Copy to artifact folder


The build process is complete. The shortest time required for build is about 10-20 minutes, including a simple test in the Xamarin Test Cloud. In the future, even when creating a full-scale test scenario, it will be possible to complete it within 2 hours.

Current time for build pipeline


Setting up the release pipeline

For this project, SUMITEM decided to release to the store manually for the present. Therefore, after approval of the team leader, applications that have completed the beta test are copied to Azure Blob storage for the following reasons:

  • They need to notify customers beforehand when they’re going to release the latest version.
  • A press release may be needed depending on the version.
  • The release needs to be synchronized with the release of the server-side application.

Release pipeline


In this project, the conditions for approval by the project leader are defined in the following list. For now, these are the initial conditions, and they will gradually be changed as they continue to operate in the future.

  • Set the beta testing period to 10% of the total lead time at the maximum.
  • Crash reports generated during beta testing aim at 100% resolution.
  • Feedback is reviewed at regular meeting.
  • Team leader confirms the release with Kanban.
  • Team leader approves the release by using Visual Studio Team Services.

Conclusion

I would like to look back on the VSM here. The lead time (LT) and process time (PT) before introducing Team Services were as follows:

  • Step 1: Design LT = 10 days, PT = 3 days
  • Step 2: Coding LT = 20 days, PT = 16 days
  • Step 3: Unit test LT = 6 days, PT = 5 days
  • Step 4: Integrated test/Load test LT = 6 days, PT = 4 days
  • Step 5: Release LT = 0.4 days, PT = 0.4 days

The total lead time was 42 days. The total process time was 29 days.

By automating the build process of Web Apps and Xamarin, it is expected to improve as follows:

  • Total lead time = 33 days (improved 22%)
  • Total process time = 23 days (improved 24%)
  • Work efficiency (PT/LT) = 69%

Result of hackfest


As we described, it is possible to realize an efficiency of 23% as a whole by merely automating tests.

Of course, this is not the end. We still have the impression that the designing and coding time is too long. Therefore, the next two challenges the developers should address are as follows:

  • Reduce iteration
  • Activate communication

Now they can easily share results and problems by using Visual Studio Team Services, which should shorten the lead time. Their current work efficiency is 69%. To improve this to 90%, it is necessary to shorten the lead time from 33 days to 26 days. This goal seems to be difficult at first glance, but activating communication should help us.

General lessons

SUMITEM was already using Web Apps, Azure SQL Database, Azure Storage, Xamarin, and Visual Studio 2015, and hopes to use Visual Studio Team Services with CI/CD to Azure. During the hackfest, they learned how to use Team Services in only three days.

One of their challenges was that there was no evidence from the tests. By using Microsoft tests and Visual Studio Team Services build pipelines, they gained valuable experience with test-driven development (TDD).

They are considering what is the best way to improve mobile development using Xamarin. One of the answers is to use Xamarin Test Cloud, and they tried this in the hackfest. In the near future, they want to use Visual Studio Mobile Center as well. Mobile Center is a service that integrates Test Cloud, HockeyApp, and CI/CD of iOS/Android apps. They hope Mobile Center has a feature to integrate with Team Foundation Version Control on Team Services; currently, it only has GitHub integration.

At last we got feedback from SUMITEM about Visual Studio Team Services. Their project members are not only developers but also project managers and some approvers on the business side. They hope to use Team Services in Japanese. Especially, they said that the Approver feature of Team Services release management was very useful and important for their development process, but currently, there is only an English UI. They mentioned to me that some approvers in their company feel that Team Services is difficult to use.