NAVITIME JAPAN, the leading provider of navigation technology and services in Japan, keeps improving their development process by using cutting-edge technologies. During this hackfest, we tackled several technology areas: microservices, mobile DevOps, and advanced DevOps practices.

Their navigation services comprises several parts. Their services are interconnected via REST APIs, and their user interfaces include web, Android, and iOS. Now they want to move from on-premises to microservices on Azure and reduce their lead time.

Also, they tended to spend a long time in decision-making before creating their user story. We recommended using hypothesis-driven development instead of big upfront discussions.

Key technologies

Core team

Photo of hackfest members

Test in production

  • Tomoya Takahashi
  • Ayami Nakazawa
  • Shigeru Tsukada
  • Go Maniwa

Automated testing and release management

  • Tomonori Yoshinaga
  • Koichi Goto
  • Konosuke Murata
  • Kyoko Morimoto
  • Naoki Ishii

Microservices architecture

  • Shigehisa Watabe
  • Kouki Kikuchi
  • Naoki Sekizawa
  • Soichiro Hayashi
  • Katsuhide Kayashima
  • Kazuki Tanaka

Microsoft and Creationline supporters

  • Tsuyoshi Ushio (@sandayuu) – Technical Evangelist for DevOps, Microsoft
  • Naoki Sato (@satonaoki) – Senior Technical Evangelist for Azure, Microsoft
  • Junichi Anno – Principal Technical Evangelist for DevOps and Security, Microsoft
  • Yoshio Terada – Technical Evangelist for Java, Microsoft
  • Drew Robbins (@DrewRobbins) – Technical Evangelist, Microsoft

  • Tadahiro Yasuda – CEO, Creationline
  • Mitsutoshi Kiuchi – Evangelist for Microservices and DC/OS, Creationline

Customer profile

Tokyo-based NAVITIME JAPAN is a leading provider of navigation technology and services. NAVITIME is a total navigation solution that enables users to plan travel routes by using various transportation methods, including trains, subways, buses, and taxis.

Creationline is a Microsoft partner with expertise in DevOps and microservices.

Problem statement

The team had already moved to Agile development and automation for web services using Jenkins. However, they want to shorten their lead times.

They encountered several problems when they integrated their services: Their system is interconnected via REST APIs, but they can’t deploy separately. Their branching strategy causes a big-bang integration problem. To decide whether to implement a new feature requires four months. And their mobile deployment process has a lot of manual processes.

Also, they were running their service on on-premises servers, which meant inflexible resource allocation.

Solution, steps, and delivery

We solved these problems by implementing the following four steps.

1. Value stream mapping

We started with value stream mapping. Because their services have several services, we had to draw several maps. We found a lot of kaizen points: Long lead times for planning. Manual processes for mobile development. Big-bang integration causes a lot of bugs and consumes a lot of time.

Photo of value stream map for planning and mobile development

Photo of value stream map for integration

They spent up to four months in planning a new feature because they didn’t want to make mistakes in their decision making. I recommended that they implement feature flags and test in production. Instead of discussing for a long time, we can use hypothesis-driven development, which removes the lead time for planning and can get feedback much more quickly. For this purpose, I recommended LaunchDarkly. It enabled us to implement both feature flags and A/B testing.

They develop their application using Java any wanted to use Docker to speed up their release cycle. They have several interconnected services but couldn’t deploy their services separately. We decided to use microservices practices for independent deployment among the services.

We recommended using Azure Container Services with Vamp. Vamp enables canary releasing and autoscaling for microservices. It includes service discovery, API gateways, and self-healing functionality on DC/OS.

Also, they have a lot of manual processes for mobile development, although they did use Jenkins and fastlane. I recommended HockeyApp and Xamarin Test Cloud with fastlane. These services enable us to test more quickly.

2. Getting feedback using feature flags and testing in production

Photo of the Testing in Production team at work

The “Testing in Production” team tackled feature flags and implemented LaunchDarkly on both iOS and Android during the first day of the hackfest. Using the dashboard, they did exposure control on their mobile apps.

Using LaunchDarkly to implement feature flags

They shared their knowledge with other teams to help solve the big-bang integration problem. Their feature branch was long-lived, so I recommended that they decompose the branch until it can be merged in one day. Then they can use feature flags to control exposure to their customers.

Now they can separate deployment and release because feature flags enable release anytime using the dashboard. Being able to expose their application to a limited number of customers enables alpha/beta testing and canary testing.

The following Java code (from the LaunchDarkly Setup page) uses a feature flag:

LDUser user = new LDUser("user@test.com");
boolean showFeature = ldClient.get().boolVariation("your.feature.key", user, false);
if (showFeature) {
  // application code to show the feature 
}
else {
  // the code to run if the feature is off
}

We used the Android SDK and the iOS SDK for LaunchDarkly.

However, the team didn’t finish their hacking, which surprised me. They did start to change their planning process to discuss with their upper management. Also, because feature flags are easy to implement, the team performed user testing during the hackfest. However, the important thing is how they did it: They did A/B testing and observed the users by getting telemetry through Google Analytics.

Photo during A/B testing and user observation

Screen shots of A/B testing

Screen shot of LaunchDarkly dashboard

They learned a lot from the Testing in Production team. They created a quick report to show us how well testing in production worked. It was the perfect outcome in only three days!

3. Mobile CI/CD pipeline improvement

Photo of Automated Testing team

They wanted to learn about test-driven development (TDD). I demonstrated TDD and told them about the famous TDD tutorial Bowling Game Kata by Uncle Bob. We also used the Japanese version of the tutorial Test Driven Development in Action.

Then I reviewed their CI pipeline, created in Jenkins, which couldn’t deploy mobile applications to their testers. Before the hackfest, they just uploaded their .ipa or .apk file to AWS S3 and then asked testers to download the application.

Screen shot of TDD in Eclipse

I recommend that they use HockeyApp, which they did. They also used fastlane for building and testing mobile applications. fastlane integrates with HockeyApp and enables submission of apps to the Apple Store and Google Play.

Here’s an example of using fastlane with HockeyApp:

#====================== Public =================#
lane :hockey do |options|
  version = options[:version]
  production = options[:production]

  build(
    version: version,
    scheme: "MyApp",
    output_directory: "build",
    output_name: "MyApp.ipa",
    configuration: CONFIGURATION_RELEASE,
    use_legacy_build_api: true,
    export_method: EXPORT_METHOD_ADHOC,
    include_bitcode: false,
    production: production
  )

  hockey(
    api_token: "hockeyapp_token",
    ipa: "build/MyApp.ipa",
    notes: "テスト用アップロード"
  )
end

Screen shot of HockeyApp

At the same time, they started using Xamarin Test Cloud. When they tested mobile phones, they needed to prepare actual machines. Xamarin Test Cloud enabled us to perform testing on emulators of various mobile phones. (We also tried out Xamarin Test Recorder.)

Screen shot of Xamarin Test Cloud

Screen shot of Xamarin Test Recorder

Unfortunately, we couldn’t finish hacking during the three days. Their Calabash tests didn’t work on Xamarin, so we had to wait for the response of the support team. However, they successfully improved their pipeline in three days. They especially enjoyed using HockeyApp. It is going to replaced by Visual Studio Mobile Center, but their experience will remain relevant when they move to Mobile Center after it becomes generally available.

4. Microservices using Azure Container Service with DC/OS and Vamp

Photo of Microservices team

Because they weren’t familiar with Azure, we provided a hands-on lab before starting this hackfest; now they are quickly learning how to use Azure. They had already created Docker images on their desktop; we had to enable the CI/CD pipeline for microservices.

We started with Azure Container Registry. They wanted a private repository, which easy to create and deploy on Azure.

Screen shot of setting up Azure Container Registry

For example, to push Nginx to Container Registry, you can use the following Docker commands:

> docker login {put your LOGIN NAME}.azurecr.io -u {put your USERNAME} -p {put your PASSWORD}
> docker tag nginx {put your LOGIN NAME}.azurecr.io/samples/nginx
> docker push {put your LOGIN NAME}.azurecr.io/samples/nginx

Then we start deploying Container Registry, which should have been easy but, because they had just begun using Azure, was not. For starters, we forget to increase ghe CPU core quota. The default is 20, which makes it hard to deploy a DC/OS cluster. We immediately sent a request to increase the quota and waited for the response.

Until the request has processed, we deployed Container Registry with one node. We started by installing Vamp on DC/OS. You need to open a port according to the port to which you want exposure. (See Enabling Vamp on Azure Container Service (DC/OS) for more information.)

We started with the Vamp tutorial, which worked fine. After that, we tried to deploy their actual application on Vamp.

Screen shot of Azure Container Service (DC/OS)

Screen shot of Vamp

However, that caused a disk-full error. The application is 6 GB, and /var/docker of the DC/OS agent was 6 GB, so it quickly overflowed. We needed to deploy elastic search, Vamp core, and API Gateway.

Because this is created on a VM scale set, we needed to scale up the scale-set SKU. It was quite easy to do via Azure portal; of course, we can also do it via PowerShell or REST API.

Screen shot of setting scaling

Screen shot of setting instances

Then it worked fine. When we deploy an application on Vamp, it automatically creates API Gateway for one service and port. We can easily try canary testing and blue-green deployment. It is so nice!

Screen shot of configuring canary testing

The last challenge was the database connection string. We use MySQL. For production purposes, we need to enable service discovery.

Vamp enabled us to use service discovery. We can obtain the server IP and port from environment variables without knowing where the container is actually deployed.

Here’s the blueprint for Vamp:

name: mocha:1.3
kind: blueprint
gateways:
  '9051':
    port: '9051'
    sticky: null
    virtual_hosts: []
    routes:
      mocha/webport:
        lookup_name: 41f1c6add09d6239956a5d2c7dbebec91e47938c
        weight: null
        balance: default
        condition: null
        condition_strength: null
        rewrites: []
clusters:
  mocha:
    services:
    - breed:
        name: mocha-frontend:1.2.0
        kind: breed
        deployable:
          type: container/docker
          definition: {YOUR REPOSITRY NAME}.azurecr.io/latest
        ports:
          webport: 8080/http
        environment_variables:
          BACKEND_1: $backend1.host:$backend1.ports.webport
        constants: {}
        arguments: []
        dependencies:
          backend1: mocha-backend1:1.2.0
      environment_variables: {}
      scale:
        cpu: 1.0
        memory: 3000.00MB
        instances: 1
      arguments: []
      dialects:
        marathon:
          uris:
          - file:///etc/docker.tar.gz
    gateways: {}
    dialects: {}
  backend1:
    services:
    - breed:
        name: mocha-backend1:1.2.0
        kind: breed
        deployable:
          type: container/docker
          definition: mysql/mysql-server
        ports:
          webport: 3306/tcp
        environment_variables:
          MYSQL_ROOT_PASSWORD: {YOUR ROOT PASSWORD}
          MYSQL_PASSWORD: {YOUR PASSWORD}
          MYSQL_USER: {YOUR USERNAME}
        constants: {}
        arguments: []
        dependencies: {}
      environment_variables: {}
      scale:
        cpu: 1.0
        memory: 500.00MB
        instances: 1
      arguments: []
      dialects:
        marathon:
          container:
            volumes:
            - containerPath: /var/lib/mysql
              hostPath: /home/navitime-msa-test/mocha_config/var/lib/mysql
              mode: RW
            - containerPath: /etc/my.cnf
              hostPath: /home/navitime-msa-test/mocha_config/my.cnf
              mode: RW
    gateways: {}
    dialects: {}
environment_variables: {}

However, the framework, which was created by NAVITIME, didn’t have the functionality to get the database connection string from environment variables. We started to add this functionality, but we ran out of time.

Photo of Microservices team hacking Vamp

At the same time, we reviewed and hacked the CI/CD pipeline using Visual Studio Team Services and Jenkins. They can now build, test, and push the container by using the Jenkins pipeline as code.

Screen shot of Jenkins

Screen shot of setting up Jenkins job in Visual Studio Team Services

However, we successfully got the actual IP and port. Because NAVITIME maintains the framework, the change is not so difficult.

Conclusion

We got a lot accomplished in only three days: The Testing in Production team enabled feature flags. The Mobile team improved their CI/CD pipeline, automated a lot of procedures, and made getting feedback easier. The Microservices team enabled deploying their application through Vamp using Container Service and DC/OS rather than on-premises.

I was surpised by the progress in only three days! I was impressed by the energy and talent of the NAVITIME JAPAN team, and I hope that they continue to enjoy hacking Azure and get great results from their services.

Now they are planning to apply to production what they learned during this hackfest. NAVITIME have already changed their mindset to learn from production instead of discussing for a long time in advance. They already implemented TDD with their CI/CD pipeline. (They didn’t apply it to the whole system, but they have implemented it for some important parts, such as the authentication system.) Also, they tried to implement A/B testing for their operation.

As the next step, we are planning a microservices-specific hackfest with them.

Here are some customer comments from this hackfest:

“I enjoyed the hackfest. I’ve got a lot of insight of Container Service, DC/OS, and VAMP for microservices architecture. Also, it enables us to do canary release quite easily.”

“We learned about microservices on Azure. Also, we could focus on this hackfest without distractions.”

“I can automate the CI/CD pipeline for containerized microservices.”

“We learned TDD and use it for production. It is helpful!”

“Sometimes, the tools didn’t work well. However, I enjoyed [the hackfest]. We noticed that the parts of the process were not so bad; however, integration was not so good. As a QA, I can hack the Jenkins pipeline, HockeyApp, and Xamarin Test Cloud. I can collaborate with developers very well. I feel that this hackfest was worth several months’ work.”

“It delivers value for production; it was very useful. Also, we can discuss with Microsoft folks.”

“I didn’t have the confidence to generate a decent outcome at first. However, we successfully achieved testing in production; also, we can use it on actual operations!”

Additional resources

DevOps practices

Microservices

DevOps

Mobile DevOps