In this DevOps hackfest, Microsoft teamed up with NEC Solution Innovators to hack Docker and HashiCorp products on Azure. This report describes the process and the results, including the following DevOps elements:

  • Infrastructure as code
  • Continuous integration / continuous delivery
  • Release management
  • Automated testing

Customer profile

NEC Solution Innovators, Ltd. is a leading provider of solutions for business. It delivers services and software to support IT customers worldwide in cooperation with local NEC group companies. Headquartered in Tokyo, NEC Solution Innovators has approximately 12,000 employees.

We conducted this hackfest for the company’s information systems (IS) department. In Japan, many IS departments are struggling with waterfall-based business models, red tape, strict rules, and insufficient resources. These elements prevent them from moving on to new technologies.

As a result of this hacking, NEC Solution Innovators should be able to exit this technological bind and enter the cutting-edge world of DevOps.

Figure 1. NEC Solution Innovators hackathon members

Hackathon team


The hacking team:

  • Toshiyuki Ando – Agile/Developer, NEC Solution Innovators
  • Tomohiro Fukui – Manager, NEC Solution Innovators
  • Yuzo Tokutani – Manager, NEC Solution Innovators
  • Satoru Koike – Developer, NEC Solution Innovators
  • Jyunya Yamaki – Developer, NEC Solution Innovators
  • Daisuke Kitagawa – IT Pro, NEC Solution Innovators
  • Hayato Sasaki – IT Pro, NEC Solution Innovators
  • Masahito Zembutsu – Docker/HashiCorp Evangelist, Creationline
  • Tsuyoshi Ushio – Senior Technical Evangelist DevOps, Microsoft
  • Junichi Anno – Principal Technical Evangelist Azure AD, Microsoft

Problem statement

NEC Solution Innovators was facing the following challenges:

  • Long lead time for delivery
  • Japanese culture (numerous approval requirements, documentation, strict rules)
  • Manual operations
  • No Agile adoption
  • Low utilization rate

They had a formidable number of hand-off and approval processes as well as many manual processes. Their total lead time was 8.5 months.

They also limit the size and abilities of their standard PC devices. For example, they can’t install Docker Tool Box on their PCs even if they use Docker. They are building Docker images on a Linux machine because of this limitation. The VPN firewall doesn’t allow port 80, 443. The standard PC has only 3 GB of memory, 32-bit.

This situation is not unusual in Japan. NEC Solution Innovators and many other Japanese companies struggle with these problems because of Japan’s cultural background. Its enterprise software industry is in need of a cultural breakthrough in order to move forward.

Current architecture

NEC Solution Innovators’ current architecture is as follows:

  • Docker
  • Docker Registry
  • AWS (development/acceptance testing environment)
  • Jenkins
  • vSphere (production/on-premises)
  • GitLab
  • CentOS

The company used Jenkins only for continuous build, not for testing. They have strong on-premises rules, which means they can’t use cloud services like DockerHub. (They can, however, use Visual Studio Team Services.) Testing and release management were done manually. One big surprise was their use of a committer, someone in charge of commit code. Once a programmer writes code, they send it to the committer with a request to commit. (In Japan, a lot of companies still follow such rules from the mainframe era.)

Solution, steps, and delivery

Our joint team of Microsoft and NEC Solution Innovators began the hackfest with a DevOps presentation to development, operations, and business managers. Because Japan has a hierarchical culture, we needed to involve management to get approval to change the development process.

Next, we performed a value stream mapping exercise to visualize and share the existing processes and problems. The results revealed a lot of room for improvement:

  • Many hand-offs and manual processes.
  • Requirements written as specification Excel sheets.
  • No regression testing.
  • Manual testing needed for each environment.
  • Documentation, meetings, and approvals required prior to software releases.
  • Additional documentation required after a release.
  • Lengthy approval process to spin up a virtual machine on vSphere.

With only seven days to come up with solutions, we faced quite a challenge.


Figure 2. Value stream map of NEC Solution Innovators

Value stream map

Pre-hackfest day

One challenge was to instill an Agile methodology before moving on to DevOps. The company already had an Agile coach, Toshiyuki Ando. Toshiyuki told the hack team about the scrum process and extreme programming practices. Then the team had a pre-hackfest day to learn test-driven development. Also, we set up Azure and a Visual Studio Team Services account so we could get off to a fast start.

NEC Solution Innovators has a hybrid cloud environment. They used to use AWS as a cloud platform and vSphere for on-premises. They are running Docker, so they could quickly run it on Azure. Until now, they needed to keep vSphere; however, they might move to Azure soon.

For this reason, we chose Terraform for an infrastructure as code platform—it supports both Azure and vSphere. Terraform supports only v1 (Azure Service Management), but we thought it would help before long and we were right: After this hackfest, HashiCorp announced an Azure Resource Manager driver for Terraform.

For a release management environment, NEC Solution Innovators chose Visual Studio Team Services. This will allow them to automate release management.

Figure 3. Architecture discussion

Architecture discussion


Figure 4. Architecture overview

Architecture overview

We tried first to install Docker Tool Box, but it didn’t work. That’s because the department’s standard PC was 32-bit with 3 GB of memory. We recommended they upgrade their PCs and add 16 GB of memory for developing an application.

The devs started hacking Visual Studio Team Services to automate build, test, and release. It was quite easy to implement these tasks. Also, we provisioned a VSO agent build server on CentOS to execute Docker tasks. We thought we could build on a hosted server. If so, then we might have used the VSO agent server for release management. However, that didn’t work because of the differences in return code between Linux and Windows. So instead we went with a VSO agent server for build, test, and release. At the same time, another dev hacked Flyway, a database migration tool that could be used to change the database structure as part of automating the deployment process.

Figure 5. Software Kanban

Software Kanban

Figure 6. Build definition

Build definition

Figure 7. Release definition

Release definition

Meanwhile, the Ops team hacked Vagrant to provision a virtual machine on Azure. They thought it might be easier than with Terraform. However, it didn’t work well because of the severe Nokogiri error. Because it was a native extension error and we had little time to hack, we instead tried Terraform, which is created using Go language. Go works both on Windows and Linux. With this, we successfully achieved an infrastructure as code practice.

The final challenge was to deploy a Docker container. We thought it would be easy to use DockerHub, but that wasn’t possible because of the firewall policy. We had to use Docker Registry Server instead, and that required some work. We had to provision a server, come up with a deployment strategy, and a blue-green deployment or rollback facility.

Figure 8. Hack participants

Hack participants


But how would we deploy Java application and SQL Server using Docker? We considered several architectures and finally chose a cluster solution: Serf. Serf is a decentralized solution for cluster membership, failure detection, and orchestration. It is lightweight and highly available. Installing Serf on Azure is easy because it requires a single binary. We created a Serf cluster on an App / DB / VSO agent server. Forming the cluster was quite easy—we just joined it.

self.json { "tags": { "role": "builder" }, "event_handlers": [ "query=/opt/serf/event/docker_deploy.sh" ] }docker_deploy.sh ```
!/bin/bash
echo "# DEBUG" echo "-----------------------------------------------" echo "SERFSELFNAME is ${SERFSELFNAME}" echo "SERFTAGROLE is ${SERFTAGROLE}" echo "SERFQUERYNAME is ${SERFQUERYNAME}"
read arg echo arg="$arg" echo "-----------------------------------------------"
if [ "${SERFQUERYNAME}" = images ]; then docker images
elif [ "${SERFQUERYNAME}" = run ]; then docker run $arg
elif [ "${SERFQUERYNAME}" = pull ]; then docker pull $arg
elif [ "${SERFQUERYNAME}" = rmi ]; then docker rmi $arg
elif [ "${SERFQUERYNAME}" = run ]; then docker run -itd $arg
elif [ "${SERFQUERYNAME}" = exec ]; then docker $arg
elif [ "${SERFQUERYNAME}" = compose ]; then docker-compose $arg
elif [ "${SERFQUERYNAME}" = git ]; then git $arg
else docker ps fi
exit 0 ```
start Serf agent serf agent -config-file=./serf.json &
Join the cluster serf agent -join=<master-ip-address> -name=<server-name> &


Once we fired a query by Serf on the VSO agent server, the script on the cluster worked. We also could filter the query.

This is the command we used to fire the query:

serf query -tag role=builder -node=master images

We successfully implemented a continuous delivery environment using Visual Studio Team Services build and release management, Docker, and Serf. After the hackfest, NEC Solution Innovators could implement a rollback facility quite quickly!

Figure 9. Hack team members

Hack team members

Conclusion

In Japan, starting a DevOps journey within a traditional enterprise company is a significant challenge. We included many stakeholders and used value stream mapping to showcase the issues, and successfully came to an agreement on changing their processes and rules. Also, we successfully implemented DevOps practices using an OSS stack on Azure, using Java, PostgreSQL, Docker, Terraform, Serf, Visual Studio Team Services, and Linux.

Tomohiro Fukui of NEC Solution Innovators described the outcome:

“After finishing value stream mapping, we felt like we were in the 1990s era because we had a bunch of old manual processes, rules, and old technologies. But now I feel like we’re in 2017. We are going to implement DevOps practices applying these technologies in the production environment. Then, we’ll get a short lead time soon!”

Additional resources