Microsoft joined Quarta Technologies and TechnoKom in a hackfest to develop the IoT gateway for a TechnoKom solution, used as the transport layer for iQFreeze.

In this report, we describe the process and what we did before and during the hackfest, including:

  • How we created a base prototype of software as a service (SaaS)-ready architecture for the iQFreeze solution.
  • How to achieve a working prototype of a SaaS-ready IoT solution within a few hours by using Microsoft Azure IoT Hub and Azure.
  • Learnings and architecture questions from the process.

Key technologies used

Core team

The hackfest team consisted of two groups:

  • Development
    • Two TechnoKom developers
    • Alex Belotserkovskiy – Technical Evangelist, Microsoft
  • Business and planning
    • Kirill Konyagin – Director of Business Development, Quarta
    • Victor Kiselev – Partner Business Evangelist, Microsoft

Customer profile

Quarta Technologies is a Moscow-based solution provider for emerging and innovation projects. It was established in 1997 and became a Windows Embedded distributor in 2004. It drives Windows Embedded in the Russian market and also organizes an annual Windows Embedded Day conference in Russia with attendees from local and global IoT markets.

Technokom Corporate Group has grown from a small production company to a major supplier of fleet management solutions hardware and software. The Russia-based business has a strong market presence in products and applications.

The iQFreeze solution collects, processes, and transmits information about the condition of the cargo and refrigeration units in real time using the TechnoKom AutoGRAPH fleet management system hardware as the source of data and transport to data servers. It was introduced to the market in May 2016.

At the time of the hackfest, it was a classic client-server based product with analytics with a high cost of scaling. The idea behind the project was to propose and test a platform as a service (PaaS)-based architecture for the solution to finally create a SaaS-ready solution for a simple, low-cost way to solve the scale problem and provide a new way to market and sell the product.


Figure 1. AutoGRAPH device

Figure 1. AutoGRAPH device


Pre-hackfest activity

An IoT hackfest can be a challenge because it usually involves not only software, but hardware-related questions, and it can be difficult to simulate a real environment. Proper preparation eases the challenge, and therefore we had a lot of discussions during the month before the hackfest. To understand the context, here are some of the important questions and answers that were raised.

Question: Is the system already working? What is the state of the devices? What are they capable of? What are they not capable of?
Answer: The system is not new, it has been functional for years. Because the system is intended for use with vehicles from different manufacturers/partners, the hardware is different as well. Almost all of the devices are alike in one detail—changing the firmware even on a small portion of these devices is not possible because they are already in production. It prevents using a field gateway and/or IoT client.

Question: What is the protocol the devices use to communicate with the current backend?
Answer: TechnoKom implemented their own binary protocol. Any change to the current state will break current systems.

Question: Is it possible to implement a new backend that will work without breaking changes to the current system, and change the workflow to the new version?
Answer: It is a challenge, but still possible. It will involve a thorough testing and should be out of scope of such a short activity as a hackfest.

Question: Is there a need to control devices remotely?
Answer: Not at the moment.

We decided to increase the hackfest scope by adding the IoT gateway and data processing pipeline. In short, it meant that we had to break down the current monolithic application into a more microservice-oriented architecture, scalable and PaaS-based. Because we had the real data stream from AutoGRAPH, we removed all of the device-related questions as well.

Instead of one server with the Win32 application, we decided to use cloud hosting for the IoT Protocol Gateway that listens to the TCP connections and forwards data to the data pipeline (Azure IoT Hub => Azure Stream Analytics => Azure Storage, Azure Machine Learning, and Azure SQL Database)—in five hours.

Problem statement

We had five hours and three developers to build a working prototype of a SaaS-ready IoT solution using IoT Hub and Azure. Our six tasks:

  • Assess the original architecture and propose a new Azure PaaS-based architecture.
  • Implement the test gateway (console app) as a proof of concept.
  • Assess the possible options to implement it as the cloud gateway (for example, to use Azure IoT Gateway, or Cloud Services, and so on).
  • Migrate the gateway to the cloud.
  • Develop the code to connect the gateway with the data pipeline.
  • Equip the prototype with the monitoring suite (Azure Application Insights).

Solution and steps

Architecture before

First, we discussed how to decouple the monolithic approach into the PaaS-based architecture that was implemented long ago for the solution. The old architecture was the classical two-tier architecture “client-server” with C++ built as a monolithic application running in the Windows Server environment. The server job is to receive data packets from the monitoring system devices installed on vehicles and save the received data in the local store, as well as be the frontend for external customers who want access to the data. The code has a modular structure at the level of source code files—network communication module, file storage, data decoding, and the database module.

Data between the server and the onboard devices is transferred (generally over mobile comm channels) using the proprietary binary TCP protocol (without the use of application layer protocols). The client starts with the handshake packet and the server should answer with the ACK packet.


Figure 2. Architecture - before

Figure 2. Architecture - before


The server software uses a set of system calls Win32API. Data is stored in the binary form, and each connected device has its own separate file. Data is being stored without any modification. External customers access data using the special client software.

One challenge with such architecture is that there is no way to scale it in a faster way than to install a new server, set the load balancing up, and so on.

We took our time to prototype the architecture that would provide the same functionality in a scalable and agile way.

Architecture after

During the hackfest, we noticed there was no need for the old application anymore. It served the gateway function, saved data without modification, and did some extra tasks. We decided to replace that application with Azure services. We considered a few options, including Azure IoT Gateway and a custom-made gateway. In the end, we decided to implement the custom-made gateway and host it on Azure Cloud Services.


Figure 3. Architecture - after

Figure 3. Architecture - after


A Cloud Services-hosted gateway should function the same as the Win32 app in the old architecture, but in a more scalable and agile way because it is hosted behind the load balancer and it is simple and fast to scale it, even automatically. The gateway forwards data to the IoT Hub that is connected to Stream Analytics, which sends data to Azure Storage, Machine Learning, and a SQL database.

Azure Cloud Services allowed us to implement the cloud architecture and make the legacy Win32 application more cloud-friendly, highly available, and scalable. In a few hours, we were able to decouple the complex monolithic application.

The original architecture was almost completely changed.

Technical delivery

Development

We started the development process by implementing the on-premises version of the gateway. It is important to start with a small step and not do everything in the cloud immediately. The cloud has its benefits, but if you develop something new, there are some nuances that can make the prototyping more difficult—such as setting the endpoints, troubleshooting a remote project, and deploying and re-deploying the solution, which can take up to 10 minutes each time.

We divided the task of implementing the IoT gateway into subtasks:

  • Gateway listens to the TCP port.
  • Gateway establishes the connection with the device.
  • Gateway gets the stream from the device.
  • Gateway sends the ACK packet back.
  • Gateway parses the protocol.
  • Gateway sends data to the IoT Hub.

Although nothing is new or innovative in implementing a TCP server and sending the ACK packet back, we had one issue with the local gateway. Taking into account that we were in a Microsoft office, the network perimeter could make the testing more difficult. We could not manage the network perimeter in the office, but we could in the cloud, so we wanted to avoid any settings that would not be needed in the final solution.

What did we do? We used ngrok!

Ngrok is a nice project that provides a way to implement secure tunnels to the localhost. We downloaded ngrok, registered the account (needed for TCP tunnels), and everything started to work. Just work! We got the stream and were able to send the ACK packet back with only one command.


Figure 4. Ngrok command line interface is really simple

Figure 4. Use of Ngrok command line interface


While developing, we had two additional steps—writing to the local storage, and getting the Endpoint and the Port when deployed in the cloud. Because Cloud Services has its own storage and addressing models, we decided to go a little deeper with that.

Writing to the Cloud Services local storage

Of course, Azure has better ways to persist data, but the original architecture stores data locally, so we decided at least to test it. By default, everything that Cloud Services is using (derived from the RoleEntryPoint class) will not have permission to write to the file system (no way to write to “C:/temp” or “d:/Program Files/MyCoolStorage”). To be able to do that, you should set the special local storage area first and use Azure API to write to disk in the Cloud Services role.

You will need to configure some space in Local Storage from your ServiceDefinition.csdef by giving that space a name and a size, and setting if the data should survive basic role recycles (cleanOnRoleRecycle setting). One minus: cleanOnRoleRecycle does not guarantee your data will survive.

We had some spare time to test it, and it worked. The code is pretty simple.


const string azureLocalResourceNameFromServiceDefinition = "LocalStorage1";

var azureLocalResource = RoleEnvironment.GetLocalResource(azureLocalResourceNameFromServiceDefinition);

var filepath = azureLocalResource.RootPath + "telemetry.txt";

Byte[] bytes = new Byte[514];
String data = null;

while (true)
{
	TcpClient client = server.AcceptTcpClient();
	data = null;
	int i;
	NetworkStream stream = client.GetStream();
		
	while ((i = stream.Read(bytes, 0, bytes.Length)) != 0)
	{
 		…
	 	System.IO.File.AppendAllText(filepath, BitConverter.ToString(bytes));
	 	…
 	}
 	
	client.Close();
 }


Data persisted through the recycling, so it may be a good way to store some temporary data.

Getting Endpoint and Port for listener

Taking into account the dynamic nature of Cloud Services (and almost everything in the cloud), it may be a bad idea to just say, “hey, listen to the 127.0.0.1”. Cloud Services has its own methods to get the Endpoint defined.

First, you need to define the Endpoint in the CS settings.


Figure 5. Configuration of Cloud Services Local Storage area

Figure 5. Configuration of Cloud Services Local Storage area


  IPEndPoint instEndpoint = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints\["TCPEndpoint"\].IPEndpoint;
  IPAddress localAddr = IPAddress.Parse(instEndpoint.Address.ToString());
  TcpListener server = new TcpListener(localAddr, instEndpoint.Port);


Now, even if we scale the CS, it will work and listen to the needed TCP port and Endpoint. Load Balancer will distribute the requests, so it is quite important in our case. Also important is that our scenario is stateless, so Cloud Services is a perfect candidate for implementation.

We discussed the use of Azure Service Fabric, but the timing was tight so we went with CS instead.

We made the local gateway work. Now it was time to deploy it into the cloud. The Visual Studio tooling for Azure is tightly integrated with Azure, and the deployment process is clear and simple.


Figure 6. Publish menu

Figure 6. Publish menu


Published and working!

The few issues we had were successfully solved by setting up the Remote Desktop and Application Insights.

We had no issues with forwarding messages to IoT Hub and Stream Analytics, and just used the official code samples.

Conclusion

When we planned our IoT hackfest, we worried that a single day would not be enough for developing something that really worked, especially in a situation with such a challenging environment and requirements (a lot of legacy devices, proprietary TCP/IP protocol, a partner dev team that came from another area just for the hackfest, and, of course, the core monolithic Win32 application written in C++).

What we learned is that PaaS is really a great way to:

  • Prototype the solution.
  • Make it scalable and agile from the very beginning.
  • Learn new things.

In a short time, we were able to prototype an end-to-end IoT solution using real data.

In one day, we achieved everything that we agreed on as a target and were ready to go further. Our big bet was to make the monolithic Win32 application loosely coupled, and, at the end of the day, get rid of the monolithic approach and go with one that uses more microservices and is more agile. Now we are sure that the solution is scalable and it can be done with very few actions. And that is a great outcome!

What is great as well is that the hackfest is not just a development event where geeks from two companies have some fun with code; it is a great time to communicate and go through the issues together—talk, learn, and share the experience.

We are grateful to the TechnoKom and Quarta teams for many reasons:

  • They were eager to rearchitect the solution and make it more cloud-friendly.
  • There was a lot of discussion on how to improve both their solutions and Microsoft—we were able to gather some technical feedback.
  • Their spirit is awesome.

The solution is not completely ready for production. What is important is that, in one day, we were able to engage a lot of technical and business masterminds to create a foundation for our further development—to test the solution with Azure Service Fabric, to migrate to AMQP or MQTT, and to develop the Machine Learning component and gather insights from the data. We decided to “go and fail fast” and, as we learned, we did not fail in any aspect.

Additional resources