Microsoft teamed up with GTI, an Internet of Things (IoT) platform and service provider headquartered in Beijing, to develop the “big data” platform used in its Smart City solution, which enables data-driven city operation.

Customer profile

GTI is a leading IoT platform and service provider in China. Its business includes chipset and hardware manufacturing, communication networking, application development, and the operation of IoT-dedicated cloud services, which are broadly used in scenarios centered on the concept “Smart Device, Better Life.”

Pain point

GTI Smart City solution is built for the city management in Smart first world, and covers many scenarios, like security, gas, electronic, parking, lamp, etc, which is running in on-promises datacenter powered by Huawei solution.

The key pain point is around “big data.” To best serve the customer requirements, GTI plans to build a end-to-end data platform that 1) can scalably collect and store data from a few hundred to hundreds of thousands of sensors and devices in real-time, 2) has efficient tools for hot and cold data analysis and presentation of insights, and 3) can be accessed by future platform partners.

Another pain point is the cost of storing and analyzing historical data generated by the solution and platform operations. All data produced before the latest month needs to be stored in a persistent state. GTI wants a one-stop data warehouse that can be accessed from multiple locations in China.

Solution

  1. Use Azure IoT Hub to collect data generated by anywhere from a few hundred to millions of sensors and devices and send that data to the uniform management platform for daily monitoring. Such monitoring enables real-time notifications and alerts, which can speed up the response time for urgent events and lower the costs on the operation side.

  2. Use Azure Storage, especially DocumentDB, to store all historical data generated by all channels and customers, which helps to lower storage costs and enable future data insights.

In the Smart City solution, GTI will use the following Microsoft technologies on Azure:

  • Azure IoT Hub
  • Stream Analytics
  • SQL Database
  • DocumentDB

Architecture

The GTI Smart City solution architecture can be represented as follows:

  • Sensors will be deployed in each customer project for monitoring and controlling the selected business scenarios, such as illumination management, parking, facility monitoring, and water monitoring. The data collected include device status, temperature, presure, speed, and customized information.

  • A Gateway is the hub that connects all sensors via private protocol (for security needs) and sends the sensor data to the Azure IoT Hub every 5, 10, or 15 seconds, depending on the definition of each business scenario. This means that GTI needs to 1) maintain reliable communicaiton when the number of sensors grows quickly and 2) support a broad set of protocols, such as MQTT, for future expansion.

  • Stream Analytics will transfer the data to various storage options, including SQL Server, DocumentDB, and even files.

  • DocumentDB will also be used to store the historical data transferred from SQL Server. The data will be uploaded weekly and stored as JSON documents; typical volume for a set of 100 sensors is 500,000 documents.

GTI Smart City overview architecture

Smart City overview architecture

GTI Smart City data platform architecture

Smart City data platform architecture

Device used and code artifacts

The v-team, composed of the Microsoft China DX TE team and the GTI dev team, split the engagement into four segments:

  • GTI Gateway IoT SDK integration
  • Historical data storage in DocumentDB
  • Deployment of IoT Hub, Stream Analytic, and DocumentDB
  • Query sample in Stream Analytics

Gateway IoT SDK integration

In this segment, the GTI team focused on the Azure IoT Hub SDK integration with GTI Gateways, which is based on the Raspberry Pi 3 running Linux. The Microsoft China DX team helped the GTI hardware engineer to code the Azure IoT Hub SDK in their client, set up and register the gateway with the Azure IoT Hub, connect with the Azure IoT Hub, and send all sensor data to the Azure IoT Hub. The C code is as follows:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <termios.h>
#include <fcntl.h>

#include "azure_c_shared_utility/platform.h"
#include "azure_c_shared_utility/threadapi.h"
#include "azure_c_shared_utility/crt_abstractions.h"
#include "iothub_client.h"
#include "iothub_message.h"
#include "iothubtransportamqp.h"


#ifdef MBED_BUILD_TIMESTAMP
#include "certs.h"
#endif // MBED_BUILD_TIMESTAMP

static const char* connectionString = "HostName=imsehAccess.azure-devices.cn;DeviceId=Device_Raspberry;SharedAccessKey=XXXXXX";

static int callbackCounter;
static bool g_continueRunning;
static unsigned char msgText[1024];
static char propText[1024];
static int confirmBuf[1024];
static int length;
static int serialFd;
#define MESSAGE_COUNT       5
#define DOWORK_LOOP_NUM     3

typedef struct EVENT_INSTANCE_TAG
{
    IOTHUB_MESSAGE_HANDLE messageHandle;
    size_t messageTrackingId;  // For tracking the messages within the user callback.
} EVENT_INSTANCE;

// 主程序入口
void iothub_client_sample_amqp_run(void)
{
    IOTHUB_CLIENT_LL_HANDLE iotHubClientHandle;

    g_continueRunning = true;
    srand((unsigned int)time(NULL));
    double avgWindSpeed = 10.0;

    callbackCounter = 0;
    int receiveContext = 0;

    (void)printf("Starting the IoTHub client sample AMQP...\r\n");

    if (platform_init() != 0)
    {
        printf("Failed to initialize the platform.\r\n");
    }
    else
    {
        // IoT Client 初始化
        if ((iotHubClientHandle = IoTHubClient_LL_CreateFromConnectionString(connectionString, AMQP_Protocol)) == NULL)
        {
            (void)printf("ERROR: iotHubClientHandle is NULL!\r\n");
        }
        else
        {
            bool traceOn = true;
            IoTHubClient_LL_SetOption(iotHubClientHandle, "logtrace", &traceOn);

#ifdef MBED_BUILD_TIMESTAMP
            // 安装证书
            if (IoTHubClient_LL_SetOption(iotHubClientHandle, "TrustedCerts", certificates) != IOTHUB_CLIENT_OK)
            {
                printf("failure to set option \"TrustedCerts\"\r\n");
            }
#endif // MBED_BUILD_TIMESTAMP

            // 设置回调函数
            if (IoTHubClient_LL_SetMessageCallback(iotHubClientHandle, ReceiveMessageCallback, &receiveContext) != IOTHUB_CLIENT_OK)
            {
                (void)printf("ERROR: IoTHubClient_SetMessageCallback..........FAILED!\r\n");
            }
            else
            {
                (void)printf("IoTHubClient_SetMessageCallback...successful.\r\n");

                // 消息发送
                size_t iterator = 0;

                do
                {

                    printf("the length of msg text is %d\n",length);
                    if (length <= 1)
                    {
                        printf("broken data,drop it\n");
                        sleep(1);
                        continue;	
                    }

                    IOTHUB_MESSAGE_HANDLE messageHandle = NULL;

                    // 消息转化成字节流
                    if ((messageHandle = IoTHubMessage_CreateFromByteArray((const unsigned char*)msgText, length)) == NULL)
                    {
                        (void)printf("ERROR: iotHubMessageHandle is NULL!\r\n");
                    }
                    else
                    {
                        MAP_HANDLE propMap = IoTHubMessage_Properties(messageHandle);
                        (void)sprintf_s(propText, sizeof(propText), "PropMsg_%zu", iterator);
                        if (Map_AddOrUpdate(propMap, "PropName", propText) != MAP_OK)
                        {
                            (void)printf("ERROR: Map_AddOrUpdate Failed!\r\n");
                        }

                        // 异步发送消息
                        if (IoTHubClient_LL_SendEventAsync(iotHubClientHandle, messageHandle, SendConfirmationCallback, messageHandle) != IOTHUB_CLIENT_OK)
                        {
                            (void)printf("ERROR: IoTHubClient_SendEventAsync..........FAILED!\r\n");
                        }
                        else
                        {
                            (void)printf("IoTHubClient_SendEventAsync accepted data for transmission to IoT Hub.\r\n");
                        }
                    }

                    IoTHubClient_LL_DoWork(iotHubClientHandle);
                    ThreadAPI_Sleep(1000);

                    iterator++;
                } while (g_continueRunning);

                (void)printf("iothub_client_sample_mqtt has gotten quit message, call DoWork %d more time to complete final sending...\r\n", DOWORK_LOOP_NUM);
                for (size_t index = 0; index < DOWORK_LOOP_NUM; index++)
                {
                    IoTHubClient_LL_DoWork(iotHubClientHandle);
                    ThreadAPI_Sleep(1);
                }
            }
            IoTHubClient_LL_Destroy(iotHubClientHandle);
        }
        platform_deinit();
    }
}

// 串口数据读取
void* read_config()
{

	while(1)
	{
        printf("begin to read serialFd data... ...\n");
        length = read(serialFd, msgText, sizeof(msgText));
        printf("the length of data recv serial: %d \n",length);
        if(length == -1)
        {  
            perror("read");
        }
        printf("recv of data from serial:");
        int num = 0;            
        for(num = 0; num < length; num++)
        {           
            printf("%hhx ",msgText[num]);
        }
        printf("\n");
    
        printf("read serialFd data done !!! \n");
		sleep(2);
	}
}

// 串口初始化
int serial_config()
{
    struct termios opt; 
    serialFd = open("/dev/ttyAMA0", O_RDWR | O_NOCTTY);
    if(serialFd== -1)
    {
        perror("return of open serial is 0\n");
	close(serialFd);
        exit(0);
    }

    tcgetattr(serialFd, &opt);      
    cfsetispeed(&opt, B115200);
    cfsetospeed(&opt, B115200);
    
    if(tcsetattr(serialFd, TCSANOW, &opt) != 0 )
    {     
       perror("tcsetattr error");
       close(serialFd);
       return -1;
    }
    
    opt.c_cflag &= ~CSIZE;  
    opt.c_cflag |= CS8;   
    opt.c_cflag &= ~CSTOPB; 
    opt.c_cflag &= ~PARENB; 
    opt.c_iflag &= ~INPCK;
    opt.c_cflag |=  HUPCL;
    opt.c_cflag |= (CLOCAL | CREAD);
 
    opt.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG); 
    opt.c_oflag &= ~OPOST;

    opt.c_oflag &= ~(ONLCR | OCRNL); 
 
    opt.c_iflag &= ~(ICRNL | INLCR);
    opt.c_iflag &= ~(IXON | IXOFF | IXANY);  

    opt.c_cc[VTIME] = 1;
    opt.c_cc[VMIN] = 255;
    
    tcflush(serialFd, TCIOFLUSH);
 
    printf("serial configure complete\n");
    
    if(tcsetattr(serialFd, TCSANOW, &opt) != 0)
    {
        perror("serial configure error");
        close(serialFd);
        return -1;
    }
    return serialFd;
}

Historical data storage in DocumentDB

In this segment, the GTI architecture builds the one-stop data warehouse for the persistent storage of historical data, which are transfered from existing data sources and platforms. The Microsoft China DX team helped the architecture consultant set up and configure DocumentDB, and coded the data to be transferred to DocumentDB. The C# code is as follows:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using System.Net;
using Microsoft.Azure.Documents;
using Microsoft.Azure.Documents.Client;
using Newtonsoft.Json;
using DocumentDB;
using System.Threading;
using System.Configuration;

namespace TestDocumentDB
{
    class Program
    {
        private static string EndpointUri;
        private static string PrimaryKey;
     
        static void Main(string[] args)
        {
            try
            {
                //获取连接字符串
                GetConfig();
            }
            catch (Exception ex)
            {

                Console.WriteLine("请先检查连接字符串是否正确");
            }
            try
            {
                DocumentManager doc = new DocumentManager();
                //创建数据和集合
                doc.GetStartedDemo(EndpointUri,PrimaryKey).Wait();
                //上传数据
                int i = 0;
                while (true)
                {
                    DataLog datalog = new DataLog() { ID = Guid.NewGuid(), NodeOID = "1234"+i, SensorName = "温湿度", DValue =new Random().Next(0,45), CreateTime = DateTime.Now };//实际使用中调用接口获取最新的历史数据
                    doc.CreateDocument<DataLog>(datalog,datalog.ID.ToString()).Wait();

                    i ++;
                    Thread.Sleep(1000);
                }

              //  p.GetStartedDemo().Wait();
            }
            catch (DocumentClientException de)
            {
                Exception baseException = de.GetBaseException();
                Console.WriteLine("{0} error occurred: {1}, Message: {2}", de.StatusCode, de.Message, baseException.Message);
            }
            catch (Exception e)
            {
                Exception baseException = e.GetBaseException();
                Console.WriteLine("Error: {0}, Message: {1}", e.Message, baseException.Message);
            }
            finally
            {
                Console.WriteLine("End of demo, press any key to exit.");
                Console.ReadKey();
            }
        }
      
        private static void GetConfig()
        {
            if ( !string.IsNullOrEmpty( ConfigurationManager.AppSettings["EndpointUri"]))
            {
                EndpointUri = ConfigurationManager.AppSettings["EndpointUri"];
            }
            if (!string.IsNullOrEmpty(ConfigurationManager.AppSettings["PrimaryKey"]))
            {
                PrimaryKey = ConfigurationManager.AppSettings["PrimaryKey"];
            }
        }

    }
}

Deployment of Azure IoT Hub, Stream Analytics, and Azure SQL Database

In this segment, the GTI team deployed Azure IoT Hub, Stream Analytics, and Azure SQL Database on Azure. As a proof of concept for Azure IoT Services, GTI created one free Azure IoT Hub, deployed Stream Analytics, and created its device-status database on Azure SQL Database. In the future, more outputs, such as Azure Blob storage, will be added depending on the business growth.

Azure IoT Hub configuration in Azure

Azure IoT Hub Configuration in Azure

Stream Analytics input streaming configuration in Azure

SA input streaming configuration in Azure

Stream Analytics output streaming configuration in Azure

SA output streaming configuration in Azure

DocumentDB configuration in Azure

DocumentDB configuration in Azure

Query sample in Stream Analytics

In this segment, a query sample is attached to capture some typical data fields in real-time.

//查询设备的实时状态值
SELECT
    nodeid,sensorid,ovalue,value,updatetime
INTO
    imsehAccessOutput
FROM
    imsehAccessInput

Opportunities created and going forward

Technical cooperation going forward

GTI is planning to begin building the micro-device IoT platform in 2017, which will provide “building block” capabilities for developers or ISVs who want to set up end-to-end IoT solutions easily. This platform will be a kind of “PaaS + SaaS” model and use the same “big data” solution described earlier.

Micro-device IoT platform

Micro-device IoT platform

Business opportunities driven

The teams of GTI and Microsoft China set up a scalable working model for broader customer engagement and to help users in China with digital transformation. As of December 2016, two opportunities are underway.

Co-selling to a real-estate customer

Co-selling to a Microsoft customer

Government opportunity engagment

Government opportunity engagement

Great team

The executives meeting

Executives meeting

The GTI hackfest team

GTI hackfest team

The Smart City solution in the Microsoft Technical Center

Smart City solution in Microsoft Technical Center

Special thanks to the GTI team, the Microsoft China DX Technical Evangelist team, and the Microsoft Audience Evangelism team.

This project team included the following participants:

  • Haibo Lv – CTO, GTI
  • Lin Zhang – Tech VP, GTI
  • Junjie Bai – Backend Architect, GTI
  • He Li – Hardware Engineer, GTI
  • Yan Zhang – Audience Evangelism Manager, Microsoft
  • Michael Li – Technical Evangelist, Microsoft
  • Shijun Liu – Technical Evangelist, Microsoft
  • Tory Xu – Sr. Technical Evangelist, Microsoft

Additional resources