Smart Helmet, an IoT solution built by Cybercom, helps prevent accidents and make construction sites safer by providing live, wireless monitoring, panic buttons, and multiple sensors (such as temperature or location). The solution sends telemetry from a low-power wide-area network (LPWAN) to the cloud and processes the data in Microsoft Azure.

The Smart Helmet project involved the following individuals:

  • Cybercom
    • Rafał Rudnicki
    • Andrzej Nielipiński
    • Piotr Artychowski
    • Adam Makowski
    • Michal Walędziak
    • Marcin Goluch
    • Maciej Bożyk
    • Piotr Kuszewski
    • Mariusz Ragus
  • Microsoft
    • Tomasz Kopacz

During scrum meeting

Photo during scrum meeting

Discussion about the Azure IoT Gateway SDK

Photo of discussion about Azure IoT Gateway SDK

Problem statement

According to the Occupational Safety and Health Administration (OSHA) in the United States, one in ten construction-site workers are injured every year. The Bureau of Labor Statistics logs 150,000 construction-site accident injuries each year. Although many of these injuries are caused by simple accidents, others could have been prevented if we could detect dangerous behaviors or situations on construction sites. Injuries are a huge cost because, according to WorkersCompensation.com, about 15 percent of costs are spent on workers who were injured while at a construction site.

In the European Union, according to Eurostat and many independent scientific studies, the rate of fatal injuries is almost steady, but the general number of accidents, unfortunately, is rising. Quoting Titas Dėjus in Accidents on construction sites and their reasons:

“594 accidents have occurred during a year in building companies, 448 of which are easy, 73 serious and 33 mortal.… The organization of works should be engineered in the technological project of works, including particular technical equipment, which saves workers from dangerous factor[s], because just using right technical equipment (not organizational) helps to avoid [an] accident or make it not as harmful as it could be. Only applying progressive complex tools—organizational, technical and technological, also preparation for working dangerous works could allow reducing the number of accidents.… During building process about 50 percent of serious and mortal accidents happened to employees whose work experience is less than one year.”

Company profile

Cybercom is an innovative IT consulting company enabling companies and organizations to benefit from the opportunities of the connected world, to enhance their competitiveness, and to achieve efficiency gains. Cybercom Poland is a group of more than 200 engineers in Łódź and Warsaw and more than 1000 people all over the world.

Solution

Together with a client, Cybercom has developed the connected helmet, the next generation of construction-site head protection, to improve safety and quality. Having the helmet connected online provides a wide range of important features, such as being able to ensure the right person with the right training is in the right place, that the wearer has the right certification and training to operate machinery and equipment, is in a safe zone and has evacuated an area at risk, is wearing the helmet, and much more.

“Health and safety are key considerations when developing this kind of IoT solution. We will discuss issues and challenges at the demonstration. The helmet is connected to a workforce management platform for data collection and presentation of statistics on a dashboard in real time.” —Rafal Rudnicki, Business Solutions & Partnership Manager, Cybercom

Communication is based on the LoRa protocol (see What is LoRa?), which allows covering an entire construction site with a single router and does not require a lot of energy. Smart Helmet can run for a long time on a single battery. Therefore, it is possible to have live reporting from wearable devices, with real-time monitoring and alarming. In short: better workforce control. Thanks to GPS tracking, the construction manager will know exactly where employees are. This functionality can be used for electronic time logging as well.

The following table summarizes the business benefits of the Smart Helmet solution.

Category KPI Use case
Safety Reduce risk and prevent injury • Proper helmet usage monitoring
• Alarm button
• High-risk activities notification
Health Reduce risk of inability to work • Temperature monitoring inside helmet
• Shock/falls events notification
Operation/processes Workforce surveillance and management procedures improvement • Estimated location of site workers
• Ad-hoc access to personnel files
• Real-time events reporting
• Training and work permissions management
Machinery and vehicles Reduce amount of accidents involving machinery • Warning machinery driver about people/workers nearby
• Warning helmet users about accessing danger zone

Architecture of the solution

Diagram of solution architecture

What is LoRa?

LoRa is one of many LPWAN or low-power network (LPN) wireless telecommunication networks designed to allow long-range communications at a low bit rate. LoRa was patented by Semtech and is used by LoRaWAN, Haystack Technologies, and Symphony Link.

Usually, LoRa provides a bitrate of 0.3–35 Kbps, but with a long range for a single router—up to 5 km.

LoRaWAN network architecture is typically laid out in a star-of-stars topology in which gateways provide a transparent bridge relaying messages between end devices and a central network server.

In our case, on construction sites, we have a simple router (based on the Intel NUC) that communicates with LoRa devices on the helmets and with a LoRa gateway based on Azure.

Technical challenges

  • How to translate messages between a helmet and Azure
  • How to apply another LPWAN stack (or even another device, such as a smartphone)
  • How to build a “fast patch” to inform about critical alert (like panic button)
  • How to store data for future long-term analysis
  • Which technologies will be more convenient (easy to use, but not too expensive) to build a real software as a service application—multiple end customers, many construction sites, separate access rights, and so forth
  • How to build a portal with “live” data
  • How to build a relatively easy mobile app to immediately deliver important information (such as worker is in wrong place, environmental conditions are wrong, or serious incident occurred)

Key components and their roles

  • Smart Helmet device with LoRa stack

    Early prototype

    Early prototype

    Almost-final device

    Almost-final device

    Final device on a mannequin during MWC 2017

    Final device on a mannequin during MWC 2017

  • LoRa router (based on Intel NUC)

    Intel NUC

    Intel NUC

  • LoRa server – in Azure, routing messages to MQTT
  • Azure IoT Gateway SDK – for receiving messages from MQTT, doing categorization, forwarding to IoT Hub
  • Azure IoT Hub – as a telemetry storage and tool for managing devices
  • Azure Stream Analytics – initial aggregation, overheating alerts in longer terms, and so on
  • Azure Data Lake Store and Azure Data Lake Analytics – long-term data storage and processing
  • Web Apps feature of Azure App Service and Azure DocumentDB – as a tool for creating UI for Works Management

Implementation

LoRa, IoT Hub, and MQTT

The LoRa server running in Azure uses the MQTT protocol. (Technically, this server is also an MQTT server.) However, it is unable to send data directly to IoT Hub. Therefore, we agreed that the best solution is to utilize the Azure IoT Gateway SDK, an additional component that can read MQTT, attach necessary information, and then forward everything to IoT Hub.

The Azure IoT Gateway SDK is a very extensible platform. We need simply clone it from its GitHub repository and compile (using cmake and Visual Studio 2017 as the C++ compiler). Because we are also using C#, it is necessary to specify that, during the build, we want to create dotnet.exe (binding for .NET). To do that, in \tools\build.cmd (the script used to compile the gateway) we need to set CMAKE_enable_dotnet_binding to ON.

Compilation of Azure IoT Gateway SDK

Compilation of Azure IoT Gateway SDK

The Azure IoT Gateway SDK can use modules built in C, Java, and Node.js (and .NET, of course).

High-level IoT gateway architecture

High-level IoT gateway architecture

What we need to do is the following:

  1. Build an ingester for the MQTT protocol (module).
  2. Add (using the standard identity-map module) information about helmet registration. Each LoRa device has a MAC address. IoT Hub allows naming of individual devices (separate keys, twins, and so on), and we need to translate the MAC address to deviceKey to do a proper authorization.
  3. Add a filter (another module) to translate important messages from LoRa (MQTT) metadata to IoT Hub metadata (enabling us to use IoT Hub routing later).
  4. Configure output (using a standard IoT Hub module).

The following code is a core part of the ingester module. (Full examples are in tkopacz/2017CyberCom_Demo on GitHub.)

public class MQTTReader : IGatewayModule, IGatewayModuleStart
{
    Broker m_broker;
    private MqttClient m_mqtt;

    public void Create(Broker broker, byte[] configuration)
    {
        Console.WriteLine("CREATE");
        m_broker = broker;
    }

    public void Destroy()
    {
    }

    public void Receive(Message received_message)
    {
    }

    public void Start()
    {
        Console.WriteLine("Starting");
        m_mqtt = new MqttClient("localhost");
        m_mqtt.MqttMsgPublishReceived += M_mqtt_MqttMsgPublishReceived;

        //Subscription for LoRa events

        m_mqtt.Subscribe(new string[] { "lora/events/#" }, new byte[] { MqttMsgBase.QOS_LEVEL_AT_MOST_ONCE });
        m_mqtt.Connect(Guid.NewGuid().ToString());
        if (!m_mqtt.IsConnected) throw new ArgumentException("No MQTT?");
        Console.WriteLine("Started");

    }

    private void M_mqtt_MqttMsgPublishReceived(object sender, uPLibrary.Networking.M2Mqtt.Messages.MqttMsgPublishEventArgs e)
    {
        //Publish messages 

        var msg = JsonConvert.DeserializeObject<TKMessage>(Encoding.UTF8.GetString(e.Message));
        Dictionary<string, string> msgProp = new Dictionary<string, string>();

        msgProp.Add("source", "tktelemetry");
        msgProp.Add("macAddress", msg.macAddress);

        Message messageToPublish = new Message(e.Message, msgProp);

        //to the next step in pipeline

        this.m_broker.Publish(messageToPublish);

    }
}

The filter is another module that we need to build. It simply decodes the message and, if the “panic” property is detected (which means that a worker pushed the panic button on a helmet), adds a “panic” attribute as metadata.

public class TKFilter : IGatewayModule, IGatewayModuleStart
{
    private Broker m_broker;
    private string m_configurationStr;

    public void Create(Broker broker, byte[] configuration)
    {
        this.m_broker = broker;
        this.m_configurationStr = System.Text.Encoding.UTF8.GetString(configuration);
    }

    public void Destroy() { }

    public void Receive (Message received_message)
    {
        var str = Encoding.UTF8.GetString(received_message.Content);

        JObject cfg = JObject.Parse(str);

        //Parse message
        if (cfg.TryGetValue("panic", out JToken jpanic))
        {

            //If panic exist, add metadata

            received_message.Properties.Add("panic", "1");
        }

        //And publish everything to the next step in the pipeline.

        m_broker.Publish(received_message);
    }

    public void Start() {  }
}

The following code configures the gateway. (Comments are in C# style.)

{
  "modules": [
    {
      //Logger – native module
      "name": "logger",
      "loader": {
        "name": "native",
        "entrypoint": {
          "module.path": "logger.dll"
        }
      },
      "args": {
        "filename": "log.txt"
      }
    },
    {
      //MQTT Reader (from LoRa server), written in C#, therefore we need dotnet loader
      "name": "dotnet_ModuleMQTTReader",
      "loader": {
        "name": "dotnet",
        "entrypoint": {
          "assembly.name": "ModuleMQTTReader",
          "entry.type": "ModuleMQTTReader.MQTTReader"
        }
      }
    },
    {
      "name": "identity_map",
      "loader": {
        "name": "native",
        "entrypoint": {
          "module.path": "identity_map.dll"
        }
      },
      "args": [
        {
          "macAddress": "01:02:03:04:05:06",
          "deviceId": "mqttlora01", //first helmet
          "deviceKey": "[...]"
        },
        {
          "macAddress": "01:02:03:04:05:07",
          "deviceId": "mqttlora02", //second helmet
          "deviceKey": "[...]"
        },
        {
          "macAddress": "01:02:03:04:05:08",
          "deviceId": "mqttlora03", //third helmet
          "deviceKey": "[...]"
        }
      ]
    },
    {
      //IoT Hub sender
      "name": "iothub",
      "loader": {
        "name": "native",
        "entrypoint": {
          "module.path": "iothub.dll"
        }
      },
      "args": {
        "IoTHubName": "[iot_dns_name_from_portal]",
        "IoTHubSuffix": "azure-devices.net",
        "Transport": "AMQP"
      }
    }

  ],
  "links": [
    {
      //All messages, from all pipeline stages will be sent to logger
      "source": "*",
      "sink": "logger"
    },
      //Messages generate by MQTT will be delivered to identity map
    {
      "source": "dotnet_ModuleMQTTReader",
      "sink": "identity_map"
    },
    {
      //And messages from identity map will be sent to Iot Hub module
      "source": "identity_map",
      "sink": "iothub"
    }
  ]
}

To run the gateway, we need the following files (also compiled from the Azure IoT Gateway SDK):

  • Main IoT gateway (required files)
    • aziotsharedutil.dll
    • gateway.dll
    • nanomsg.dll
  • Standard modules
    • identity_map.dll (identity map to translate MAC address into deviceKey)
    • logger.dll (helper logger to debug pipeline and messages)
    • dotnet.dll (binding for .NET)
    • iothub.dll (communication module with IoT Hub)
    • iothub_client.dll (C client for IoT Hub)
  • Our custom ingester and filter modules

By the way, on Azure Virtual Machines (which host both the LoRa server and our IoT gateway) we need the appropriate libraries from the Microsoft Visual C++ Redistributable for Visual Studio 2017.

Thanks to the Azure IoT Gateway SDK, Cybercom can easily extend the solution for additional protocols. It is enough to build additional ingester modules—and nothing more!

Because software can usually be created much faster than hardware, we also created a very simple “LoRa” emulator (to speed up prototyping), using the M2Mqtt library.

public class TKMessage
{
    public DateTime dt { get; set; }
    public string macAddress { get; set; }
    public double val1 { get; set; }
    public double val2 { get; set; }
}
class Program
{
    static void Main(string[] args)
    {
        int cnt=0;
        MqttClient client = new MqttClient("anymqttserver;could be mosquitto");
        client.Connect(Guid.NewGuid().ToString());
        if (!client.IsConnected) Console.WriteLine("Not Connected!");
        //Simulation of many helmets
        string[] mac = new string[] { "01:02:03:04:05:06", "01:02:03:04:05:07", "01:02:03:04:05:08" }; 
        Random rnd = new Random();
        while (true)
        {
            TKMessage msg = new TKMessage {
                dt = DateTime.Now, val1 = rnd.NextDouble(), val2 = rnd.NextDouble(),
                macAddress = mac[rnd.Next(mac.Length)]
            };
            //Publish messages
            client.Publish("lora/events",Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(msg)),MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE, false);
            Thread.Sleep(100);
        }
    }
}

To sum up: We just described a solution that gets information from helmets and sends it to IoT Hub (device to cloud). And the same pipeline can process cloud-to-device messages.

Alerts and “fast” patches: IoT Hub configuration, Azure Service Bus, and Azure Functions

In the next step, we need to design mechanisms that allow us to do fast processing of selected messages in the case of critical situation, like alerts. For that purpose, we are using two mechanisms. One is based on the routing mechanism in IoT Hub.

First, we need to add an endpoint to IoT Hub. The most convenient receiver is a Service Bus topic. (Topics and subscriptions provide a “one-to-many” form of communication, using a publish/subscribe pattern.)

Adding new endpoint to IoT Hub (Service Bus topic)

Adding new endpoint to IoT Hub (Service Bus topic)

Service Bus topics can have many subscriptions, which allows delivering a single message to multiple receivers. In the case of emergency, we can trigger multiple alerts in a different application. Because we are using the AMQP protocol here, it can be almost any “standard” application. (Some construction sites already have specialized solutions to call an ambulance immediately.)

Service Bus Explorer and multiple subscriptions for single topic

Service Bus Explorer and multiple subscriptions for single topic

Next, we need to define the rules, such as “if message contains ‘panic’ attribute, forwarded message directly to Service Bus topic.”

Adding new route to IoT Hub to get “panic” messages

Adding new route to IoT Hub to get "panic" messages

From the application perspective (to show a message on the dashboard), we need to subscribe to “alert”; when the message is delivered, we need to show some visual indicator:

class Program
{
    //Create topic importantiotmessages and subscription all (with default rule - 1=1)
    static SubscriptionClient m_clientAll = SubscriptionClient.Create("importantiotmessages", "all");
    static void Main(string[] args)
    {
        CancellationTokenSource cts = new CancellationTokenSource();
        CancellationToken ct = cts.Token;
        Task.Run(() => processAll(ct));
        Console.WriteLine("Enter = End");
        Console.ReadLine();
        cts.Cancel(); //Cancel background processing
    }
    static async void processAll(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var msg = await m_clientAll.ReceiveAsync(TimeSpan.FromSeconds(5));
            if (msg != null)
            {
                Console.WriteLine($"ALERT: {msg.MessageId}");
                await m_clientAll.CompleteAsync(msg.LockToken); //Yes, we processed "ALERT"
            }
        }
    }
}

A Service Bus topic is, in fact, a transactional queue, and we can confirm whether the message was correctly processed.

However, for some messages the logic is much more complicated. For example, if we need to check whether a worker is in an allowed area, or if some combination of telemetry is showing dangerous behaviors, we need something more sophisticated than normal attribute-based routing.

For that purpose, after a long discussion between Microsoft and Cybercom, we decide to use Azure Functions (“serverless” computing with practically unlimited scalability). As a trigger, we used IoT Hub (binding to the Event Hub layer). So, for each message, the function is called and complex logic is executed.

In our case, a special application running on the construction manager’s smartphone should receive information about possible problems. Therefor, as output for Azure Functions, we are using Azure Notification Hubs (special Azure services for delivering push notifications to Android, iOS, and Windows Mobile).

Redirecting output from Azure Functions to Notification Hubs

Redirecting output from Azure Functions to Notification Hubs

Stream Analytics, reports, and long-term events storage

For reporting and analytical purposes, we need to store events for a longer period than is possible inside IoT Hub. (The maximum retention for device-to-cloud messages is 7 days.) Furthermore, we need to calculate statistics, such as the distance each worker traveled daily. The ideal tool for doing that is Azure Stream Analytics.

Stream Analytics is a fully managed, cost-effective real-time event processing engine that helps to unlock deep insights from data. Stream Analytics makes it easy to set up real-time analytic computations on data streaming from devices, sensors, websites, social media, applications, infrastructure systems, and more. In our case, the Stream Analytics query to calculate the average temperature for a 1-hour period looks like this—

select max(Dt) as maxdt, avg(Temperature) as avgTemperature into [lakea] from [iot] timestamp by Dt group by TumblingWindow(minute,60)

—where iot is a streaming input (IoT Hub), Dt is a timestamp for events, and lakea is a Data Lake Store output where we want to store aggregated information.

Another use case for Stream Analytics is the ability to send data directly to Power BI (a tool for building sophisticated reports). From the perspective of Power BI, output from Stream Analytics looks like a table, so there is no difference in report design. (After the report is refreshed, we see new data.)

Data from Stream Analytics treated as a normal Power BI table

Data from Stream Analytics treated as a normal Power BI table

Selected messages are also forwarded to DocumentDB using Stream Analytics because they need to be easily accessible from the mobile application.

DocumentDB, portal, and fast prototyping

To speed up prototyping, Cybercom decided to use one of the samples in Azure IoT Suite (Remote Monitoring) as a starting point. The Portal is running on Azure App Services and will be integrated with Azure Active Directory. All information is stored in DocumentDB, which was chosen for two main reasons. As a NoSQL container, DocumentDB is very flexible and can keep any document or schema. We can store information about construction sites (with different metadata), workers (with diverse characteristics), and customers. Another reason is that connecting directly to a database on an Android device is easy, thanks to support for MongoDB. Therefore, some information is delivered by REST API from App Services and some is delivered directly from the database by using classic MongoDB client libraries on Android.

UI for general construction information

UI for general construction information

List of workers (actually helmets)

List of workers (actually helmets)

Detailed information about permissions, certificates, and current worker state

Detailed information about permissions, certificates, and current worker state

Architecture challenges

The last interesting part of the solution is the ability of Azure to build a software as a service application, where a single codebase is used simultaneously by many customers with full data isolation.

As for the database, Cybercom decided that each construction site will get own DocumentDB collection. They have no business need for queries that cover multiple construction sites.

Each helmet registered in IoT Hub can also have a device twin: JSON metadata saved together with the device registration, which (in our case) includes detailed information about the owner (customer), construction site, and any limitations for workers. To update a device twin we need to run the following:

RegistryManager rm = RegistryManager.CreateFromConnectionString(m_ownerConnection);
var helmet = await rm.GetDeviceAsync("mqttlora01");
Debug.WriteLine($"helmet: { helmet.ConnectionState}");

//Check if device is online – this is also very important signal for potential alert! 
if (helmet.ConnectionState != DeviceConnectionState.Connected) return;
Debug.WriteLine("Connected!");
var twinHelmet = await rm.GetTwinAsync("mqttlora01");

//Set (update) twin
var patchHelmet = new {
  properties = new {
    tags = new
    {
      devicetype = "SmartHelmet",
      location = "Warsaw",
      constructionSite = "FIN23",
      owner = "SKANSKA"
    }
};
await rm.UpdateTwinAsync("mqttlora01", JsonConvert.SerializeObject(patchHelmet), twinHelmet.ETag);

Then in any place where we get a device ID (helmet ID), we can get access to the entire metadata. Of course, it can be saved in DocumentDB as well, but having everything inside Iot Hub is more convenient. Thanks to that, we can also clearly separate devices between owners and construction sites.

Reading device twins is also not very complicated:

RegistryManager rm = RegistryManager.CreateFromConnectionString(m_ownerConnection);
//Read device twin 
var twinHelmet = await rm.GetTwinAsync("mqttlora01");

//Get all tags:
foreach (var item in twinHelmet.Tags)
{
  Debug.Write($"{item},");
}

In its next release, Cybercom wants to provide communication from cloud to helmet. The value of device twin tags is that they can be used inside the IoT Hub query language—for example, to send messages only to helmets on a single construction site.

Authorization and authentication will be based on Azure Active Directory, so we can use classic Active Directory access control lists inside Data Lake Store to additionally protect data.

Conclusion and next steps

“Thanks to cooperation with Microsoft, we can speed up the development process. If we need support, we can get answers to our questions fast and also get real help in design/code promptly. From a technology point of view, IoT Hub, Stream Analytics, Service Bus, and Azure Functions allow us to build very fast, very complex, and extensible routers for Smart Helmet. Thanks to routing capabilities, we can easily manage alerts. Using transactional capabilities in Service Bus, we can guarantee that we will know when alerts were processed. Stream Analytics allow us to calculate many live statistics based directly on received telemetry.” —Rafał Rudnicki, Cybercom

In the next version, we also will automatically update firmware and change the sampling rate for telemetry to dynamically adjust the data stream during daily operations. Also, Stream Analytics can save all data to Data Lake Store, which will enable a more-advanced analytics solution using Azure Machine Learning and the U-SQL language (part of Data Lake Analytics).