Prodata Logo

Prodata and Microsoft worked together to architect and build an IoT backend in the cloud to process telemetry and financial events from Prodata’s bus ticketing system in São Paulo, Brazil.

Key technologies

Core team

  • Marcelo Ceccon – Software Architect, Prodata
  • Eduardo Bergantini Pinto – Developer, Prodata
  • Allan Targino – Technical Evangelist, Microsoft Brazil

Customer profile

Prodata Mobility is a Brazilian company that operates in the electronic ticketing segment. It was founded in 1991 as a spin-off of the Belgian company Prodata Mobility Systems, which was founded in 1971 and was a pioneer in providing contactless card systems.

The company’s focus is on delivering ticketing and fleet management solutions to public transportation organizations. To date, it has sold more than 100,000 components of ticketing equipment, and is the leader in this market in Brazil and South America. It is present in 13 Brazilian state capitals and more than 100 cities, as well as in Colombia, Argentina, and Ecuador. It has offices in several cities and a manufacturing plant in Hortolândia, Sao Paulo.

Problem statement

Ticketing is used worldwide in public transportation for the advance purchase of recorded travel credits in special devices, especially in cards. It is an integrated solution of hardware and software developed to control and manage the collection of the costs charged in public passenger transportation systems. It helps the customer to minimize fraud, get more retention, and provide security to end users.

Prodata is responsible for delivering the ticketing system for São Paulo’s public transportation system. This system’s backend is mostly on-premises, hosted in Prodata’s datacenter.

Bus ticketing devices have two basic operations:

  • Financial transactions (such as charging for a ride or refilling the passenger card)
  • Monitoring and telemetry

In Sao Paulo, most of the population goes to work between 6:00 AM and 8:00 AM, and returns home about 5:00 PM or 6:00 PM. This behavior is shown in the following chart.


Average load of passengers per day

Average load of passengers per day


A great part of the population receives, as a benefit from their jobs, electronic credits to use for public transportation. Between the 31st of the month and the 5th of the following month, this benefit is delivered to employees, who can refill their cards inside the buses. Following is a chart displaying the average load of refills during a specific month.


Average load of card refills per month

Average load of card refills per month


The following table contains some additional numbers about the current situation in the Sao Paulo system.

Current situation Number
Vehicles with refill system 6,000
Monthly financial transactions 900,000
Messages from financial transactions 3,600,000
Messages from monitoring 8,640,000
Monthly financial volume (estimated) R$ 50,000,000.00 (~US$ 15,000,000.00)


Public transportation companies are now both expanding their operation and modernizing the current system, which causes more vehicles to be handled by Prodata’s ticketing backend. The estimated situation in six months is shown in this table.

Next 6 months estimated situation Number
Vehicles with refill system 14,000
Monthly financial transactions 2,250,000
Messages from financial transactions 9,000,000
Messages from monitoring 21,600,000
Monthly financial volume (estimated) R$ 120,000,000.00 (~US$ 37,000,000.00)


As we can see, it is expected that the volume of messages will be 2.5 times the current value. The current architecture and infrastructure suit their needs well, but the estimated volume increase in the next six months is critical, and they are unsure if they will be able to meet it with an on-premises solution. Even if they do, cost implications could be prohibitive. This led them to explore a cloud platform as a more efficient alternative.

Solution and steps

Based on this scalability requirement, we recreated the critical and sensitive parts of their infrastructure on Azure. Today the ingestion of data is done by using a TCP Gateway, which parses the received messages in a proprietary protocol and converts them into HTTP. This is to enable the communication with their own backend using REST and also with the public transportation company backend.

This TCP Gateway on the cloud has been designed as shown in the following diagram.


Azure architecture diagram

Azure architecture diagram


The green elements are the unchanged Prodata devices/APIs. Azure IoT Hub and Azure Functions were used to deliver the scalability need, replacing specific parts of their original architecture, while the table and Power BI were used to record logs and deliver the presentation layer for the telemetry dashboard. The gray elements represent the error handling structure.

The following requirements had to be considered to complete this change successfully:

  • Guarantee device communication directly with the cloud.
  • Reuse the existing Prodata code for parsing and sending messages to the backend.
  • Ensure that the delay between the request made by the device and its response is less than 1 second (today it is around 400 ms).
  • Allow bi-directional communication with all devices.


Technical delivery

The devices used on the buses are a proprietary board with an adapted Linux distro running on it. Every board has a 3-GB modem connected to it, which makes the device capable of connecting directly to the Internet, so we didn’t need to use a gateway to connect them. At the time of the hackfest, we didn’t have a physical device to test and develop the application with IoT Hub, so we decided to use simulated devices by using the Azure IoT Device SDK written in C#.

The technical capabilities for the devices include the following.

General:

  • CPU Freescale i.MX6 Cortex-A9 Dual Core @ 1 GHz
  • 1-GB RAM
  • 1-GB Flash storage

Network connectivity:

  • 2 GB (GSM/GPRS/EDGE - 850/900/1800/1900 MHz)
  • 3 GB (UMTS/HSPA - 800/850/900/1700/1900/2100 MHz)
  • IEEE 802.11 b/g/n 2.4 GHz
  • IEEE 802.3 10/100/1000 Mbps

IO connectivity:

  • EIA/TIA RS232-C
  • EIA/TIA RS485 Full/Half Duplex
  • CAN
  • USB
  • Contact-less: ISO14443 + ICode: ISO14443 Type A/B e ISO15693, freq. 1356 MHz, modulation ASK 100%

Other capabilities:

  • GPS

The refill requests can happen at any time of the day. Monitoring messages are sent every five minutes to show the ticketing machine status and health. Following are some details about message sizes and average bandwidth consumption.

Message Message size Average bandwidth
Refill request 200 bytes 180 MB/month
Refill response 200 bytes 180 MB/month
Monitoring message 50 bytes 432 MB/month


Because messages are small, we built a system where multiple devices were simulated at once by using the Azure IoT Device SDK with a threads pool that we implemented. We chose MQTT to be the application protocol due to its great performance, security, and reliability.

The concern with security in this scenario is critical because it is fundamental to protecting the charging system against financial fraud. Today the customer has already implemented a security system in their infrastructure that involves the use of encryption in conjunction with hardware security modules (HSMs). This ensures that the message payload is encrypted all the way to its final destination. Additionally, when using IoT Hub with MQTT, the Azure IoT SDK itself is responsible for generating temporary authentication tokens for the devices.

After the device has registered itself and sent a message to IoT Hub, we need to handle it and do all the backend processes. Because we didn’t need hot path processing and we need a request/response pattern, we chose not to use Azure Stream Analytics, even if it could be extended with User-Defined Functions (UDF) in JavaScript. We used Azure Functions instead, so that whenever a message arrives at IoT Hub, it triggers the communication with Prodata’s backend. Using Functions also guarantees the proper scalability for the data ingestion process—a basic requirement for this migration.

Currently, Azure Functions can only be triggered by Event Hubs and not directly by IoT Hub. Because IoT Hub has an interface compatible with Event Hubs, the integration is possible.

To make an easy integration between IoT Hub and Azure Functions, you can just replace the field “Hostname” with “Endpoint” in your connection string that you are going to use in Functions.

For example, if you have the following:

`HostName=yoururl;SharedAccessKeyName=iothubowner;SharedAccessKey=yourkey`

Change it to:

`Endpoint=yoururl;SharedAccessKeyName=iothubowner;SharedAccessKey=yourkey`


After configuring an event hub as a trigger source for the function, it was able to read the message payload but not its header and other properties because, by default, the Azure Functions input read data from the event hub is plain text. Because we needed to read all this information, deserialize JSON, and send cloud-to-device messages, we added the following dependencies on the Azure Functions definition:

Azure Functions definition

    {
      "frameworks": {
        "net46": {
          "dependencies": {
            "Microsoft.Azure.Devices": "1.2.5",
            "Newtonsoft.Json": "10.0.2",
            "WindowsAzure.ServiceBus": "4.0.0"
          }
        }
      }
    }


By using the WindowsAzure.ServiceBus assembly, we could read all the messages delivered to IoT Hub’s internal event hub. A typical internal message (and its schema) is shown in the following snippet. The respective documentation can be found at EventData Class.

Typical Event Hubs message

    {
        "SerializedSizeInBytes": 54,
        "Offset": "2880",
        "PartitionKey": null,
        "SequenceNumber": 9,
        "EnqueuedTimeUtc": "2017-04-11T17:29:10.075Z",
        "Properties": {},
        "SystemProperties": {
            "x-opt-sequence-number": 9,
            "x-opt-offset": "2880",
            "x-opt-enqueued-time": "2017-04-11T17:29:10.075Z",
            "iothub-connection-device-id": "6c08208b-bda2-4219-ae9a-84807cd08c5b",
            "iothub-connection-auth-method": "{ \"scope\": \"device\", \"type\": \"sas\", \"issuer\": \"iothub\" }",
            "iothub-connection-auth-generation-id": "636275198245423531",
            "EnqueuedTimeUtc": "2017-04-11T17:29:10.075Z",
            "SequenceNumber": 9,
            "Offset": "2880",
            "correlation-id": "083dc261-7504-49e7-9d9e-c3a433a75498"
        }
    }


We were especially interested in SystemProperties, which the IoT Hub SDK fills automatically when sending a message. The two properties we were going to use were:

  • iothub-connection-device-id”: The device ID that sent the current message. We are going to use this value to send a response to the device through the same function that has received the request.
  • correlation-id”: This property can be interpreted as the message ID. It can be used to confirm to which message the response belongs when the cloud sends a message to the device.

When writing the function code body by using C# and trying to reuse Prodata existing code, we faced some problems. First, we could not use .cs files, only .csx files, a by-design implementation of Azure Functions in C#. Second, Functions does not support namespaces. To solve that, we made a simple PowerShell script to remove all namespaces from the old classes (with the premise that there wouldn’t be any conflict between models) and save it as a big .csx file. Problem solved!

The result function is shown here. Comments over each important step were also added:

Azure Function body

    //Uses the legacy external file:
    #load "ProdataBackend.csx"

    using System;
    using System.Text;
    //Important reference to use the complete Event Hub functionally:
    using Microsoft.ServiceBus.Messaging;
    //Another important one, used to send Cloud to Devices messages from IoT Hub:
    using Microsoft.Azure.Devices;
    using Newtonsoft.Json;

    static string connectionString = "HERE GOES YOUR CONNECTION STRING";
    static ServiceClient serviceClient;
    //Some other state variables were omitted

    public static void Run(EventData inputEventData, ICollector<LogRegistry> outputTable, TraceWriter log)
    {
        //Initialize:
        console = log;
        table = outputTable;
        eventData = inputEventData;

        //Verbose:
        Log("Event Data", LogType.Information);

        try
        {
            //Get input data payload from Event hub:
            var bodyBytes = inputEventData.GetBytes();
            //Thiese properties has information about the device ID and message ID sent to IoT Hub:
            var deviceId = inputEventData.SystemProperties["iothub-connection-device-id"].ToString();
            var correlationId = inputEventData.SystemProperties["correlation-id"].ToString();

            //Call Prodata backend:
            var result = ProdataBackendUtil.Process(bodyBytes);

            //Send the response to device:
            serviceClient = ServiceClient.CreateFromConnectionString(connectionString);
            SendCloudToDeviceMessageAsync(deviceId, correlationId, result).Wait();
        }
        //Error handling and Logging was collapsed to simplify the view
        catch{}
    }

    private async static Task SendCloudToDeviceMessageAsync(string deviceId, string correlationId, byte[] messageBytes)
    {
        try
        {
            await serviceClient.SendAsync(deviceId, new Message(messageBytes) { CorrelationId = correlationId });
        }
        catch{}
        finally
        {
            await serviceClient.CloseAsync();
            serviceClient.Dispose();
        }
    }

    public static void Log(string message, LogType type)
    {
        var eventSerialized = JsonConvert.SerializeObject(eventData);

        //Console log:
        //console.Info($"Message: {message}");
        //console.Info($"Event: {eventSerialized}");

        //Azure Table log:
        table.Add(
            new LogRegistry()
            {
                PartitionKey = "AzureFunctionLogs",
                RowKey = Guid.NewGuid().ToString(),
                LogType = type.ToString(),
                Message = message,
                EventData = eventSerialized
            }
            );
    }

    public enum LogType
    {
        Information,
        Warning,
        Critical
    }

    public class LogRegistry
    {
        public string PartitionKey { get; set; }
        public string RowKey { get; set; }
        public string LogType { get; set; }
        public string Message { get; set; }
        public string EventData { get; set; }
    }


For error handling, because we didn’t have write permission on this internal event hub, we could not mark the message as read (resetting the offset to a previous state). The solution for this was to put the message with errors in a dedicated event hub used just for this purpose. Another function could be triggered by that to do the retry pattern (with some time-to-live/retry count) and apply this behavior.

Conclusion

The architecture provided both efficiency and scalability, achieving the initial requirements for a partial cloud migration. All the tests that used this architecture were realized with resources provisioned on Azure’s West US datacenter. Because IoT Hub is not available in Brazil South yet, this is creating an issue for Prodata due to the delay requirements. As an immediate alternative, we tested an ASP.NET Web API on Brazil South to do the Functions role. It was a very interesting alternative as a message ingestion system for their devices, especially due to its intrinsic synchronous nature—even if the Web API doesn’t have all the capabilities that IoT Hub has for device management, managed scalability, and message retention.

Load tests were done from a single computer using about 40 threads, simulating both telemetry and refill requests. The Visual Studio Team Services load testing service was also used to check the performance of the proposed architecture. Both tests yielded good results in terms of performance and scalability.

In addition to the functional tests performed during the hackfest, a complete test that used the Prodata homologation environment is necessary and will be conducted shortly. One future task will also be to create a Power BI dashboard to allow visualization of collected and processed data.

Partner feedback

“Even our brief experience with Microsoft IoT Hub and Azure during the IoT hackfest was enough to show not only the vast potential of the tools but also the leverage of Microsoft’s expertise available through direct contact with their team. It has inspired the developers to research every potential application of these technologies to apply to our product line and business operation.” —Marcelo Ceccon, Software Architect, Prodata


Photo from our team at the end of hackfest: We did it!

Team Photo