N Solid

Blue Frog Robotics is a 2-year-old startup based in Paris that designs and develops Buddy, which its creators call the first affordable companion robot for the family. They say Buddy is a social robot that connects, protects, and interacts with each family member.

During its Crowdfunding campaign in 2015, Blue Frog reached its goal of $100,000 within 24 hours, and raised $620,000 by the end of the campaign. The presales have continued on the Blue Frog website, bringing the total amount to approximately $1.5 million.

Buddy’s distribution will be overseen by Blue Frog Robotics but via a different store network for each country. It is important, therefore, for Blue Frog to put in place a means of knowing the actual deployment of Buddy, as well as its uses, throughout the world in order to ensure the optimum service level according to this volume.

A cloud solution will take care of storing the collected data (usage, sensors, and so on) and hosting the different services need for the proper functioning of the first 2,200 robots that will be deployed next year. It is mandatory to build a robust and scalable back end to ensure the usability of the robot, but it is also important to design the architecture in order to have the most economic solution with the optimum level of services.

Blue Frog Robotics and Microsoft DX France worked together during a hackfest to design and build the solution with a production perspective.

The project team included the following:

  • Jean-Michel Mourier – CTO, Blue Frog Robotics
  • Benoit Pironnet – R&D Engineer, Blue Frog Robotics
  • Valentin Macheret – Engineer, Blue Frog Robotics
  • Marc Hullcellz – Engineer, Blue Frog Robotics
  • Benjamin Guinebertière – Technical Evangelist, Microsoft DX France
  • Benoit Le Pichon – Technical Evangelist, Microsoft DX France

About BUDDY

Buddy

Buddy is:

  • 22 inches/56 cm tall, 5 kg in weight, and with a battery life of 8-10 hours.
  • Outfitted with a brain that is an integrated, 8-inch smart tablet with built-in Wi-Fi and Bluetooth.
  • Operating on an Arduino platform designed to make all the mechanical parts of the robot respond to the “brain” inquiries.
  • Fully mobile, with three wheels and a plethora of sensors, allowing the robot to travel, learn, and interact with the world around it.
  • Using 3D vision. A conventional camera, IR camera, and IR laser projector allows Buddy to easily track and interpret hand, arm, and head movements, distinguish among objects, faces, animals, plants, etc., and gauge the depth of objects in sight.
  • Equipped with state-of-the-art artificial intelligence technology, including face and object detection, recognition, and tracking.
  • Able to hear, speak, and make coinciding head movements.
  • Full of personality and reacts to his environment through a range of expressions that allow him to better engage with his family. He has a high level of kawaii, or cuteness, whether he’s happy, grumpy, angry, or sad. If it’s too cold, his teeth may even chatter!

Buddy was purposely built with popular development tools such as Arduino, OpenCV, and Unity 3D to allow as many developers as possible to collaborate with Blue Frog. They are also developing a complete API to enable the developer community to start working with Buddy. In addition to software, developers will be able to build custom hardware-based solutions that Buddy can interact with.

Buddy also comes with a mobile companion app for members of the family in order to let them contact Buddy from a distance, with features like CCTV, video call, or remote control. Developed with Unity 3D and C# language, the app will be available on iOS, Android, and Windows.

Hardware

Data typology

Buddy sends two types of data to the cloud:

  • Technical and usage data
  • Context and profile data

Context and profile data represents the set of data generated by Buddy or by the companion app used for synchronization or backup purposes (cartography data, user profile, and so on). This data is configured for reading and writing with a low solicitation rate, and stored in JSON format.

Technical and usage data represents all the data collected by the tablet and the Arduino card on the technical indicators of the robot such as the battery level, the location, or the activity of the servo motors. This data is sent to a high frequency, is non-editable, and is stored in JSON format. The data is also encoded in base64 to reduce the length of the message.

Designing the architecture

The main criteria for architectural designs are price, scaling, and security. Price is a main focus in our case because Buddy is sold at a fixed price and no surcharge is applied subsequently to the consumer. It is therefore mandatory that the cost of infrastructure be as low as possible.

Scaling must be considered for all technical bricks that require it in order to avoid performance problems at the time of the global deployment of the robots.

With regard to security, the Blue Frog team considers the safety of the family’s personal data to be a top priority. On Buddy’s software side, different types of data encryption and isolation have been implemented. To guarantee this protection on the cloud side, the transferred data was anonymized and we must ensure that data can be read or written only by authorized systems.

Ascend

The implementation of the chosen architecture was divided into three main domains:

  • Blob storage: How to save and restore data to/from blob storage through shared access signatures (SAS).
  • Message collection: Collect messages such as log events, commands, and monitoring, and create Power BI reports.
  • Automation: Creating all Azure Resource Manager files and automation scripts to allow continuous deployment of the infrastructure.

Message collection implementation: from the robot to Power BI

  • Event Hubs has been chosen as the data ingestor for the solution.

To be allowed to send messages to the event hub, each Buddy must obtain a shared access signature. The SAS is delivered by an Azure Function.

Here is the code of the Azure Function:

#r "Microsoft.ServiceBus"

using System.Net;
using Microsoft.ServiceBus;
using System.Configuration;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("Generate Shared Access Signature for Event Hub");

    // parse query parameter
    string publisherName = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "publisherName", true) == 0)
        .Value;

    string tokenTimeToLiveParam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "tokenTimeToLive", true) == 0)
        .Value;

    // Get request body
    dynamic data = await req.Content.ReadAsAsync<object>();

    // Set name to query string or body data
    publisherName = publisherName ?? data?.publisherName;
    tokenTimeToLiveParam = tokenTimeToLiveParam ?? data?.tokenTimeToLive;

    if( publisherName == null)
        return req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a publisherName on the query string or in the request body");

    TimeSpan tokenTimeToLive;

    if( tokenTimeToLiveParam == null)
        {tokenTimeToLive = TimeSpan.FromMinutes(60);}
    else
        {tokenTimeToLive = TimeSpan.FromMinutes(double.Parse(tokenTimeToLiveParam));}

    var appSettings = ConfigurationManager.AppSettings;
    var sas = CreateForHttpSender(
                                appSettings["EH_1_SAS_PolicyName"],
                                appSettings["EH_1_SAS_Key"],
                                appSettings["EH_1_SAS_Namespace"], 
                                appSettings["EH_1_SAS_HubName"], 
                                publisherName, 
                                tokenTimeToLive);

    if (string.IsNullOrEmpty(sas))
            {return req.CreateResponse(HttpStatusCode.NoContent, "No SaS found!");}
        else
            {return req.CreateResponse(HttpStatusCode.OK, sas);}
}

public static string CreateForHttpSender(string senderKeyName, string senderKey, string serviceNamespace, string hubName, string publisherName, TimeSpan tokenTimeToLive)
{ 
            var serviceUri = ServiceBusEnvironment.CreateServiceUri("https", serviceNamespace, String.Format("{0}/publishers/{1}/messages", hubName, publisherName))
                .ToString()
                .Trim('/');
            return SharedAccessSignatureTokenProvider.GetSharedAccessSignature(senderKeyName, senderKey, serviceUri, tokenTimeToLive);
}

The robot sends data to Event Hub.

Via the Unity 3D app, the robot gets a SAS token that is given by the Azure Function created above.

    ```       
   /// Methode call with StartCoroutine();
    private IEnumerator GetSaS(string publisherName)
    {
        WWW wwwSas = new WWW(string.Format("https://tbrblobsasfunapp.azurewebsites.net/api/EventHubSasTokenCSharp?code=YourAccessKey&publisherName={0}", publisherName));

        yield return wwwSas;

        // check for errors
        if (wwwSas.error == null)
        {
            SaS = wwwSas.text.Trim(new Char[] { ' ', '\"' });
            Debug.Log("Get SaS OK: " + wwwSas.text.Trim(new Char[] { ' ', '\"' }));
        }
        else
        {
            Debug.Log("Get SaS Error: " + wwwSas.error);
        }
    }
    ```

The function has the event hub management key:

The data is sent through a Web Request, too, from Unity. Here is the equivalent in curl:

The data is received by Azure Event Hub:

It is then processed by Azure Stream Analytics:

Azure Stream Analytics sends

A copy of all the data to blob storage

The data can be searched with Azure Data Lake Analytics:

A summary of the data to Power BI

Estimating the infrastructure cost

To estimate the cost of the solution, we have to determine some key elements:

  • Deployed robots count: 2,200
  • Active companion app count per robot: 2
  • Technical and usage data average size: 100ko
  • Context and profile data average size: 3Mo
  • Technical and usage message sending frequency: 5 min
  • Context and profile data sync per days per robot/device: 1
  • SAS tokens expiration: 1 hour
  • Azure Functions average duration: 100ms
  • Azure Functions memory consumption: 512 MB

We also considered an activity 24/7 for a month of 31 days. So, we are looking at the worst case in terms of consumption.

Blob storage cost for context and profile data

We decide to use hot locally redundant storage for all blob storage.

This data set is composed of an average of five files of various sizes and formats (.json, .png, cartography). Azure storage pricing is composed of two elements: storage and access. In our case, the storage price is $0.024 per GB per month. So we have to calculate the amount of data stored for all robots:

Storage price = deployed robots count * context and profile data average size * storage price per GB = 2,200 * 0.003 * 0.024 = $0.1584.

Access price is different per operation type and it is really difficult to determine the exact nature and count of operations each month. So we decided to apply the most expensive price.

Estimated number of operations per robot or device: 10

Total number of operations per month = deployed robots count * (active companion app count per robot + 1) * estimated number of operations per robot/device * 31 = 2,200 * 4 * 10 * 31 = 2,728,000 operations.

Total access price = total number of operations per month / 10,000 * 0.054 = $14.7312.

Blob storage for context and profile data cost = $14.8896

Event Hubs cost

Event Hubs is designed to ingest millions of events per second so you can process and analyze massive amounts of data from connected devices and applications.

Event Hubs pricing is based on number of deployed throughput units and number of ingress events. To determine the number of throughput units, we have to calculate the MB/s ingress sending by the robots.

MB/s ingress = technical and usage data average size in MB * deployed robots count / technical and usage message sending frequency in seconds = 0.1 * 2,200 / 300 = 0.7333 MB/s.

For one throughput unit, the limit is 1 MB/s ingress. Our ingress is 27% lower than the limit, so we only need 1 throughput unit. We also have to determine the ingress events count sending by the robots.

Ingress events count = minutes per days / technical and usage message sending frequency * deployed robots count * days = 1,440 / 5 * 2,200 * 31 = 19,641,600 events

Ingress events price = ingress events count / 1,000,000 * 0.028 = $0.55.

One standard throughput unit costs ~ $22 per month, so the Event Hubs total cost is:

Event Hub cost = 22 + 0.55 = $22.55.

Azure Functions cost

We used Azure Functions to deliver shared access signatures for accessing Blob storage for context and profile data and Event Hub.

Azure Functions pricing is based on execution time and total executions. Our SAS token expires after 1 hour, so each function is called 24 times per day per robots or app. Functions for blob access is called by robots and app. And functions for Event Hub access is called only by robots.

Total executions = ((deployed robots count + deployed robots count * active companion app count per robot) * 24 * days) + (deployed robots count * 24 * days) = ((2,200 + 2,200 * 2) * 24 * 31) + (2,200 * 24 * 31) = 4,910,400 + 1,636,800 = 6,547,200.

Total executions cost = (6,547,200 - 1,000,000) / 1,000,000 * 0.20 = $1.1094.

On the Azure portal, the monitor tool for Azure Functions helped us to determine the average execution duration. In our case, it is 80 ms. And the memory size of our functions is 512 MB. With this information, we can calculate the execution time cost:

Resource consumption (seconds) = executions * execution duration (seconds) = 6,547,200 * 0.08 = 523,776 s.

Resource consumption (GB-s) = resource consumption converted to GB’s * execution time (seconds) = 512/1 024 * 523,776 = 261,888 GB-s.

Billable resource consumption = resource consumption - monthly free grant = 261,888 - 400,000 = 0 GB-s.

Monthly resource consumption cost = $0 !!!

Azure Functions cost = total executions cost + monthly resource consumption cost = $1.1094.

Stream Analytics cost

Stream Analytics is priced by volume of data processed and the number of streaming units required to process it.

Volume of data processed by the streaming job = technical and usage data average size in MB * deployed robots count * technical and usage message sending frequency per days * days = 0.1 * 2,200 * 288 * 31 = 1,964,160 MB = 1,964.16 GB.

Volume of data processed cost = 1,964.16 * 0.001 = $1.9641.

We just need 1 streaming unit for this volume of robots:

Streaming unit cost = 0.031 * 24 * 31 = $23.064.

Stream Analytics cost = volume of data processed cost + streaming unit cost = $25.0281.

Blob storage cost for technical and usage data

This Blob storage contains only data sent by Stream Analytics. The total size of the blob file in storage is equal to the volume of data processed by Stream Analytics.

Storage price = volume of data processed by the streaming job * storage price per GB = 1,964.16 * 0.024 = $47.1398.

To determine access price, we consider 1 access per message processed by ingress events in Event Hubs for update access.

Access price = ingress events count * operations cost (per 10,000) = 19,641,600 / 10,000 * 0.004 = $7.60.

Blob storage for technical and usage data cost = storage price + access price = $54.7398

Hidden cost: bandwidth

Bandwidth refers to data moving in and out of Azure datacenters other than those explicitly covered by the content delivery network or ExpressRoute pricing. We have to consider the download of the data by the mobile application. We suppose that each application synchronized the entire file twice a month.

Volume of data downloaded = context and profile data average size in GB * 2 * active companion app count per robot * deployed robots count = 0.003 * 3 * 2 * 2,200 = 39.6 GB.

Outbound data transfers cost = (volume of data downloaded - monthly free grant) * transfers cost per GB = (39.6 - 5) * 0.087 = $2,5752.

There are a lot of small data transfers between Azure and the robot—for example, getting the SAS key. We decided to augment this cost by 50% to cover these transfers.

Bandwidth cost = outbound data transfers cost + 50% = 2.5752 + 1.2876 = $3.8628

Total cost

  • Blob storage for context and profile data cost = $14.8896
  • Event Hub cost = $22.55
  • Azure Functions cost = $1.1094
  • Stream Analytics cost = $25.0281
  • Blob storage for technical and usage data cost = $54.7398
  • Bandwidth cost = $3.8628

Total cost = $122.1797

Total cost per Buddy per month = 122.1797 / 2,200 = $0.05554

Conclusion

In just three days we were able to set up a complete back end, allowing Blue Frog Robotics to oversee the deployment and commissioning of the robots that will soon be deployed in production.

The implemented architecture offers great flexibility for future development because it allows the engineering team to add new intermediate services, but also by automating the entire deployment and architecture in many regions.

And most importantly, we were able to prove that the cost of the back-end solution was very modest per robot, which allows Blue Frog Robotics to offer an affordable robot for many without compromising its business model.

If you are interested in deploying the same architecture, or just want to pick one part, all the source code is available on GitHub here: https://github.com/DXFrance/BlueFrogHackfest.

Additional resources