“Hey oven, what’s the temperature of the chicken inside you?”

Solutions that allow you to talk with your kitchen appliances are becoming more and more common these days. Iomote has developed “plug and play” hardware that stores data gathered by sensors and enables users to communicate with their kitchens. The test customer is an important Italian kitchen manufacturer. The project consists of a smart kitchen equipped with several sensors that make it possible for people to interact by voice.

Customer profile

Iomote is a leading company in Internet of Things (IoT) hardware in Italy, designing and producing professional IoT solutions for European companies. Iomote decided to use Microsoft Azure as the cloud back-end provider for all their devices to provide their customers with reliable end-to-end solutions. From hardware to the cloud, they leverage the flexibility and wide range of tools available on the Azure platform. Most Iomote customers are system integrators who are looking for flexible IoT devices. They want a solution that can scale from pilot projects to volumes.

In this specific project, Iomote decided to use Windows 10 IoT Core because of the team’s expertise in .NET programming. Regarding the cloud, Azure was the best option thanks to the ease of use of Azure IoT Hub and also because the team wanted to leverage Language Understanding Intelligent Service (LUIS), a powerful natural-language processing service.

Problem statement

This project had two goals: to store the data gathered by sensors in the kitchen, and to record the interaction between people using their voices and the “smart kitchen.” The intent was to have the application answer a question like “What’s the temperature?” using voice via a Raspberry Pi 3 and the speech recognition APIs of Cognitive Services.

The proof of concept was developed during a hackfest in Madrid. For reference, Iomote used only one temperature sensor, which was connected to the custom gateway. In the final version, they plan to have several different sensors and actuators so that they can provide a useful quantity of data gathered from smart plugs, temperature and humidity sensors, and motion sensors.

The final functionalities managed through the app will include the following:

  • Domotic safety: motion sensors to monitor drawers and cupboards (containing knives and detergents, for example) that are dangerous for children, and sensors to detect gas and water leaks
  • Domotic comfort: RGB light bulbs that can light up in a selected color in the event of an alarm, with the app turning on the lights when something is wrong
  • Domotic energy saving: plugs on kitchen appliances

Photo of Iomote kitchen

Sensors and gateway

The sensor used during the Hackfest was a PAT02-B temperature sensor produced by Philio Technology. It was connected to an Iomote custom field gateway, which is certified for Azure IoT Suite. Iomote decided to certify this hardware before and independently from the development of this specific project to ensure their customers the security of and seamless integration into the Azure IoT ecosystem. To obtain the certification, Iomote simply followed the standard process required by Microsoft.

Data was gathered from the sensor and then sent to IoT Hub. This part of the project was developed quickly thanks to the certification and was deployed before the event to generate some data for immediate use in testing.

To run the Universal Windows Plaform (UWP) app and use both the speech recognition APIs and LUIS, a Raspberry Pi 3 running Windows 10 IoT Core was used.

Challenges

The main challenge in the development of the UWP app was to enable voice-mediated interaction between the user and the sensors.

Some hardware and software integration was necessary for the development of the UWP app running on the Raspberry Pi 3, including the following:

  • Speech-to-Text and Text-To-Speech APIs, used to recognize the question asked by the user and to use a voice to provide the answer.
  • Connection with LUIS, so that the request received could be understood.
  • Interaction with telemetry data, so that the correct answer could be given.

Another interesting aspect that arose during the hackfest was the creation of a dashboard to register the devices and to manage things like firmware updates, rebooting, and the current status. This dashboard will be useful for both Iomote and its customers because it will give Iomote a general overview of all the devices so that it can plan their management. Meanwhile, customers will have a different view of the current status thanks to the single sensors installed in their own kitchens, such as battery level, values like temperature, and error messages.

Solution and steps

Telemetry transmission

All the standard IoT architecture for the telemetry data—such as temperature—from the sensors to the database was developed in advance. Therefore, the focus during the hackfest was the development of the UWP app that would interact with the user.

Diagram of Iomote telemetry transmission

Voice interaction: UWP application

At first the app was developed locally on a PC. The team developed a simple GUI with a button as trigger to test the speech recognition APIs, both Speech-to-Text and Text-to-Speech. After the initial tests, they removed the button. They tested whether the application was listening and waiting for a specific command as a trigger, such as “Hey, Azure.”

The real scenario requires a headless Raspberry Pi 3. The GUI in the following image is useful only for testing purposes.

Screen shot of Iomote voice-interaction test

The GUI of the UWP app GUI is in Italian. The source code is available in a public repository; see the Additional Resources section at the end of this document for the link.

Understanding the question: LUIS

The use of LUIS made it possible for the app to understand the commands. The model was trained to enable the app to recognize two intents: humidity and temperature. After that, the LUIS APIs were added to the UWP application, and everything said after the trigger command—“Hey, Azure”—was sent to and analyzed by LUIS. The JSON answer was parsed locally. To determine the correct answer for the application to give the user, the intent was recognized and a threshold was set up for the confidence level. Because Iomote was working on a project for an Italian customer, the Italian language was used for modeling and testing.

Screen shot of intents in LUIS

As you can see from the following code, LUIS is called inside the MainPage.xaml.cs code-behind page of the UWP, receiving the input and responding with an output. The output is based on the confidence of the LUIS response; a command is sent to IoT Hub to get the temperature of the sensor.

private async Task luisTask(string queryString)
{
    int evalScore = 0;
    await speechRecognizer.ContinuousRecognitionSession.StopAsync();
    Debug.WriteLine("[LUIS] - Stop recognition");

    using (var client = new HttpClient())
    {
        string uri =
          "https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/[your app id]?subscription-key=[your subscription]&verbose=true&q=" + queryString;
        HttpResponseMessage msg = await client.GetAsync(uri);
        if (msg.IsSuccessStatusCode)
        {
            var jsonResponse = await msg.Content.ReadAsStringAsync();
            var luisData = JsonConvert.DeserializeObject<LUISResponse>(jsonResponse);
            string[] mTypes = { "Misura", "Ambiente" };
            evalScore = evalString(luisData, "LetturaSensore", mTypes);
        }

    }
    if (evalScore > 1)
    {
        Debug.WriteLine("[IOTHUB] - Executing command! :-)");

        var response = await HTTPHelper.Send(HttpMethod.Get, "http://localhost:4530/api/orchestrator/LetturaSensore");

        speakSafe("Here's what I got. " + response);
        speakSafe("Ho eseguito il comando");
    }
    else
    {
        Debug.WriteLine("[IOTHUB] - NOT Executing command, low score! :-(");
        speakSafe("Mi spiace, non ho capito");
    }
}

Retrieving the value: device-management web app

During the testing process, Iomote developed a web app that showed all the sensors connected to IoT Hub. The aim of this web app was to make it easier to manage the devices: rebooting, firmware updates, and so on. Device twin REST APIs and direct methods were used.

Screen shot of Iomote device-management web app

To retrive information about the devices, a web API was called:

private async Task<IEnumerable<Device>> GetDevices()
{
    IEnumerable<Device> iotHubDevices;
    List<Device> devicesDictionary = new List<Device>();

    registryManager = RegistryManager.CreateFromConnectionString(connString);

    // var x = await registryManager.GetDeviceAsync("Device00000001");
    iotHubDevices = await registryManager.GetDevicesAsync(1000000);

    foreach (var device in iotHubDevices)
    {
        devicesDictionary.Add(device);
    }

    return devicesDictionary;
}

The same API has a method to update information and data to the devices using device twin REST APIs:

public async Task Put(string id, [FromBody]DesiredProperties patch)
{
    var input = Newtonsoft.Json.JsonConvert.SerializeObject(patch).Replace("\"", "'");
    var p = @"{'properties': {'desired': " + input + "}"; //{ "send_time":"10","log_time":"5","power_mode":"0","digital_in_mode":"0"}
    var twin = await registryManager.GetTwinAsync(id);
    await registryManager.UpdateTwinAsync(twin.DeviceId, p , twin.ETag);
}

The source code is available in a public repository; see the Additional Resources section at the end of this document for the link.

Answering the question

To answer the question “What’s the temperature now?” the UWP app interacted with the web app, which was responsible for sending a direct method to the specific device. The UWP app retrieved the current temperature directly from the device and was able to answer the question, providing the current temperature value to the user through the Text-to-Speech API.

In the following diagram, you can see the whole architecture of the project.

Architecture diagram

Security

Security was provided by IoT Hub. This allowed the registration and management of every device in the solution. In fact, the web app was developed for management of the devices and used the device twin REST APIs and the IoT Hub direct methods to ensure a friendly and secure connection and communication among users, devices, and the back end.

Conclusion

The customers appreciated the usefulness of the hackfest in Madrid. In four days, they developed a complete proof of concept with the support of the Microsoft technical evangelists. The customers achieved their goal and found the connection with the product team helpful. They will now be able to identify the best solution and give feedback.

Iomote has built strong skills in a short time. They can use these skills to accelerate the development of this project as well as future ones.

Customer quote

“Leveraging on the Microsoft Azure and AI services, we are reinventing the way users interact with products and processes. Having such advanced technologies available as a service on the cloud is a game changer for any industry.”

Gabriele Allegria presenting the Iomote project during the Madrid hackfest

Photo of presentation during hackfest

Going forward

The real scenario will include more and different types of sensors and actuators. The architecture will be adjusted to include all of them and to ensure efficiency. The LUIS model will be improved by adding new intents and identifying potential phrases that can be recognized by the system.

The UWP app will run on a Raspberry Pi 3 installed in the kitchen. The app will be able to communicate with all the sensors and actuators and interact with the user by voice.

The command that triggers interaction with the application will be adjusted according to the customer’s preferences. The whole solution will be perfectly integrated in the “smart kitchen.”

Additional resources

The GitHub repository guendas/IomoteIoTHackfest contains the source code for the UWP application (Speech 01) and the web app (DeviceManagement). The console application is used to simulate more devices (DeviceSimulator and DeviceSimulatorNode) and the web API for the communication between the web app and the UWP app (Orchestrator).

For more information about the company, go to the official Iomote website.

Technologies used

  • PAT02-B Z-Wave temperature sensor
  • Microphone for listening to commands
  • Custom gateway, produced by Iomote and certified for Azure IoT Suite
  • Raspberry Pi 3 running Windows 10 IoT Core
  • UWP app running on Raspberry Pi 3, using speech recognition APIs to interact with sensors and people
  • Azure IoT Hub, for managing the various devices and storing the telemetry data retrieved from the sensors
  • Azure Stream Analytics, for performing simple manipulations of the data and sending it to the database
  • Azure SQL Database
  • Cognitive Services: LUIS was used to understand the commands received from the Raspberry Pi 3

Project team

  • Gabriele Allegria – CTO, Iomote s.r.l.
  • Guenda Sciancalepore – Technical Evangelist, Microsoft Italy
  • Valery Jacobs – Technical Evangelist, Microsoft Netherlands