During March 2017 at Microsoft offices in Lysaker, Norway, Microsoft teamed up with Powel to create a brand-new bot named André. The aim of the hackfest was to create a voice-driven chatbot that could assist field engineers within the space of electrical grids to answer questions and provide reporting capabilities during inspections to free up their hands. The solution was built using tools from Microsoft leveraging chatbots and conversations as a platform (CaaP) technologies.

Photo of field engineer working atop substation transformer

In this hackfest, we used the following key technologies:

Additionally, the team expressed a need for getting more hands-on experience with

  • Test automation
  • Continuous integration/continuous delivery (CI/CD) scenarios

The hackfest occurred after multiple calls in which the problem area was demystified and ambitions for the hackfest were scoped.

Core team

  • Powel
    • Damian Plaza, Software Engineer
    • Jakub Hiszczyn, Software Engineer
    • Karol Stosik, Software Engineer
    • Maksymilian Jastrzebski, Software Engineer
    • Tor Hovland, Software Engineer
    • Simen Karlsen, Smart Grid-Enabler
    • Øystein Askeland, Interaction Designer
  • Microsoft
    • Pedro Dias, Sr. Technical Evangelist
    • Anders Gill, Technical Evangelist

Solution overview

Powel envisions a future in which on-site engineers can use natural voice as the main driver for computer-assisted support during inspections of power facilities. During an inspection of an electrical station or power line, using André, a Xamarin-based phone application, will free the engineers hands, providing a more fluent process than having to fill out a paper form or record data in a device application. Using the capabilities of a wearable device, such as a mobile phone, the application will automatically understand which facility is being inspected by obtaining the user’s GPS location and provide the necessary checkpoint lists and dialogs appropriate to that facility.

The hackfest focused on two distinct inspection targets:

  • Power stations
  • Power lines

The goal was to be able to design two distinctly different conversation dialogs with enough functionality to be tested in the field. Based on these dialogs, Powel would then have amassed enough experience with CaaP to enrich André and devise similar conversational applications for use in other areas.

We used Xamarin.Forms, Microsoft Bot Framework, LUIS, and Cognitive Services to develop a bot that would allow the engineer to talk to the Xamarin.Forms Android app. The app uses the Google Speech recognition engine to transform audio into text; the text would then be sent to LUIS to determine user intent at any point during the conversation. Based on the detected intent and entities, the Bot Framework would send the appropriate response back to the client app, which would read the response to the engineer, helping him stay safe by keeping him hands-free.

One feature in particular that really extends the usefulness of this bot was the integration of the Bot Framework into the Powel back-end system. We successfully integrated the bot to the systems that would, for instance, allow engineers to extract valuable information, such as

  • history of inspections at specific facilities
  • specific parts prone to constant maintenance
  • extended information about facilities
  • the quickest route to the next critical prioritized facility that needs remediation
  • a checklist of possible rectification methods

“Microsoft Bot Framework and Cognitive Services will help us in Powel to empower our customers by making their working day simpler, safer, and more efficient.” —Simen Karlsen, Smart Grid-Enabler, Powel

Customer profile

Powel logo

Powel spans Europe with a broad and sustainable customer base and a long history as a trusted supplier of software solutions for cities, municipalities, counties, the energy industry, and the contracting sector. Powel generates around US$50M revenue each year and was founded in Norway in 1996. Powel has grown to be an international corporation with staff numbering 460 with offices in these six countries in addition to Norway: Sweden, Denmark, Switzerland, Chile, Turkey, and Poland.

Powel business segments

Photo montage illustrating Powel business segments: smart enerby, asset performance, metering, water and community, construction

Problem statement

The number-one aspect that every company enforces is worker safety. Powel is no exception. Some jobs, though, are more dangerous than others. Powel has some of the largest customers in Norway that maintain and develop the electrical grids. These inspections must be done once per year for each facility. Other more in-depth technical inspections have to be done once every 10 years for all the components in the electrical grid. These inspection periods are enforced by law.

To perform an inspection, an engineer might have to climb tall masts that can create dangerous situations depending on weather and other factors. These inspections take place especially in wintertime when frost on the ground hinders the development of new grids. This forces the companies to focus on maintenance, which is also an important job. For this reason, it would be very inconvenient to keep a cell phone or a notebook at hand when you could potentially be hanging many meters above the ground.

As a result, an engineer has to remember what to check for, what potential issues might exist, whether other maintenance has been performed on this particular mast or station, what types of remedies counter any issues discovered, and so on. Based on this data, the engineer has to travel back to the main office to hand the information to his colleagues or to digitize it himself. During this transit, valuable information and specific context gathered during the inspection could be lost.

Powel requires a solution that allows the engineers to stay hands-free while still adding value to the conversation. A chatbot triggered by a simple “Hello, André” could potentially relieve the engineer of having to remember all the tasks that he is to execute during his inspection. The great thing about such a chatbot is that it could be connected to all other systems that are relevant and could provide information that the engineer needs. This was one of our goals for the hackfest: being able to develop a simple chatbot that can converse dialogues to external systems and return valuable insight that can increase efficiency during the inspection.

An inspection could consist of anything from assessing facilities and returning data regarding the types of issues discovered (presently based on a checklist) to actually fixing the issues. Common issues range from woodpeckers pecking the masts or graffiti to more-serious problems like high-voltage electrical components failing.

In summary, the chatbot will

  • avoid security issues by staying hands-free
  • provide specific insight related to particular facilities
  • manage handover by taking notes of current inspection so that no data is lost during the inspection
  • support the engineer with other relevant information that might be significant to know about or that he might have forgotten about during the inspection

Solution and steps

The hackfest was initiated by Pedro, who introduced the team to the Bot Framework, showing examples of a working bot, explaining features, and ensuring that the whole team was up to speed before we started developing.

The goal of the three-day hackfest was to create a client app in Xamarin.Forms for Android that could visualize the conversation that the engineer would be having using the Bot Framework using LUIS and the cognitive services. We created a user story in Visual Studio Team Services that consisted of multiple tasks.

The user story was as follows:

As a field worker, I want to be able to submit inspections that are OK to a bot so that I don’t have to fill out a form for that.

We defined features and tasks to make it easier to distribute work across the team. We created two repositories: one for the ASP.NET bot application, and one for the Xamarin.Forms application. We also configured CI on the Bot Framework, so that it builds and deploys to Microsoft Azure after each commit.

Kanban board

Screen shot of Kanban board

We decided to use Android for the visualization part because the Powel developers all had Android devices. Besides developing a working app, the aim was also to educate the Powel developers on Bot Framework and CaaP, so that they could bring their knowledge back to the drawing board when developing and extending their apps for scenarios that could leverage the Bot Framework and Cognitive Services. The proof-of-concept (PoC) bot developed during the hackfest would be used as a foundation that would help Powel move their own bot into production after further development.

Sketch of how a field engineer usually would inspect a power station

Whiteboard diagram of power-station inspection process

The following was achieved by the end of the hackfest:

  • Power station inspection
    • Checkpoint lists and dialogs appropriate to a specific facility
    • Selection of substations
    • Selection of discrepancy type
    • Registration of discrepancies with comments, integrated with Powel back end

The following was not implemented but was in scope for the hackfest:

  • Power line inspection
  • Translation of text from Norwegian to English
  • Upload image from mobile app
  • Triggering of conversation through voice without the use of a button

The current PoC chatbot allows the field engineer to execute an inspection by initiating a conversation through natural voice, ask for its capabilities, select a substation, and register a discrepancy of a specific type with a comment. This fulfills the feedback loop by sending the data directly to the Powel back end so that the field engineer no longer has to physically be present at the offices to register the same information.

Architecture diagram

Architecture diagram

Technical delivery

Setting up the development environment

To get started with the project, we had to install and configure the following:

Visual Studio

Language Understanding Intelligent Services (LUIS)

To use LUIS, go to the LUIS homepage and create an account. You can try it free of charge for a generous number of messages per day.

You now have everything you need to create a bot application. For a complete reference on getting started with Bot Builder, see Bot Builder SDK for .NET.

Achievements day 1

André is a Xamarin.Forms application, so we needed to implement native services to have the mobile app listen and speak through the Google speech-recognition interface. We also started work on the chat components for the app.

With LUIS, we had 10 intents described and trained with utterances. We started writing imaginary dialogs between a user and the app to get a better understanding of how the conversation would flow. We also started to experiment with LUIS entities and having intents recognize and use these entities.

For the Bot Framework, we created a bot application in the dev.framework.com space and a site in Azure for the web API.

Setting up the bot in Azure

The Bot Framework essentially operates like any ASP.NET app service. It has a controller that handles the conversation messages, so in essence, all you need to do is the following:

  1. Create or register a new bot on the Bot Framework developer site. This will give you a bot handle (GUID) and an app secret for your bot.

    Take a note of the app ID and password in the same portal.

  2. Deploy the bot code to Azure App Service.
  3. Configure your settings to include the app ID and password so that you can connect the web API to the Bot Framework.

(For a complete reference to this process, see Deploy a bot to the cloud.)

We opted to used the CloudConfigurationManager class to read the settings directly from App Service.

We set up CI in Visual Studio Team Services to build and deploy the bot to Azure on every commit. We then added LUIS integration to start working with the intents. At this stage, we had two intents implemented.

The team working hard

Photo of team working in conference room

Achievements day 2

We finalized the direct connection through the use of the WebSockets portable class library (PCL) but ended up going for HTTP because we could not make the PCL work. This was largely because of the challenge of getting WebSockets to work on Android: We weren’t sure at this stage whether we had a quick fix, so, in the interest of time, we opted out and focused our efforts on getting everything to work.

We successfully connected the bot application to the Powel back end (an API named Condition Monitoring) through the use of APIs.


A note about the back-end APIs

We connected the back-end APIs of Powel through some NuGet packages internal to Powel, and called the methods on the objects within.

Accessing these NuGet packages through Visual Studio Team Services for CI was a breeze. Because of the sensitivity of the code, we cannot share specific code examples, but in essence we used the APIs to get installation IDs by GPS location (longitude/latitude) and for submitting the report data.


We connected the mobile app to the bot and made it work. We also spent some time designing the Xamarin.Forms app and making sure that the interface and user experience were good.

The following image is a rough initial sketch of the design interface, made by using PowerPoint Storyboarding.

Low-fidelity UI storyboard

We further worked on getting NuGet packages from the Powel core APIs (specifically, the Condition Monitoring API) to be included in the build for the bot. We worked on some conversation flow plans for later implementation, and spent some more time on working with the intents and training LUIS to recognize them.

We spent a lot of time on the PromptDialog objects because we forgot to set the dialog implementation as [Serializable].

Other pain points we experienced:

  • Some issues with the .NET Framework version not matching the Bot Framework
  • WebApiConfig from the bot template made all parameter names in JSON camel-cased. This was not accepted by the internal Powel API, which took some time to figure out.

The configuration initially looked like this.

JsonConvert.DefaultSettings = () => new JsonSerializerSettings()
{
     ContractResolver = new CamelCasePropertyNamesContractResolver(),
     Formatting = Newtonsoft.Json.Formatting.Indented,
     NullValueHandling = NullValueHandling.Ignore,
}; 

We simply removed the ContractResolver line, and that seemed to fix the issue.

We ended the day by testing the bot outside to see how it handled background noise and whether it would be able to capture the intent of the field engineer. The location chosen was a noisy environment (a highway) that could very much be an actual inspection site where a field engineer would have to inspect an overhead line or a power mast. The testing succeeded with positive results:

Photo of testing mobile app directly over active highway

Achievements day 3

The final part of the hackfest setting up build automation for the Xamarin.Forms project using Visual Studio Mobile Center, as well as finding possibilities for making a conversation that flows through different dialogs.

The following diagram show the final flow of the PoC chatbot. (Sadly, we did not implement text translation because of time constraints, but we included that part in the following image).

Flow diagram

A field worker talks to André (the mobile app) in his natural language. The speech is transformed into text by the Google speech recognition engine and returned to the mobile app. The app sends the text for translation and receives fully translated text in return. The app then passes the translated text to LUIS, which captures the intent and entities and sends the response back to the mobile app. The mobile app sends the detected intent and entities (including the field worker ID) to the Bot Framework, which sends a request to the Powel back end. The appropriate response is returned to the mobile app, which translates the text into the field worker’s language and speaks the text aloud.

Handling dialogs

In the Bot Framework, a dialog works as the “brains” of the chatbot.

A dialog provides a couple of neat features that manage the state of the conversation and session management, as well as persistence that relies on an implementation of an interface named IDialog. This interface manages the state transition using IDialogContext. These principals have been included in the way we worked with the bot during the hackfest and form an integral part of the bot, enabling it to steer a conversation using the underlying stack, where it saves data to the session object, which can be accessed throughout the course of the conversation.

An example: When LUIS acknowledges the user’s intent to be happy, the Bot Framework will proceed to that particular method with the intent and respond with the appropriate response.

[LuisIntent("happy")]
public async Task Happy(IDialogContext context, LuisResult result)
{
    await context.PostAsync("That's so kind of you!");
    context.Wait(MessageReceived);
}

Another example is when LUIS acknowledges that the user wants to select a substation.

[LuisIntent("selectSubstation")]
public async Task SelectSubstation(IDialogContext context, LuisResult result)
{
    AssetService assetService = new AssetService(new AssetRepository());
    string extractedName;
    string message = "I'm sorry, I didn't catch the name of the substation.";

    if (TryExtractingSubstationName(result, out extractedName))
    {
        var asset = await assetService.GetByName(extractedName);
        substationName = asset != null ? extractedName : null;
        message = asset != null ? $"The substation {substationName} has been chosen." : $"Sorry, I didn't find a substation called '{extractedName}'.";
    }

    await context.PostAsync(message);
    context.Wait(MessageReceived);
}

More-complex dialogs were also developed in which the Bot Framework communicates with the appropriate Powel back end and lists the possible discrepancy types that the field engineer can choose from.

[LuisIntent("getDiscrepancyType")]
public async Task SelectControlPoints(IDialogContext context, LuisResult result)
{
    if (String.IsNullOrEmpty(substationName))
    {
        await context.PostAsync($"You need to tell what substation you want to select.");
        context.Wait(MessageReceived);
    }
    else
    {
        DiscrepancyTypeService discrepancyService = new DiscrepancyTypeService(new DiscrepancyTypeRepository(new InspectionRepository(new AssetRepository()), new AssetRepository()));
        AssetService assetService = new AssetService(new AssetRepository());
        InspectionService inspectionService = new InspectionService(new InspectionRepository(new AssetRepository()));
        var asset = await assetService.GetByName(substationName);
        var inspection = await inspectionService.GetNotPerformedByAssetId(asset.ObjectId);
        var discrepancyTypes = await discrepancyService.Get(asset.ObjectId, inspection.Id);

        await context.PostAsync($"You have your discrepancy types here:\n {JsonConvert.SerializeObject(discrepancyTypes)}");
        context.Wait(MessageReceived);
    }
}

Prompt dialog

The Bot Framework includes several built-in prompts that can collect input from a user. This type of prompt was used throughout the bot project, such as when André asks the field engineer whether he is satisfied with the inspection and wants to finish. This is based on the recognition of a LUIS intent of assetAllGood.

[LuisIntent("assetAllGood")]
public async Task AssetAllGood(IDialogContext context, LuisResult result)
{
    if (String.IsNullOrEmpty(substationName))
    {
        await context.PostAsync($"You need to tell what substation you want to select.");
        context.Wait(MessageReceived);
    }
    else
    {
        PromptDialog.Confirm(context, AfterConfirmAsync, "Are you sure that you want to finish this inspection?");
        await Task.FromResult<Object>(null);
    }
}

In the preceding scenario, the field engineer is asked to confirm with either “Yes” or “No” (by either tapping the appropriate button or using natural voice to respond), but many other types of data-collection inputs can be chosen:

  • Prompts.text Asks the user to enter a string of text
  • Prompts.confirm Asks the user to confirm an action
  • Prompts.number Asks the user to enter a number
  • Prompts.time Asks the user for a time or date
  • Prompts.choice Asks the user to choose from a list of choices
  • Prompts.attachment Asks the user to upload a picture or video

FormFlow

As we have seen, dialogs work as a powerful mechanism to guide a conversation, but they can get complex when the puzzle has many small pieces, such as ordering a sandwich. To simplify a guided conversation, the Bot Framework offers a much easier way of building a dialog, named FormFlow. A FormFlow dialog guides the user through filling in the form while providing help and guidance along the way. The form dialog was used when the field engineer needs to submit a discrepancy, for which he has to fill in the remaining fields, including the type of discrepancy and the comment.

[Serializable]
public class DiscrepancyFormDialogBuilder
{
    public string DiscrepancyType;
    public string Comment;

    public static IForm<DiscrepancyFormDialogBuilder> BuildForm()
    {
        return new FormBuilder<DiscrepancyFormDialogBuilder>()
            .AddRemainingFields()
            .Build();
    }

    public static IFormDialog<DiscrepancyFormDialogBuilder> Build(FormOptions options = FormOptions.PromptInStart)
    {
        // Generated a new FormDialog<T> based on IForm<BasicForm>
        return FormDialog.FromForm(BuildForm, options);
    }
};

Connector

The Bot Framework Connector is a component that provides a single API for the bot to communicate across multiple client services, such as Skype, email, and Slack.

Because we were already using Slack for communication throughout the hackfest, we decided to set up integration with Slack so that we could talk to the bot efficiently.

(Read about the steps for configuring bot channels in the “Configuring Channels” section of Getting Started with the Connector.)

In the following image, you can see how testing the conversation using Slack looks.

Screen shot of conversation with bot in Slack

We managed to set up the Slack integration by registering the bot in the dev.botframework space and choosing the connector type without the need of any coding.

Bot communication

Direct Line is a simple REST API for connecting directly to the bot. The main purpose of this API is to let developers connect custom agents to the bot to enable conversations. As mentioned during earlier, we initially set out to implement the connectivity through WebSockets in the mobile app developed in Xamarin.Forms but decided to go for HTTP because we couldn’t make the PCL work. The following code shows a simple send task.

public async Task Send<T>(T item)
{
    var client = new HttpClient();
    client.DefaultRequestHeaders.Accept.Clear();
    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _conversationStartedResponse.Token);
    var content = JsonConvert.SerializeObject(item);

    var response = await client.PostAsync($"https://directline.botframework.com/api/conversations/{_conversationStartedResponse.ConversationId}/messages", new StringContent(content, Encoding.UTF8, "application/json"));
}

Other measures implemented

In the Bot Framework, you have to directly provide BotId, MicrosoftAppId, and MicrosoftAppPassword in the web.config file to set the appropriate app settings. Because of security issues, we did not want to do this. To provide the settings in a more secure way, we got the settings values from CloudConfigurationManager or a local JSON file that has the bot settings.

We wrote the following code for the settings reader.

public string this[string index]
{
    get
    {
        var settingValue = CloudConfigurationManager.GetSetting(index);

        if (string.IsNullOrEmpty(settingValue))
            settingValue = GetByLocalJsonFile(index);

        return settingValue;
    }
}

private string GetByLocalJsonFile(string index)
{
    var path = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), "FieldWorkerBotSettings.Json");
    if (!File.Exists(path))
        return string.Empty;

    var fileContent = File.ReadAllText(path);
    if (string.IsNullOrEmpty(fileContent))
        return string.Empty;

    var json = JObject.Parse(fileContent);

    if (!json.HasValues)
        return string.Empty;

    return json[index].Value<string>();
}

This allows us to create a new LuisModelAttribute object through we pass appId and appKey in the message controller.

private static readonly LuisService _service;

static MessagesController()
{
    var settings = new SettingsReader();
    var appId = settings["LUIS:AppId"];
    var appKey = settings["LUIS:AppKey"];
    var model = new LuisModelAttribute(appId, appKey);
    _service = new LuisService(model);
}

Because there is too much code to display and explain in this report, you can find all the source code for the hackfest in the GitHub repository readyforchaos/Powel-power-station-inspection-bot.

Agent UI

The following animated GIF shows the client mobile app on the left developed in Xamarin.Forms, and the Powel Condition Monitoring app on the right displaying the status of a power station.

Animated screen shot of Agent UI

Conclusion

The results of the hackfest were brilliant, and it is mesmerizing to think that the developers from Powel went from knowing nothing about the Bot Framework to successfully building an integrated chatbot that could handle conversations across multiple systems and provide much-needed insight for the field engineers. Powel is dedicated to continuing the development of the chatbot during the next few months to include analytics and historical view of specific parts before going live, but also to extend the concept to areas and application domains of Powel other than just the power-business segment.

From a developer standpoint, all the necessary parts are there, and integrating them to create a solution felt almost too easy! The bumps that we encountered were not directly related to the Bot Framework itself, but more specific to Xamarin-related challenges.

The guidance provided for the Bot Framework and LUIS have the potential to improve, but it is sufficient to get you started, and once you’ve got the hang of it, you will spend the majority of your time designing conversational flows and less time on code.

Having all the information within the Powel back end accessible through natural speech within an app empowers the engineer to make decisions based on knowledge for which they can simply ask through the app, leading to more accurate inspection reports.

Being able to register discrepancies on the fly, ask for inspection details, and obtain other related information allows engineers to stay paperless during inspections so that they are less likely to forget an inspection point, and don’t have the extra overhead of having to return to the office to digitize the findings after the fact.

The cool part about this hackfest is that we managed to build something that actually works and could be put into production in almost no time. With LUIS and the Bot Framework, we can now give the field worker the support he needs when he needs it and without the use of hands.

“This is a totally new user experience, heavily influenced by Microsoft, an experience we can bring into our customer conversations across the domains within Powel.” —Øystein Askeland, Interaction Designer, Powel

Overall, the hackfest turned out to be a success for all parties involved and many valuable lessons were learned.

Going forward

We did not manage to include all the features that we scoped for the three-day hackfest. The scoped features that were left out were the following:

  • Translating text
  • Uploading images from the client app
  • Triggering André by natural voice instead of a button
  • Figuring out the WebSockets PCL and fixing it

Powel intends to iron out these issues shortly after the hackfest to produce a working prototype to be tested in the field as soon as possible.

Additional resources

Want to try out Bot Framework right now? Start here: Microsoft Bot Framework

Want to try out LUIS and Microsoft Cognitive Services? Go to
Microsoft Cognitive Services

Source code from the hackfest can be found on GitHub: readyforchaos/Powel-power-station-inspection-bot