Telecommunications and Internet service provider Three has experimented informally with several chatbot services on a variety of platforms but is keen to ensure as much code reusability as possible and reduce maintenance overhead. Its existing webchat has proven very successful but there are still unresolved user queries via this existing interface. To solve this problem, Three worked with Microsoft to build a bot that guides users through several self-service scenarios (that don’t require a human agent) such as activating a SIM card and answering general questions with a view to moving this quickly into production.

The final solution was a chatbot that answers general questions at any point during a conversation. It also provides two self-service flows that enable users to activate their SIM cards and to port their numbers using the bot—without ever having to leave the chat interface. Finally, the bot also gave users information on how to cancel or upgrade their contract with Three. Underlying the whole bot, we used Azure Application Insights to track telemetry of all the dialogs that we hope to later use as feedback into the bot for conversational flow/UX improvements.

Key technologies

Customer profile

Three UK - Maidenhead, United Kingdom

Three is a telecommunications and Internet service provider operating in the United Kingdom. The company launched in March 2003 as the UK’s first commercial video mobile network. It provides 3G and 4G services through its own network infrastructure.

Three wants to make life easier for its customers by helping them get the most out of their mobile devices, offering real value from the services it provides and by removing the barriers that frustrate them.

Problem statement

For the past 18 months, Three’s existing webchat functionality has proven very successful but there are still unresolved customer queries via this existing interface. These include:

  • Cancel or upgrade a contract.
  • New customer to Three.
  • Purchase a new phone/contract.
  • Report a stolen or lost device/SIM card.
  • Add/remove services and contract add-ons.
  • Report no Internet service or no/poor mobile coverage.
  • Top up service usage.
  • Amend user details.
  • Account balance queries.

The webchat built in to the Three website currently handles more than 100,000 messages per month, which leads to around 20,000 service interactions with live agents. Three also receives more than 1 million customer service calls per month. Of all of those questions and interactions with live agents, thousands can be self-served. This is a significant problem and requires a lot of human resource. An intelligent bot that can answer general queries and guide users through the self-service steps would enable significant time and financial savings.

Solution and steps

Prerequisites

Three’s success measurements

  • Reduced number of webchats related to topics for which there are adequate online self-help resources.
  • Increased number of visits to self-service journeys selected to solve customer queries.
  • Internal validation within the bot; for example, the number of people who say the bot session was helpful and meant they did not have to call/chat and so on.
  • Ascertain effort required by Three resources to set up, maintain, and optimize user cases and responses from the bot.

Solution and architecture

The final solution in the hackfest was a chatbot that:

  • Answers frequently asked questions (FAQs) using the Microsoft QnA Maker service.
  • Enables users to activate their Three SIM card.
    • Users fill in a form through a conversation with the bot.
  • Enables users to port their existing phone number over to Three.
  • Enables users to upgrade or cancel their contract with Three.
  • Tracks telemetry of all the dialogs using Azure Application Insights, which will later be used for feedback into the bot for conversational flow/UX improvements.

Main Menu

Activate SIM Flow

Activate SIM Flow

Port Number Flow

Upgrade Flow

Cancel Flow

The bot starts off with a welcome message that is triggered when a user adds a bot to their contacts or opens a conversation with the bot. From here the user can:

  • Use free text to ask a question (which gets handled by the QnA Maker).
    • For example: “What is 3G?”
  • Use free text to tell the bot to do something (which triggers the relevant dialog).
    • For example: “I want to port my number.”
  • Just click a button to trigger the relevant dialog.
    • For example: Clicking the “Activate SIM” button.

All the dialogs within the bot incorporate telemetry (using Azure Application Insights) to track where the users are navigating within the bot’s conversational flow—what sort of questions they’re asking and how long it takes to complete a request.

All the technical implementation for the above can be found in the “Technical delivery” section of this document.

Bot Architecture Diagram

The main flows that Three wanted to build out during the hack for the initial bot solution were:

  • Activate SIM
  • Port a number
  • Cancel or upgrade a contract

Technical delivery

This section describes the solution implementation details.

Core bot capabilities

Activate SIM

This bot flow is made up of several dialogs:

  • ActivateSIM
  • ActivateSIMForm
  • CommonFormIntro
  • ActivateSIMSubmit

ActivateSIM

Users are first asked which profile best suits them—this will help the Three bot fill in the correct form depending on the type of customer it is interacting with. This is done through a choice prompt. Depending on the response from the user, different dialogs are called such as CommonFormIntro or ActivateSIMForm. A lot of the following dialogs will use a similar prompt, as shown below, to guide the users.

builder.Prompts.choice(
    session, 
    'OK\n which of these best describes you?', 
    [
        'I ordered a replacement for a missing or broken SIM', 
        'I ordered a different size SIM', 
        'I have just upgraded', 
        'I am a new customer'
    ], 
    {listStyle: builder.ListStyle.button}
);


Activate SIM

ActivateSIMForm

This dialog asks users a series of questions that are required to complete the form needed to activate their SIM. This is a useful method to easily create a form within the bot framework.

function (session, args) {
    // Save entity data to dialogData
    if (args.entityData) {
    session.dialogData.entityData = args.entityData
    }
    session.dialogData.index = args.index ? args.index : 0
    session.dialogData.form = args.form ? args.form : {}

    // Check if entityData exists
    if (session.dialogData.entityData) {
    // If the entityData exists and it possesses the property for this question, send a confirm prompt
    if (session.dialogData.entityData.hasOwnProperty(questions[session.dialogData.index].field)) {
        var prompt = questions[session.dialogData.index].prompt
        prompt = prompt.replace('{' + questions[session.dialogData.index].field + '}', session.dialogData.entityData[questions[session.dialogData.index].field])
        builder.Prompts.confirm(session, prompt)
    } else {
        // If the entityData exists but the property for this question doesn't, send a text prompt
        builder.Prompts.text(session, questions[session.dialogData.index].question)
    }
    } else {
    // If there is no entityData, proceed as normal
    builder.Prompts.text(session, questions[session.dialogData.index].question)
    }
},
function (session, results, next) {
    // Check if the user responding via a Confirm or Text prompt
    if (results.response === true) {
    // If the confirm prompt is true then we save the entity to the form object and increment the index
    var field = questions[session.dialogData.index++].field
    session.dialogData.form[field] = session.dialogData.entityData[field]
    } else if (results.response === false) {
    // If the confirm prompt is false then we delete the entity from the entityData object but we do NOT increment the index
    field = questions[session.dialogData.index].field
    delete session.dialogData.entityData[field]
    } else {
    // If the user replied via text then we save the user response
    field = questions[session.dialogData.index++].field
    session.dialogData.form[field] = results.response
    }

    // Check for end of form
    if (session.dialogData.index >= questions.length) {
    // Process form for submission
    session.privateConversationData.simForm = session.dialogData.form
    session.beginDialog('ActivateSIMSubmit')
    } else {
    // Next field
    session.replaceDialog('ActivateSIMForm', session.dialogData)
    }
}


Activate SIM

CommonFormIntro

This dialog introduces the user to the form and describes the information required.

var requirements = ['You will need the following information: \n'];
for (var requirement in session.dialogData.formRequirements) {
    requirements.push('\n * ' + session.dialogData.formRequirements[requirement])
}
requirements = requirements.join('');
session.send(requirements);


Activate SIM

Activate SIM

ActivateSIMSubmit

This dialog is called once the ActivateSIMForm is completed, and submits the form to Three using request to complete a REST POST call to the Three API. The image below displays what the user sees after successfully submitting a form.

Activate SIM

Port a number

This dialog is very similar to ActivateSIMForm and prompts the user with questions required to submit a form that will port the customer’s number. The submission is also made using a POST call to Three.

var questions = [
    {field: 'mobileNumber', question: 'What is the existing number you want to keep?'},
    {field: 'mobileNumber2', question: 'What is your temporary new Three number?'},
    {field: 'pac', question: 'What is your PAC number?'},
    {field: 'emailPayMonthly', question: 'What is your email address?'},
    {field: 'fullName', question: 'What is your full name?'},
    {field: 'dob', question: 'What is your birthday (e.g. 01/01/1901)?'},
    {field: 'address1', question: 'What is the first line of your address?'},
    {field: 'postcode', question: 'What is your postcode?'}
];

function (session, args) {
    session.dialogData.index = args ? args.index : 0;
    session.dialogData.form = args ? args.form : {};

    builder.Prompts.text(session, questions[session.dialogData.index].question);
},
function (session, results) {
    // Save users reply
    var field = questions[session.dialogData.index++].field
    session.dialogData.form[field] = results.response

    // Check for end of form
    if (session.dialogData.index >= questions.length) {
    session.privateConversationData.portForm = session.dialogData.form
    session.beginDialog('PortNumberConfirm');
    } else {
    session.replaceDialog('PortNumberForm', session.dialogData);
    }
}


Activate SIM

Cancel or upgrade

This flow contains three dialogs:

  • UpgradeOrCancel
  • Cancel
  • Upgrade

UpgradeOrCancel

This dialog simply asks users whether they want to upgrade or cancel and then calls the relevant dialog depending on the user’s choice. This is implemented using a choice prompt.

Upgrade or Cancel

Cancel

This asks the users, using a choice prompt, whether they want a PAC code or to know the end date of their contract. Depending on the response, it replies with the phone number to call or the website they need to navigate to.

Cancel Date

Cancel PAC

Upgrade

This dialog does the same as the ‘cancel’ dialog. It asks for the type of contract and then, depending on the answer, it gives the user the website they need in order to upgrade.

Upgrade

Bot intelligence

LUIS

In this project, we primarily relied on using regex to determine the user’s intent. However, we also wanted to use the Microsoft Language Understanding Intelligent Service (LUIS) as a fallback in case the user decided to enter free text queries. We created a LUIS model to handle the intents for the three flows that we had built within this bot (activate SIM, port a number, cancel or upgrade).

Example regex used:

matches: /^Activate SIM/i
matches: /^Port my number/i
matches: /^Upgrade or cancel/i


LUIS model

First, we configured the LUIS recognizer inside config.js:

// Import LUIS Model
var recognizer = new builder.LuisRecognizer(process.env.LUIS_MODEL_URL);
bot.recognizer(recognizer);


We then set up trigger actions for each main flow dialog so that if the LUIS model recognized the intent, it would trigger the required dialog.

For example, if the user says: “I want to activate my SIM card,” LUIS would pick this up as an ActivateSIM intent, which would then trigger the start of the ActivateSIM dialog because the intent matches the triggerAction keyword (see below). We also set the intentThreshold for the LUIS intent trigger so that only matches above a 0.5 confidence rating would trigger the dialog.

.triggerAction({
    matches: 'ActivateSIM',
    intentThreshold: 0.5
  })


You can easily learn how to make your own LUIS model at the LUIS website.

Trigger QnA

QnA Maker

Three used the Microsoft Cognitive Services QnA Maker service to answer simple customer questions that are available online (Three FAQ Page). This was quick to implement and helped triage simple customer queries away from the direct human assistance.

This is the QnA dialog that handles all FAQs. We call endDialog so that the bot returns to the previous dialog the user was in when they asked the question.

module.exports = function () {
  bot.dialog('QnA', (session, results) => {
    console.log(process.env)
    var client = restify.createJsonClient('https://westus.api.cognitive.microsoft.com')
    var options = {
      path: '/qnamaker/v2.0/knowledgebases/' + process.env.QNA_KB_ID + '/generateAnswer',
      headers: {
        'Ocp-Apim-Subscription-Key': process.env.QNA_SUBSCRIPTION_KEY
      }
    }

    var question = {'question': results.question}

    client.post(options, question, (err, req, res, obj) => {
      if (err == null && obj.answers.length > 0) {
        for (var i in obj.answers) {
          if (parseInt(obj.answers[i].score) > 0.80) {
            session.endDialog(obj.answers[i].answer)
          } else {
            session.endDialog('Sorry, I couldn\'t find an answer in our FAQs. Don\'t forget, you can type \'help\' if you need assistance')
          }
        }
      } else {
        session.endDialog('Sorry, there was an error!')
      }
    })
  })
}


This dialog is then called through the UniversalBot. This means that the QnA maker is surfaced globally throughout the bot. If the user query is not recognized by either LUIS or by one of the regex expressions, then it will be sent to the QnA dialog.

global.bot = new builder.UniversalBot(connector, function (session) {
    session.send('I\'ll just check that for you...', session.message.text)
    session.replaceDialog('QnA', { question: session.message.text })
});


The QnA Maker functionality is also triggered when the user says help, quit, problem, or support and doesn’t want the main menu:

if (results.response && results.response.entity === 'no') {
    builder.Prompts.text(session, 'Ok, why don\'t you try asking your query here and I\'ll search our FAQs');
}
.
.
.
function (session, results, next) {
    session.replaceDialog('QnA', { question: session.message.text });
}


Trigger QnA

Azure Application Insights

We set up Application Insights to capture telemetry within the bot. This is used to trace the conversational flows that the user has gone through within the bot and to also track metrics on how long it has taken to complete a particular task.

First, we set up the Application Insights client in the config file:

global.telemetryModule = require('./telemetry-module')
const appInsights = require('applicationinsights')
appInsights.setup(process.env.APPINSIGHTS_INSTRUMENTATION_KEY).start()
global.appInsightsClient = appInsights.getClient()


Then we created a telemetry module to handle the telemetry work:

exports.createTelemetry = function (session, properties) {
  var data = {
    conversationData: JSON.stringify(session.conversationData),
    privateConversationData: JSON.stringify(session.privateConversationData),
    userData: JSON.stringify(session.userData),
    conversationId: session.message.address.conversation.id,
    userId: session.message.address.user.id
  }
  if (properties) {
    for (var property in properties) {
      data[property] = properties[property]
    }
  }
  return data
}


Finally, we created different telemetry objects in each dialog to trace that the user visited the dialog. For example:

// Store entity data in dialogData
session.dialogData.entities = data;
// Create a new telemetry module with session data
session.dialogData.telemetry = telemetryModule.createTelemetry(session, { setDefault: false });
// Track that the user has been to the activate sim dialog
appInsightsClient.trackTrace('activateSIM', session.dialogData.telemetry);


We can then retrieve the trace results within the Analytics portal in Azure Application Insights:

Trace

To trace how long it took for a form to be submitted, the following code was used:

// Setup a telemetry module
session.dialogData.measuredEventTelemetry = telemetryModule.createTelemetry(session);
// Start timer. We want to track how long it takes for us to submit a SIM activation request
session.dialogData.timerStart = process.hrtime();
.
.
.
// Submittion has been made, calculate how long it took.
var timerEnd = process.hrtime(session.dialogData.timerStart);
// Save the time it took to 'metrics' within the measuredEventTelemetry module
session.dialogData.measuredEventTelemetry.metrics = (timerEnd[0], timerEnd[1] / 1000000);
// Track the above metric as 'timeTaken'
appInsightsClient.trackEvent('timeTaken', session.dialogData.measuredEventTelemetry);


Metric

Conclusion

We began this project with the aim of building a bot that can guide users through several self-service scenarios such as activating a SIM card and answering general FAQs with a view to moving this quickly into production.

As a result, we have developed a compelling bot that should save Three significant time and money. It will make the job easier for customer service agents because they will no longer have to deal with commonly asked simpler/general questions, therefore freeing up their time to deal with more specific, trickier queries. The bot should also provide customers with a simpler, faster method to get their queries answered through a guided conversation experience. Three thinks this will change the behavior of customers and agents alike, making them more productive.

Challenges

  • The documentation for the Node.js SDK can be tricky to follow in its current form and feels limited in comparison to the C# SDK documentation. This sometimes made it difficult to find the information needed to put the bot together. However, a lot of useful code samples are available.
  • Error feedback from LUIS and QnA services is limited, or non-existent in the case of QnA Maker. This made it difficult to debug and figure out why things didn’t work as expected.
  • Clear guidance on development best practices for the bot framework is still evolving. When developing a bot that is intended for production, it would be useful to see the best practices for doing certain things such as:
    • Determining project structure.
    • Asking the user for feedback at the end of a dialog.
    • Using LUIS versus regex.

Learnings

Welcome messages don’t work with Slack

Unlike other channels, onConversationUpdate is not triggered when a bot is added to Slack. Also, contactRelationUpdate is not triggered when a user is added to Slack. This means there is currently no way to welcome a Slack user and introduce them to the bot, without them interacting with the bot first. We did discover that the native Slack channel does deal with this. If using a web socket, the Slack framework sends a bot_added event to let you know when a user has added a bot to their Slack channel. However, the Microsoft Bot Framework is not using web sockets with Slack and doesn’t get notification of this event.

The pre-built QnA Maker package is ‘semi-permanent’

If you start the QnA dialog, there is no obvious way out of this dialog flow. The user would be stuck using the QnA service and won’t be able to continue using the bot’s other dialogs, unless you use the global restart command. The solution is to call the QnA API directly and end the dialog after the question is answered. This way, users can ask a question in the middle of any other dialog, get their question answer by the QnA service, and then continue with the dialog they were in last. The implementation for this can be found in the “Technical delivery” section of this document (under QnA Maker) or in this sample GitHub repo.

How to implement safe words

Using regex and actions, we are able to set up global commands that users can use at any point in the bot. This allows users to return out of a dialog when they become stuck in a conversational flow.

bot.endConversationAction('goodbye', 'Goodbye :)', { matches: /^bye/i }); 
bot.beginDialogAction('home', '/start', { matches: /^home/i });
bot.beginDialogAction('help', '/help', { matches: /^help/i }); 


Using LUIS intents through triggerActions and setting the intentThreshold

Instead of using the usual method of matching LUIS intents:

intents = new builder.IntentDialog({recognizers: [recognizer]});

intents
.matches('LUIS_Intent', '/dialog')
.matches('LUIS_Intent', '/dialog')
.onDefault(builder.DialogAction.send("I'm sorry. I didn't understand."));


We found it was a lot cleaner to match LUIS intents to dialogs through triggerActions (which you simply add at the end of a dialog). This also led us to discover you can set a threshold for which LUIS intents should meet before they get triggered using intentThreshold. The default is to trigger actions even if the confidence of the match is less than 0.1.

.triggerAction({
    matches: 'ActivateSIM',
    intentThreshold: 0.5
  })


You also can add an intentThreshold at a global level, so that it applies to all LUIS matches:

intents = new builder.IntentDialog({ recognizers: [recognizer], intentThreshold: 0.5 });


An easy and clean way to fill out forms

We developed a clean way of asking users questions to fill out a form using dialog recursion.

First, create an array of questions:

var questions = [
    {field: 'mobileNumber', question: 'What is the existing number you want to keep?'},
    {field: 'mobileNumber2', question: 'What is your temporary new Three number?'},
    {field: 'pac', question: 'What is your PAC number?'},
    {field: 'emailPayMonthly', question: 'What is your email address?'},
    {field: 'fullName', question: 'What is your full name?'},
    {field: 'dob', question: 'What is your birthday (e.g. 01/01/1901)?'},
    {field: 'address1', question: 'What is the first line of your address?'},
    {field: 'postcode', question: 'What is your postcode?'}
]


Then, loop through each question by calling the same dialog and passing in an index counter, saving the user’s response after each question is asked:

bot.dialog('FillOutForm', [
function (session, args) {
    session.dialogData.index = args ? args.index : 0
    session.dialogData.form = args ? args.form : {}

    builder.Prompts.text(session, questions[session.dialogData.index].question)
},
function (session, results) {
    // Save users reply
    var field = questions[session.dialogData.index++].field
    session.dialogData.form[field] = results.response

    // Check for end of form
    if (session.dialogData.index >= questions.length) {
    session.privateConversationData.portForm = session.dialogData.form
    session.beginDialog('EndofForm')
    } else {
    session.replaceDialog('FillOutForm', session.dialogData)
    }
}


You can find a follow code sample of this here: Simple Form Sample for Node SDK

Plans for production

The plan for this bot is to roll it out into A/B testing in the coming weeks so that it can be tested with real customers. Following the one-week A/B trial, the plan is to release this bot into production within months.

Looking further forward, Three recognizes it can provide a richer experience by delivering on the following:

  • Rather than linking out to resources on the Three website, the user can ask the bot questions based on the topic of the page and receive answers back based on page content.
  • Integration with APIs—there are network-based APIs around coverage and outage problems and reporting an issue, which could be worked into a bot experience.
  • Posting to forms—for example, requesting a SIM or porting a number to Three can be facilitated through the bot.
  • Handing off to a call center/live agents.
  • Personality and personalization—addressing the user by name and personalizing content to suit the user.

“Thanks all for inviting us to this awesome week. It’s been really fun and we have more than the bones of a real product to take to market. We’re all gutted to be back to real work tomorrow!”

— Justin Beasley, Lead Digital Development Manager

“Thanks @microsoft-simon and all other MS folk. We had a great time and learned a lot!”

— Stuart Brown, Digital Development Manager

Additional resources

Documentation:

Code:

Team

Microsoft:

Three:

  • Justin Beasley – Lead Digital Development Manager
  • Nick Bishop – Digital Development Manager
  • Stuart Brown – Digital Development Manager
  • Thomas Barton – Scrum Master
  • Dimos Fountoukos – Software Developer