Instaply logo

Developers from Instaply and Microsoft built a custom Direct Line client solution to introduce intelligent conversation. This chatbot integrates with Instaply’s preexisting customer-facing messaging platform to add an intelligent automation layer. The bot will be able to answer basic questions about such things as business hours, addresses, and whether a location is open.

The bot will also be able to handle more complicated requests (such as making reservations) by handing the request to a customer service representative. Using Microsoft Bot Framework and LUIS, this feature will serve as the first point of contact for customer conversations.

Key technologies

The technologies outlined and included in this solution are:

Partner profile

Instaply is a customer service/software as a service (SaaS) platform mainly focused on SMS communication. The San Francisco, California-based company has an established product that allows a subscriber (a company) to manage and respond to SMS/Facebook Messenger conversations within its platform.

Instaply bridges the gap between businesses and their customers by creating a text-messaging channel that helps provide accessible customer service and support.

Platform UI

Problem statement

Every conversation prompt going to Instaply is currently being handled by a human. When a conversation is initiated, the platform responds to the sender with a canned message (set by the subscriber) notifying the sender that someone will respond shortly. The platform currently does not have a bot that can automatically respond to commonly asked questions or inquiries.

“As a company that’s entirely dedicated to communication efficiency, we need to find a bot solution to offer our clients, to continue to help them reduce labor costs, by automating text responses to frequently asked questions.”

— Don Voogd, Director of Business Development, North America, Instaply

Instaply’s platform and API

Instaply’s platform has a unique feature that allows a subscriber’s internal employees to communicate within the organization to resolve an issue. The internal conversation is kept separate from the customer. There can be two separate threads of conversation for a customer engagement: public thread and invisible thread. Each public thread has three possible, mutually exclusive statuses: ‘waiting,’ ‘replied,’ and ‘resolved.’

In its developer portal, Instaply documents its current bot API, which allows a third party to interact with its public thread. Instaply’s bot API uses three key components during this engagement:

  • Webhook. On every message received (regardless of the sender), Instaply can send a predefined payload to a configured endpoint. The payload (JSON object) has several essential properties: fromCustomer, invisible, customerThreadId, and businessId.
  • Send Message. The platform receives a POST call and expects the customerThreadId from the webhook to add a message to the public thread. This in turn sends the message through the appropriate channel (for example, SMS) back to the customer.
  • Change Thread Status. The platform receives a POST call to change the thread (specified by customerThreadId) to one of its statuses.

Requirements and goals

For this technical engagement, we defined these milestones before the hackfest:

  • When a customer sends a text to an Instaply number, the customer will see a greeting from the bot.
  • When the customer asks for the hours, the bot will respond with the hours of operation for the store (subscriber).

Solution overview

Many different technologies all came together to make this project successful: the Microsoft Bot Framework SDK, integrating with a custom messaging platform using Direct Line client, LUIS, and Docker containers using Azure Container Registry. Below is an example user experience scenario and overall view of the architecture diagram followed by each part broken down with greater detail.

Architecture Diagram

Scenario experience walkthrough

These steps will follow the progression denoted in the architecture diagram above.

  1. A user messages Instaply’s platform with their mobile device (SMS).
  2. Instaply’s webhook sends a payload containing fromCustomer, threadId, and query string.

    a. If the webhook is from a customer, we continue. Messages from the sales representative are filtered here.

    b. The threadId is checked in the repository.

    • If new, record in repository and create connection with bot.
    • If exists, check connection with bot and continue.

    c. The message is sent from the Direct Line client to the bot.

  3. The bot sends LUIS the message to determine the intent and entities.
  4. Intent is returned to the bot.

    • If the intent is none and confirmed more help is required, we change the status to waiting.
    • If intent is found, we execute the corresponding dialog.
  5. We query Instaply’s knowledge base via APIs and send the response back to the client.

    The message is received back at the Direct Line client through websockets.

  6. The message is forwarded to Instaply’s platform.
  7. Instaply receives the bot’s response and forwards to the user’s mobile device (SMS).

If the bot changes Instaply’s thread status to waiting, this initiates the bot-to-human handoff. A human will enter the conversation at this time.

Technical delivery

Prerequisites

Implementation steps

  1. Create a new project with the Bot Framework using Node.js. (See Create a bot with the Bot Builder SDK for Node.js.)
  2. Create the Direct Line client using Direct Line 3.0 REST API. (See “Creating a custom Instaply channel: Direct Line client” below.)
  3. Deploy the Direct Line client and bot application to the Web Apps feature of Azure App Service. (See “Docker containers” below.)
  4. Add LUIS to your chatbot. (See “Adding intelligence with LUIS and Bot Builder Node.js” below.)

Microsoft Bot Framework

The Bot Framework Node.js SDK provides all the tools necessary to develop and implement a fully conversational chatbot to connect with multiple preexisting channels through the Bot Connector. The Bot Connector has a Direct Line client that can be used to integrate with any preexisting messaging platform; that’s perfect for our problem statement and scope requirements.

Instaply Chat

Creating a custom Instaply channel: Direct Line client

This Direct Line client is the key piece to bridging the gap between Instaply’s custom messaging platform and the Microsoft Bot Framework. The Bot Connector has many different connections to predefined channels for easy click deployment such as Skype, Microsoft Teams, Twilio (SMS), Slack, GroupMe, and Facebook Messenger. Because we plan to integrate with Instaply’s channel, we are required to create a custom connection. This is where the Direct Line channel comes in.

The Microsoft Bot Connector has documentation on Direct Line 3.0 REST, which is a set of defined APIs that are sent between the custom Direct Line client and the Bot Connector. These are the API calls we used to broker our connection and send activity messages back and forth to our chatbot.

These are the steps that happen within the client:

1. Receive the webhook payload

Instaply’s webhook payload contains the threadId, message, and whether the message is from a customer. This is important for determining whether the bot should respond. We do not want the bot responding to the sales representative, only the customer.

const initialBody = req.body;

const threadId = initialBody.customerThreadId; //unique to user
const businessId = initialBody.businessId; //store id so we know location
const msg = initialBody.messageBody; // message user is sending
const fromCustomer = initialBody.fromCustomer; // from customer or sales rep?

// if from sales rep, we ignore this webhook call
if (!fromCustomer) 
{
    res.json({message: 'ignored'});
    return;
}


2. Create mapping of threadId

Once the webhook is received, we need to determine whether the bot has had any existing conversation with this user based on their threadId. To do this, we check our repository.

We decided to use Map() to see what’s in the repository:

var repository = new Map();
var conversationMapping = new Map(); // {threadId: {conversationId, muteBot, isConnected, watermark}}
var businessMapping = new Map(); // {conversationId : {threadId, businessId} }


Two things are possible: Either we have an existing connection associated with this threadId or there is no result in our repository. In order to see which is true, the Bot Connector initiates a conversation and a websocket connection to listen for the bot response.

If there is already a conversation and connection established, we skip the setup steps and just send the connector a message through an API call.

3. Ensure conversation is started

If no connection is established, we initiate a new conversation.

We start a new conversation by passing our Direct Line secret in the authorization header. This will let the Bot Connector know it is initiating a conversation with a trusted service client. We do not have to use JSON Web Tokens because we are not exposing the secret to a public client.

After the call has been made, we receive a URL that we connect via websockets.

function startConversation (threadId, businessId) {
  // initiate the new conversation
  return fetch(directLineBase + `/v3/directline/conversations`, {
    method: 'post',
    headers: {
      authorization: `bearer ${process.env.BOT_DIRECTLINE_SECRET}`
    }
  })
    .then(res => res.json())
    // store information about this conversation in our repository so we have a record
    .then(data => {
      var repositoryData = {
        conversationId: data.conversationId,
        muteBot: false,
        isConnected: false,
        businessId: businessId
      };
      repository.set(threadId, repositoryData);

      // we now map the conversationId to the threadId and businessId (from the webhook)
      conversationMapping.set(data.conversationId, {threadId, businessId});
      
      // we now have the url so we can connect our websocket and listen for a response.
      return startConnection({url: data.streamUrl, threadId, businessId});

    }).then(() => {
      return {id: 'directline', threadId, businessId}
    })
}


4. Ensure websocket connection is established

Next, we set up the websocket connection with the URL we received from the Bot Connector. This function returns a JS promise that will resolve after the websocket connection is established. When we receive a response from the bot through the websocket, it will include a watermark. This watermark (as defined in the documentation) helps to keep track of the conversation state to ensure no messages get lost. We take this watermark and update our repository mapped to the threadId. Once we receive a message through the websocket, we make another API call back to Instaply with the bot’s message so it can be displayed to the waiting user.

function startConnection ({url, threadId, businessId}) {
  const ws = new WebSocket(url)
  const resultPromise = new Promise((resolve) => {
    
    // WHEN OPEN WS
    ws.on('open', () => {
      var conversationObject = repository.get(threadId);
      conversationObject.isConnected = true;
      repository.set(threadId,conversationObject);
      resolve()
    })
  })

  // POST MESSAGE TO INSTAPLY
  ws.on('message', (messageStr) => {
    const message = messageStr !== '' ? JSON.parse(messageStr) : {}
    if (message.activities) {
      // update conversation watermark
      const activity = message.activities[0]
      if (activity.from.name) {
        var {threadId} = conversationMapping.get(activity.conversation.id)

        var conversationObject = repository.get(threadId);

        //ID related to conversation
        conversationObject.watermark = message.watermark;
        repository.set(threadId,conversationObject);

        const msg = activity.text
        // take the message from the bot and respond back to Instaply to post to their user
        postToApi(threadId, msg)
      }
    }
  })

  ws.on('disconnect', () => {
    // console.log('WS DISCONNECT')
  })
  return resultPromise
}


5. Send message to Bot Connector

The final step is to send the activity message from the Direct Line client to the bot. This is done once the conversation and websocket connection has been established (Steps 3 and 4).

To do this, we send a separate API REST call containing the conversationId and message. We pass the businessId value in the channelData property, so we have access to this value in the bot logic.

function sendMessageToBotConnector (threadId, message, businessId) {
    var conversationObject = repository.get(threadId);
    var convoId = conversationObject.conversationId;

    const endpoint = directLineBase + `/v3/directline/conversations/${convoId}/activities`
    // console.log('send message to bot connector endpoint', endpoint)
    return fetch(endpoint, {
        method: 'post',
        headers: {
          'Content-Type': 'application/json',
          'authorization': `bearer ${process.env.BOT_DIRECTLINE_SECRET}`
        },
        body: JSON.stringify({
          type: 'message',
          from: {
            id: threadId
          },
          text: message,
          channelData: {
            businessId,
            threadId
          }
        })
      })
}


There’s one more thing for the Direct Line client to do if the threadId has an existing connection history, but the websocket connection has disconnected. There is a specific API call (REST) for reconnecting purposes. We send the REST call with the conversationId; the connector will reply with a new URL for us to connect to.

Preexisting knowledge base

Prior to the hackfest engagement, Instaply created two APIs the bot could call to access the knowledge base we needed to answer the use case questions:

  • Get Hours API takes in the specific businessId (based on location) and returns the full Monday through Sunday schedule for us to display.
  • IsOpen API takes in a datetime parameter and returns a Boolean, which represents whether the store is open at that specific datetime.

These two new APIs minimize the amount of dialog logic we had to implement. By abstracting the knowledge base, we were able to focus more on the conversation flow than the actual information gathering and useful information display. This approach also allows for easy dialog expansion.

Adding intelligence with LUIS and Bot Builder Node.js

We have the basic “Hello World” bot template working with our custom Direct Line client channel, so now we can add some intelligence!

Because the Bot Connector abstracts the channels from the dialog implementation, we can treat this portion of the project the same way we would for a regular botbuilder SDK chatbot. Below is an outline of the LUIS dialog and state for our conversation flow.

LUIS Dialog


This conversation is scoped to handle two main use cases (which will be built out for production after the hackfest is complete):

  • Hours of operation.
  • Ask for ‘More Help,’ which will initiate bot-to-human handoff.

We expect the user to initiate the conversation (through SMS), which will prompt the bot to greet the user.

  • The user will have to be able to ask a question or make a statement.
  • The bot will take the user input and send it to LUIS for analysis against the current language model.
  • LUIS will send the appropriate intent and entities found within the user query. This will dictate which intent-based dialog gets called.
bot.dialog('/',dialog);

dialog.matches('Greeting', [
  function(session, args, next) {
    session.beginDialog('/greeting');
  }
]);

dialog.matches('GetHours', [
  function(session, args, next) {
    session.beginDialog('/getHours',args);
  }
]);

dialog.matches('GetHoursBoolean', [
  function(session, args, next) {
    session.beginDialog('/getHoursBoolean',args);
  }
]);

dialog.matches('EndConvo', [
  function(session, args, next) {
    session.beginDialog('/endConvo');
  }
]);

dialog.onDefault([
  function(session, args, next) {
    session.send("I'm sorry, I didn't understand. Let me show what I can do");
    session.beginDialog('/mainMenu');
  }
]);


GetHours

This dialog will display the establishment’s hours of operation Monday through Sunday. No entities are captured in this intent. Example utterances are:

  • “When are you open?”
  • “What are your store hours?”
  • “When can I visit?”
return fetch(process.env.INSTAPLY_ME_ENDPOINT, {
                method: 'post',
                headers: {
                    'Authentication-token': process.env.INSTAPLY_ME_TOKEN,
                    'Content-Type': 'application/json'
                },
                body: JSON.stringify({
                     "query": "get-business-out-of-hours-query",
                     "variables": {
                        "businessId": "125510"
                     }
                })
            })
            .then(res => res.json())
            .then(data => {
                
                var result = "";

                var days = data.data.businessOutOfHours.days;
                days.forEach(function(element) {
                    console.log("day: "+element.day);
                    result = result + element.day+" : "+element.hours[0].end+" - "+element.hours[1].start+"\n";
                }, this);

                session.send(result);
                session.send("Can I help you with another Question?");
                session.endDialog();
            })


GetHoursBoolean

This intent is called when the user asks a question and expects a yes-or-no response.

The user gives the bot a specific date, time, or datetime; we must identify this data as entities and determine if the facility is open. LUIS provides builtin.datetime function, which we use to resolve the entity into a consistent datetime format for us to parse.

Examples of utterances are:

  • “Are you open right now?

    Resolves to current datetime.

  • “When are you open on Friday?”
    • Resolves to datetime: 6T
    • 6th day of the week (starting Sunday)
  • “Are you open Tuesday at 1:30 PM?”
    • Resolves to datetime: 3T13:30
    • 3rd day of the week
    • 13:30 military time (24-hour clock)
var dateTimeArray = args.entities.filter(function(entity) {
    return entity.resolution.time && ( entity.resolution.time === 'PRESENT_REF' || entity.resolution.time.length >= 13) //'XXXX-WXX-1T19
})

var timeArray = args.entities.filter(function (entity) {
    return entity.resolution.time && entity.resolution.time.length >= 2 && entity.resolution.time.length <= 6 //'T20:12
})

var dateArray = args.entities.filter(function(entity) {
    return entity.type === 'builtin.datetime.date'
})

if (dateTimeArray.length > 0) {
    session.send('you have date time')
}

if (timeArray.length > 0) {
    session.send('you have just time')
}

if (dateArray.length > 0) {
    session.send('you asked about a day')
}


None

If LUIS does not recognize what the user’s intent is based on their input, we will prompt the user with a main menu. This will explicitly list what the bot is capable of helping the user with.

For this scope, we present the user with two options: ‘View Store Hours’ and request ‘More Help.’

  • If the user selects ‘View Store Hours,’ the ‘getHours’ dialog will execute and display the Monday-Sunday schedule as outlined above.
  • If the ‘More Help’ choice is selected, the bot will initiate the bot-to-human handoff. A human representative will take control of the conversation, and the chatbot will mute itself from the conversation. For training and intelligence purposes, the bot will still collect conversation data, but will not respond.
bot.dialog('/mainMenu', [
  function (session, args, next) {
    builder.Prompts.choice(session,'What would you like to do',['View Store Hours','Need More Help'])
  },
  function (session,args,next) {
    switch (args.response.index) {
      case 0:
        session.send('View Store Hours');
        session.beginDialog('/getHours');
      break;
      case 1:
        session.send('Need More Help');
        session.beginDialog('/humanHandoff');
      break;
    }
  }
]);

Bot-to-human handoff

Instaply’s platform uses different predefined statuses to determine the state of each conversation thread between their different users.

  • Replied: A sales rep (or bot) has responded to the user.
  • Waiting: The user has asked a question and is currently waiting for a response.
  • Resolved: The user is satisfied with the conversation and no longer needs assistance.

When the bot responds to the conversation thread, the status is automatically set to replied. The bot needs to make two separate API calls to Instaply to change the thread status to waiting or resolved. As outlined in the section above, the bot will change the status to waiting when the user navigates to the main menu and selects ‘More Help.’ Once the API call has been sent, and the thread changes status to waiting, a human will be able to view and respond to the conversation without any further input from the bot.

bot.dialog('/humanHandoff', [
    function (session, args, next) {
        session.send('Let me get someone that can further help you. One moment please!');
        
        // Make API call to set thread status to waiting
        return fetch(process.env.INSTAPLY_WAITING_POST_ENDPOINT, {
            method: 'post',
            headers: {
            token: process.env.INSTAPLY_WAITING_POST_TOKEN,
            'Content-Type': 'application/json'
            },
            body: JSON.stringify({
              status: 'waiting',
              customerThreadId
            })
        })

        session.endDialog();
    }
])


Docker containers

Because Instaply might make a unique bot for every customer, making the bot creation and deployment process as efficient as possible was a key goal for this engagement.

We decided to use Docker containers. And we used Azure Container Registry to store the Docker images.

Azure Container Registry


We then created a web app on Linux service and pointed to the image with the latest tag.

Web App on Linux


Continuous deployment with CircleCI

When using Docker containers, there is additional processing.

For every code commit, deploying the container requires the app to rebuild the Docker image and push it to the container registry. This is tedious and can be automated. Instaply uses a continuous deployment solution called CircleCI.

The file circle.yml is CircleCI’s configuration file. The file outlines the commands needed to log on to Azure Container Registry, build the Docker image, and push the image to the registry.

CircleCI Yaml


CircleCI listens to GitHub’s webhook and kicks off the build process on every push. This image shows that the build was successful (showing green) after the code change was pushed to GitHub.

CircleCI Build Successful


Authentication

Even though this is a customer- and public-facing chatbot, we did not have to manage who could talk to it.

Because Instaply’s platform handles service representative logons so they can track who is able to respond to users, there is no need for bot authentication. Instaply also maintains data on user’s SMS numbers mapped to the threadId so when the user returns, we have context about the user from previous conversations.

The only authentication aspects we had to consider and implement were for the Direct Line channel communication between the client and the connector.

The Direct Line 3.0 REST documentation outlines the available and required REST calls to initiate conversation and send message activities to the bot.

We had to handle the authentication handshakes coming from our Direct Line client. We had the choice of requesting a JSON Web Token or passing our Direct Line secret for authentication. The documentation recommended we use our secret (generated from the Bot Connector) because we have a service-to-service scenario. Our secret will not be exposed through the Direct Line client and Bot Connector.

Architectural decisions

One-app architecture

Future Instaply bots will be tailored to each of their customers’ needs. Because every Instaply bot needs a Direct Line client, it would be great if Instaply could merge the Direct Line client and the bot into one app. Instaply would only need to deploy one app per bot, and would not have to worry about deploying a separate Direct Line client app along with the bot.

Muting the bot

The bot receives a webhook from every Instaply message.

When Instaply receives a message from the customer service rep or the bot, the platform will fire a webhook. The Direct Line client currently processes the messages from the customer by inspecting the fromCustomer property in the webhook payload. This isn’t sufficient in a real-world scenario. For example, a customer is having a conversation with the bot when the customer service rep jumps into the conversation and the customer responds to the rep. The bot would still process the message from the customer meant for the rep, and respond with a message (likely from the ‘None’ intent).

In order to have the expected bot behavior, the Instaply platform needs an explicit indication to mute the bot even when the message is from a customer. The muteBot parameter would be included in the webhook payload, and the Direct Line client would inspect it much like that of the fromCustomer property.

Channel data

In order to make various API calls to Instaply’s knowledge base, the bot needs to know which business is requesting the information. All of this information is contained in the Direct Line client webhook payload. We used the channelData payload to pass the information from the Direct Line client to the bot. That way, data is passed to the right conversation, and the bot can retrieve the data easily via the session object.

When we send the message from the Direct Line client to the Bot Connector, we include the payload in the channelData property.

fetch(endpoint, {
  method: 'post',
  headers: {
    'Content-Type': 'application/json',
    'authorization': `bearer ${process.env.BOT_DIRECTLINE_SECRET}`
  },
  body: JSON.stringify({
    type: 'message',
    from: {
      id: threadId
    },
    text: message,
    channelData: {
      businessId, // for requesting hours API in the bot
      threadId //for changing the thread status in the bot
    }
  })
})


On the bot side, the value can be accessed in session.message.sourceEvent.

Here is an example when the bot is calling a REST API using the data:

bot.dialog('/getHours', [
    function (session,args, next) {
    //session.send('getHours');
      console.log("channelData:",session.channelData);
      const businessId = session.message.sourceEvent.businessId //accessing the channelData passed from directline client
      return fetch(process.env.INSTAPLY_ME_ENDPOINT, {
        method: 'post',
        headers: {
            'Authentication-token': process.env.INSTAPLY_ME_TOKEN,
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
              "query": "get-business-out-of-hours-query",
              "variables": {
                businessId
              }
        })
    })
   ...
])

Business impact

Since Instaply is a SaaS platform, it has its own customers. Because of the per-customer bot integration feature, Instaply will add tremendous value. Creating a bot with bot-to-human handoff will drastically reduce the customer service load for Instaply’s customers. Microsoft’s Bot Framework along with LUIS fills this need.

Other SaaS companies will find the code story valuable because they can use similar strategies to build a per-customer bot integration on top of their respective platforms.

“Creating a bot would allow our current clients to redeploy valuable human resources to revenue-generating business, as opposed to answering basic questions. Being able to end a conversation, mark it resolved, and automatically send a survey would be huge.”

— Don Voogd, Instaply

Partner technical engagement feedback

When asked if Microsoft technologies and partnership would add value to their business, Don Voogd of Instaply replied, “Absolutely. Our Microsoft team was clinical in their focus and execution of this project. The team clearly understood our desire to move quickly, iterate and test, and they were instrumental in filling the clear dev gaps that prevented us from being able to complete this project on our own.”

“This engagement is wonderfully refreshing. We are currently a nimble startup that helps other companies to be more efficient in their customer service. With the bot integration and LUIS, it really puts us in a different selling category.”

— Don Voogd, Instaply


“It was a great experience for us. For me on the technical side, it exceeded my expectations. It was really nice to be able to finish the project during our time together. In terms of productivity, it was well planned both days. The first day we tackled the Direct Line and the second day we worked with LUIS. In general, it was a good experience and we plan to use LUIS in production very soon.”

— Gerard Espona, Chief Technology Officer, Instaply

Conclusion

Instaply offered a complete manual solution providing a way for its customers to interact with their users to answer questions and communicate. Under heavy user loads, this human system became highly inefficient, especially because most queries were derived from the same questions. Instaply identified two common ones, which made up about 15K requests.

Microsoft’s Bot Framework enabled Instaply to not just automate its responses, but provide its users with an intelligent conversation. The Bot Framework SDK provides a way to bridge Instaply’s custom messaging platform with the Bot Connector.

Instaply ended the engagement by stating that the experience was very profitable and they were glad we were able to work together to build this solution.

Future possibilities

After the engagement, we learned that Instaply’s team continued to dig deeper into Microsoft’s Cognitive Services. They are especially interested in text and language analysis. Their team has discussed how it can help them in production.

“Sentiment analysis makes sense for us as a customer support tool. It can give more overview to our customers on how they are doing with support.”

— Gerard Espona, Instaply

This hackfest was a catalyst for Instaply. We look forward to seeing what Instaply will build in this new environment.

Key learnings

Here are several key findings from this engagement:

  • Arbitrary payload from the Direct Line client can be sent to the bot using the channelData. The value can be retrieved by the bot on the session object (specifically session.message.eventSource.[payloadObject]).
  • There is no way to test the Direct Line client locally, even with ngrok tunneling and setting the ngrok URL in the bot messaging endpoint. In order to test the Direct Line client, the app had to be deployed in the cloud.

Development team

Team Collage


  • Don Voogd – Director of Business Development, North America, Instaply
  • Brett von Halle – Business Intelligence Manager, Instaply
  • Gerard Espona – Chief Technology Officer, Instaply
  • Hao Luo – Senior Technical Evangelist, Microsoft
  • Kevin Leung – Technical Evangelist, Microsoft