Travelers want unique experiences, but finding local food, sightseeing spots, and such can be challenging. With typical travel apps, travelers can find famous restaurants and locations but not local information. So travelers ask local people, which can be difficult given language barriers. For example, many travelers know sushi as Japanese food, but they don’t know great local food such as shirasu-don and motsu-yaki.

NAVITIME wants to solve the problem, so they implemented a chatbot in the Kamakura Travel Guide app. Users can use the app to find information about famous attractions. In addition, users can communicate with the bot to find local food and interesting places.

Technologies used:

NAVITIME Travel project members:

  • Ikuo Odanaka – Developer Manager, NAVITIME JAPAN
  • Shinichi Tanabe – Senior Software Development Engineer, NAVITIME JAPAN
  • Makoto Yoshihama – Software Development Engineer, NAVITIME JAPAN
  • Naoya Sasaki – Software Development Engineer, NAVITIME JAPAN
  • Ayako Omori (@ayako_omori) – Technical Evangelist, Microsoft Japan
  • Hiroyuki Watanabe (@hiwatan007) – Technical Evangelist, Microsoft Japan
  • Naoki Sato (@satonaoki) – Senior Technical Evangelist, Microsoft Japan
  • Daiyu Hatakeyama (@dahatake) – Principal Software Developer, Microsoft Japan
  • Masayuki Ota (@masota0517) – Technical Evangelist, Microsoft Japan

Customer profile

Company logo

NAVITIME JAPAN is a leading provider of navigation technology and services. They offer mainly business-to-business (B2B) and business-to-consumer (B2C) navigation applications.

For B2B, they offer navigation apps for businesspeople, consulting for transportation, and advertising. For B2C, they offer navigation apps for traveling by train, bus, car, bicycle, or on foot and started a travel business named NAVITIME Travel.

Problem statement

Language barriers can challenge even the most experienced travelers. Navitime wanted to address this challenge before the Tokyo Olympics in 2020, when foreign travelers will visit Japan.

Solution and steps

Architecture

Architecture diagram

We used the Bot Framework to implement a chatbot, and we also used the Direct Line API to communicate with the chatbot from our iOS app. (Android and Windows apps can also use Direct Line.) For extracting intent and entities from user input, we used LUIS. We supported spelling correction by using the Bing Spell Check API before passing text to LUIS. After extracting intent and entities, we call Azure Search to fetch information about local food and sightseeing spots stored in Cosmos DB.

Users can freely input text, which means that sometimes LUIS can’t identify entities. For this reason, we also use the Text Analytics API to get the key phrase. (See Detect sentiment, key phrases, and languages in the Azure Text Analytics Quick Start Guide.) We then pass the key phrase to Azure Search.

Users can also communicate with the bot by using photos. To support this scenario, we implemented image recognition by using the Bing Images Search API and Custom Vision API.

DevOps and continuous improvement are also important when developing bots. For managing bot versions, we used API Management. For storing the user-input log, we call Azure Functions and save data in Cosmos DB. We also use Azure Search, Azure Storage, and Power BI for monitoring search terms. We can get user demands from the text logs and brush up the bot again and again.

User experience by app and chatbot combination

We implemented the bot in an iOS app because we thought that a chatbot could extend the app’s user experience. We show a notification message (the green window in the following image) in the app and invite users to use the chatbot in two situations:

  1. When a user starts the app for the first time, the notification message says, “Welcome! Feel free to ask me for travel tips and spot information!”
  2. If the app detects that the user has seen several recommended articles but has not selected one, it concludes that the user can’t find suitable information and says, “Not interested in our recommended articles? You can also…”

Screen shot of notification messages

Users can tap the chat button (fourth from left) in the menu list to start communication with the chatbot. Users can input free text in English and Japanese, such as “I want to eat soba.” The bot replies “Do you want…?” and recommends a restaurant. The app also shows a menu from which users can select the next action.

Screen shot of conversation with bot

The user can tap “See More Results” to see another recommendation or tap “Try Another Keyword” to enter different keywords. Tapping the Map button displays a map; by tapping “Go to the spot (Get Direction)” a user can both see the map and how to get to the location. Because NAVITIME has own routing technology, the bot redirects the user to the NAVITIME routing web app.

Screen shot of navigation web app

Travelers tend to upload photos to social network services such as Instagram and Facebook. When other users see those photos, they want to go to that place or eat that food—but they don’t know where it is or how to get it. Users can sent photos to our bot, which helps to recognize images and tell users what it is and how to get it.

Screen shot of image recognition in bot conversation

Technical delivery

This section describes how to implement a similar bot by using Bot Framework, Cognitive Services, and Azure.

Prerequisites

Develop a bot with Bot Framework

After the prerequisite steps, we started to implement our own bot by using the MultiDialogSample code on GitHub. Because we wanted to release an iOS app, we used the Direct Line API.

Extract intent and entities with LUIS and the Bing Spell Check API

For extracting intent and entity from user messages, we used LUIS. Because users sometimes mistype and LUIS can’t extract intent, we also use the Bing Spell Check API to detect and correct typos before sending messages to LUIS with steps below.

  1. From the Azure portal, add the Bing Spell Check API.
  2. In the LUIS menu, choose My keys.
  3. Choose External Keys and Add a new key to add a Bing Spell Check API key to LUIS.
  4. Go to your LUIS app and choose Publish App in the left menu.
  5. At the bottom, choose the Add Key Association button to bind the Bing Spell Check API to LUIS.
  6. Select Enable Bing spell checker, which changes the endpoint URL. Use this URL to call LUIS with the Bing Spell Check API.

Save NoSQL data in Cosmos DB

We used the fully managed NoSQL database service Cosmos DB to store NAVITIME location information, which is formatted as JSON. We also use Cosmos DB for storing logs of communication between users and bots.

If you want to learn how to initialize Cosmos DB and write code for it, start with the tutorial Azure CosmosDB: Develop with the DocumentDB API in .NET. You need to import data for testing and production by following the instructions in How to import data into Azure Cosmos DB for the DocumentDB API.

Although LUIS can extract intents well, users sometimes input just words, not sentences, so we also use Azure Seach. With the following architecture, we can handle sentences and words and reply with the correct information.

To use Azure Search, we wrote C# code like the following.

[Serializable]
public class AzureSearchService
{
    private static readonly string QueryString = $"https://{WebConfigurationManager.AppSettings["SearchName"]}.search.windows.net/indexes/{WebConfigurationManager.AppSettings["IndexName"]}/docs?api-key={WebConfigurationManager.AppSettings["SearchKey"]}&api-version=2015-02-28&";

    public async Task<SearchResult> SearchByName(string name)
    {
        using (var httpClient = new HttpClient())
        {
            string nameQuey = $"{QueryString}search={name}";
            string response = await httpClient.GetStringAsync(nameQuey);
            return JsonConvert.DeserializeObject<SearchResult>(response);
        }
    }

    public async Task<FacetResult> FetchFacets()
    {
        using (var httpClient = new HttpClient())
        {
            string facetQuey = $"{QueryString}facet=Era";
            string response = await httpClient.GetStringAsync(facetQuey);
            return JsonConvert.DeserializeObject<FacetResult>(response);
        }
    }

    public async Task<SearchResult> SearchByEra(string era)
    {
        using (var httpClient = new HttpClient())
        {
            string nameQuey = $"{QueryString}$filter=Era eq '{era}'";
            string response = await httpClient.GetStringAsync(nameQuey);
            return JsonConvert.DeserializeObject<SearchResult>(response);
        }
    }
}

You can use it from the Bot Framework with the following code. If you want to see a sample project, go to ryanvolum/AzureSearchBot on GitHub. This project describes how to set up Azure Search and Cosmos DB and how to call Azure Search from a bot.

public virtual async Task MessageRecievedAsync(IDialogContext context, IAwaitable<IMessageActivity> result)
{
    var message = await result;
    try
    {
        SearchResult searchResult = await searchService.SearchByName(message.Text);
        if(searchResult.value.Length != 0)
        {
            CardUtil.showHeroCard(message, searchResult);
        }
        else{
            await context.PostAsync($"No musicians by the name {message.Text} found");
        }
    }
    catch(Exception e)
    {
        Debug.WriteLine($"Error when searching for musician: {e.Message}");
    }
    context.Done<object>(null);
}

Recognize images with Bing Image Search and Custom Vision Service

We can use Bing Image Search to find similar images and “best representative queries.” For example, if you send a photo of the temple seen earlier in this article to Bing Image Search, it replies with URLs of similar images and “Tsuruoka Hachimangu Temple, Kamakura” as bestRepresentativeQuery. Bing has enormous knowledge of images, and we can use it to recognize famous places and foods with the following code.

public async Task<ImageResult> GetSimilarImagesAsync(string url)
{
    using (var httpClient = new HttpClient()){
        httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", ApiKey);
        string apiUrl = BingApiUrl + $"&imgUrl={HttpUtility.UrlEncode(url)}";

        var text = await httpClient.GetStringAsync(apiUrl);
        var response = JsonConvert.DeserializeObject<BingImageResponse>(text);

        ImageResult result = new ImageResult();
        if(response.bestRepresentativeQuery.displayText != null)
        {
            result.suggestedText = response.bestRepresentativeQuery.displayText;
        }

    result.similarImages = response
    ?.visuallySimilarImages
    ?.Select(i => new SimilarImage{
        HostPageDisplayUrl = i.hostPageDisplayUrl,
        HostPageUrl = i.hostPageUrl,
        Name = i.name,
        ThumbnailUrl = i.thumbnailUrl,
        WebSearchUrl = i.webSearchUrl
    }).ToList();

    return result;
    }
}

You can find a sample project in the GitHub repo NT-D/suggesttriplocationBot and learn about how to post images to a bot and how to use Bing Image Search to recognize images and get bestRepresentativeQuery. You can see more detail in the Bing Image Search API Reference.

Although Bing Image Search helps us to find famous places and foods, it’s difficult to recognize local or lesser-known places and foods. Therefore we decided to use Custom Vision Service. To build, test, and use this API, we started with the getting-started document Build a classifier using Custom Vision Service machine learning.

Because it is important to improve the bot logic with real input from users, we want to see user input and search logs easily.

To save user searches, we call Azure Functions and store data in Cosmos DB. Azure Functions natively support Cosmos DB bindings and can save data by using only a little code, such as the following.

#r "Newtonsoft.Json"

using System;
using System.Net;
using Newtonsoft.Json;

public static HttpResponseMessage Run(HttpRequestMessage req, TraceWriter log, out string outputDocument)
{
    log.Info($"Webhook was triggered!");
    string jsonContent = req.Content.ReadAsStringAsync().Result;
    //Store Data in Cosmos DB
    outputDocument = jsonContent;

    dynamic data = JsonConvert.DeserializeObject(jsonContent);

    //Return HTTP Response (BadRequest or OK)
    if (data.first == null || data.last == null) {
        return req.CreateResponse(HttpStatusCode.BadRequest, new {
            error = "Please pass first/last properties in the input object"
        });
    }

    return req.CreateResponse(HttpStatusCode.OK, new {
        greeting = $"Hello {data.first} {data.last}!"
    });
}

You can use Power BI to access the data in Cosmos DB by following the information in Data sources in Power BI Desktop. To analyze search logs, see Analyzing your data with Power BI.

Improve response performance

We deployed the bot in the West US region to minimize response time. (The Bot Framework State service and LUIS are hosted in the West US region, and the Direct Line endpoints are in Eastern Asia, Europe, and North America.)

At first we deployed our bot in the Japan East region, which caused slow response becuase when the bot calls LUIS and the State service, the traffic round-trips the Pacific Ocean again and again. If you experience slow response, you can improve the performance by moving your bot to the West US region.

Diagram of problem assessment

Conclusion

Travelers want to have an unique expeience in trip, but hard to find local food, sightseeing spot and more with normal app. It’s also difficult to find it by communicating with local people because of language barrier.

The chatbot is now live in the Kamakura Travel Guide app. Users can find both famous and obscure local information by using both the app and its built-in bot.

“It’s difficult to make NLP logic in multi-languages, but LUIS solves this problem. LUIS is really good for getting intents in the messege.” —Ikuo Odanaka, Developer Manager, NAVITIME JAPAN

“It’s [Cosmos DB] a very cool data store. We can save NoSQL data in it and can fetch data with SQL-like queries, so it is very easy to use. Read/write speed is very quick; integration with Azure Seach is seamless.” —Shinichi Tanabe, Senior Software Developer, NAVITIME JAPAN

“It is the smoothest project for me. PaaS such as Cosmos DB and Azure Search can help us make new things rapidly.” —Shinichi Tanabe, Senior Software Developer, NAVITIME JAPAN

Additional resources