Gigseekr worked with Microsoft to produce a solution that would help users get information about artists, live music events, and live music venues for specific locations or dates.

The solution used the Microsoft Bot Framework together with LUIS from Microsoft Cognitive Services. This allows consumers to use natural language to find information on artists, where and when artists are playing, recommended artists, and other information. Users also can find nearby live music venues as well as live music events featuring given artists, gig styles, music styles, or locations.

The gigseekr bot is written using the Microsoft Bot Framework and published on Skype and Facebook Messenger. The goal also is to take advantage of other channels supported by the Bot Framework at a later stage.

Key technologies

Customer profile

Gigseekr provides information about live music events and venues as well as artists performing within the UK and Ireland. It is a brand name of Dam Good Media (DGM), a UK-based company that provides data solutions for the live entertainment industry. DGM is the creator of the popular Pepper mobile app, which focuses on social live music discovery.

DGM has a longstanding partnership with Microsoft, having started as a Microsoft BizSpark company. DGM now uses Microsoft Azure for all of its hosting and online cloud services as well as providing gig and music event data for Bing search services. DGM also worked very closely with Microsoft on the development of the Pepper app for Windows and Windows phone. Therefore, selecting Microsoft as a development partner for their bot implementation was a natural fit for DGM.

“We’re a Microsoft house in terms of development—everything is done in C# anyway, so there is a big attraction there. It is great that the Bot Framework supports so many channels—I’m not having to develop for Facebook and then Skype, etc. I like idea of creating something once!”

— David Hamilton, CEO, gigseekr

Problem statement

Using social media and conversational user interface was a natural fit for the sort of queries that gigseekr wants to allow users to ask.

While there is an existing gigseekr website, most of the traffic to the website is referred from Bing search and therefore the gigseekr brand is not well known or promoted. DGM sees the bot as the first promotable consumer service that carries the gigseekr brand identity.

Gigseekr chose to start with a bot because it is a relatively simple service to design and develop compared to a fully featured, cross-platform mobile app as well as being suited to the sort of conversational user interface that is required.

Prior to the hack, gigseekr planned to tackle the following scenarios:

  • Search for an artist
    • Search for an artist > Artist details
    • When are they performing next?
    • Where are they performing next?
    • When are they performing next near me?
    • Search for similar artist
  • Search for a location
    • Locations near me > Location details with address
    • Artists performing near me with a timeframe; for example, “which bands are playing near me this weekend?”
  • Proactive notifications of favorite artists
    • Tell me when ‘artist’ is playing near me next > Bot proactively alerts user
  • Bot plumbing
    • Implementation of a help dialog
    • Implementation of a triage dialog and routing between various dialogs
  • Cognitive Services
    • LUIS for natural language processing, intent, and entity extraction

Solution and steps

The high-level solution is a Microsoft Bot Framework bot that leans heavily on LUIS to understand the user’s intent and help navigate them to either an artist, event, or venue (referred to as an ‘entity’ by gigseekr). From this point the user can invoke several other intents relating specifically to that entity, such as buy tickets or get recommended artists.

The majority of the bot involved relatively standard use of dialogs and dialog stack management. However, research was done in several areas and new patterns and practices were discovered. (See the Dialog Flow section for details on how the dialogs were implemented.)

In terms of the bot’s architectural implementation, users access the bot via social media channels such as Skype or Facebook. This uses the Bot Connector service to interface with the bot’s web app, which in turn uses Microsoft Cognitive Services and gigseekr’s own API to provide NLP, recommendations, and other supporting data back to the bot to be displayed to the user.

The following image shows the overall architecture of the final bot:

Dialog flow

Gigseekr chose to implement a level of abstraction of the basic dialogs provided by the BotBuilder SDK.

The following diagram shows a high-level view of the dialog abstractions:

User interface considerations

A long-term goal of gigseekr is to use the bot with digital assistants such as Cortana and Alexa. Therefore, we made a design constraint to keep the user interface as text-based as possible without compromising it. Buttons and text have been used throughout the bot but richer interfaces such as carousels and cards have not been used because they are not likely to be compatible with Cortana and other text or voice-only digital assistants.

This is an example of the simple yet intuitive bot UI shown in Skype:

edsheeran

LUIS architecture: using multiple models concurrently

The nature of the gigseekr bot mandated that users are likely to traverse from one logical location to another without following a linear flow. As an example, a user might follow a flow like this:

  1. “Tell me about Metallica” > Bot takes the user to the Metallica details dialog.
  2. “When are they playing next?” > Bot lists upcoming events for Metallica.
  3. User clicks an event from the list > Bot takes the user to the event details page.
  4. “Tell me more about the venue” > Bot takes the user to the venue details page for the venue where the event will take place.
  5. “Who else is playing at this venue?” > Bot lists other upcoming events at this venue.
  6. User clicks another event (for example, a Korn gig) > Bot takes the user to the event details page.
  7. “Tell me more about Korn” > Bot take user to the artist details page for Korn.
  8. The flow proceeds like this endlessly with no clear exit point.

There was a requirement to support these endless flows between the six primary logical locations listed below:

  • Artist
  • List of Artists
  • Event
  • List of Events
  • Venue
  • List of Venues

Each of these logical locations has its own list of LUIS intents that should be supported, some of which are unique to a logical location (for example, “show me related artists” applies only to the artist location, “buy tickets” applies only to the event location). Additionally, the primary entity is known by the time the user is at one of these locations; therefore, knowledge of the artists, event, and venue LUIS entities is not required.

While supporting these entity-based flows, the bot also had a requirement to respond to base commands such as “help,” “abuse,” and “start again” or allowing the user to jump to a new search out of context of the existing path. In the Metallica example above, the user may at any point choose to ask about a completely different artist, venue, or event and therefore start again. There is no specific interface that teaches users that these global commands are available, but if they were to ask for “help,” it would show them.

This was a tricky conceptual challenge in terms of LUIS, and during the hack we discussed the LUIS architecture in some detail.

We finally created a base LUIS model that supports various entities and intents relating to base (global) commands and initial searches. We also implemented a LUIS model for artists, venues, and events separately, each with their own unique intents and entities.

The LuisDialog class in the C# Bot Builder SDK is also problematic in this scenario because it only supports a single LUIS model at any one time. This means that users can only use artist/venue/event-specific intents or base ones. This causes a significant constraint on the flow of the bot and severely impacts its usefulness.

The solution used was to implement a custom class that did the call to the LUIS API directly and allowed a cascade of multiple models. Depending on the logical location, the class would call either the artist, venue, or event-specific models first, but would fall back to the base model, thus supporting all the required commands—both base ones and those specific to the logical location.

The following image shows the architecture of the different LUIS dialogs:


The custom Base dialog deals with some basic message routing and supports various commands that help with debugging as well as handling generic interactions such as ‘please’ and ‘thank you’ or handling abusive queries. The following code provides further implementation details:

protected async virtual Task OnAbuseAsync(IDialogContext context, string reply = "")
{
    await this.SendTypingAsync(context);
    if (String.IsNullOrEmpty(reply))
        reply = "There is no need to speak to me like that! I may not actually have feelings, but I know it isn't nice.";
    await this.PostAndWaitAsync(context, reply);
}

protected async virtual Task OnThanksAsync(IDialogContext context, string reply = "")
{
    await this.SendTypingAsync(context);
    if (String.IsNullOrEmpty(reply))
        reply = "I'm glad I can be helpful - go on, ask me something else...";
    await this.PostAndWaitAsync(context, reply);
}


The custom LuisBase class inherits from base and adds support for calling and processing results from LUIS. This replaces the LuisDialog class provided by the Bot Framework. The LuisBase class enabled the bot to pass the user message through multiple LUIS models or alternatively bypass LUIS entirely.

protected override async Task ProcessMessageAsync(IDialogContext context, IAwaitable<IMessageActivity> activity)
{
    if (await BypassLuis(context, activity))
        return;

    if (_usablePriority == null)
        SetLuisModelPriority();

    if (_models == null)
    {
        return;
    }

    if (_models.Count == 0)
    {
        return;
    }

    var msg = await activity;
    string query = msg.Text;

    List<DGM.Utilities.Bot.Microsoft.Luis.Models.IntentRecommendation> acceptedIntents = null;
    List<DGM.Utilities.Bot.Microsoft.Luis.Models.EntityRecommendation> entities = null;
    foreach (DataModels.Luis.Enabled l in _usablePriority)
    {
        //if the item is set to must use, or if there are no intents yet found, or the only intent found is none then process the luis model
        bool shouldUse = false;
        if (acceptedIntents == null || l.MustUse)
            shouldUse = true;

        if (!shouldUse && acceptedIntents != null)
        {
            List<DGM.Utilities.Bot.Microsoft.Luis.Models.IntentRecommendation> noneIntents = acceptedIntents.Where(x => x.Intent != "none").ToList();
            if (noneIntents == null || noneIntents.Count == 0)
                shouldUse = true;
        }

        if (shouldUse && _models.ContainsKey(l.Id))
        {
            DGM.Utilities.Bot.Microsoft.Luis.Models.Result result = await GetLuisResult(query, _models[l.Id]);
            List<DGM.Utilities.Bot.Microsoft.Luis.Models.IntentRecommendation> intents = result?.Intents.Where(x => x.Score.HasValue && x.Score > MIN_TRUST && x.Intent != "none").ToList();
            if (intents?.Count > 0)
            {
                if (acceptedIntents == null)
                    acceptedIntents = new List<DGM.Utilities.Bot.Microsoft.Luis.Models.IntentRecommendation>();
                acceptedIntents.AddRange(intents);
            }

            if (result?.Entities?.Count > 0)
            {
                if (entities == null)
                    entities = new List<DGM.Utilities.Bot.Microsoft.Luis.Models.EntityRecommendation>();
                entities.AddRange(result.Entities);
            }
        }
    }
    await OnLuisResultsAsync(context, query, acceptedIntents, entities);
}


The custom ApplicationBase class inherits from LuisBase and deals with routing messages to the default intents set up in the Base class. There are further classes that deal with handling specific actions for artists, events, venues, and more.

LUIS phrase lists

The search LUIS model has the following entities that map to a known data type in the gigseekr APIs:

  • Artist: for example, Metallica, Muse, Frank Turner
  • ArtistType: for example, Band, Solo Artist, Singer/Songwriter
  • EventType: for example, Gig, ClubNight, Residency
  • Genre: for example, Pop, Folk, Country
  • Location: for example, London, Birmingham, Glasgow (see more on why this exists in the LUIS Geography section)
  • Venue: for example, Cavern Club, The Live Lounge, The Deaf Institute

During the initial testing for LUIS, we struggled to have LUIS differentiate between entities with any usable accuracy, especially artist and venue, because many artist and venue names sound the same. For example, even humans may assume that ‘The Purple Turtle’ is a band name, but it is actually a venue in Reading, UK.

To resolve this, we started to use phrase lists to give LUIS training indicators. Phrase lists can contain up to 5,000 comma-separated terms that map to an entity with a maximum of 10 phrase lists per model.

We used the top 1,500 artists, venues, and locations (see more on why this exists in the LUIS Geography section) from gigseekr’s database as phrase lists and in doing so, we saw a huge jump in accuracy of entity identification.

To prepare the phrase lists, we found this Convert Column to Comma Separated List tool very useful. It converts columns of data into comma-separate text blocks.

Here is an example of the training utterance used to train LUIS for some of these entities:

    {
      "text": "i'm looking for rock gigs in worcester on saturday night",
      "intent": "EventSearch",
      "entities": [
        {
          "entity": "Genre",
          "startPos": 16,
          "endPos": 19
        },
        {
          "entity": "EventType::Gig",
          "startPos": 21,
          "endPos": 24
        },
        {
          "entity": "Location",
          "startPos": 29,
          "endPos": 37
        }
      ]
    },


LUIS hierarchical entities

The EventType entity maps to one of five types of event:

  • Gig
  • Residency
  • ClubNight
  • OpenMic
  • Showcase

This was a great use of hierarchical entities in LUIS. This is an entity type that has a list of ‘child’ entities (up to 10 children), enabling LUIS to much more accurately train for the entity because it knows it must map to one of the child entities.

The JSON for this type of entity is as follows.

  "entities": [
    {
      "name": "EventType",
      "children": [
        "Gig",
        "Residency",
        "ClubNight",
        "OpenMic",
        "Showcase"
      ]
    },


LUIS geography

LUIS has a range of built-in entities for common things such as DateTime, Currency, and Ordinals. One of the built-in entities that we wanted to use was ‘Geography,’ which helps identify names of towns, cities, postal codes (zip codes), and latitude and longitude coordinates.

Geography was key to gigseekr’s requirement in order to support location-based queries such as:

  • “Who is playing in Worcester tonight”
  • “Are there any singer/songwriters in Birmingham on Saturday”

Upon testing the built-in Geography entity, we found that it worked well for locations in the United States and major cities such as London, but failed to recognize even large UK towns and cities such as Leeds, Oxford, and Bristol. This meant the built-in Geography entity was useless for gigseekr because we needed to be able to identify even small towns and villages within the UK.

To address this problem, we created our own ‘Location’ entity, which we trained with a phrase list of the top 1,000 towns and cities in the UK (sorted by population).

You can download the CSV file we used for the phrase list, which contains the top 1,000 UK place names.

Proactive bot messages and notifications

To enable users to subscribe to notifications about artists, gigseekr has a requirement to support utterances such as:

  • “Let me know when Metallica are touring”
  • “Let me know when Metallica are playing near me”

If a user subscribes, the bot will proactively message the user in response to an event in the back-end systems. The key thing about this requirement is that the messages are generated by some back-end system that in gigseekr’s case will be the same system that handles their mobile app notifications.

There was no proven pattern for these kinds of capabilities, so Microsoft team members Tomi Paananen and Lilian Kasem worked on this problem as part of the hack.

Tomi documents the solution here:

Conclusion and next steps

The gigseekr team has been able to make a great start with their bot and hopes to have it available via Skype, Facebook, and Cortana in the near term.

Some of the key learnings from the hack that could be applied to other scenarios are as follows:

  • LUIS architecture: Using multiple models concurrently to enable combinations of global and logical location-specific utterances.
  • LUIS phrase lists to help train LUIS.
  • LUIS ‘Geography’ doesn’t work well in the UK so use a custom entity with phrase lists instead.
  • Proactive bot messages/notifications are a great mechanism to encourage users to come back into the bot and reinitiate the conversation.

The next step for gigseekr is to further extend the solution, publish it via Skype and Facebook, and promote to their users via the website and social media channels.

Gigseekr also plans to integrate the bot with digital personal assistants such as Cortana and Alexa as and when the technology to enable this becomes available to realize proactive/insight-driven scenarios such as: “Hi, I see Adele has just announced a tour and has dates in Oxford, which is near where you live. Would you like me to get you some tickets?”

Core team