Microsoft and LISA teamed up to develop a universal logging service for all LISA application components and resources. Azure Log Analytics serves as the foundation for the LISA App universal logging service. Using Azure Functions, the LISA App universal logging service is accessible to all the application components.

Core team

  • Vince Mease – Director of Product Management, LISA
  • Martin Schray – Senior Technical Evangelist, Microsoft
  • David Giard – Senior Technical Evangelist, Microsoft

In our solution, we used the capabilities of Azure Log Analytics and the development and deployment environment of Azure Functions to build the LISA App’s universal log service. The universal log service provides all application components easy access to logging and a unified application logging view for developers and administrators.

Customer profile

LisaApp Logo

The LISA App focuses on delivering beauty and massage services in both B2B and B2C environments in the Chicago area. With its app, LISA has created an open marketplace where clients can get these services at a place and price they want. LISA also wants those who provide those services to be able to work when and where they want, at the price they set and advertise on the platform. LISA’s beta launched in Q2 2016.

Pain point

Like many development teams, the LISA App team used several application logging methods. There were multiple ways to search and view logs, and applications log data could be logged to different locations. Because there were multiple ways to search, log, and store applications, it was difficult to view the relationships between all the system components, interactions, and logging methods.

The LISA App team wanted their universal logging service to not only log all relevant information but also to search, sort, filter, and view their logged data. Although it was important to be able to create, easily view, and search their logs, a universal logging service wasn’t a high enough development priority for anyone to work on it. It wasn’t a customer-facing feature and its development seemed too time-consuming, but the capabilities of a universal logging service would provide great benefit to the developers.

Solution, steps, and delivery

The LISA App team wanted their universal logging service to have the following features:

  • Ability for any application component (from shell scripts to application code in different programming languages) to easily access the service’s logging capabilities.
  • Easy deployment from version control.
  • Elastic scaling as need and use grows.
  • Ability to capture all incoming log data for archival and future analysis.

LISA decided Azure had the components they needed to quickly build their universal logging service—an end-to-end logging and log-viewing solution. Working with Microsoft, the LISA App team decided that Azure Log Analytics could ingest log data, store and archive that data, and provide easy-to-use but sophisticated log data searching, sorting, and filtering.

The team started by designing the following architecture on Azure:

LISA App universal logging service architecture

LisaApp Universal Logging Service Architecture

Technical details

Because LISA is a fast-moving and growing company, the team decided to hold several mini-hackfests instead of one large one. This way we could design and create the universal logging service but not impact the LISA Apps’ overall product development roadmap.

Azure Functions

Even though Azure Log Analytics has a well thought-out and documented REST API, the LISA App team wanted to make it even easier to use. Because the team did not want their developers to build their own language-specific wrapper to log Analytics REST API, the team decided to build a REST endpoint to take care of the Azure Log Analytics service messages authorization and packaging. Developers like this because it takes care of Azure specific authentication and signature generation for them. They can use the Azure Functions endpoint for their logging needs.

The LISA App team chose Azure Functions because they can use it with no upfront cost; they pay only when their code runs and they can scale their usage as needed. The LISA App team also liked Azure Functions’ support for writing functions in languages such as C#, F#, Node.js, Python, and PHP.

Azure Functions language choices

Azure Function Language Choices

Azure functions allowed nearly immediate productivity because the team was able to start writing code (in languages like C#, Node.js, PHP, Python, and Bash) without setting up a server infrastructure or application framework. The Azure Functions environment allows you to write, test, and monitor code. Azure Functions also lets you quickly:

  • Bind to various triggers such as schedules, REST or webhooks, Blob storage, events, queues, timers, and Service Bus queues or topics.
  • Read from inputs such as Blob storage, tables, and DocumentDB.
  • Bind to various output tables, DocumentDB, and push notifications.

You can think of triggers, inputs, and outputs this way:

  • Triggers are what will start your function. For example, you can have a trigger be a new queue message or HTTP request. Your function must always have a trigger.
  • Input bindings provide additional data to your function when triggered. For example, you can fetch data from Table storage or receive data from your HTTP request. Inputs are optional.
  • Output bindings allow you to output data from your function. For example, you can add a new item to a queue, Blob storage, HTTP response, and DocumentDB or other outputs. Outputs are optional.

As shown below, Azure Functions provides a great editor for selecting triggers and input and output bindings.

Azure Functions triggers, inputs, and outputs

Azure Functions trigger, Inputs and Outputs

Once you define the triggers, the inputs and outputs are stored in the function directory with the file name of function.json.

Here is an example function.json file with an HTTP trigger and an HTTP output binding. It doesn’t include an input binding.

{
  "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
  ],
  "disabled": false
}

Azure Functions allowed the team to easily incorporate shared code. The team chose to write our Azure Functions in Node.js. The next example shows where we used the standard Node.js approach (for example, require) to pull in functionality from the AzureLogAnalyticsHelper.

For our work, we used the Azure Functions HTTP trigger. This allows an HTTP request to trigger our code. The code below is the Azure function that receives an HTTP request and, after validation, calls writeLogEntry to use Azure Log Analytics to write a log message.

In the following example, note that a parameter named context is passed in along with the HTTP request. The Azure Functions runtime uses a context object to pass data to and from the function and to let you communicate with the runtime. The context object is the first parameter to a function and must be included because its methods (such as context.done and context.log) are required to correctly use the runtime. You can name the context object whatever you like (such as ctx or c), but it is called context by default.

// Import the AzureLogAnalyticsHelper.js library
var logger = require("../shared/AzureLogAnalyticsHelper.js");
  
module.exports = function (context, req) {
    context.log(`JavaScript HTTP trigger function processed a request with req of ${req.body.test}.`);
 
    if (req.query.msg || (req.body && req.body.msg)) {
 
        context.log(`Precessing ${req.body.msg}.`);
 
        // write to logging message to Azure Log Analytics Service
        logger.writeLogEntry(context, req.body.msg);

        res = {
            // status: 200, /* Defaults to 200 */
            body: "Hello " + (req.query.msg || req.body.msg)
        };
    }
    else {

        res = {
            status: 400,
            body: "Please pass in msg in JSON format in on the query string or in the request body"
        };
    }
    context.done(null, res);
};

To trigger an HTTP-triggered Azure function, you need to retrieve the URL to use, which the following image illustrates.

Azure function URL

Azure Function URL

You can include npm packages in your JavaScript function by creating or uploading a package.json file to your function’s folder in the function app’s file system. For file upload instructions, see the How to update function app files section of the Azure Functions developer guide.

For the Log Analytics Helper to operate, you need to provide three pieces of information:

  • OPERATIONS_MANAGEMENT_WORKSPACE
  • OPERATIONS_MANAGEMENT_KEY
  • LOG_ANALYTICS_APPNAME

You can provide these settings via app settings for your Azure Function (see How to manage a function app in the Azure portal) or the AzureLogAnalyticsHelper library initialize function.



/**
 * If you don't set CustomerId, SharedKey and LogName in code or via the environment call this function with these arguments to get everything 
 * setup and ready for use of the Azure Log Analytics
 * @param {*} theCustomerId the Operations Management Workspace ID
 * @param {*} theSharedKey  the Operateions Management Key for the Azure Log Analytis Service
 * @param {*} theLogName    the log name to use when logging to the Azure Log Analytics Service 
 */
const aliasInitialize = function initialize(theCustomerId, theSharedKey, theLogName) {

    customerId = theCustomerId;
    
    sharedKey = theSharedKey;
    
    logName = theLogName;

} 

Things to note:

  • The Azure Analytics Workspace, or OPERATIONS_MANAGEMENT_WORKSPACE, is the “name” (really a GUID) that shows up under the Azure portal overview after you’ve created the Azure Log Analytics service. Make sure to create your Azure Log Analytics workspace.
  • The OPERATIONS_MANAGEMENT_KEY can be obtained after you have created your Operations Management workspace.
  • The LOG_ANALYTICS_APPNAME should be the name you want to appear for your app in the Log Analytics service. Don’t uses spaces or dashes because these will cause a runtime error from the service.

The source code of the AzureLogAnalyticsHelper can be found on GitHub.

Azure Log Analytics

Azure Log Analytics is a service in the Operations Management Suite that helps you collect and analyze data generated by resources in your cloud and on-premises environments. You can get real-time insights by using integrated search and custom dashboards to analyze millions of records across all of your workloads and servers regardless of their physical location.

Azure Log Analytics offers a simple interface and powerful query language to let you search your log entries.

Azure Log Analytics

Azure Log Analytics

Architecture

We leveraged a number of Azure components in building this solution, including:

  • Azure Functions
  • Azure Log Analytics
  • Azure Storage
  • Azure resource groups

The architecture can be seen below.

LISA App universal logging service architecture

LisaApp Universal Logging Service Architecture

Key learnings

  • Azure Functions provides serverless compute (no need to set up a server infrastructure); we used this capability to quickly build a REST endpoint to the Azure Log Analytics service. We defined an HTTP trigger allowing any application component that can do an HTTP request to use the LISA App universal logging service.
  • Azure Functions allows you to select from predefined triggers and input and output bindings to quickly define:
    • What triggers (for example, HTTP request, timer, new queue entry, new blog) your function.
    • What inputs (blob, queue, and so on) your function can access.
    • What outputs (for example, blobs, queues, tables, DocumentDB) your function can use.
  • Azure Functions makes it quick and easy to develop new capabilities without worrying about servers, virtual machines, or infrastructure.
  • When Azure continuous deployment completes a deployment, you can easily fire webhooks as a notification mechanism.
  • Azure Log Analytics offers a powerful service for log storage and discoverability.
  • Azure Log Analytics offers a REST API for its various capabilities.
  • Azure Functions app setting pulled the application setting out of application code and store. The application could then read these configuration settings for the runtime environment.

Conclusion

Insights

  • Using Azure Functions to expose a REST API that all application components could use allowed the LISA App team to quickly create a universal logging service.
  • Azure Functions configured in capacity plan mode allows you to pay only for what you use.
  • Because Azure Functions provides access to many programming languages, developers can pick their most productive language and get started immediately.
  • Azure Functions provides a rich environment for creating, modifying, testing, and monitoring your collection of Functions.
  • Using managed services such as Azure Functions and Azure Log Analytics allows you to quickly add functionality to your application.
  • Our hackfest team was made up of local and remote team members in different geographic locations. Approaches, desired goals, and outcomes were carefully and thoughtfully defined before we started; designing small, testable tasks and establishing predefined team check-ins for the distributed hackfest team were very effective and productive.

How the learnings and insights can be applied elsewhere

If you expose your APIs via RESTful approaches, the API can be used from any programming language that supports HTTP requests. For a universal application service like logging, you could build a language-specific library (such as Node.js) to be used from that language, but providing the service via a REST endpoint means that any application component in any language can use the service.

Jointly designing the goals and desired outcomes and determining the preliminary research and prerequisites before a hackfest can be one of the most important success steps. Vince Mease and the Microsoft team had planning calls leading up to the hackfests. During these calls, we focused on designing a small but complete API to create a universal logging service.

We kicked off the hackfests by creating the required Azure resources such as the Operations Management workspace and Azure Functions app. After an initial effort to create a working universal logging service, we refined and cleaned up code and added documentation, making the service easier to understand, maintain, and use. We learned that although a hackfest can quickly create working code, we needed to allocate additional time to make the code easy to read, well-documented, and robust so that it can be used in a production application.

As an agile startup, LISA needed to quickly build and test the universal logging service. Building on managed services allowed them to have the universal logging service ready for use by various application components in a short period of time. We learned that managed services can greatly accelerate development because its capabilities mean you don’t have to spend extensive time and effort creating and maintaining infrastructure. For example, we didn’t have to create a user interface and query language for log discovery because Azure’s Log Analytics service provided these capabilities.

Closing

We were able to build and deploy the LISA App universal logging service during the hackfests using Azure managed services.

Vince Mease, director of product management for LISA, used an unmet application need of universal logging as an opportunity to build needed new capabilities on Azure. Managed services in Azure such as Azure Functions and Azure Log Analytics greatly accelerated what we were able to build and accomplish. These managed services allowed us to quickly create and immediately begin using services needed in our solution. As beneficial as quickly creating and using the services can be, the real benefit is realized because the upkeep, patching, and maintenance is handled by each managed service. Additionally, these managed services can be easily scaled or even auto-scaled as demand grows and there is a need for additional capacity.

“I was surprised at how quickly and easily we were able to put the universal log service together, building it on top of Azure’s managed services like Azure Functions and Azure Log Analytics.”

— Vince Mease, director of product management, LISA

Resources