Logging and Monitoring with .NET/C#

This is the playbook for "code-with" customer or partner engagements


Logging and Monitoring with .NET/C#

Logging

.NET Core introduced a generic logging interface, ILogger, which is well integrated in the framework and ecosystem. The common usage scenario is to inject the concrete ILogger into the application using dependency injection, while not having any class dependent on the actual log provider (as we might change the provider in the future).

There are mainly two ways to use ILogger:

  • By injecting into the class constructor, which makes writing unit test simpler. It is recommended if instances of the class will be created using dependency injection (like mvc controllers). Not intended to be used in classes that will be instantiated directly, because it would require us to pass ILogger through the call stack (i.e. var msg = new Message(this.logger)).
public class MyController
{
    ILogger logger;
    public MyController(ILogger<MyController> logger)
    {
        this.logger = logger;
        this.logger.LogDebug($"New {nameof(MyController)} created");
    }
}
  • Using a utility class to make logging available to all library classes, without having to add ILogger to the constructor of every class that logs data. It makes unit testing a less clean as we need to provide a concrete implementation onto the static class.
internal static class ApplicationLogging
{
    internal static ILoggerFactory LoggerFactory { get; set; }
    internal static ILogger CreateLogger<T>() => LoggerFactory.CreateLogger<T>();
    internal static ILogger CreateLogger(string categoryName) => LoggerFactory.CreateLogger(categoryName);
}

public class MyController
{
    static ILogger logger = ApplicationLogging.CreateLogger<MyController>();

    public MyController()
    {
        logger.LogDebug($"New {nameof(MyController)} created");
    }
}

Logging Levels for ILogger

Logging levels for ILogger are listed below, in order of high to low importance:

Level Description Example
Critical When the application reaches a scenario where immediate attention is required. It can often cause it to end abnormally No memory or disk space
Error An unexpected exception has happened, most of the time aborting the current operation Cannot reach a REST API or failed to update a database record
Warning When something unexpected happens that requires attention, however the application remains working Configuration file was not found
Information Track general application flow Request received, file opened, user created
Debug Track important information during development or troubleshooting production system Using API located at http://myapi:8080 or Listening on port 8080
Trace Track important for development purposes, might contain sensitive information Using connection string: server=dbserver;UserID=sa;password=mysecret

Logging Providers

Even though .NET Core has built-in logging providers (debug, console, event source, etc) it is often recommended to use a specialized library that offers more configuration and sink options. Most popular libraries are:

Library Semantic Log Sinks
Serilog Yes Log Analytics, Application Insights, File, DataDog and many others
NLog Yes Application Insights, Elastic Search and many others
Log4Net No File, SQL and [many others] (https://logging.apache.org/log4net/release/config-examples.html)

A complete logging providers list can be found here.

Logging in a Library

When developing a class library we most often cannot choose the logging provider. Our responsibility instead is to provide logging support which includes deciding a category strategy:

  • One category per class: each class has a unique category (“MyLibrary.MyClass”) which allows log control at the class level. Library wide rules are still supported by defining a rule using the main namespace as category (i.e. “MyLibrary”)
  • Custom categories: allows grouping classes with same category name (“MyLibrary.Models”, “MyLibrary.Repositories”)
  • Single category: defining a user defined category valid for all library classes (“MyLibrary”)

A common pattern is to receive the logger in the class constructor (either a ILogger<T> or ILoggerFactory). ILogger<T> is a shortcut to ILoggerFactory.Create(typeof(T).Name). ILoggerFactory.Create(string category) creates a logger for the specified category as the example below illustrates:

namespace MyLibrary
{
    // This class will have the log category "MyLibrary.UsingGenericILogger"
    public class UsingGenericILogger
    {
        public UsingGenericILogger(ILogger<UsingGenericLogger> logger)
        {
        }
    }

    // This class will have the log category "MyLibrary"
    public class UsingILoggerFactory
    {
        public UsingILoggerFactory(ILoggerFactory loggerFactory)
        {
            this.logger = loggerFactory.CreateLogger("MyLibrary");
        }
    }
}

Another way to implement logging in libraries without taking dependency on ILogger can be found in the LibLog repository.

Logging in a ASP.NET Core Application

When developing an ASP.NET Core application we are responsible for choosing a log provider. By default, logging is enabled with debug and console providers as you can see in the WebHost.CreateDefaultBuilder implementation.

...
ConfigureLogging((hostingContext, logging) =>
{
    logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
    logging.AddConsole();
    logging.AddDebug();
})
...

WebHost.CreateDefaultBuilder also reads logging configuration from the Logging section in appsettings.json file. (the file appsettings.Development.json is used in addition when running the application in the Development environment). The example below defines the following rules:

  • Categories System and Microsoft: log level “Information”
  • All others: log level “Debug”
{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Information"
    }
  }  
}

An option to handle application errors in a single place is through app.UseExceptionHandler. A few more resources can be found here:

ASP.NET Core and Application Insights

Application Insights integrates well with ASP.NET Core. With little effort request, error, dependency, traces, metrics and many more information becomes available on Application Insights Portal. If you double down on Application Insights as your Log Management system you can use a sink that output application logs to it. This way application and web logs will be available in a single searchable database.

Adding a custom provider to ASP.NET Core Project

This section demonstrates how to add Serilog to an ASP.NET Core project. Adding another provider requires similar steps. Detailed instructions can be found here. For a quick implementation based on configuration files follow the steps below (assuming you start from a ASP.NET Core project using default WebHost builder):

  1. Add required nuget packages
Serilog.AspNetCore
Serilog.Settings.Configuration
Serilog.Sinks.Console (optional, in case console sink is needed)
  1. Remove the console and debug providers by modifying the Program.cs file and clearing the registered providers.
 public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>()
                .ConfigureLogging((logging) => {
                    logging.ClearProviders();
                });  // this will remove Console and Debug loggers
  1. Add Serilog to the services collection
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    app.UseMvc();

    var logger = new LoggerConfiguration()
        .ReadFrom.Configuration(this.Configuration)
        .CreateLogger();
    loggerFactory.AddSerilog(logger);
}
  1. Modify the appsettings.json to setup Serilog (in this case adding console sink). For more examples check here.
{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "Serilog": {
    "MinimumLevel": "Debug",
    "Using": [ "Serilog.Sinks.Console" ],
    "WriteTo": [
      {
        "Name": "Console"
      }
    ]
  }
}

Logging in a .NET Core Console Application

Unlike ASP.NET Core, .NET Core Console apps don’t have dependency injection by default. Adding it is simple and allows using ILogger as in the web application. The code below illustrates a way to add dependency injection to a console app:

public class Program
{
    static void Main(string[] args)
    {
        var services = new ServiceCollection();
        ConfigureServices(services);
        var serviceProvider = services.BuildServiceProvider();
        var app = serviceProvider.GetService<Application>();
        app.Run();

    }

    private static void ConfigureServices(ServiceCollection services)
    {
        ILoggerFactory loggerFactory = new LoggerFactory();

        services.AddSingleton(loggerFactory);
        services.AddLogging(); // Allow ILogger<T>

        var configuration = GetConfiguration();


        services.AddSingleton<IConfigurationRoot>(configuration);
        services.AddTransient<Application>();
    }

    private static IConfigurationRoot GetConfiguration()
    {
        return new ConfigurationBuilder()
            .AddJsonFile("appsettings.json", optional: true)
            .Build();
    }
}

The actual application code resides in the Application class. The Program class only has infrastructural code to setup DI and logging.

public class Application
{
    private readonly ILogger logger;

    public Application(ILogger<Application> logger)
    {
        this.logger = logger;
    }

    internal void Run()
    {
        this.logger.LogInformation("Application {applicationEvent} at {dateTime}", "Started", DateTime.UtcNow);

        Thread.Sleep(2000);

        this.logger.LogInformation("Application {applicationEvent} at {dateTime}", "Ended", DateTime.UtcNow);

        if (System.Diagnostics.Debugger.IsAttached)
        {
            Console.WriteLine("PRESS <ENTER> TO EXIT");
            Console.ReadLine();
        }
    }
}

Keep in mind that the default console log flushes content of a separated thread. If your console application is quick you might not see any log. This issue is tracked here

High Performance Logging

For high performance workloads it is recommended to use the LoggerMessage pattern. It has less computational and memory requirements compared to using ILogger extension methods.

Monitoring

As we previously discussed, monitoring allow us to validate the performance and health of a running system through key performance indicators.

In .NET a great option to add monitoring capabilities is Application Insights. By adding Application Insights to an ASP.NET Core application we get out of the box requests, errors, dependencies and many more metrics.

A few additional features of Application Insights: