Securely log to blob storage using NLog with connection string in key vault.

If you do a simple google search on how to log to blob storage using NLog, you can find examples from the project page as well as posts from other developers. However, in most of the examples I have found, the connection string for the blob storage are directly embedded in the nlog.config file, which is not ideal. In this post, I show you another example of using NLog to log to azure blob storage, with the connection string coming from an azure key vault.

About NLog

In case you are new to NLog, it’s one of the most popular logging libraries as of this writing. It’s relatively straightforward to get started with NLog in your .NET core application. Literally, all I had to do to get started with logging to a file is importing a few nuget packages, add a nlog.config file and a few lines of codes in Program.cs. You can find simple tutorial to get started here.

NLog is a flexible and free logging platform for various .NET platforms, including .NET standard. NLog makes it easy to write to several targets. (database, file, console) and change the logging configuration on-the-fly

NLog project

With NLog, you can write an extension to log to any target you desire. In addition, several extensions to log to popular targets are available, one of which is the AzureStorage extension for logging to an azure storage including blob, queue, and table storage.

NLog gdc layout renderer

Gdc stands for Global Diagnostics Context. It’s just a dictionary object to hold variables so you can use them in the nlog.config file.

Use the Global Diagnostics Context when you want to make certain information available to every logger in the current process.

Gdc layout renderer

Azure Key Vault

In case you are not familiar with azure key vault, it has all the awesome security features built in to protect your secrets. You can store passwords, connection strings, certificates etc … in the vault, and stop worrying about leaking those secrets in the source codes (for the most part).

I have written a few posts on how to setup and integrate azure key vault with an asp.net core application. Checkout some of my posts below to learn more on setting up and integrating with azure key vault.

Three ways of accessing azure key vault.

Access azure key vault using certificate.

Access azure key vault using system assigned managed identity

Logging to azure blob storage using connection string from key vault.

The approach is simple. We are going to do the following in order:

1. Retrieve the connection string from key vault before loading nlog.config file.
2. Make the connection string accessible from nlog.config using gdc layout renderer.
3. Load nlog configurations.

I put all of the logic in setting things up in Program.cs as the extension method to use NLog is callable from an IWebHostBuilder object.

First, make sure you install the following nuget packages:

<PackageReference Include="Microsoft.Extensions.Configuration.AzureKeyVault" Version="3.1.0" />
<PackageReference Include="NLog.Extensions.AzureBlobStorage" Version="2.2.0" />
<PackageReference Include="NLog.Web.AspNetCore" Version="4.9.0" />

In your Program.cs, import the following packages:

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using NLog;
using NLog.Web;

The following method in my Program.cs describes the order of the calls.

public static IWebHostBuilder CreateWebHostBuilder(string[] args) {
 var webhostBuilder = WebHost.CreateDefaultBuilder(args)
  .ConfigureAppConfiguration((hostingContext, config) => {
   config.SetBasePath(hostingContext.HostingEnvironment.ContentRootPath);
   if (!hostingContext.HostingEnvironment.IsLocal() && !hostingContext.HostingEnvironment.IsTest()) {
    // 1. Load secrets from key vault.                    
    config.SetupKeyVault(hostingContext.HostingEnvironment);
   }
   // 2 & 3 Set connection string via gdc and load nlog.config file. 
   UpdateNLogConfig(config.Build(),hostingContext.HostingEnvironment);
  })
  .ConfigureLogging((host, logging) => {
   logging.ClearProviders()
    .AddConfiguration(host.Configuration.GetSection("Logging"));
  })
  .UseNLog()
  .UseStartup < Startup > ();
 return webhostBuilder;
}

 

In the above snippets, SetupKeyVault is an extension method to connect to key vault using managed identity.

public static IConfigurationBuilder SetupKeyVault(this IConfigurationBuilder builder,
      IWebHostEnvironment env)
   {
       var configuration = builder.Build();
       var keyVaultOptions = configuration.GetSection("KeyVault").Get<KeyVaultOption>();
       if (!env.IsLocal())
       {

           // use Identity Management
           var azureServiceTokenProvider = new AzureServiceTokenProvider();
           var keyVaultClient = new KeyVaultClient(
               new KeyVaultClient.AuthenticationCallback(
                   azureServiceTokenProvider.KeyVaultTokenCallback));
           builder.AddAzureKeyVault(keyVaultOptions.URL, keyVaultClient, new DefaultKeyVaultSecretManager());
       }
       return builder;
   }

After retrieving the connection string in key vault, set the value in gdc and then load nlog.config file

private static void UpdateNLogConfig(IConfiguration configuration, IWebHostEnvironment env) {
 var storageConnectionString = configuration.GetSection("Storage:ConnectionString").Get < string > ();
 GlobalDiagnosticsContext.Set("StorageConnectionString", storageConnectionString);
 var configFile = env.IsLocal() ? $ "nlog.{env.EnvironmentName}.config" : "nlog.config";
 LogManager.Configuration = LogManager.LoadConfiguration(configFile).Configuration;
}

In nlog.config file, you also need to load the extensions.

<extensions> <add assembly="NLog.Extensions.AzureBlobStorage" /> <add assembly="NLog.Web.AspNetCore" /> </extensions>

Add a AzureBlobStorage target and set the connectionString value to the value you set via gdc.

<target xsi:type="AzureBlobStorage" name="azure" layout="${layout}" connectionString="${gdc:item=StorageConnectionString}" container="nlog" blobName="${date:universalTime=true:format=yyyy-MM-dd}/${date:universalTime=true:format=HH}.log" ></target>

For reference, below is how my nlog.config looks like:

<?xml version="1.0" encoding="utf-8" ?>
<nlog
  xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      internalLogLevel="info"
      internalLogFile="D:\Home\LogFiles\internal-nlog.log"
      throwExceptions="true">
  <extensions>
    <add assembly="NLog.Extensions.AzureBlobStorage" />
    <add assembly="NLog.Web.AspNetCore"/>
  </extensions>
  <variable name="logDirectory" value="D:\Home\LogFiles" />
  <variable name="layout" value="${longdate}|${level:uppercase=true}|${logger}|${message}|${exception:format=tostring:innerFormat=tostring:maxInnerExceptionLevel=1000}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" />
  <!-- the targets to write to -->
  <targets>
    <target xsi:type="AzureBlobStorage"
              name="azure"
              layout="${layout}"
              connectionString="${gdc:item=StorageConnectionString}"
              container="nlog"
              
              blobName="${date:universalTime=true:format=yyyy-MM-dd}/${date:universalTime=true:format=HH}.log" ></target>
    <!-- all logs. Uses some ASP.NET core renderers -->
    <target xsi:type="File" name="allLogs" fileName="${logDirectory}\nlog-all.log"
            layout="${layout}"
            archiveFileName="${logDirectory}\archives\nlog-all-{#}.log"
            archiveNumbering="Date"
            archiveDateFormat="yyyyMMdd"
            maxArchiveFiles="30"
            archiveEvery="Day"/>
    <target xsi:type="Debugger" name="debugger" layout="${layout}" />
  </targets>
  <!-- rules to map from logger name to target -->
  <rules>
    <logger name="*" minlevel="Debug" writeTo="debugger" />
    <logger name="*" minlevel="Debug" writeTo="azure" />
    <logger name="*" minlevel="Debug" writeTo="allLogs" />
  </rules>
</nlog>

If everything works, you should see outputs in the internal nlog file, like the following:

2020-03-19 21:43:23.4229 Info Loading assembly: NLog.Extensions.AzureBlobStorage
2020-03-19 21:43:23.4229 Info Loading assembly: NLog.Web.AspNetCore
2020-03-19 21:43:23.5335 Info Adding target BlobStorageTarget(Name=azure)

That’s it. Hope this post is helpful to you. Let me know in the comments if you have any questions or feedback.

References

NLog AzureStorage extension

Gdc layout renderer

NLog getting started with ASP.NET core 3