Observability in ASP.NET Core with Elasticsearch and Serilog

This post will explain what observability is and how it can be done with the Elasticsearch Platform and Serilog for ASP.NET Core applications.

What is observability?

Observability is the quality of a system that allows its internal state to be captured and examined by external tools.

Having the ability to examine the internal state of a system, when it fails, can help us to reduce the MTTR or, mean-time-to-repair. It can also help us to increase the MTBF or, mean-time-between-failures.

According to Arfan Sharif there are three pillars to observability that when analyzed together, provides a holistic view of a system's internal state, viz.:

1. Logs which are historical records of events and errors;
2. Metrics which is the numerical measurements of system performance and behaviour;
3. Traces which are the transactions or requests through the system.

How can we capture and examine the internal state of a system?

The Elasticsearch Platform is a great tool for capturing and examining the internal state of a system through its observability solution. It collects logs, metrics, traces and application performance monitoring (APM) data and unifies the data for a holistic view of the system.

Logs via Serilog

Even though the observability solution of the Elastic Platform can ingest, parse, filter and aggregate logs, it cannot generate the log stream. The logs need to be generated by the system and then ingested into Elasticsearch. The logs can either be ingested directly into Elasticsearch, via Filebeat or, the Elastic Agent.

To generate and ship the logs in an ASP.NET Core application we can use a logging library such as Serilog and configure it with the Elasticsearch Sink. This will then ship the logs from the application to Elasticsearch.

Configuring Serilog is fairly simple. Below is an extract of the appsettings.json file from the example Github repository:

"Serilog": {
  "Using": [ "Serilog.Sinks.Elasticsearch" ],
  "MinimumLevel": "Debug",
  "WriteTo": [
  {
      "Name": "Elasticsearch",
      "MessageTemplate": "{ElasticApmTraceId} {ElasticApmTransactionId} {Message:lj} {NewLine}{Exception}",
      "Args": {
      "nodeUris": "https://es-server-1:9200", // ; separated
      "indexFormat": "logs-datastream",
      "numberOfReplicas": 0, // set this to 1 for production
      "connectionGlobalHeaders": "Authorization=ApiKey base64(api-key-id:api-key)",
      "emitEventFailure": "WriteToSelfLog",
      "customFormatter": "Elastic.CommonSchema.Serilog.EcsTextFormatter, Elastic.CommonSchema.Serilog",
      "typeName": null,
      "batchAction": "Create",
      "detectElasticsearchVersion": true,
      "registerTemplate": false
  }
  }]
}

In the example above it's important to note that the indexFormat property is the name of the data stream in Elasticsearch that the logs will be written to.

The connectionGlobalHeaders property sets the Authorization header with an Elasticsearch API key which is the base64 value of api-key-id:api-key.

A customFormatter is also set to the EcsTextFormatter, which outputs a JSON serialized log entry formatted according to the Elastic Common Schema.

The other sink options can be found in the Elasticsearch Sink wiki.

Metrics and Traces via Elastic APM

Elastic APM is an application performance monitoring system that collects transaction information that comes in and out of an application. It also collects data of any unhandled exceptions and errors.

Configuring the Elastic APM NuGet package to collect the transaction information is very straight forward.

In the example appsettings.json file below, we set the name of the service and environment as we would like to see it reported in Kibana. We then provide the server URL of the Elastic APM server and a corresponding token (if one is required by the configuration).

"ElasticApm": {
  "ServiceName": "example_webapi_serilog",
  "SecretToken": "secret-token",
  "ServerUrl": "http://apm-server:8200",
  "Environment": "Development"
}

Tying the logs, metrics and traces together

Naturally, we would want to tie the logs and the APM data together for a holistic view of the internal state of our application. That's exactly what the configuration below achieves through the WithElasticApmCorrelationInfo method call.

However, there's a small caveat where we have to set the service.name property of the log entry to same value as what was set for APM. We achieve that simply by enriching the log entries with the service.name property.

builder.Host.UseSerilog((hostingContext, services, loggerConfiguration) =>
{
  var httpAccessor = hostingContext
      .Configuration
      .Get<HttpContextAccessor>();

  loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration)
	.Enrich.FromLogContext()
	.Enrich.WithElasticApmCorrelationInfo()
	.Enrich.WithEcsHttpContext(httpAccessor!)
	.Enrich.WithProperty(
      "service.name", 
      hostingContext.Configuration["ElasticApm.ServiceName"]
    );
});

Conclusion

There are three pillars we need to have in place to get a holistic view of our system, viz., logs, metrics and traces. In this post we have shown how to configure Serilog and the Elastic APM NuGet package. We have also shown how to tie the logs and the APM information together to provide us with the necessary insights to reduce the MTTR and increase the MTBF of our system.

An example repository is available on GitHub.