HTTP Response Body Capture

Render the posted body, e.g. FORM or Ajax POST

Platforms Supported: All

Remarks

• Will buffer the input-stream (if allowed) and capture the posted-body as string-value in the HttpContext Items Dictionary.
• .NET6 introduces UseHttpLogging that also supports logging of HTTP Response Body.

ASP.NET Core Requirements

• NLog.Web.AspNetCore 5.1.0+
• Register NLogResponseBodyMiddleware as middleware in the Configure method of Startup class

Description

If you are using NLog.Web.AspNetCore, you may need to capture the HTTP response body. The library already supports the capture of the HTTP request body via the included NLogRequestPostedBodyMiddleware and the aspnet-request-posted-body layout renderer. We can capture the response body in a similar method with a similar middleware.

We have decided to include this in source code form only, as this requires the use of a NuGet package that NLog.Web.AspNetCore does not normally use, Microsoft.IO.RecyclableMemoryStream. The reason this library is utilized is in order to use a pool of MemoryStreams instead of constantly creating and destroying MemoryStream instances where one instance is required per response body capture.

The middleware, and its corresponding options class, are provided as attachments. There is an explanation of the options and usage below.

  // needed for capture of the HTTP response body with an API Controller.
  // The options default to only logging a maximum of 30KB.
  // Also, only content types starting with ‘text/‘, or those ending with ‘xml’, ‘html’, ‘json’, or content types
  // that have the word ‘charset’ are logged, since we desire to log strings and not binary content
  // Those can be overridden in the options if necessary.  But typically the default options should be adequate.
  app.UseMiddleware<NLog.Web.NLogResponseBodyMiddleware>
        (new NLog.Web.NLogResponseBodyMiddlewareOptions());

Full Example:

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }
        else
        {
            app.UseExceptionHandler("/Home/Error");
            // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
            app.UseHsts();
        }
        app.UseHttpsRedirection();
        app.UseStaticFiles();
        app.UseRouting();
        app.UseAuthorization();

        // needed for HTTP response body with an API Controller.
        app.UseMiddleware<NLog.Web.NLogResponseBodyMiddleware>(
            new NLog.Web.NLogResponseBodyMiddlewareOptions());

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllerRoute(
                name: "default",
                pattern: "{controller=Home}/{action=Index}/{id?}");
        });

NLogResponseBodyMiddleware Options

• int MaxContentLength

  • The maximum request posted body that will be captured. Defaults to 30 KB.

• IList<KeyValuePair<string,string>> AllowContentTypes

  • The Key is the prefix value such as ‘text’.
  • The Value is the suffix value such as ‘html’.
  • The default contents of this property are:
    • 'application/', 'json'
    • 'application/', 'xml'
    • 'application/', 'html'
    • 'text/', ''
    • '', 'charset'

• Predicate<HttpContext> ShouldCapture()

  • This indicates if the response body should be saved to a MemoryStream before continuing to the caller
  • By default this method returns true
  • At this point we do not know the response body content length since the logic that creates the response can not been executed at this point.

• Predicate<HttpContext> ShouldRetain()

  • This indicates if the contents of the MemoryStream, once captured, should be saved into a key/value pair in HttpContext.Items to be logged later.
  • The key is 'nlog-aspnet-response-body'
  • By default this method returns true if
    • The content length is not null and is <= MaxContentLength
    • The content type must match one of the entries in the AllowContentTypes List.
    • For a match the ContentType must StartsWith the Key, case insensitive, and also have an IndexOf the Value, case insensitive.
    • An empty string for the Key or for the Value acts as a wildcard match.
    • The method has the HttpContext as the parameter, so a user wishing to override this property should have all the properties required to implement an override.

Source Code

using System.IO;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.IO;
using NLog.Common;

namespace NLog.Web
{
    /// <summary>
    /// This class is to intercept the HTTP pipeline and to allow additional logging of the Response Body
    ///
    /// Usage: app.UseMiddleware&lt;NLogResponseBodyMiddleware&gt;(); where app is an IApplicationBuilder
    ///
    /// Inject the NLogResponseBodyMiddlewareOption in the IoC if wanting to override default values for constructor
    ///
    /// Then add {aspnet-item:variable=nlog-aspnet-response-body} in your NLog.config
    ///
    /// This class requires the NuGet Package Microsoft.IO.RecyclableMemoryStream to build
    /// and also NLog, and NLog.Web.AspNetCore NuGet Packages
    /// </summary>
    public class NLogResponseBodyMiddleware
    {
        /// <summary>
        /// Use {aspnet-item:variable=nlog-aspnet-response-body} in your NLog.config
        /// </summary>
        public static readonly string NLogResponseBodyKey = "nlog-aspnet-response-body";
		
        private readonly RequestDelegate _next;
		
        private readonly NLogResponseBodyMiddlewareOptions _options;

        // Using this instead of new MemoryStream() is important to the performance.
        // According to the articles, this should be used as a static and not as an instance.
        // This will manage a pool of MemoryStream instead of creating a new MemoryStream every response.
        // Otherwise we will end up doing new MemoryStream 1000s of time a minute
        // This requires the NuGet Package Microsoft.IO.RecyclableMemoryStream
        private static readonly RecyclableMemoryStreamManager MemoryStreamManager = new RecyclableMemoryStreamManager();

        /// <summary>
        /// Initializes new instance of the <see cref="NLogResponseBodyMiddleware"/> class
        /// </summary>
        /// <remarks>
        /// Use the following in Startup.cs:
        /// <code>
        /// public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        /// {
        ///    app.UseMiddleware&lt;NLog.Web.NLogResponseBodyMiddleware&gt;();
        /// }
        /// </code>
        /// </remarks>
        public NLogResponseBodyMiddleware(RequestDelegate next, NLogResponseBodyMiddlewareOptions options = default)
        {
            _next = next;
            _options = options ?? NLogResponseBodyMiddlewareOptions.Default;
        }

        /// <summary>
        /// This allows interception of the HTTP pipeline for logging purposes
        /// </summary>
        /// <param name="context">The HttpContext</param>
        /// <returns></returns>
        public async Task Invoke(HttpContext context)
        {
            if (ShouldCaptureResponseBody(context))
            {
                using (var memoryStream = MemoryStreamManager.GetStream())
                {
                    // Save away the true response stream
                    var originalStream = context.Response.Body;

                    // Make the Http Context Response Body refer to the Memory Stream
                    context.Response.Body = memoryStream;

                    // The Http Context Response then writes to the Memory Stream
                    await _next(context).ConfigureAwait(false);
					
                    // Should we retain the response based on the content length and content type
                    // By default content length should be <=30KB
                    if (_options.ShouldRetain(context))
                    {
                        // This next line enables NLog to log the response
                        var responseBody = await GetString(memoryStream).ConfigureAwait(false);

			// Only save the response body if there is one
			if (!string.IsNullOrEmpty(responseBody))
			{
			    context.Items[NLogResponseBodyKey] = responseBody;
			}
                    }
					
                    // Copy the contents of the memory stream back to the true response stream
                    await memoryStream.CopyToAsync(originalStream).ConfigureAwait(false);
                }
            }
            else
            {
                if (context != null)
                {
                    await _next(context).ConfigureAwait(false);
                }
            }
        }

        private bool ShouldCaptureResponseBody(HttpContext context)
        {
            // Perform null checking
            if (context == null)
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: HttpContext is null");
                return false;
            }

            // Perform null checking
            if (context.Response == null)
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: HttpContext.Response is null");
                return false;
            }

            // Perform null checking
            if (context.Response.Body == null)
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: HttpContext.Response.Body stream is null");
                return false;
            }

            // If we cannot write the response stream we cannot capture the body
            if (!context.Response.Body.CanWrite)
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: HttpContext.Response.Body stream is non-writeable");
                return false;
            }

            // Use the ShouldCaptureResponse predicate in the configuration instance that takes the HttpContext as an argument
            if (!_options.ShouldCapture(context))
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: _configuration.ShouldCapture(HttpContext) predicate returned false");
                return false;
            }

            return true;
        }

        /// <summary>
        /// Convert the stream to a String for logging.
        /// If the stream is binary please do not utilize this middleware
        /// Arguably, logging a byte array in a sensible format is simply not possible.
        /// </summary>
        /// <param name="stream"></param>
        /// <returns>The contents of the Stream read fully from start to end as a String</returns>
        private async Task<string> GetString(Stream stream)
        {
            string responseText;

            // If we cannot seek the stream we cannot capture the body
            if (!stream.CanSeek)
            {
                InternalLogger.Debug("NLogRequestPostedBodyMiddleware: HttpApplication.HttpContext.Request.Body stream is non-seekable");
                return null;
            }

            // Save away the original stream position
            var originalPosition = stream.Position;

            try
            {
                // This is required to reset the stream position to the beginning in order to properly read all of the stream.
                stream.Position = 0;

                // The last argument, leaveOpen, is set to true, so that the stream is not pre-maturely closed
                // therefore preventing the next reader from reading the stream.
                // The middle three arguments are from the configuration instance
                // These default to UTF-8, true, and 1024.
                using (var streamReader = new StreamReader(
                           stream,
                           Encoding.UTF8,
                           true,
                           1024,
                           leaveOpen: true))
                {
                    // This is the most straight forward logic to read the entire body
                    responseText = await streamReader.ReadToEndAsync().ConfigureAwait(false);
                }
            }
            finally
            {
                // This is required to reset the stream position to the original, in order to
                // properly let the next reader process the stream from the original point
                stream.Position = originalPosition;
            }

            // Return the string of the body
            return responseText;
        }
    }
}

using System;
using System.Collections.Generic;
using Microsoft.AspNetCore.Http;
using NLog.Common;
using NLog.Web.Internal;

namespace NLog.Web
{
    /// <summary>
    /// Contains the configuration for the NLogResponseBodyMiddlewareOptions
    /// </summary>
    public class NLogResponseBodyMiddlewareOptions
    {
        /// <summary>
        /// The default configuration
        /// </summary>
        internal static readonly NLogResponseBodyMiddlewareOptions Default = new NLogResponseBodyMiddlewareOptions();

        /// <summary>
        /// The default constructor
        /// </summary>
        public NLogResponseBodyMiddlewareOptions()
        {
            ShouldCapture = DefaultCapture;
            ShouldRetain = DefaultRetain;

            AllowContentTypes = new List<KeyValuePair<string, string>>
            {
                new KeyValuePair<string, string>("application/", "json"),
                new KeyValuePair<string, string>("text/", ""),
                new KeyValuePair<string, string>("", "charset"),
                new KeyValuePair<string, string>("application/", "xml"),
                new KeyValuePair<string, string>("application/", "html")
            };
        }

        /// <summary>
        /// The maximum response body size that will be captured. Defaults to 30KB.
        /// </summary>
        /// <remarks>
        /// Since we must capture the response body on a MemoryStream first, this will use 2x the amount
        /// of memory that we would ordinarily use for the response.
        /// </remarks>
        public int MaxContentLength { get; set; } = 30 * 1024;

        /// <summary>
        /// Prefix and suffix values to be accepted as ContentTypes. Ex. key-prefix = "application/" and value-suffix = "json"
        /// The defaults are:
        ///
        /// text/*
        /// */charset
        /// application/json
        /// application/xml
        /// application/html
        /// </summary>
        public IList<KeyValuePair<string,string>> AllowContentTypes { get; set; }

        /// <summary>
        /// If this returns true, the response body will be retained
        /// for capture
        /// Defaults to true
        /// </summary>
        /// <returns></returns>
        public Predicate<HttpContext> ShouldCapture { get; set; }

        /// <summary>
        /// If this returns true, the response body will be captured
        /// Defaults to true if content length &lt;= 30KB
        /// This can be used to capture only certain content types,
        /// only certain hosts, only below a certain request body size, and so forth
        /// </summary>
        /// <returns></returns>
        public Predicate<HttpContext> ShouldRetain { get; set; }

        /// <summary>
        /// The default predicate for ShouldCaptureResponse. Returns true
        /// Since we know nothing about the response before the response is created
        /// </summary>
        private bool DefaultCapture(HttpContext context)
        {
            return true;
        }

        /// <summary>
        /// The default predicate for ShouldRetainCapture.  Returns true if content length &lt;= 30KB
        /// and if the content type is allowed
        /// </summary>
        private bool DefaultRetain(HttpContext context)
        {
            var contentLength = context?.Response?.ContentLength ?? 0;
            if (contentLength <= 0 || contentLength > MaxContentLength)
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: HttpContext.Response.ContentLength={0}", contentLength);
                return false;
            }

            if (!context.HasAllowedContentType(AllowContentTypes))
            {
                InternalLogger.Debug("NLogResponsePostedBodyMiddleware: HttpContext.Request.ContentType={0}", context?.Request?.ContentType);
                return false;
            }

            return true;
        }
    }
}