The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.

[santoshpatro]

Hi Daniel,

Thanks a lot for sharing with us a great library which has a very good documentation also : https://github.com/thepirat000/Audit.NET/blob/master/src/Audit.WebApi/README.md for implementation of Audit Trail.

I need your help for a scenario detailed at : https://stackoverflow.com/questions/79362056/property-value-exceeds-the-maximum-allowed-size-64kb-if-the-property-value-is

Any help on this request is much appreciated

Thanks,
Santosh

[thepirat000]

Consider that with your current configuration, the middleware will log all requests, including those that do not reach an action method (e.g., unresolved routes or parsing errors). Additionally, it logs all request headers, response headers, and the response body.

You can probably optimize your middleware configuration by applying more filtering and using an OnSaving global custom action to modify audit events before saving. This can help reduce the size of your audit logs. Here's an example:

app.UseAuditMiddleware(_ => _
    .FilterByRequest(r => 
        !r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase) 
        && r.Path.StartsWithSegments("/api"))
    .WithEventType("{verb}:{url}")
    .IncludeHeaders()
    .IncludeResponseHeaders()
    .IncludeResponseBody());

// Add a custom action to process and trim large audit data
Audit.Core.Configuration.AddCustomAction(ActionType.OnEventSaving, scope =>
{
    var action = scope.GetWebApiAuditAction();

    // Truncate excessively large headers
    foreach (var headerKey in action.Headers.Keys)
    {
        if (action.Headers[headerKey]?.Length > 1024)
        {
            action.Headers[headerKey] = "too long...";
        }
    }

    // Truncate excessively large response headers
    foreach (var headerKey in action.ResponseHeaders.Keys)
    {
        if (action.ResponseHeaders[headerKey]?.Length > 1024)
        {
            action.ResponseHeaders[headerKey] = "too long...";
        }
    }

    // Truncate excessively large response bodies
    // NOTE: The action.ResponseBody.Length is derived from the Content-Length response header. If the server does not send this header, it will be null.
    if (action.ResponseBody is { Value: not null, Length: > 16384 })
    {
        action.ResponseBody.Value = "too long...";
    }
});

You might also consider setting up a fallback mechanism to log failed audit events in an alternative location for debugging purposes.

One approach is to use the Audit.NET.Polly library. For instance:

using Audit.AzureStorageTables.Providers;
using Audit.Polly;
using Audit.Core.Providers;

var azureTableStorage = new AzureTableDataProvider(config => config
    .Endpoint(new Uri("..."))
    .TableName(evt => "...")
    .ClientOptions(...)
    .EntityBuilder(...));

var fallbackStorage = new DynamicDataProvider(config => config
    .OnInsert(auditEvent =>
    {
        Console.WriteLine(auditEvent.ToJson());
    }));

// var fallbackStorage = new FileDataProvider(config => config.Directory(@"C:\Logs"));

Audit.Core.Configuration.Setup()
    .JsonSystemAdapter(options)
    .UsePolly(polly => polly
        .DataProvider(azureTableStorage)
        .WithResilience(resilience => resilience
            .AddFallback(new()
            {
                ShouldHandle = new PredicateBuilder().Handle<Exception>(),
                FallbackAction = args => args.FallbackToDataProvider(fallbackStorage)
            })));

Key Points:

1. Request Filtering: The FilterByRequest method ensures only specific requests (e.g., non-GET requests starting with /api) are logged.
2. Custom Action: The OnEventSaving custom action processes audit events, truncating oversized headers and body content to maintain manageable log sizes.
3. Fallback Mechanism: Configure a fallback mechanism to log failed audit events for debugging purposes.

[santoshpatro]

Thanks @thepirat000 for your response. I implemented the code changes suggested by you but still getting the error. One observation I see the custom action method : AddCustomAction is not getting triggered in this case

Here are the options that I tried

Option 1 :

public sealed class AuditTrailConfiguration(IOptions<PurgeEntitiesOptions> purgeEntitiesOptions) : IAuditTrailConfiguration
{
	private readonly IOptions<PurgeEntitiesOptions> _purgeEntitiesOptions = purgeEntitiesOptions.IsNotNull();
	/// <summary>
	/// Add the audit middleware to the pipeline
	/// </summary>
	public void AuditSetupMiddleware(IApplicationBuilder app)
	{
		// Add the audit Middleware to the pipeline
		app.UseAuditMiddleware(_ => _.FilterByRequest(r => !r.Path.Value!.EndsWith("favicon.ico") && !r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase)).WithEventType("{verb}:{url}").IncludeHeaders().IncludeResponseHeaders().IncludeResponseBody());
		// Add a custom action to process and trim large audit data
		Audit.Core.Configuration.AddCustomAction(ActionType.OnEventSaving, scope =>
		{
			var action = scope.GetWebApiAuditAction();
			// Truncate excessively large headers
			foreach (var headerKey in action.Headers.Keys)
			{
				if (action.Headers[headerKey]?.Length > 1024)
				{
					action.Headers[headerKey] = "too long...";
				}
			}

			// Truncate excessively large response headers
			foreach (var headerKey in action.ResponseHeaders.Keys)
			{
				if (action.ResponseHeaders[headerKey]?.Length > 1024)
				{
					action.ResponseHeaders[headerKey] = "too long...";
				}
			}

			// Truncate excessively large response bodies
			// NOTE: The action.ResponseBody.Length is derived from the Content-Length response header. If the server does not send this header, it will be null.
			if (action.ResponseBody is { Value: not null, Length: > 16384 })
			{
				action.ResponseBody.Value = "too long...";
			}
		});
	}

	/// <summary>
	/// Setups the audit output
	/// </summary>
	public void AuditSetupOutput(IApplicationBuilder app)
	{
		var options = new JsonSerializerOptions()
		{
			DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingDefault,
			WriteIndented = true
		};
		Configuration.Setup().JsonSystemAdapter(options).UseAzureTableStorage(config => config.ConnectionString("Connectionstring").TableName(evt => $"{_purgeEntitiesOptions.Value.TargetTableName}{DateTime.UtcNow:MMMyyyy}").ClientOptions(new TableClientOptions() { Retry = { MaxRetries = 3 } }).EntityBuilder(builder => builder.PartitionKey(auditEvent => auditEvent.Environment.UserName).RowKey(auditEvent => Guid.NewGuid().ToString("N")).Columns(col => col.FromDictionary(auditEvent => new Dictionary<string, object>() { { "EventType", auditEvent.EventType }, { "UserName", auditEvent.Environment.UserName }, { "EventDuration", auditEvent.Duration }, { "DataSize", auditEvent.ToJson().Length }, { "Data", auditEvent.ToJson() } }))));
		// Include the trace identifier in the audit events
		var httpContextAccessor = app.ApplicationServices.GetRequiredService<IHttpContextAccessor>();
		Configuration.AddCustomAction(ActionType.OnScopeCreated, scope =>
		{
			scope.SetCustomField("TraceId", httpContextAccessor.HttpContext?.TraceIdentifier);
		});
	}
}

Option 2:

public sealed class AuditTrailConfiguration(IOptions<PurgeEntitiesOptions> purgeEntitiesOptions) : IAuditTrailConfiguration
{
	private readonly IOptions<PurgeEntitiesOptions> _purgeEntitiesOptions = purgeEntitiesOptions.IsNotNull();
	/// <summary>
	/// Add the audit middleware to the pipeline
	/// </summary>
	public void AuditSetupMiddleware(IApplicationBuilder app)
	{
		// Add the audit Middleware to the pipeline
		app.UseAuditMiddleware(_ => _.FilterByRequest(r => !r.Path.Value!.EndsWith("favicon.ico") && !r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase)).WithEventType("{verb}:{url}").IncludeHeaders().IncludeResponseHeaders().IncludeResponseBody());
	}

	/// <summary>
	/// Setups the audit output
	/// </summary>
	public void AuditSetupOutput(IApplicationBuilder app)
	{
		var options = new JsonSerializerOptions()
		{
			DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingDefault,
			WriteIndented = true
		};
		// Add a custom action to process and trim large audit data
		Audit.Core.Configuration.AddCustomAction(ActionType.OnEventSaving, scope =>
		{
			var action = scope.GetWebApiAuditAction();
			// Truncate excessively large headers
			foreach (var headerKey in action.Headers.Keys)
			{
				if (action.Headers[headerKey]?.Length > 1024)
				{
					action.Headers[headerKey] = "too long...";
				}
			}

			// Truncate excessively large response headers
			foreach (var headerKey in action.ResponseHeaders.Keys)
			{
				if (action.ResponseHeaders[headerKey]?.Length > 1024)
				{
					action.ResponseHeaders[headerKey] = "too long...";
				}
			}

			// Truncate excessively large response bodies
			// NOTE: The action.ResponseBody.Length is derived from the Content-Length response header. If the server does not send this header, it will be null.
			if (action.ResponseBody is { Value: not null, Length: > 16384 })
			{
				action.ResponseBody.Value = "too long...";
			}
		});
		Configuration.Setup().JsonSystemAdapter(options).UseAzureTableStorage(config => config.ConnectionString("Connectionstring").TableName(evt => $"{_purgeEntitiesOptions.Value.TargetTableName}{DateTime.UtcNow:MMMyyyy}").ClientOptions(new TableClientOptions() { Retry = { MaxRetries = 3 } }).EntityBuilder(builder => builder.PartitionKey(auditEvent => auditEvent.Environment.UserName).RowKey(auditEvent => Guid.NewGuid().ToString("N")).Columns(col => col.FromDictionary(auditEvent => new Dictionary<string, object>() { { "EventType", auditEvent.EventType }, { "UserName", auditEvent.Environment.UserName }, { "EventDuration", auditEvent.Duration }, { "DataSize", auditEvent.ToJson().Length }, { "Data", auditEvent.ToJson() } }))));
		// Include the trace identifier in the audit events
		var httpContextAccessor = app.ApplicationServices.GetRequiredService<IHttpContextAccessor>();
		Configuration.AddCustomAction(ActionType.OnScopeCreated, scope =>
		{
			scope.SetCustomField("TraceId", httpContextAccessor.HttpContext?.TraceIdentifier);
		});
	}
}

Any help on this issue is much appreciated

[thepirat000]

Is your other custom action ActionType.OnScopeCreated triggering?

If it is, then you might not be reaching the line where ActionType.OnEventSaving is being added. Try moving Audit.Core.Configuration.AddCustomAction(ActionType.OnEventSaving, ...) to the same place where you're setting up the other custom action

Or maybe the request was just filtered out by your configuration r => !r.Path.Value!.EndsWith("favicon.ico") && !r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase) and it's not being audited by the middleware, try with r => true.

[santoshpatro]

Thanks much @thepirat000 for your response. The below configuration worked out for me. But still I wanted to get confirmation from you from implementation point of view about its validity. Also I wanted to know what exactly the values 16384 point to you. My assumption is bytes. From the error : maximum number of characters should be 32K or less , hence in that case the condition needs to be modified accordingly ?. Also can you please help me to know how to handle any error occurs. In short assuming an error occured during action method execution, in that case I want to handle the error with some logging and let allow the method to complete its execution. Right now with the current implementation in case any error occurs while the action method is executed it completely blocks the method execution. Any help on this request is much appreciated.

public sealed class AuditTrailConfiguration(IOptions<PurgeEntitiesOptions> purgeEntitiesOptions) : IAuditTrailConfiguration
{
	private readonly IOptions<PurgeEntitiesOptions> _purgeEntitiesOptions = purgeEntitiesOptions.IsNotNull();
	/// <summary>
	/// Add the audit middleware to the pipeline
	/// </summary>
	public void AuditSetupMiddleware(IApplicationBuilder app)
	{
		// Add the audit Middleware to the pipeline
		app.UseAuditMiddleware(_ => _.FilterByRequest(r => !r.Path.Value!.EndsWith("favicon.ico") && !r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase)).WithEventType("{verb}:{url}").IncludeHeaders().IncludeResponseHeaders().IncludeResponseBody());
	}

	/// <summary>
	/// Setups the audit output
	/// </summary>
	public void AuditSetupOutput(IApplicationBuilder app)
	{
		var options = new JsonSerializerOptions()
		{
			DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingDefault,
			WriteIndented = true
		};
		Configuration.Setup().JsonSystemAdapter(options).UseAzureTableStorage(config => config.ConnectionString("Connectionstring").TableName(evt => $"{_purgeEntitiesOptions.Value.TargetTableName}{DateTime.UtcNow:MMMyyyy}").ClientOptions(new TableClientOptions() { Retry = { MaxRetries = 3 } }).EntityBuilder(builder => builder.PartitionKey(auditEvent => auditEvent.Environment.UserName).RowKey(auditEvent => Guid.NewGuid().ToString("N")).Columns(col => col.FromDictionary(auditEvent => new Dictionary<string, object>() { { "EventType", auditEvent.EventType }, { "UserName", auditEvent.Environment.UserName }, { "EventDuration", auditEvent.Duration }, { "DataSize", auditEvent.ToJson().Length }, { "Data", auditEvent.ToJson().Length >= 32000 ? ManageLargeAuditEventData(auditEvent) : auditEvent.ToJson() } }))));
		// Include the trace identifier in the audit events
		var httpContextAccessor = app.ApplicationServices.GetRequiredService<IHttpContextAccessor>();
		Configuration.AddCustomAction(ActionType.OnScopeCreated, scope =>
		{
			scope.SetCustomField("TraceId", httpContextAccessor.HttpContext?.TraceIdentifier);
		});
	}

	private static string ManageLargeAuditEventData(AuditEvent auditEvent)
	{
		var action = auditEvent.GetWebApiAuditAction();
		// Truncate excessively large headers
		foreach (var headerKey in action.Headers.Keys)
		{
			if (action.Headers[headerKey]?.Length > 1024)
			{
				action.Headers[headerKey] = "too long...";
			}
		}

		// Truncate excessively large response headers
		foreach (var headerKey in action.ResponseHeaders.Keys)
		{
			if (action.ResponseHeaders[headerKey]?.Length > 1024)
			{
				action.ResponseHeaders[headerKey] = "too long...";
			}
		}

		// Truncate excessively large response bodies
		// NOTE: The action.ResponseBody.Length is derived from the Content-Length response header. If the server does not send this header, it will be null.
		if (action.ResponseBody?.Value?.ToString()?.Length > 16384)
		{
			action.ResponseBody.Value = "too long...";
		}

		return JsonSerializer.Serialize(action);
	}
}

[thepirat000]

Your approach looks good to me. As for 16384, I just used a "random" number.

The audit middleware will log the request (unless filtered out), even if the action method throws an exception.

Refer to https://github.com/thepirat000/Audit.NET/blob/0bbb103b44da7ca9f05299cbdbaabb78c1479a7f/src/Audit.WebApi/AuditMiddleware.cs#L78C12-L93C14

I suspect that something else is occurring after the exception is thrown; otherwise, you should see the exception details in the AuditApiAction.Exception property.

.Columns(col => col
    .FromDictionary(auditEvent => new Dictionary<string, object>()
    {
        { "Exception", auditEvent.GetWebApiAuditAction().Exception },
        ...

[santoshpatro]

Thanks @thepirat000 for your response. The current Audittrail implementation works almost 90% cases in lower environment. I tested it through in DEV and QA. Once it is PRODUCTION I see where the data content produced as part of API response is more it fails with the error : The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.. Now in this case I wanted to know is it possible to handle the error by just logging the exception and let allow the Web API method to execute successfully without any issues.

[thepirat000]

To ensure the Web API method executes successfully even if audit saving fails, you can use Audit.NET.Polly to configure a fallback mechanism in case of an exception. For example:

var azureTableDataProvider = new AzureTableDataProvider(config => config
    .Endpoint(new Uri("..."))
    .TableName(evt => "...")
    .ClientOptions(...)
    .EntityBuilder(...));

var fallbackDataProvider = new DynamicDataProvider(c => c.OnInsert(auditEvent =>
{
    Console.WriteLine(auditEvent.ToJson());
}));

Audit.Core.Configuration.Setup()
    .UsePolly(polly => polly
        .DataProvider(azureTableDataProvider)
        .WithResilience(resilience => resilience
            .AddFallback(new()
            {
                ShouldHandle = new PredicateBuilder().Handle<Exception>(),
                FallbackAction = args => args.FallbackToDataProvider(fallbackDataProvider)
            })));

[santoshpatro]

Thanks @thepirat000 for your response. Finally it got resolved :). Thanks a ton once again for your help and support on this request.