I've been working on implementing an Audit Trail in an ASP.NET Core 8 Web API, utilizing the Audit.NET library, specifically with the AzureStorageTableDataProvider for storing audit events. The library's documentation can be found here:
https://github.com/thepirat000/Audit.NET/blob/master/src/Audit.WebApi/README.md
All looks good in lower environments (DEV, QA). On deploying it PRODUCTION I am seeing error with bad request :
The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:xxxxx-yyyy-xxxx-yyyy-xxxxxxxx
Time:2025-01-16T04:59:02.9744140Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge
Here goes the code details :
public void AuditSetupMiddleware(IApplicationBuilder app)
{
// Add the audit Middleware to the pipeline
app.UseAuditMiddleware(_ => _
.FilterByRequest(r => !r.Path.Value!.EndsWith("favicon.ico") && !r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase))
.WithEventType("{verb}:{url}")
.IncludeHeaders()
.IncludeResponseHeaders()
.IncludeResponseBody());
}
public void AuditSetupOutput(IApplicationBuilder app)
{
var options = new JsonSerializerOptions()
{
WriteIndented = true
};
Configuration.Setup()
.JsonSystemAdapter(options)
.UseAzureTableStorage(config => config
.Endpoint(new Uri(_appOptions.Value.AuditTrailStorageAccountUrl?.Replace(Constants.Constant.Blob, Constants.Constant.Table)!), new ManagedIdentityCredential(_appOptions.Value.UserAssignedClientId))
.TableName(evt => $"{_purgeEntitiesOptions.Value.TargetTableName}{DateTime.UtcNow:MMMyyyy}")
.ClientOptions(new TableClientOptions() { Retry = { MaxRetries = 3 } })
.EntityBuilder(builder => builder
.PartitionKey(auditEvent => auditEvent.Environment.UserName)
.RowKey(auditEvent => Guid.NewGuid().ToString("N"))
.Columns(col => col
.FromDictionary(auditEvent => new Dictionary<string, object>()
{
{ "EventType", auditEvent.EventType },
{ "UserName", auditEvent.Environment.UserName },
{ "EventDuration", auditEvent.Duration },
{ "DataSize", auditEvent.ToJson().Length },
{ "Data", auditEvent.ToJson().Length >= 32000 ? CompressAuditEventData(auditEvent.ToJson()): auditEvent.ToJson()}
}))));
// Include the trace identifier in the audit events
var httpContextAccessor = app.ApplicationServices.GetRequiredService<IHttpContextAccessor>();
Configuration.AddCustomAction(ActionType.OnScopeCreated, scope =>
{
scope.SetCustomField("TraceId", httpContextAccessor.HttpContext?.TraceIdentifier);
});
}
/// <summary>
/// Compress the AuditTrail data in case characters count is about 32 K or less to fix the error produced
/// as part of Azure Table Storage limitation: The property value exceeds the maximum allowed size (64KB).
/// If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
/// </summary>
/// <param name="auditTrailUnCompressedData"></param>
/// <returns></returns>
/// <exception cref="ArgumentNullException"></exception>
private static string CompressAuditEventData(string auditTrailUnCompressedData)
{
if (string.IsNullOrWhiteSpace(auditTrailUnCompressedData))
{
throw new ArgumentNullException(nameof(auditTrailUnCompressedData));
}
byte[] dataToCompress = Encoding.UTF8.GetBytes(auditTrailUnCompressedData);
byte[] compressedData = Compress(dataToCompress);
return Encoding.UTF8.GetString(compressedData);
}
AuditTrailFilter.cs
internal static MvcOptions AuditSetupFilter(this MvcOptions mvcOptions)
{
mvcOptions.AddAuditFilter(config => config
//.LogRequestIf(r => (r.Method != "GET") && !((r.HttpContext.Request.HasFormContentType)))
.LogRequestIf(r => !(r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase)) && !((r.HttpContext.Request.HasFormContentType)))
//.LogAllActions()
.WithEventType("{verb} {controller}.{action}")
.IncludeHeaders(ctx => !ctx.ModelState.IsValid)
.IncludeModelState()
.IncludeResponseHeaders()
.SerializeActionParameters()
);
// Ignore ActionParameters
Audit.Core.Configuration.AddCustomAction(ActionType.OnEventSaving, scope =>
{
scope.GetWebApiAuditAction().ActionParameters = null;
});
return mvcOptions;
}
The above filter is called from the Program.cs as mentioned below :
builder.Services.AddControllers(config =>
{
config.AuditSetupFilter();
});
I already tried to compress the data in case characters count is about 32 K or less to fix the error produced as part of Azure Table Storage limitation but still it is not working for me
Can anyone help me here with some code sample which will serve as a reference for my implementation?
Consider that with your current configuration, the middleware will log all requests, including those that do not reach an action method (e.g., unresolved routes or parsing errors). Additionally, it logs all request headers, response headers, and the response body.
You can probably optimize your middleware configuration by applying more filtering and using an OnSaving global custom action to modify audit events before saving. This can help reduce the size of your audit logs. Here's an example:
app.UseAuditMiddleware(_ => _
.FilterByRequest(r =>
!r.Method.Equals(nameof(HttpMethod.Get), StringComparison.OrdinalIgnoreCase)
&& r.Path.StartsWithSegments("/api"))
.WithEventType("{verb}:{url}")
.IncludeHeaders()
.IncludeResponseHeaders()
.IncludeResponseBody());
// Add a custom action to process and trim large audit data
Audit.Core.Configuration.AddCustomAction(ActionType.OnEventSaving, scope =>
{
var action = scope.GetWebApiAuditAction();
// Truncate excessively large headers
foreach (var headerKey in action.Headers.Keys)
{
if (action.Headers[headerKey]?.Length > 1024)
{
action.Headers[headerKey] = "too long...";
}
}
// Truncate excessively large response headers
foreach (var headerKey in action.ResponseHeaders.Keys)
{
if (action.ResponseHeaders[headerKey]?.Length > 1024)
{
action.ResponseHeaders[headerKey] = "too long...";
}
}
// Truncate excessively large response bodies
// NOTE: The action.ResponseBody.Length is derived from the Content-Length response header. If the server does not send this header, it will be null.
if (action.ResponseBody is { Value: not null, Length: > 16384 })
{
action.ResponseBody.Value = "too long...";
}
});
You might also consider setting up a fallback mechanism to log failed audit events in an alternative location for debugging purposes.
One approach is to use the Audit.NET.Polly library. For instance:
using Audit.AzureStorageTables.Providers;
using Audit.Polly;
using Audit.Core.Providers;
var azureTableStorage = new AzureTableDataProvider(config => config
.Endpoint(new Uri("..."))
.TableName(evt => "...")
.ClientOptions(...)
.EntityBuilder(...));
var fallbackStorage = new DynamicDataProvider(config => config
.OnInsert(auditEvent =>
{
Console.WriteLine(auditEvent.ToJson());
}));
// var fallbackStorage = new FileDataProvider(config => config.Directory(@"C:\Logs"));
Audit.Core.Configuration.Setup()
.JsonSystemAdapter(options)
.UsePolly(polly => polly
.DataProvider(azureTableStorage)
.WithResilience(resilience => resilience
.AddFallback(new()
{
ShouldHandle = new PredicateBuilder().Handle<Exception>(),
FallbackAction = args => args.FallbackToDataProvider(fallbackStorage)
})));
FilterByRequest method ensures only specific requests (e.g., non-GET requests starting with /api) are logged.OnEventSaving custom action processes audit events, truncating oversized headers and body content to maintain manageable log sizes.