Added HostRecorder (part 1)

docs/decisions/0070-logging-crashes-auditing-measuring-usage.md

@@ -6,15 +6,19 @@
 
 # Context and Problem Statement
 
-Every developer understanding logging, and every developer turns on logging for their software (at some point), because every developer (at some point) needed logging output to diagnose issues in a running system. However, not every developer correctly takes care of crash reporting, when the system dies, nor thinks about auditing of the use of the software (for legal purposes), nor measuring certain aspects of the software, nor capturing usage metrics about how the software is actually used, until very late in the business. And often, only when it is too late to have any past data to learn from.
+Every developer understands the basic idea behind logging, and every developer turns logging ON for their software (at some point), because every developer (at some point) needed logging output to diagnose issues in a running system.
 
-All of these needs are present in every SaaS business, and collecting (and managing) that data from day one is just another thing that adds value later on when you need it.
+However, not every developer correctly takes care of crash reporting when the system dies, nor thinks about auditing of the use of the software (for legal purposes), nor measuring certain aspects of the software (e.g., perf or usage), nor capturing usage metrics about how the software is actually used by end-users, until very late in the business. And often, only when it is too late to have any past data to learn from.
 
-Rather than separating these concerns and having 5 different disparate ways of capturing this data (some data which is duplicated between the 5 mechanisms) we wish to have one abstraction to take care of it all. Why? because:
+All of these needs are ever-present in every SaaS business from day one, and collecting (and managing) that data is just another thing that adds value later on when you really need it (in the future).
+
+Logging, specifically, can create a significant load on a cloud-based system, if not handled effectively and can slow down the performance of each HTTP request that uses it (imagine how much more time and risk of failure it would be to post each logging message to a remote logging service). It is not unusual to record 10-100 log messages for any specific API call (depending on its level of complexity). That API call may also require 1-5 audits, and raise 1-10 usages. Most modern logging frameworks deal with this kind of load asynchronously in a background thread, however, handling any of the other 4 mechanisms (i.e., Crashes, Auditing, Usage, Measures) requires significant engineering to offload to asynchronous mechanisms so that the API caller is not delayed while this data is processed.
+
+Rather than separating these concerns and having 5 different disparate ways of capturing all these different kinds of data (some data which is duplicated between the 5 mechanisms) we wish to have one abstraction to take care of it all. Why? because:
 
 1. Developers of SaaS systems need to be aware of these 5 things at all times while designing their software. So having it in one abstraction reminds them of those 5 things.
-2. There needs to be an easy and consistent way to capture these 5 kinds of data (and reuse it between these 5 different mechanisms).
-3. A single well known abstraction will need to work across any cloud platform (e.g. Azure, AWS, GoogleCloud) in any deployment topology (e.g. Monolith, Microservices).
+2. There needs to be an easy and consistent way to capture these 5 kinds of data (and reuse it between these 5 different mechanisms). Rather than engineering 5 different ways, and injecting 5 different things into the code.
+3. A single well-known abstraction will need to be provided to work across any cloud platform (e.g., Azure, AWS, GoogleCloud) in any deployment topology (e.g., Monolith, Microservices, etc).
 
 ## Considered Options
 
@@ -30,4 +34,5 @@ The options are:
 
 - `IRecorder` is a single interface to access these 5 services
 - We can plugin in the standard .Net `Ilogger` to take care of diagnostic logging.
-- We can use ports and adapters to plugin any 3rd party system to handle all 5 mechanisms
+- We can use ports and adapters to plugin any 3rd party system to handle all 5 mechanisms
+- We will need to use a reliable messaging mechanism (i.e. queues) to offload this load on HTTP requests

src/.run/AllHosts.run.xml

@@ -1,7 +1,8 @@
 <component name="ProjectRunConfigurationManager">
-  <configuration default="false" name="AllHosts" type="CompoundRunConfigurationType">
-    <toRun name="ApiHost1: ApiHost1-Development" type="LaunchSettings" />
-    <toRun name="TestingStubApiHost: TestingStubApiHost-Development" type="LaunchSettings" />
-    <method v="2" />
-  </configuration>
+    <configuration default="false" name="AllHosts" type="CompoundRunConfigurationType">
+        <toRun name="ApiHost1: ApiHost1-Development" type="LaunchSettings" />
+        <toRun name="TestingStubApiHost: TestingStubApiHost-Development" type="LaunchSettings" />
+        <toRun name="WebsiteHost: WebsiteHost-Development" type="LaunchSettings" />
+        <method v="2" />
+    </configuration>
 </component>

src/AncillaryApplication.UnitTests/AncillaryApplication.UnitTests.csproj

@@ -6,13 +6,12 @@
     </PropertyGroup>
 
     <ItemGroup>
-        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
+        <ProjectReference Include="..\AncillaryApplication\AncillaryApplication.csproj" />
+        <ProjectReference Include="..\UnitTesting.Common\UnitTesting.Common.csproj" />
     </ItemGroup>
 
     <ItemGroup>
-        <ProjectReference Include="..\AncillaryApplication\AncillaryApplication.csproj" />
-        <ProjectReference Include="..\Application.Interfaces\Application.Interfaces.csproj" />
-        <ProjectReference Include="..\UnitTesting.Common\UnitTesting.Common.csproj" />
+        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
     </ItemGroup>
 
 </Project>

src/AncillaryApplication.UnitTests/AncillaryApplicationSpec.cs

@@ -2,7 +2,6 @@
 using AncillaryDomain;
 using Application.Interfaces;
 using Application.Persistence.Shared;
-using Application.Services.Shared;
 using Common;
 using Common.Extensions;
 using Domain.Common.Identity;
@@ -98,7 +97,7 @@ public async Task WhenDeliverUsageAsync_ThenDelivers()
         {
             ForId = "aforid",
             EventName = "aneventname",
-            Context = new Dictionary<string, string>
+            Additional = new Dictionary<string, string>
             {
                 { "aname", "avalue" }
             }

src/AncillaryApplication/AncillaryApplication.cs

@@ -5,7 +5,6 @@
 using Application.Persistence.Interfaces;
 using Application.Persistence.Shared;
 using Application.Resources.Shared;
-using Application.Services.Shared;
 using Common;
 using Common.Extensions;
 using Domain.Common.Identity;
@@ -126,7 +125,7 @@ private async Task<Result<bool, Error>> DeliverUsageAsync(ICallerContext context
             return Error.RuleViolation(Resources.AncillaryApplication_MissingUsageEventName);
         }
 
-        await _usageReportingService.TrackAsync(context, message.ForId!, message.EventName!, message.Context,
+        await _usageReportingService.TrackAsync(context, message.ForId!, message.EventName!, message.Additional,
             cancellationToken);
 
         _recorder.TraceInformation(context.ToCall(), "Delivered usage for {For}", message.ForId!);

src/AncillaryApplication/AncillaryApplication.csproj

@@ -8,6 +8,7 @@
         <ProjectReference Include="..\AncillaryDomain\AncillaryDomain.csproj" />
         <ProjectReference Include="..\Application.Common\Application.Common.csproj" />
         <ProjectReference Include="..\Application.Persistence.Common\Application.Persistence.Common.csproj" />
+        <ProjectReference Include="..\Application.Persistence.Shared\Application.Persistence.Shared.csproj" />
         <ProjectReference Include="..\Application.Services.Shared\Application.Services.Shared.csproj" />
     </ItemGroup>
 

src/AncillaryApplication/IAncillaryApplication.cs

@@ -1,6 +1,6 @@
 using Application.Interfaces;
-using Application.Resources.Shared;
 using Common;
+using Audit = Application.Resources.Shared.Audit;
 
 namespace AncillaryApplication;
 
@@ -13,12 +13,15 @@ Task<Result<bool, Error>> DeliverUsageAsync(ICallerContext context, string messa
         CancellationToken cancellationToken);
 
 #if TESTINGONLY
+    Task<Result<Error>> DrainAllAuditsAsync(ICallerContext context, CancellationToken cancellationToken);
+#endif
 
-    Task<Result<SearchResults<Audit>, Error>> SearchAllAuditsAsync(ICallerContext context, string organizationId,
-        SearchOptions searchOptions, GetOptions getOptions, CancellationToken cancellationToken);
-
+#if TESTINGONLY
     Task<Result<Error>> DrainAllUsagesAsync(ICallerContext context, CancellationToken cancellationToken);
+#endif
 
-    Task<Result<Error>> DrainAllAuditsAsync(ICallerContext context, CancellationToken cancellationToken);
+#if TESTINGONLY
+    Task<Result<SearchResults<Audit>, Error>> SearchAllAuditsAsync(ICallerContext context, string organizationId,
+        SearchOptions searchOptions, GetOptions getOptions, CancellationToken cancellationToken);
 #endif
 }

src/AncillaryApplication/IRecordingApplication.cs

@@ -0,0 +1,15 @@
+using Application.Interfaces;
+using Common;
+
+namespace AncillaryApplication;
+
+public interface IRecordingApplication
+{
+    Task<Result<Error>> RecordMeasurementAsync(ICallerContext context, string eventName,
+        Dictionary<string, object?>? additional,
+        CancellationToken cancellationToken);
+
+    Task<Result<Error>> RecordUsageAsync(ICallerContext context, string eventName,
+        Dictionary<string, object?>? additional,
+        CancellationToken cancellationToken);
+}

src/AncillaryApplication/RecordingApplication.cs

@@ -0,0 +1,43 @@
+using Application.Common;
+using Application.Interfaces;
+using Common;
+using Common.Extensions;
+using Task = System.Threading.Tasks.Task;
+
+namespace AncillaryApplication;
+
+public class RecordingApplication : IRecordingApplication
+{
+    private readonly IRecorder _recorder;
+
+    public RecordingApplication(IRecorder recorder)
+    {
+        _recorder = recorder;
+    }
+
+    public Task<Result<Error>> RecordMeasurementAsync(ICallerContext context, string eventName,
+        Dictionary<string, object?>? additional,
+        CancellationToken cancellationToken)
+    {
+        _recorder.Measure(context.ToCall(), eventName, (additional.Exists()
+            ? additional
+                .Where(pair => pair.Value.Exists())
+                .ToDictionary(pair => pair.Key, pair => pair.Value)
+            : null)!);
+
+        return Task.FromResult(Result.Ok);
+    }
+
+    public Task<Result<Error>> RecordUsageAsync(ICallerContext context, string eventName,
+        Dictionary<string, object?>? additional,
+        CancellationToken cancellationToken)
+    {
+        _recorder.TrackUsage(context.ToCall(), eventName, (additional.Exists()
+            ? additional
+                .Where(pair => pair.Value.Exists())
+                .ToDictionary(pair => pair.Key, pair => pair.Value)
+            : null)!);
+
+        return Task.FromResult(Result.Ok);
+    }
+}

src/AncillaryDomain.UnitTests/AncillaryDomain.UnitTests.csproj

@@ -6,13 +6,12 @@
     </PropertyGroup>
 
     <ItemGroup>
-        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
+        <ProjectReference Include="..\AncillaryDomain\AncillaryDomain.csproj" />
+        <ProjectReference Include="..\UnitTesting.Common\UnitTesting.Common.csproj" />
     </ItemGroup>
 
     <ItemGroup>
-        <ProjectReference Include="..\AncillaryDomain\AncillaryDomain.csproj" />
-        <ProjectReference Include="..\Application.Interfaces\Application.Interfaces.csproj" />
-        <ProjectReference Include="..\UnitTesting.Common\UnitTesting.Common.csproj" />
+        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
     </ItemGroup>
 
 </Project>

src/AncillaryDomain/Validations.cs

@@ -0,0 +1,5 @@
+namespace AncillaryDomain;
+
+public static class Validations
+{
+}

src/AncillaryInfrastructure.IntegrationTests/AncillaryInfrastructure.IntegrationTests.csproj

@@ -6,12 +6,12 @@
     </PropertyGroup>
 
     <ItemGroup>
-        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
+        <ProjectReference Include="..\ApiHost1\ApiHost1.csproj" />
+        <ProjectReference Include="..\IntegrationTesting.WebApi.Common\IntegrationTesting.WebApi.Common.csproj" />
     </ItemGroup>
 
     <ItemGroup>
-        <ProjectReference Include="..\ApiHost1\ApiHost1.csproj" />
-        <ProjectReference Include="..\IntegrationTesting.WebApi.Common\IntegrationTesting.WebApi.Common.csproj" />
+        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
     </ItemGroup>
 
     <ItemGroup>

src/AncillaryInfrastructure.IntegrationTests/AuditsApiSpec.cs

@@ -1,7 +1,6 @@
 using AncillaryInfrastructure.IntegrationTests.Stubs;
 using ApiHost1;
 using Application.Persistence.Shared;
-using Application.Services.Shared;
 using Common;
 using Common.Extensions;
 using FluentAssertions;

src/AncillaryInfrastructure.IntegrationTests/RecordingApiSpec.cs

@@ -0,0 +1,89 @@
+using System.Text.Json;
+using ApiHost1;
+using Common;
+using FluentAssertions;
+using Infrastructure.Web.Api.Common.Extensions;
+using Infrastructure.Web.Api.Operations.Shared.Ancillary;
+using IntegrationTesting.WebApi.Common;
+using IntegrationTesting.WebApi.Common.Stubs;
+using Microsoft.Extensions.DependencyInjection;
+using Xunit;
+using Task = System.Threading.Tasks.Task;
+
+namespace AncillaryInfrastructure.IntegrationTests;
+
+[Trait("Category", "Integration.Web")]
+public class RecordingApiSpec : WebApiSpec<Program>
+{
+    private readonly StubRecorder _recorder;
+
+    public RecordingApiSpec(WebApiSetup<Program> setup) : base(setup, OverrideDependencies)
+    {
+        EmptyAllRepositories(setup);
+        _recorder = setup.GetRequiredService<IRecorder>().As<StubRecorder>();
+        _recorder.Reset();
+    }
+
+    [Fact]
+    public async Task WhenRecordUseWithNoAdditional_ThenRecords()
+    {
+        var request = new RecordUseRequest
+        {
+            EventName = "aneventname",
+            Additional = null
+        };
+        await Api.PostAsync(request, req => req.SetHmacAuth(request, "asecret"));
+
+        _recorder.LastUsageEventName.Should().Be("aneventname");
+        _recorder.LastUsageAdditional.Should().BeNull();
+    }
+
+    [Fact]
+    public async Task WhenRecordUse_ThenRecords()
+    {
+        var request = new RecordUseRequest
+        {
+            EventName = "aneventname",
+            Additional = new Dictionary<string, object?>
+            {
+                { "aname1", "avalue" },
+                { "aname2", 25 },
+                { "aname3", true }
+            }
+        };
+        await Api.PostAsync(request, req => req.SetHmacAuth(request, "asecret"));
+
+        _recorder.LastUsageEventName.Should().Be("aneventname");
+        _recorder.LastUsageAdditional!.Count.Should().Be(3);
+        _recorder.LastUsageAdditional!["aname1"].As<JsonElement>().GetString().Should().Be("avalue");
+        _recorder.LastUsageAdditional!["aname2"].As<JsonElement>().GetInt32().Should().Be(25);
+        _recorder.LastUsageAdditional!["aname3"].As<JsonElement>().GetBoolean().Should().Be(true);
+    }
+
+    [Fact]
+    public async Task WhenRecordMeasure_ThenRecords()
+    {
+        var request = new RecordMeasureRequest
+        {
+            EventName = "aneventname",
+            Additional = new Dictionary<string, object?>
+            {
+                { "aname1", "avalue" },
+                { "aname2", 25 },
+                { "aname3", true }
+            }
+        };
+        await Api.PostAsync(request, req => req.SetHmacAuth(request, "asecret"));
+
+        _recorder.LastMeasureEventName.Should().Be("aneventname");
+        _recorder.LastMeasureAdditional!.Count.Should().Be(3);
+        _recorder.LastMeasureAdditional!["aname1"].As<JsonElement>().GetString().Should().Be("avalue");
+        _recorder.LastMeasureAdditional!["aname2"].As<JsonElement>().GetInt32().Should().Be(25);
+        _recorder.LastMeasureAdditional!["aname3"].As<JsonElement>().GetBoolean().Should().Be(true);
+    }
+
+    private static void OverrideDependencies(IServiceCollection services)
+    {
+        services.AddSingleton<IRecorder, StubRecorder>();
+    }
+}

src/AncillaryInfrastructure.IntegrationTests/Stubs/StubUsageReportingService.cs

@@ -1,5 +1,5 @@
 using Application.Interfaces;
-using Application.Services.Shared;
+using Application.Persistence.Shared;
 using Common;
 
 namespace AncillaryInfrastructure.IntegrationTests.Stubs;

src/AncillaryInfrastructure.IntegrationTests/UsagesApiSpec.cs

@@ -1,7 +1,6 @@
 using AncillaryInfrastructure.IntegrationTests.Stubs;
 using ApiHost1;
 using Application.Persistence.Shared;
-using Application.Services.Shared;
 using Common;
 using Common.Extensions;
 using FluentAssertions;

src/AncillaryInfrastructure.UnitTests/AncillaryInfrastructure.UnitTests.csproj

@@ -6,14 +6,12 @@
     </PropertyGroup>
 
     <ItemGroup>
-        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
+        <ProjectReference Include="..\AncillaryInfrastructure\AncillaryInfrastructure.csproj" />
+        <ProjectReference Include="..\UnitTesting.Common\UnitTesting.Common.csproj" />
     </ItemGroup>
 
     <ItemGroup>
-        <ProjectReference Include="..\AncillaryInfrastructure\AncillaryInfrastructure.csproj" />
-        <ProjectReference Include="..\Application.Interfaces\Application.Interfaces.csproj" />
-        <ProjectReference Include="..\Infrastructure.Web.Api.Operations.Shared\Infrastructure.Web.Api.Operations.Shared.csproj" />
-        <ProjectReference Include="..\UnitTesting.Common\UnitTesting.Common.csproj" />
+        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
     </ItemGroup>
 
 </Project>

src/AncillaryInfrastructure.UnitTests/Api/Audits/DeliverAuditRequestValidatorSpec.cs

@@ -35,6 +35,6 @@ public void WhenMessageIsNull_ThenThrows()
 
         _validator.Invoking(x => x.ValidateAndThrow(_dto))
             .Should().Throw<ValidationException>()
-            .WithMessageLike(Resources.AnyMessageValidator_InvalidMessage);
+            .WithMessageLike(Resources.AnyQueueMessageValidator_InvalidMessage);
     }
 }