Skip to content

Commit

Permalink
Merge pull request #30 from Avanade/v2-2-0
Browse files Browse the repository at this point in the history
Improved, major, version update.
  • Loading branch information
chullybun authored May 5, 2023
2 parents 9c933b3 + b8466be commit e9eb66f
Show file tree
Hide file tree
Showing 92 changed files with 1,149 additions and 1,258 deletions.
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,18 @@

Represents the **NuGet** versions.

## v2.0.0
- **Note:** This version contains a number of breaking changes and generated artefact file name changes; all existing generated artefacts should be deleted and re-generated. Where possible it is recommended to create new using the template solution and copy across existing non-generated artefacts, such as the NTangle YAML etc.
- *Enhancement:* Removed need for `UseSqlServer` for `CodeGenConsole` as this is the only supported database option. New `UseDeploymentOption` to enable deployment option override; defaults to `DeploymentOption.DbEx`.
- *Enhancement:* Leverage `EventDataFormatter` and `IEventPublisher.CreateValueEvent` for improved _CoreEx_ consistency.
- *Enhancement:* `VersionTracking` table has had `Object` column replaced with `Schema` and `Table` columns to be explicit/consistent.
- *Enhancement:* Improved configuration override capabilities using `appsettings.json`.
- *Enhancement:* Template solution has refactored baseline code output.
- *Enhancement:* Updated `CoreEx` (`v2.10.1`) and `DbEx` (`v2.3.4`) dependencies.
- *Fixed:* Create and update events not published where a subsequent delete occurs; i.e. the entity is physically deleted. The create and update cannot be published as the entity contents are no longer available, and there is not way to reliable way to construct.
- *Fixed:* Previously where a create (optional updates) and delete occured within a batch no events were published; now the delete event is published as a minimum to be consistent with the preceeding fix.
- *Fixed:* Code-generated entities correctly invoke `IdentifierMapping` asynchronously using an `await`.

## v1.1.0
- *Enhancement:* Updated `CoreEx` (`v2.5.1`) and `DbEx` (`v2.3.2`); this results in minor breaking changes that will need to be addressed.
- [*Issue 25:*](https://github.com/Avanade/NTangle/issues/25) Resolved code generation error where a database table name resulted in an alias of `r`.
Expand Down
19 changes: 9 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

_nTangle_ is a Change Data Capture (CDC) code generation tool and corresponding runtime. Unlike other CDC-based technologies which replicate changes to rows, _nTangle_ is designed to replicate business entity (aggregate) changes.

For example, if a database contains a `Person` and one-to-many related `Address` table, a traditional CDC replicator would leverage the CDC-capabilities of the database as the data source and replicate all changes from both tables largely distinct from each other. Additional logic would then be required within the downstream systems to aggregate these distinct changes back into a holistic business entity, if possible.
For example, if a database contains a `Person` and one-to-many related `Address` table, a traditional CDC replicator would leverage the CDC-capabilities of the database as the data source and replicate all changes from both tables largely distinct from each other. Additional logic would then be required within the downstream systems to aggregate these distinct changes back into a holistic business entity where required, if possible.

_nTangle_ tackles this differently by packaging the changes at the source into an aggregated entity which is then replicated. With _nTangle_ the CDC-capabilities of the database are leveraged as the trigger, with a corresponding query across all related tables to produce a holistic business entity. Therefore, if a change is made to `Person` or `Address` this will result in the publishing of the entity. Where transactional changes are made to both `Person` and `Address` a single holistic business entity will be published including all changes.

Expand Down Expand Up @@ -39,7 +39,7 @@ SalesOrder // Parent

The CDC capability is used specifically as a trigger for change (being `Create`, `Update` or `Delete`). The resulting data that is published is the latest, not a snapshot in time (CDC captured). The reason for this is two-fold:
1. Given how the CDC data is batch retrieved there is no guarantee that the CDC captured data represents a final intended state suitable for publishing; and,
1. This process is intended to be running near real-time so getting the latest version will produce the most current committed version as at that time.
2. This process is intended to be running near real-time so getting the latest version will produce the most current committed version as at that time.

To further guarantee only a single event for a specific version is published the resulting _entity_ is JSON serialized and hashed; this value is checked (and saved) against the prior version to ensure a publish contains data that is actionable. This will minimize redundant publishing, whilst also making the underlying processing more efficient.

Expand All @@ -49,7 +49,7 @@ To further guarantee only a single event for a specific version is published the

This official [documentation](https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server) describes the Microsoft SQL Server CDC-capabilities.

Although throughout references are made to Microsoft SQL Server, the intention of _nTangle_ is that it is largely agnostic to the database technology, and therefore support for other databases will (or may) be supported in the future based on demand.
Although throughout references are made to Microsoft SQL Server, the intention of _nTangle_ is that it is largely agnostic to the database technology, and therefore support for other databases will (or may) be supported in the future based on demand, and their capabilities.

<br/>

Expand All @@ -70,7 +70,7 @@ _nTangle_ has been created to provide a seamless means to create CDC-enabled agg

### Code-generation

The code-generation is managed via a console application using the [`CodeGenConsole`](./src/NTangle/Console/CodeGenConsole.cs) to manage. This internally leverages [OnRamp](https://github.com/Avanade/onramp) to enable.
The code-generation is managed via a console application using the [`CodeGenConsole`](./src/NTangle/Console/CodeGenConsole.cs) to manage. This internally leverages [OnRamp](https://github.com/Avanade/onramp) to enable the underlying code-generation capabilities.

Additionally, the code-generator inspects (queries) the database to infer the underlying table schema for all tables and their columns. This is used as a source in which the configuration references to validate, whilst also minimizes configuration where the inferred schema information can be used. The code-generation adopts a gen-many philosophy, therefore where schema changes are made, the code-generation can be executed again to update accordingly.

Expand All @@ -97,10 +97,9 @@ Documentation related to each of the above are as follows:

An example [ntangle.yaml](./samples/SqlServerDemo/SqlServerDemo.CodeGen/ntangle.yaml) configuration file exists within the [`SqlServerDemo`](./samples/SqlServerDemo) sample. The [`SqlServerDemo.CodeGen`](./samples/SqlServerDemo/SqlServerDemo.CodeGen) sample also demonstrates how to invoke the code generator from the underlying [`Program`](./samples/SqlServerDemo/SqlServerDemo.CodeGen/Program.cs).

The code-generator will output a number of generated artefacts ; these will be either database-related (see [`SqlServerDemo.Database`](./samples/SqlServerDemo/SqlServerDemo.Database) sample) or corresponding .NET runtime components (see [`SqlServerDemo.Publisher`](./samples/SqlServerDemo/SqlServerDemo.Publisher) sample).
The code-generator will output a number of generated artefacts; these will be either database-related (see [`SqlServerDemo.Database`](./samples/SqlServerDemo/SqlServerDemo.Database) sample) or corresponding .NET runtime components (see [`SqlServerDemo.Publisher`](./samples/SqlServerDemo/SqlServerDemo.Publisher) sample).


The following [`NTangle`](./src/NTangle) namespaces provide code-generation capabilties:
The following [`NTangle`](./src/NTangle) namespaces provide the code-generation capabilties:

Namespace | Description
-|-
Expand All @@ -114,14 +113,14 @@ Namespace | Description

Generally, a runtime publisher is required to orchestrate the CDC-triggered aggregated entity publishing process (see [`SqlServerDemo.Publisher`](./samples/SqlServerDemo/SqlServerDemo.Publisher) sample). This in turn takes a dependency on the _nTangle_ runtime to enable.

The following [`NTangle`](./src/NTangle) namespaces provide runtime capabilties:
The following [`NTangle`](./src/NTangle) namespaces provide the runtime capabilties:

Namespace | Description
-|-
[`Cdc`](./src/NTangle/Cdc) | CDC-orchestration capabilities, primarily [`EntityOrchestrator`](./src/NTangle/Cdc/EntityOrchestrator.cs).
[`Data`](./src/NTangle/Data) | Database access capabilities to support the likes of batch tracking, identifier mapping and versioning.
[`Events`](./src/NTangle/Events) | Event capabilities, extending the capabilities enabled by [`CoreEx`](https://github.com/Avanade/CoreEx/tree/main/src/CoreEx/Events).
[`Services`](./src/NTangle/Services) | Service hosting capabilities, primarily [`HostedService`](./src/NTangle/Services/CdcHostedService.cs) and [`EventOutboxHostedService`](./src/NTangle/Services/EventOutboxHostedService.cs).
[`Events`](./src/NTangle/Events) | Event capabilities, leveraging and extending the capabilities enabled by [`CoreEx`](https://github.com/Avanade/CoreEx/tree/main/src/CoreEx/Events).
[`Services`](./src/NTangle/Services) | Service hosting capabilities, primarily the [`HostedService`](./src/NTangle/Services/CdcHostedService.cs).

<br/>

Expand Down
Loading

0 comments on commit e9eb66f

Please sign in to comment.