⭐🚀[Concept] Identification of new twins created since the last sync of dtr

[mkanal]

Overview

The goal is to develop a concept for Trace-X that ensures only newly added or to be syncted digital twins (assets) are processed, reducing daily processing volume and conserving system resources. This involves tracking the state of each twin, defining when they require reprocessing, and managing the transaction flow to ensure system resilience during scaling.

Explain the topic in 2 sentences

The Trace-X system currently lacks a mechanism to filter and process only those twins that are either newly added since the last sync or need updating due to their TTL expiration. By introducing a "Twin Process Handler" that operates based on the state of each twin, we can streamline operations and optimize resources. This handler will classify twins into various statuses (NEW, IN_PROGRESS, DONE, RESYNC, ERROR, FAIL) and define conditions for each. For instance, a twin marked as DONE will move to RESYNC if its TTL expires, triggering necessary processing. A retry logic will manage errors, and transaction integrity will support scaling and rollback for system consistency.

What's the benefit?

This approach reduces unnecessary processing, optimizing resource usage by focusing on new or stale twins, which increases overall efficiency.

What are the Risks/Dependencies ?

Detailed explanation

Current implementation

Currently, Trace-X processes twins received from DTR (Digital Twin Registry) daily without a targeted approach to filter out only new or outdated assets. This results in higher resource use and processing redundancy.

Proposed improvement

Implement the "Twin Process Handler" to:

Identify and process only new twins or those requiring updates based on TTL or error states.
Introduce a robust state model to track each twin's status and prevent redundant processing.
Ensure transactional consistency across multiple instances to support scaling and maintain system integrity.
Incorporate TTL for each twin, streamlining when resyncs occur based on system-defined time or condition thresholds.

Open questions

• (?) How will the assIdentifier be defined, and how will we track if an Asset/Twin has already been processed?
• (?) What is an appropriate TTL for an Asset/Twin? When is it considered outdated and requires resynchronization?
• (?) Under what conditions should an already processed Asset/Twin be resynced?
• (?) How should old Assets be handled, and how frequently should they be resynced, if at all?

Feature Team

Contributor

• Cofinity-X @rogocof
• Cofinity-X @mkanal

Committer

• @ds-jhartmann
• @ds-mwesener
• @ds-lcapellino

User Stories

As a customer
I want

     Trace-X to identify those Assets/Twins received in DTR sync which were added since last sync 

     AND 

     Trace-X identify those Assets/Twins which have to be updated after a given time 

so that only new added twins or twins to be updated will be processed which reduced the number of Twins to be processed on a daily basis is significantly, and system resources are preserved.

Acceptance Criteria

• A concept must be developed that governs the processing of Assets/Twins and identifies only those that need to be synchronized and processed. (Twin Process Handler)
• A state model will be implemented that tracks the following statuses ()
  • NEW: The Twin/Asset has been newly registered and needs to be processed.
  • IN_PROGRESS: The Asset/Twin is being processed in IRS.
  • DONE: Processing is complete, and the Twin will be resynced after the TTL expires.
  • RESYNC: Twin will be reset to RESYNC after TTL expired of twin in state DONE
  • ERROR: There was an error during processing, and retry logic will be applied. (3 times )
  • FAIL: Resync of asset with errors after 3 retries. → Asset status will be set to FAILED
• The Twin Process Handler must support transaction handling to ensure Trace-X functions correctly after scaling on multiple instances. Transactions must be secured, and any errors should trigger a rollback.
• Each Asset/Twin will have a Time-To-Live (TTL) defined, after which it will be resynced.
• An Asset/Twin will only be synchronized if its TTL has expired or its status indicates a need for reprocessing.
• Twin Process Handler is documented in ARC42 software documentation

Test Cases

Test Case 1

Steps

1. Do something
2. Click something
3. Add something

Expected Result

1. Expectation
2. Expectation
3. Expectation

Architectural Relevance

The following items are ensured (answer: yes) after this issue is implemented:

• This feature aligns with our current architectural guidelines
  • Data Sovereignty: All data sharing activities across company boundaries follow the Catena-X Regulatory Framework, in particular the Data Exchange Governance, and the Dataspace Protocol via a compliant Connector (like the tractusx-edc or similar, see Connector KIT)
    • CX-0010 Business Partner Number
    • CX-0013 Identity of Member Companies
    • CX-0018 Data Space Connectivity (EDC)
    • CX-0049 DID Document Schema
    • CX-0050 Framework Agreement Credential
    • CX-0149 Verified Company Identity
  • Interoperability: Digital Twins are used (compliant to the Digital Twin KIT and the Industry Core KIT)
    • CX-0001 EDC Discovery API
    • CX-0002 Digital Twins in Catena-X
    • CX-0018 Data Space Connectivity (EDC)
  • Data Format:
    • The data model is based on a published Semantic Model
• The impact on the overall system architecture has been assessed. The Feature does not require changes to the architecture or any existing standard? Please have a look here on the overarching architecture
• Potential risks or conflicts with existing architecture has been assessed

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification)

Additional information

• I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)

[stephanbcbauer]

Some hints from Release Management (@ther3sa) and Tractus-X Project Lead (@stephanbcbauer)

• Status currently in Backlog -> Since for Backlog some requirements are missing ... please update them
• Please add missing sections from the feature template, or fill them out
• Please add assignee (overall responsible person who drives the feature)