Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of the Industry Core (Data Provisioning) Testbed #1159

Open
19 tasks
gerbigf opened this issue Jan 22, 2025 · 11 comments
Open
19 tasks

Implementation of the Industry Core (Data Provisioning) Testbed #1159

gerbigf opened this issue Jan 22, 2025 · 11 comments
Labels
ic represents features/dependencies to the industry core Prep-R25.06

Comments

@gerbigf
Copy link

gerbigf commented Jan 22, 2025

Overview

Explain the topic in 2 sentences

The industry core test bed shall allow companies to test and validate if their implementation fulfils all requirements for productive data provisioning (and consumption in the second stage). It shall contain a set of test cases that can be executed to e.g. check if the EDC is configured correctly, digital twins are created according to standards or if the data provided follows the right semantic models. The test cases can be based on each other and executed one after the other with increased complexity. The testbed shall tell the user where the test failed and what is wrong in a human understandable way.

The testbed shall be hosted by the association but it shall also be possible to e.g. deploy it locally.

What's the benefit?

  • Supports self-tests of new Catena-X partners of their own implementation
  • Supports future automatic tests for certification

What are the Risks/Dependencies ?

  • Feasibilty with regard to costs
  • Dependencies to IC reference implementation, CX association (test management, central testbed), EDC

Detailed explanation

Framework Conditions:

  • Any result (OSS Software, Designs, Architecture, Configuration etc.) will be provided in Tractus-X as Open Source Code.
  • Operations of the TestBed is out of scope for this story
  • It shall be possible to operate the TestBed as a stand-alone solution for self-testing
  • Any service implemented shall be compatible to the Industry Core Hub Reference Implementation as their might be shared services (e.g. DT Pull, Event API...)
  • Technical guidelines and constraints must be specified by the association (e.g. All Open Source? Dockerized? Runtime? Programming Language if any? Integration in Jira?)
  • It shall be possible to extend the testbed with other use-cases
  • The testbed must not have company specific domain knowledge (e.g. structure of BMW PartIDs)
  • The testbed "simulates" a data consumer. Later versions could also simulate a data provider.
  • It shall be possible to host the testbed in various environments (local/MXD - see tutorials, PREPROD, Association Env.)
  • It shall be possible to spin up all components that are required for the testbed together via helm-charts
  • It shall be possible to configure an existing EDC into the testbed.
  • The Testbed shall not be dependent on implementation details of the data provider (as long as the data provider follows all relevant standards and implements them correctly)
  • The testbed is not meant to verify the correctness of EDC or DTR implementations. There are separate tools for this task. E.g. https://projects.eclipse.org/projects/technology.dataspacetck
  • The testbed shall be able to test if digital twins of the type PartInstance and PartType were created successfully (tbd how this is defined)
  • TestBed needs to support all valid releases (e.g., two major releases in parallel)
  • Gherkin and Cucumber shall be used as the languages to define and write tests and test cases

Scope:

  • The first implementation of the Testbed is scoped to IndustryCore excluding UniqueID Push and Notifications

Open Questions/Topics:

  • Shall the testbed integrate with TestCases in Xray Jira? Is there an "open source standard" to describe and define testcases and steps?
  • An interface might be needed to use/export test cases into different test tools?
  • Define final feature list per scope, finalize scope for first PI (R25.06)
  • Align with reference implementation team, test management, CX association
  • Define detailed test cases (with alternatives) for Industry Core Scope
  • Find a better name for IC testbed
  • Potential Test Cases:

Scope 1. - Positive Tests that everything is set up correctly.
IC Independent Scope

  • Test if the catalogue is reachable
  • Test if the DTR asset was created according to a standard
  • Test if the DTR can be accessed through the EDC data plane

IC Specific Scope

  • Test if a "generic" - based on test data - or "real" - based on actual data - digital twin was created correctly (Specific Asset IDs).
    • Based on the type of twin that was created (PartInstance, PartType), check for different things.
  • Test if any combination of the following submodels SerialPart, Batch, JustInSequencePart und SingleLevelBomAsBuild were created correctly (reference to semantic models etc.)
  • Test data transfer of a submodel through the EDC and an EDC asset
  • Validate content of the data according to semantic models (SerialPart, Batch, JustInSequencePart und SingleLevelBomAsBuild)

Scope 2. - Negative Tests
IC Independent Scope

  • Test if an access policy is set up correctly and "hides" an offer for a specific partner
  • Test if the DTR submodels are configured correctly and are hidden for specific partners
  • Test if usage policies are set up correctly and e.g. deny access if a certain credential is missing
  • Test successful and correct registration at discovery services (maybe scope 1 like EDC basic test)

IC Specific Scope

  • SingleLevelUsage
  • Test of notiifcation (Unique ID Push, but also use case notifications)

Current implementation

Currently, testing is done between partners manually during the onboarding of use-cases.

Proposed improvements

Image

The testbed should contain different microservices:

  • An optional UI for simple integration. Written in React.js, reusing the already existing shared portal resources
  • A test-case orchestrator that executes the test cases, calls the various services and compiles the test-results
  • A DT Pull Service to interact with the data provider EDC, DTR and Submodel Server
  • An EDC
  • Extendable Service X (e.g. a DTR)

Feature Team

Contributor

  • BMW
  • Schaeffler

Committer

User Stories

  • Issue 1, linked to specific repository
  • Issue 2, linked to another specific repository

Acceptance Criteria

  • Criteria 1
  • Criteria 2
  • Criteria 3

Test Cases

Test Case 1

Steps

  1. Do something
  2. Click something
  3. Add something

Expected Result

  1. Expectation
  2. Expectation
  3. Expectation

Architectural Relevance

The following items are ensured (answer: yes) after this issue is implemented.

In the context of the standards 126 and 127, typically only one is applicable, depending on the specific use case. Please cross out one of the two standards that does not apply.

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification)

Additional information

  • I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)
@gerbigf gerbigf changed the title Implementation of an Industry Core Testbed WIP: Implementation of an Industry Core Testbed Jan 22, 2025
@gerbigf
Copy link
Author

gerbigf commented Jan 22, 2025

@markuskeidl
Copy link

markuskeidl commented Jan 22, 2025

Constraints

  • TestBed needs to support all valid releases (e.g., two major releases in parallel)

Scope 2

  • Test of notiifcation (Unique ID Push, but also use case notifications)
  • Registration at discovery services (maybe scope 1 like EDC basic test)

What's the benefit?

  • Supports self-tests of new Catena-X partners of their own implementation
  • Supports future automatic tests for certification

Risks/Dependencies

  • Feasibilty with regard to costs
  • Dependencies to IC reference implementation, CX association (test management, central testbed), EDC

TODOs.

  • Define final feature list per scope, finalize scope for first PI (R25.06)
  • Align with reference implementation team, test management, CX association
  • Define detailed test cases (with alternatives) for Industry Core Scope
  • Find a better name for IC testbed

@eckardg eckardg added ic represents features/dependencies to the industry core Prep-R25.06 labels Jan 22, 2025
@gerbigf gerbigf changed the title WIP: Implementation of an Industry Core Testbed Implementation of the Industry Core (Data Provisioning) Testbed Jan 28, 2025
@matbmoser
Copy link
Contributor

I have detailed here the idea of the test bed and what points it needs to cover:

1 to 7

Image

@matbmoser
Copy link
Contributor

matbmoser commented Jan 28, 2025

Three Test Agents:

  • Provider Internal Agent (MVP)
  • Consumer Internal Agent
  • Extern Consumer Agent

Image

@gerbigf
Copy link
Author

gerbigf commented Jan 29, 2025

I've updated the description of the feature with results from yesterday ("Gherkin & Cucumber") and @markuskeidl comments.

@eckardg
Copy link

eckardg commented Jan 29, 2025

Based on the picture, I created a list of the seven test cases as a first proposal. This is similar to parts of the description, so feel free to discard it.

1. EDC Connection
Access Provider's EDC from Consumer and check if the EDC catalog can be retrieved. Result should contain at least one asset which is the DTR (to be configured like this by the provider)
2. Contract negotiation
do the contract negotiation for the DTR successfully
3. DTR Connection
Access Provider's DTR based on the information in the corresponding EDC asset (Test 1. has to be successful for this) and get the DT catalog. This should contain at least one DT (to be configured like this by the provider)
4. Digital Twin check
Access one digital twin in the DTR and check if the digital twin has been created correctly.
5. Submodel check
Look for submodels for this digital twin and check if the submodels have been created correctly. (Check if all submodels needed for that digital twin are available)
6. Aspect model asset check
For a specific aspect, search the corresponding submodel of the digital twin, retrieve the EDC asset and do contract negotiation
7. Aspect model semantic check
retrieve the EDC asset belonging to that aspect model and check if the semantic is ok. Check against the semantic definition

@gerbigf
Copy link
Author

gerbigf commented Jan 29, 2025

I had an exchange with one of the developers from the rollout-team that was involved in getting our 100 partners onboard in 2024. He confirmed that we can’t expect most companies to be able to even use a postman-collection or any sort of code/script.

Having said that, I would propose to build a simple user-interface that interacts with the Testbed Backend APIs to execute certain test cases. The UI can be used to pass parameters such as the specific test-case, Connector URL, BPNL, Digital Twin ID etc as well as the means to provide the test results as output.

More sophisticated users can of course interact with the API directly to trigger test-cases.

What remains open is how we secure the Testbed UI & API against misuse. We could e.g. protect it with a simple password or connect it to the Cofinity IdP?

What do you think?

@matbmoser
Copy link
Contributor

I had an exchange with one of the developers from the rollout-team that was involved in getting our 100 partners onboard in 2024. He confirmed that we can’t expect most companies to be able to even use a postman-collection or any sort of code/script.

Having said that, I would propose to build a simple user-interface that interacts with the Testbed Backend APIs to execute certain test cases. The UI can be used to pass parameters such as the specific test-case, Connector URL, BPNL, Digital Twin ID etc as well as the means to provide the test results as output.

More sophisticated users can of course interact with the API directly to trigger test-cases.

What remains open is how we secure the Testbed UI & API against misuse. We could e.g. protect it with a simple password or connect it to the Cofinity IdP?

What do you think?

Do it modular in React.js (reusing the shared portal components) so you can be fast and we can integrate in the Industry Core Hub, so people can test their "dataspace components".

The only question is how much more work would be to create an UI? Do you have developers that can do it?
But totally agree that it would be beneficial.

@gerbigf
Copy link
Author

gerbigf commented Jan 29, 2025

Do it modular in React.js (reusing the shared portal components)

Makes perfectly sense for me. I'll add that as a prerequisite for the UI part.

@markuskeidl
Copy link

I agree that only having an API without an frontend might be a challenge for a lot of new partners. But you also said it yourself, even with the UI you need to specify BPN, Connector URL, digital twin IDs and so on. The target group of the testbed are IT people and these should be able to use APIs with Postman or REST API clients.
If the target group of our testbed are business people, that will not work IMHO.

@gerbigf
Copy link
Author

gerbigf commented Jan 30, 2025

We can make the UI an optional component, depending on how far we get with the resources and time we have. It will definitely help to explain and showcase the testbed and deliver the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ic represents features/dependencies to the industry core Prep-R25.06
Projects
Status: Inbox
Development

No branches or pull requests

4 participants