Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data transformers #2547

Open
6 tasks
byte-bandit opened this issue Jan 28, 2025 · 0 comments
Open
6 tasks

Data transformers #2547

byte-bandit opened this issue Jan 28, 2025 · 0 comments
Assignees
@byte-bandit
Labels
devops
Milestone
next

Comments

@byte-bandit
Copy link

byte-bandit commented Jan 28, 2025 •
edited
Loading

Background

Transformers are composed of one or more components of the CDP service, parsing raw data provided by the indexer, and transforming it to fit the data model defined in #2545.

⏰ Estimate: 3 days

Done when

  • one or more components periodically iterate over fresh raw data provided by the Chain indexer #2546
  • iteration is controlled (recovered panics, retries, etc...)
  • data extracted from the raw dump is transformed to match the data model defined in Persistence Layer #2545
  • transformed data is inserted into the respective tables
  • processed samples are marked for deletion
  • transformers are written in a way that enables concurrent instances to process multiple batches of raw data
@byte-bandit byte-bandit added this to the next milestone Jan 28, 2025
@byte-bandit byte-bandit self-assigned this Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@byte-bandit byte-bandit

Labels
devops
Projects
None yet
Milestone
next
Development

No branches or pull requests
1 participant
@byte-bandit