Supporting Upsertion of Event Files through Change Data Capture (CDC) #816
DevashishRaghav
started this conversation in
General
Replies: 1 comment
-
@rhwarrier @Sunbird-cQube/sunbirdcqube_architectureworkinggroup |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Dear cQube Community,
We hope this message finds you well. We're excited to initiate a discussion on a crucial topic that can significantly enhance cQube's capabilities, i.e., supporting upsertion of event files through Change Data Capture (CDC).
Background
As many of you are aware, cQube is a dynamic and evolving data analytics platform that serves educational institutions and organizations across the globe. One of the core challenges we face is efficiently managing and processing event data from various sources.
The Challenge
Traditionally, event data is ingested into cQube through batch processing. While this approach has served us well, it has its limitations, particularly in scenarios where near-real-time data updates are essential.
This is where Change Data Capture (CDC) comes into play. CDC is a technique that identifies and captures changes made to data in the source system. It allows us to detect and process inserts, updates, and deletes efficiently. By implementing CDC in cQube, we could achieve:
Your Input Matters
We value the insights and experiences of our community members. Before proceeding with the implementation of CDC for upsertion of event files, we want to hear from you:
Join the Discussion
Your participation in this discussion is crucial. We invite you to share your thoughts, experiences, and suggestions on this topic. Your input will shape the direction we take and help us ensure that cQube continues to meet the evolving needs of our community.
Please feel free to comment below with your insights, or if you have any technical details or resources to share, they would be greatly appreciated.
Let's work together to make cQube even more powerful and efficient. Thank you for your continued support and collaboration.
Beta Was this translation helpful? Give feedback.
All reactions