Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: round robin general ledger stream partitions #6

Open
pnadolny13 opened this issue Nov 8, 2024 · 0 comments
Open

feat: round robin general ledger stream partitions #6

pnadolny13 opened this issue Nov 8, 2024 · 0 comments

Comments

@pnadolny13
Copy link
Contributor

The general ledger stream uses the stream partition feature to iterate the list of modules. Theres a desire to avoid waiting until an entire module is completed before moving on to the next one. An approach could be to have a maximum number of records to paginate before pausing and starting on the next partition, ultimately looping until all partitions are completed. This would have 2 benefits:

  1. the user gets the a variety of data while backfilling so they can start working with the data before the full sync completes (days in some cases)
  2. the offset size decreases and eventually resets which could potentially have performance benefits depending on how the intacct backend implements pagination
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant