Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement setting to use offset limit pagination #41

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

pnadolny13
Copy link
Contributor

@pnadolny13 pnadolny13 commented Aug 31, 2023

Closes #40

In my branch I let the user opt into pagination, set the batch size, and will throw an exception if a replication key isnt set if paginate is true.

I loaded up an Athena table with 100M rows to replicate the hanging issue, no records were synced after several minutes. With this branch I was able to sync about 600-900 records/second. Without the order by clause it was going much faster, closer to 1500 records/second but I think thats unsafe.

@pnadolny13 pnadolny13 changed the title implement setting to use offset limit pagination feat: implement setting to use offset limit pagination Aug 31, 2023
@sonarcloud
Copy link

sonarcloud bot commented Aug 31, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@pnadolny13
Copy link
Contributor Author

This works but it still scans the full dataset on each pagination batch so it allows the tap to function but still isnt efficient in its current state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

bug: large tables cause queries to hang
1 participant