Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: limit the amount of context data we parse #684

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

pnadolny13
Copy link
Contributor

We have too much data in the context_base table so performance is poor. The data volume is increasing with time so the last 6 months has more data than all before it. This is likely because more users are on newer versions of meltano that send our rich unstructured events and because usage has grown.

I manually truncated the context_base incremental table to remove all data before this year and made a backup table of the original. The table is transient but the backup is not so it will be properly persisted if we ever need that processed historical data. Since the context_base table will continue to grow and we'll have to manually prune it periodically, I created this PR which limits all downstream tables to filter only for 6 months of data so their performance should be relatively static even as the base table grows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant