Zero-downtime for table recalculation within dbt run #88
Comments
|
I had a shot at implementing a PoC for this feature as well, maybe that could help! My code is at iasoon@cca66da. It seems to work in simple tests, but it's not battle-tested :) |
Avinash-1394
pushed a commit
to Avinash-1394/dbt-athena
that referenced
this issue
Jan 12, 2023
* prune static s3 locations when CTA * refactor s3 deletion methods * feat: prune s3 location prior to CTA for all table_types except iceberg * test: add test for s3 path parsing * test: move test to right class * test: use pytest parametrize instead of loop * refactor: move debug logger into delete from s3 method * chore: remove logger from delete table method
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi
I'm sorry if I'm missing something - I'm new in dbt and just trying to switch the project from custom data model management to dbt
Does dbt-athena supports table recalculation without downtime while it's being recalculated?
What workaround I used to do in my own script: to give every CTAS an unique name (i.e. _) and have a helper view with a fixed name () and simple query
select * from <name>_<random>- so once new version of ctas is calculated I switch a view to the new version and drop the old one. All usages reference only view and the whole thing works without downtimeWill appreciate your thoughts and comments!
The text was updated successfully, but these errors were encountered: