Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use unique temporary table name + Check schema change #45

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tuan-seek
Copy link
Contributor

Changes in this PR:

  • Add ability to create unique temporary table names. This would enable running multiple dbt run concurrently for incremental model.
  • Implement fail_fast mode when schema changes

@Antauri
Copy link
Collaborator

Antauri commented Mar 10, 2022

I've also opened a different issue #62
This seems to do exactly that.

Who can review and merge it please?

@Antauri
Copy link
Collaborator

Antauri commented Mar 22, 2022

We'd require this for a performance boost on our queries. Can it be merged?

@Antauri
Copy link
Collaborator

Antauri commented Mar 23, 2022

I've tested this on my own fork, 12 parallel executions (12 batches in parallel for the same hour, distinct sets of minutes from the hour of data) and I confirm it works. If you're going to run DBT in parallel, on the same model, using different "vars" (like the batch number) then at the initial table creation you'll have 12 CTAS instead of 1 CTAS + 11 ITAS (insert-into-as-select) queries, but that's work-aroundable.

Lovely if we could get this merged in the main trunk. This feature helps the use of parallel queries on Athena and gets us down from 20m/hour to 4m/hour by running distinct sets of batches on the same partition (hourly in our case).

@nicor88
Copy link

nicor88 commented Nov 29, 2022

@tuan-seek and @Antauri I'm quite interested about this feature, if you are not aware, the community decided to fork Tomme/dbt-athena and have a more community friendly setup to changes, new fork is here: https://github.com/dbt-athena/dbt-athena, available in pip too.

Said so, could you tell me how in possible in your setup to have tmp tables with the same name?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants