Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tables V2 #927

Open
wants to merge 16 commits into
base: 1.10.latest
Choose a base branch
from
Open

Tables V2 #927

wants to merge 16 commits into from

Conversation

benc-db
Copy link
Collaborator

@benc-db benc-db commented Feb 4, 2025

Description

This is the next, and hopefully largest, chunk of migrating to Materializations V2: tackling table creation. As a reminder, the primary goal with Materializations V2 is to restructure materializations such that we first create the final destination, applying constraints, column comments, etc, then insert the required data. This allows us to a.) apply comments in bulk, b.) ensure that invalid data doesn't land in the final destination, and c.) allow for safe replacement strategies, where we first backup the existing table.

As with other large changes, this is gated behind a behavior flag so that it is initially opt-in; as such, there are large chunks of duplicated code so as not to disrupt the current behavior, but also prepare for eventually whole-sale deletion. Please start your review by looking at: https://github.com/databricks/dbt-databricks/blob/1ccc7d171dca4259807041505429e66bc59276e0/docs/table_flow.md, and dbt/include/databricks/macros/materializations/table.sql, where the structural changes are implemented.
The link to the markdown file should render as graphs showing the table materialization flow before and after changes.

This PR makes slight modifications to the new constraints logic, as I hit some bumps when adopting those changes.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant