Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix redshift_maintenance not working with ref #53

Merged
merged 1 commit into from
Feb 14, 2022

Conversation

jeremyyeo
Copy link
Contributor

@jeremyyeo jeremyyeo commented Feb 13, 2022

Description & motivation

See issue #52 for background.

I tested both mechanism of running redshift_maintenance:

  1. Specifying inclusions via a vacuum_tables_sql() macro:
{% macro vacuumable_tables_sql() %}
select 'analytics' as table_database
       , table_schema
       , table_name
  from analytics.information_schema.tables
 where table_type = 'BASE TABLE'
   and table_schema = 'dbt_jyeo'
{% endmacro %}
~ dbt run-operation redshift_maintenance

23:00:55  Running with dbt=1.0.1
23:00:58  12:00:58 + 1 of 1 Vacuuming "analytics"."dbt_jyeo"."my_table"
23:01:02  12:01:02 + 1 of 1 Analyzing "analytics"."dbt_jyeo"."my_table"
23:01:02  12:01:02 + 1 of 1 Finished "analytics"."dbt_jyeo"."my_table" in 3.43s
  1. Specify exclusions via runtime exclude_schemas and exclude_schemas_like args:
~ dbt run-operation redshift_maintenance --args '{exclude_schemas: ["bloom", "close", "incremental_explore", "spectrum", "trello"], exclude_schemas_like: ["dbt%", "sinter%", "fivetran%", "fishtown%", "google%", "snapshots%", "snowplow%", "github%", "integration%", "my%", "public%", "test%", "quickbooks%", "lever%", "slack%", "drew%", "erin%"]}'

23:06:42  Running with dbt=1.0.1
23:06:46  12:06:46 + 1 of 7 Vacuuming "analytics"."analytics"."fivetran_snowplow_unload"
23:06:46  12:06:46 + 1 of 7 Analyzing "analytics"."analytics"."fivetran_snowplow_unload"
23:06:46  12:06:46 + 1 of 7 Finished "analytics"."analytics"."fivetran_snowplow_unload" in 0.59s
23:06:47  12:06:47 + 2 of 7 Vacuuming "analytics"."analytics"."my_table"
23:06:47  12:06:47 + 2 of 7 Analyzing "analytics"."analytics"."my_table"
23:06:47  12:06:47 + 2 of 7 Finished "analytics"."analytics"."my_table" in 0.58s
23:06:48  12:06:48 + 3 of 7 Vacuuming "analytics"."analytics"."stg_customer_id_map"
23:06:48  12:06:48 + 3 of 7 Analyzing "analytics"."analytics"."stg_customer_id_map"
23:06:48  12:06:48 + 3 of 7 Finished "analytics"."analytics"."stg_customer_id_map" in 0.61s
23:06:49  12:06:49 + 4 of 7 Vacuuming "analytics"."analytics"."stg_dbt_users"
23:06:49  12:06:49 + 4 of 7 Analyzing "analytics"."analytics"."stg_dbt_users"
23:06:49  12:06:49 + 4 of 7 Finished "analytics"."analytics"."stg_dbt_users" in 0.57s
23:06:50  12:06:50 + 5 of 7 Vacuuming "analytics"."analytics"."stg_server_farm_cities"
23:06:50  12:06:50 + 5 of 7 Analyzing "analytics"."analytics"."stg_server_farm_cities"
23:06:51  12:06:51 + 5 of 7 Finished "analytics"."analytics"."stg_server_farm_cities" in 0.57s
23:06:51  12:06:51 + 6 of 7 Vacuuming "analytics"."analytics"."stg_trello_members"
23:06:51  12:06:51 + 6 of 7 Analyzing "analytics"."analytics"."stg_trello_members"
23:06:52  12:06:52 + 6 of 7 Finished "analytics"."analytics"."stg_trello_members" in 0.61s
23:06:52  12:06:52 + 7 of 7 Vacuuming "analytics"."analytics"."warehouse_operation"
23:06:56  12:06:56 + 7 of 7 Analyzing "analytics"."analytics"."warehouse_operation"
23:06:56  12:06:56 + 7 of 7 Finished "analytics"."analytics"."warehouse_operation" in 4.05s

For dbt Labs reviewer:

  1. Put this in your packages.yml and then dbt deps:
packages:
  - git: https://github.com/dbt-labs/redshift.git
    revision: fix/maintenance-macro-with-ref
  1. Add ref() macro override doing nothing:
{% macro ref(model_name) %}

  {% set rel = builtins.ref(model_name) %}
  {% set newrel = rel %}
  {% do return(newrel) %}

{% endmacro %}
  1. Try a vacuum by copying the above dbt run-operation redshift_maintenance --args ... example.

Checklist

  • I have verified that these changes work locally
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)

Copy link
Contributor

@joellabes joellabes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this @jeremyyeo! I think I'm due for another wave of package releases this week or so, so this should be able to go out shortly.

@joellabes joellabes merged commit 28dda80 into master Feb 14, 2022
@joellabes joellabes deleted the fix/maintenance-macro-with-ref branch February 14, 2022 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants