Merge pull request #17 from fivetran/feature/enable-public-models

[Feature] Enable User models by default
fivetran · Jan 7, 2025 · 4dde125 · 4dde125
2 parents bec86c5 + 70b4897
commit 4dde125
Show file tree

Hide file tree

Showing 21 changed files with 260 additions and 93 deletions.
diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh
@@ -19,7 +19,7 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars {'servicenow__using_roles: True'} --target "$db"
+dbt run --vars {'servicenow__using_roles: False'} --target "$db"
 dbt test --target "$db"
 
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,27 @@
+# dbt_servicenow v0.4.0
+[PR #17](https://github.com/fivetran/dbt_servicenow/pull/17) includes the following updates:
+
+## Breaking Changes
+- Enabled `servicenow__user_aggregated` and `servicenow__user_enhanced` by default by changing the default `servicenow__using_roles` value to true.
+- Staging and base models now also enabled by default include:
+  - `stg_servicenow__sys_user_grmember`
+  - `stg_servicenow__sys_user_has_role`
+  - `stg_servicenow__sys_user_role`
+  - `stg_servicenow__sys_user_grmember_base`
+  - `stg_servicenow__sys_user_has_role_base`
+  - `stg_servicenow__sys_user_role_base`
+- Because this change will introduce new end model tables to users because they were initially disabled by default and changes the default behavior of a variable, we've classified this as a breaking change. 
+- Quickstart users will be able to handle disabling of these tables by [utilizing the Quickstart data model tab](https://fivetran.com/docs/transformations/quickstart) within your ServiceNow connection, and choose to un-sync the tables required for these models.
+- Non-Quickstart users who do not have these source tables will need to define the variable in their `dbt_project.yml`. [See the README](https://github.com/fivetran/dbt_servicenow/blob/main/README.md#optional-step-4-additional-configurations) for more instructions. 
+
+## Documentation Update
+- Updated the variable configuration section of the README since `servicenow__using_roles` is now set to true by default.
+- Moved badges at top of the README below the H1 header to be consistent with popular README formats.
+
+## Under the Hood
+- Changed Buildkite scripts to run models when `servicenow__using_roles` is set to False (since it's now True by default on all models).
+- Updated validation tests to account for the new `servicenow__using_roles` variable configuration.
+
 # dbt_servicenow v0.3.0
 [PR #10](https://github.com/fivetran/dbt_servicenow/pull/10) includes the following updates:
 

diff --git a/README.md b/README.md
@@ -1,4 +1,6 @@
-<p align="center">
+# ServiceNow dbt Package ([Docs](https://fivetran.github.io/dbt_servicenow/))
+
+<p align="left">
     <a alt="License"
         href="https://github.com/fivetran/dbt_servicenow/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
@@ -10,9 +12,6 @@
         <img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
 </p>
 
-
-# ServiceNow dbt Package ([Docs](https://fivetran.github.io/dbt_servicenow/))
-
 ## What does this dbt package do?
 
 This package models ServiceNow data from [Fivetran's connector](https://fivetran.com/docs/applications/servicenow). It uses data in the format described by [this ERD](https://fivetran.com/docs/applications/servicenow#schemainformation).
@@ -73,7 +72,7 @@ Include the following ServiceNow package version in your `packages.yml` file:
 ```yml
 packages:
   - package: fivetran/servicenow
-    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ### Step 3: Define database and schema variables
@@ -104,14 +103,14 @@ To connect your multiple schema/database sources to the package models, follow t
 
 ### (Optional) Step 4: Additional configurations
 
-#### Populating the User Models
-Our user grain models are disabled by default because not everyone syncs the underlying tables: `sys_user_role`, `sys_user_has_role`, and `sys_user_grmember`. If these tables do exist in your schema, set this following variable `servicenow__using_roles` to True in order to populate the user models `servicenow__user_aggregated` and `servicenow__user_enhanced`.
+#### Enable/Disable the User End Models
+Our user grain models are enabled by default, but we do understand that not everyone syncs the underlying tables: `sys_user_role`, `sys_user_has_role`, and `sys_user_grmember`. If these tables do not exist in your schema, set this following variable `servicenow__using_roles` to False in order to disable the end models `servicenow__user_aggregated` and `servicenow__user_enhanced`.
 
 ```yml
 # dbt_project.yml
 
 vars:
-    servicenow__using_roles: True
+    servicenow__using_roles: False # Disable if you are not using user roles
 ```
 
 #### Changing the Build Schema
@@ -140,7 +139,6 @@ vars:
 ```
 </details>
 
-
 ### (Optional) Step 5: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand for details</summary>
 <br>

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,6 +1,6 @@
 config-version: 2
 name: 'servicenow'
-version: '0.3.0'
+version: '0.4.0'
 
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'servicenow_integration_tests'
-version: '0.3.0'
+version: '0.4.0'
 config-version: 2
 
 profile: 'integration_tests'
@@ -27,8 +27,6 @@ vars:
     servicenow_sys_user_has_role_identifier: "sys_user_has_role_data"
     servicenow_sys_user_grmember_identifier: "sys_user_grmember_data"
 
-    # servicenow__using_roles: true # to be turned on when generating docs so that can be generated via the manifest and docs as it is turned off by default.
-
 models:
   +schema: "servicenow_{{ var('directed_schema','dev') }}"
 

diff --git a/integration_tests/packages.yml b/integration_tests/packages.yml
@@ -1,2 +1,2 @@
 packages:
-  - local: ../
+  - local: ../
diff --git a/integration_tests/tests/consistency/user_aggregated_aggs.sql b/integration_tests/tests/consistency/user_aggregated_aggs.sql
@@ -1,7 +1,7 @@
 {{ config(
     tags="fivetran_validations",
-    enabled=var('fivetran_validation_tests_enabled', false)
-) }}
+    enabled=(var('fivetran_validation_tests_enabled', false) and var('servicenow__using_roles', true))
+) }} 
 
 with prod as (
     select

diff --git a/integration_tests/tests/consistency/user_enhanced_row_counts.sql b/integration_tests/tests/consistency/user_enhanced_row_counts.sql
@@ -1,6 +1,6 @@
 {{ config(
     tags="fivetran_validations",
-    enabled=var('fivetran_validation_tests_enabled', false)
+    enabled=(var('fivetran_validation_tests_enabled', false) and var('servicenow__using_roles', true))
 ) }}
 
 with prod as (

diff --git a/integration_tests/tests/integrity/user_match.sql b/integration_tests/tests/integrity/user_match.sql
@@ -1,7 +1,7 @@
 
 {{ config(
     tags="fivetran_validations",
-    enabled=var('fivetran_validation_tests_enabled', false)
+    enabled=(var('fivetran_validation_tests_enabled', false) and var('servicenow__using_roles', true))
 ) }}
 
 with source as (

diff --git a/models/servicenow__user_aggregated.sql b/models/servicenow__user_aggregated.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 with sys_user as (
 

diff --git a/models/servicenow__user_enhanced.sql b/models/servicenow__user_enhanced.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 with user_aggregates as (
 

diff --git a/models/staging/base/stg_servicenow__sys_user_grmember_base.sql b/models/staging/base/stg_servicenow__sys_user_grmember_base.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 {{
     fivetran_utils.union_data(

diff --git a/models/staging/base/stg_servicenow__sys_user_has_role_base.sql b/models/staging/base/stg_servicenow__sys_user_has_role_base.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 {{
     fivetran_utils.union_data(

diff --git a/models/staging/base/stg_servicenow__sys_user_role_base.sql b/models/staging/base/stg_servicenow__sys_user_role_base.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 {{
     fivetran_utils.union_data(

diff --git a/models/staging/stg_servicenow__sys_user_grmember.sql b/models/staging/stg_servicenow__sys_user_grmember.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 with base as (
 

diff --git a/models/staging/stg_servicenow__sys_user_has_role.sql b/models/staging/stg_servicenow__sys_user_has_role.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 with base as (
 

diff --git a/models/staging/stg_servicenow__sys_user_role.sql b/models/staging/stg_servicenow__sys_user_role.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('servicenow__using_roles', False)) }}
+{{ config(enabled=var('servicenow__using_roles', True)) }}
 
 with base as (