mart_gtfs.fct_vehicle_locations_grouped - incremental settings like fct_vehicle_locations

[tiffanychu90]

Description

Airflow DAG error showed missing dt. Instead of using partitions of service_date, we'll grab dt from mart_gtfs.fct_vehicle_locations and pass it through so source table and downstream derived table are partitioned the same way.

Runtime Error
[2025-02-04, 00:26:15 UTC] {pod_manager.py:435} INFO - [base]   Database Error in model fct_vehicle_locations_grouped (models/mart/gtfs/fct_vehicle_locations_grouped.sql)
[2025-02-04, 00:26:17 UTC] {pod_manager.py:435} INFO - [base]     Unrecognized name: dt at [3:17]

Follow-up to #3660

Slack thread -- are there implications for data duplication, data storage, since vehicle locations is probably some of the biggest tables

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

run table

jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (fix-fct-vehicle-locations-grouped) $ poetry run dbt run -s fct_vehicle_locations_grouped+
17:07:06  Running with dbt=1.5.1
17:07:09  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
17:07:10  Found 561 models, 1013 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 207 sources, 4 exposures, 0 metrics, 0 groups
17:07:10  
17:07:13  Concurrency: 8 threads (target='dev')
17:07:13  
17:07:13  1 of 1 START sql incremental model tiffany_mart_gtfs.fct_vehicle_locations_grouped  [RUN]
17:07:39  1 of 1 OK created sql incremental model tiffany_mart_gtfs.fct_vehicle_locations_grouped  [SCRIPT (36.0 GiB processed) in 25.90s]
17:07:39  
17:07:39  Finished running 1 incremental model in 0 hours 0 minutes and 28.48 seconds (28.48s).
17:07:39  
17:07:39  Completed successfully
17:07:39  
17:07:39  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

test table

jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (fix-fct-vehicle-locations-grouped) $ poetry run dbt test -s fct_vehicle_locations_grouped
17:08:38  Running with dbt=1.5.1
17:08:41  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
17:08:42  Found 561 models, 1013 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 207 sources, 4 exposures, 0 metrics, 0 groups
17:08:42  
17:08:42  Nothing to do. Try checking your model configs and model specification args

generate docs for new table

jovyan@jupyter-tiffanychu90 ~/data-infra/warehouse (fix-fct-vehicle-locations-grouped) $ poetry run dbt docs generate
17:09:06  Running with dbt=1.5.1
17:09:09  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.calitp_warehouse.mart.ad_hoc
17:09:10  Found 561 models, 1013 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 207 sources, 4 exposures, 0 metrics, 0 groups
17:09:10  
17:09:17  Concurrency: 8 threads (target='dev')
17:09:17  
17:10:02  Building catalog
17:10:27  Catalog written to /home/jovyan/data-infra/warehouse/target/catalog.json

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

• No action required
• Actions required (specified below)

[github-actions]

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

• For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_facilities

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_other

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_rolling_stock

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_federal

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_local

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_other

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_state

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_decommissioned_operatingfares

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_decommissioned_operatingother

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_federal

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_local

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_other

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_state

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_summary_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_drm

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_fares

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_ga

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_nvm

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_vm

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_vo

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_pmt

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_upt

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_voms

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_vrh

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_vrm

calitp_warehouse.mart.gtfs.fct_vehicle_locations_grouped

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_facilities

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_other

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_rolling_stock

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_federal

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_local

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_other

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_state

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_federal

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_local

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_other

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_state

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_drm

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_fares

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_ga

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_nvm

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_vm

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_vo

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_pmt

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_upt

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_voms

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_vrh

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_vrm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__facilities

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__other

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__rolling_stock

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_federal

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_local

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_other

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_state

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__decommissioned_operatingfares

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__decommissioned_operatingother

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_federal

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_local

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_other

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_state

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__summary_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__drm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__fares

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_ga

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_nvm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_vm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_vo

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__pmt

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__upt

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__voms

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__vrh

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__vrm

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

[vevetron]

LGTM

[vevetron]

@tiffanychu90 notes the bubble shows the wrong table. fct_monthly_route_service_by_timeofday vs fct_vehicle_locations_grouped