3243 PHMSA transmission part J

[jdangerx]

Overview

Closes #3243.

• add yearly_miles_of_pipe_by_decade to file_map, page_map, page_part_map, skiprows, skipfooter
• add raw_phmsagas__yearly_miles_of_pipe_by_decade asset to ETL
• add column map file

Testing

How did you make sure this worked? How can a reviewer verify this?

I ran the ETL, then ran:

from dagster import AssetKey
from pudl.etl import def
asset = defs.load_asset_value(AssetKey("raw_phmsagas__yearly_miles_of_pipe_by_decade"))

And looked at some of the data:

The spike in gathering miles in 2022 appears to be due to "Onshore Type C" pipes finally getting reported starting in 2022:

Not sure if there's any specific tests I should write about the raw data.

To-do list

Preview Give feedback

1. Make sure full ETL runs & make pytest-integration-full passes locally
2. For major data coverage & analysis changes, run data validation tests
3. If updating analyses or data processing functions: make sure to update or write data validation tests
4. Update the release notes: reference the PR and related issues.
5. Review the PR yourself and call out any questions or issues you have

[e-belfer]

I've been using onshore_type_a rather than onshore_a, so we should pick one to be consistent (there will definitely be one last clean up PR, if we just want to make note of it in an issue I'm ok with that too). Otherwise all looks good here.

[jdangerx]

I'll make it onshore_type_a, it matches the PDF more closely!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (eb739ed) 92.7% compared to head (5df0503) 92.7%.
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3266     +/-   ##
=======================================
- Coverage   92.7%   92.7%   -0.0%     
=======================================
  Files        144     144             
  Lines      13091   13091             
=======================================
- Hits       12135   12134      -1     
- Misses       956     957      +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.