-
-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Map PHMSA Natural Gas Transmission Part M columns #3270
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3270 +/- ##
=====================================
Coverage 92.7% 92.7%
=====================================
Files 144 144
Lines 13091 13091
=====================================
Hits 12134 12134
Misses 957 957 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Do we want to call the main table
yearly_transmission_failures_leaks_repairs
? - Columns in this table definitely go back all the way to 1990. See Part C/D/E in the 2005 form and the 1988 form as well.
- Just a note that I've been using either
other
orclass_3_4_only
to refer tononhca_nonmca
sections. See the advantages of both but we should pick one to use consistently. Otherwise column names look good.
@e-belfer do you know that class 3 & 4 are always non-mca/non-hca? Or is it a square and rectangle thing? I tried to read about it and couldn't find a clear a == b although they are clearly related - in that they are about how important/regulated the land is where the infrastructure is located. But this part M has class 1 & 2/non-mca/non-hca. |
src/pudl/package_data/phmsagas/column_maps/yearly_miles_of_transmission_pipe_by_nps.csv
Outdated
Show resolved
Hide resolved
This has almost certainly changed overtime, but more recently it seems like operators can either 1) blanket define all Class 3&4 locations as HCAs, or use a formula provided by PHMSA (which would explain the existence of non-MCA/HCA Class 3&4 locations. Class 1 & 2 locations could also be HCAs or MCAs if they meet the criteria (which I think essentially boils down to "many human lives but slightly further away from the pipeline than would the criteria for a Class 3/4". (Source is p.8 here: https://www.phmsa.dot.gov/sites/phmsa.dot.gov/files/docs/technical-resources/pipeline/gas-transmission-integrity-management/62271/faqsgas-transmission-integrity-management201904.pdf) |
it seems like there is mostly overlap but I am starting on part Q now and it has all classes 1-4 associated with in HCA, in MCA or not in HCA or MCA. So I don't think it is a good idea to assume there is a perfect overlap. |
or @e-belfer is your suggestion that we use |
Yes using "other" instead of "nonhca_nonmca" is my suggestion, I would definitely not conflate the classes and MCA/HCA. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One column mapping I would like you to fix, a question re doe
, and a response to your question. Otherwise thank you for handling this giant table, the asset generates in dagster as expected.
src/pudl/package_data/phmsagas/column_maps/yearly_transmission_failures_leaks_repairs.csv
Outdated
Show resolved
Hide resolved
src/pudl/package_data/phmsagas/column_maps/yearly_transmission_failures_leaks_repairs.csv
Outdated
Show resolved
Hide resolved
src/pudl/package_data/phmsagas/column_maps/yearly_transmission_failures_leaks_repairs.csv
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great, thanks! One question I'll punt to #3277
report_year,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,yr,report_year,report_year,report_year,report_year,report_year,report_year,report_year,report_year,report_year,report_year,report_year,report_year,report_year | ||
filing_date,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,doe,,,,,,,,,,,,, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't keep filing_date
in the other tables other than A-D, I believe. But I also see an argument for keeping it. Let's merge this for now and take this up in #3277, as it'll affect all the tables we've integrated up until now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm yea good point! but agreed let's punt it. I'll add a quick comment over there about all of the date fields.
Overview
Closes #3263.
What problem does this address?
What did you change?
Testing
How did you make sure this worked? How can a reviewer verify this?
To-do list