Add EIA AEO fuel cost projections #3656

jdangerx · 2024-05-27T21:18:00Z

Overview

Closes #3649.

promotes core_eiaaeo__yearly_projected_fuel_cost_in_electric_sector_by_type resource + updates fields
adds real_cost_basis_year field so we know what the real cost actually means

Testing

How did you make sure this worked? How can a reviewer verify this?

Materialized assets via Dagster, asset checks passed, and made some graphs as seen in #3649 .

To-do list

Give feedback

If updating analyses or data processing functions: make sure to update or write data validation tests (e.g., test_minmax_rows())
Update the release notes: reference the PR and related issues.
Ensure docs build, unit & integration tests, and test coverage pass locally with make pytest-coverage (otherwise the merge queue may reject your PR)
Review the PR yourself and call out any questions or issues you have
For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
Options

src/pudl/metadata/resources/eiaaeo.py

jdangerx · 2024-05-27T21:21:22Z

src/pudl/metadata/resources/eiaaeo.py

@@ -266,6 +266,7 @@
                "fuel_type_eiaaeo",
                "fuel_cost_per_mmbtu",
                "fuel_cost_real_per_mmbtu_eiaaeo",
+                "real_cost_basis_year",


Added this so we know what year the real cost refers to.

src/pudl/metadata/resources/eiaaeo.py

src/pudl/transform/eiaaeo.py

jdangerx · 2024-05-27T21:29:47Z

FYI @anamileva and @gschivley, two people who have been curious about this.

aesharpe

Generally looks good!

It looks like you added the resource metadata for this table in another PR? I have one comment about the table description (in metadata/resources/eiaaeo.py) containing the phrase: Projected fuel prices to the electric power sector -- what does "to the electric power sector" mean -- should this be "for"? It also doesn't quite match the description you put in the release notes that says "generation and capacity" and leaves out cost.

aesharpe · 2024-05-31T19:17:49Z

src/pudl/metadata/fields.py

@@ -1644,6 +1644,15 @@
        "description": "Average cost of fuel delivered in the report year per reported fuel unit (USD).",
        "unit": "USD",
    },
+    "fuel_cost_real_per_mmbtu_eiaaeo": {


why do we need the suffix eiaaeo?

Also this is a nit and not blocking but here we say: cost_real but in the other column we say real_cost. Maybe we should pick one?

We wanted to add the suffix eiaaeo because it's possible that other datasets mean something different by "real fuel cost" - maybe it is always based in the same year, for example. It's easier to remove a suffix and merge two fields into one than it is to have to tease apart the different use cases later.

I think this column would be named "Fuel Cost (Real 2022 USD/MMbtu)" and the other would be named "Real Cost Basis Year" (not "Cost (real) Basis Year") in English - which is why I put them that way. I think it's slightly less consistent but more readable the way it is, but happy to switch out if you feel strongly about it.

mmm yeah there is no great way to deal with this -- could say real_cost_fuel? but that is inconsistent with how we report fuel_cost in other columns too 🤷. I'll leave this up to you.

jdangerx · 2024-06-10T16:57:03Z

It looks like you added the resource metadata for this table in another PR?

Yeah, see #3588 !

These resource schemas are pretty closely coupled to each other, so I wanted to design them all at the same time, but also I didn't want to have to implement all the resources at once.

I added the metadatas to a "staging" area, which let @cmgosnell review the schemas & fields on their own before I did any more implementation.

Now, when implementing a new AEO asset, I just promote the resource definition from the staging area.

I think it's a nice pattern to follow, it avoids huge PRs as well as re-work when we realize the asset design was wrong.

I have one comment about the table description (in metadata/resources/eiaaeo.py) containing the phrase: Projected fuel prices to the electric power sector -- what does "to the electric power sector" mean -- should this be "for"? It also doesn't quite match the description you put in the release notes that says "generation and capacity" and leaves out cost.

Yeah, good catch :) let me fix the docs!

aesharpe

Nothing is blocking, so I'm approving this, but I have some questions!

aesharpe · 2024-06-17T19:59:04Z

src/pudl/metadata/fields.py

+    "real_cost_basis_year": {
+        "type": "integer",
+        "description": "Four-digit year which is the basis for any 'real cost' "
+        "monetary values (as opposed to nominal values).",
+    },


Is "real" vs. "nominal" cost something we should expect users to understand? TBH I don't know the difference so I don't want to assume other people do. Is this an easily googleable term, or should we go into more depth about what "real" means here. Also open to housing this explanation elsewhere (i.e.: the metadata table description, another/several column description(s)).

This is a pretty easily searchable term - just going by this docstring and searching "real cost vs nominal value" gets you the right thing as the first hit on DuckDuckGo and the fourth hit on Google after the rando AI generated BS.

aesharpe · 2024-06-17T19:59:52Z

src/pudl/metadata/resources/eiaaeo.py

+            "Projected fuel prices for the electric power sector, including "
            "electricity-only and combined-heat-and-power plants that have a "
            "regulatory status."


See comment above about describing real vs. nominal somewhere.

aesharpe · 2024-06-17T20:02:38Z

src/pudl/transform/eiaaeo.py

+check_specs = [
+    AeoCheckSpec(
+        name="gen_in_electric_sector_by_tech",
+        asset="core_eiaaeo__yearly_projected_generation_in_electric_sector_by_technology",
+        num_rows_by_report_year={2023: 166972},
+        category_counts=BASE_AEO_CATEGORIES
+        | {
+            "technology_description_eiaaeo": 13,


Is there a reason these aren't in test modules?

These are for defining asset checks, which get run during the ETL after an asset has been materialized.

jdangerx commented May 27, 2024

View reviewed changes

jdangerx requested a review from aesharpe May 27, 2024 21:23

aesharpe requested changes May 31, 2024

View reviewed changes

jdangerx force-pushed the 3649-eiaaeo-fuel-cost branch from 21faaa6 to 30c3f2a Compare June 10, 2024 18:29

jdangerx requested a review from aesharpe June 10, 2024 18:29

zaneselvans assigned jdangerx Jun 17, 2024

zaneselvans added new-data Requests for integration of new data. gridlab Work related to open modeling input data integration funded/coordinated by GridLab eiaaeo EIA Annual Energy Outlook labels Jun 17, 2024

aesharpe approved these changes Jun 17, 2024

View reviewed changes

jdangerx added 3 commits June 17, 2024 17:41

Add EIA AEO fuel cost projections

f09d3e8

Update release notes + docs

8ddbf4c

Clarify docs

15f3cf5

jdangerx force-pushed the 3649-eiaaeo-fuel-cost branch from 30c3f2a to 15f3cf5 Compare June 17, 2024 21:41

jdangerx enabled auto-merge June 17, 2024 21:41

Fix alembic history

4ebdaad

jdangerx added this pull request to the merge queue Jun 17, 2024

Merged via the queue into main with commit 4c89d53 Jun 17, 2024
12 checks passed

jdangerx deleted the 3649-eiaaeo-fuel-cost branch June 17, 2024 23:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add EIA AEO fuel cost projections #3656

Add EIA AEO fuel cost projections #3656

jdangerx commented May 27, 2024 •

edited

Loading

To-do list

jdangerx May 27, 2024

jdangerx commented May 27, 2024

aesharpe left a comment •

edited

Loading

aesharpe May 31, 2024

aesharpe May 31, 2024

jdangerx Jun 10, 2024

aesharpe Jun 17, 2024

jdangerx commented Jun 10, 2024

aesharpe left a comment

aesharpe Jun 17, 2024

jdangerx Jun 17, 2024

aesharpe Jun 17, 2024

aesharpe Jun 17, 2024

zaneselvans Jun 17, 2024

Add EIA AEO fuel cost projections #3656

Add EIA AEO fuel cost projections #3656

Conversation

jdangerx commented May 27, 2024 • edited Loading

Overview

Testing

To-do list

Choose a reason for hiding this comment

jdangerx commented May 27, 2024

aesharpe left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdangerx commented Jun 10, 2024

aesharpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdangerx commented May 27, 2024 •

edited

Loading

aesharpe left a comment •

edited

Loading