Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eia923 2022 final release q4 update Nov 21 #3073

Merged
merged 23 commits into from
Nov 29, 2023
Merged

Eia923 2022 final release q4 update Nov 21 #3073

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
e6b07f1
new eia923-2022 DOI update
robertozanchi Nov 9, 2023
5ca4d2a
Updated DOI
robertozanchi Nov 14, 2023
2f39012
Update file_map and skiprows
robertozanchi Nov 21, 2023
3b9db34
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 21, 2023
861ead1
Remove accidentally committed input-output.ts file
aesharpe Nov 22, 2023
8e174bb
Remove sandbox reference DOIs because they are now invalid since Zeno…
aesharpe Nov 22, 2023
1f7e76a
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 27, 2023
cdd4bf5
Update minmax rows for new EIA923 data
aesharpe Nov 27, 2023
69f23ba
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 27, 2023
741f2dd
Remove ref: env.GITHUB_REF in zenodo-cache-sync to avoid errors relat…
aesharpe Nov 27, 2023
992de17
Merge branch 'eia923-2022-final-release-q4' of https://github.com/rob…
aesharpe Nov 27, 2023
dd57d79
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 27, 2023
49dcd1c
Add release notes note about 923 final release and quarterly update
aesharpe Nov 27, 2023
dd18294
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 27, 2023
3a7bfa6
Fix merge failure with dev
aesharpe Nov 27, 2023
44e7ff8
Merge branch 'eia923-2022-final-release-q4' of https://github.com/rob…
aesharpe Nov 27, 2023
eb01d0c
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 27, 2023
4afc1aa
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 28, 2023
f0ec29b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 28, 2023
855b701
Update references to years of EIA923 data ingested by PUDL. Also upda…
aesharpe Nov 29, 2023
b29c042
Fix the month included in the EIA923 data years in the README
aesharpe Nov 29, 2023
8048287
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 29, 2023
2994731
Merge branch 'dev' into eia923-2022-final-release-q4
aesharpe Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/zenodo-cache-sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,6 @@ jobs:
- name: Checkout desired branch
uses: actions/checkout@v4
with:
ref: ${{ env.GITHUB_REF }}
Comment on lines -44 to -45
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this deletion about??

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This deletion was an effort to fix test failures resulting from the fact that this PR is from a forked repo. But those tests still fail due to permissions anyways, so idk whether it's worth keeping this out or not. @bendnorman @jdangerx do you have thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good point. Forgot that the GCS cache sync needs credentials 🤦 . I think we should still remove this because the default behavior uses the GITREF that started the action.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdangerx any idea why pytest.yml uses fetch-depth: 2 instead of the default fetch-depth: 1?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I bet there's something somewhere in our tox-pytest that cares about "what has changed since the previous commit." Not sure what that would be, though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth keeping it out, but point taken that the GCS credentials are obviously not going to be available in fork PRs. If we get more time to think about contributor workflows in the future we should think a bit harder about how we want our CI checks to go.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the update-conda-lockfile action there's a check for one of the actions that it's not on a forked repo. We could add that where appropriate to skip things that should run from outside the org.


- name: Install Conda environment using mamba
uses: mamba-org/setup-micromamba@v1
Expand Down
4 changes: 2 additions & 2 deletions docs/release_notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ Data Coverage

* Updated :doc:`data_sources/eia860` to include final release data from 2022.
* Updated :doc:`data_sources/eia861` to include final release data from 2022.
* Updated :doc:`data_sources/eia923` to include early release data from 2022 and
monthly YTD data as of April 2023.
* Updated :doc:`data_sources/eia923` to include final release data from 2022 and
monthly YTD data as of October 2023.
* Updated :doc:`data_sources/epacems` to switch from the old FTP server to the new
CAMPD API, and to include 2022 data. Due to changes in the ETL, Alaska, Puerto Rico
and Hawaii are now included in CEMS processing. See issue :issue:`1264` & PRs
Expand Down
20 changes: 10 additions & 10 deletions src/pudl/package_data/eia923/file_map.csv
View file
Open in desktop

Large diffs are not rendered by default.

22 changes: 11 additions & 11 deletions src/pudl/package_data/eia923/skiprows.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
year_index,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023
generation_fuel,7,7,7,7,7,7,7,7,7,7,5,5,5,5,5,5,5,5,5,5,5,6,5
puerto_rico,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,5,5,6,6,6,7,5
stocks,7,7,7,7,7,7,7,7,7,7,5,5,5,5,5,5,5,5,5,5,5,6,4
oil_stocks,-1,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,6,4
generation_fuel,7,7,7,7,7,7,7,7,7,7,5,5,5,5,5,5,5,5,5,5,5,5,5
puerto_rico,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,5,5,6,6,6,6,5
stocks,7,7,7,7,7,7,7,7,7,7,5,5,5,5,5,5,5,5,5,5,5,5,4
oil_stocks,-1,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,4
coal_stocks,-1,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,-1,-1,-1
petcoke_stocks,-1,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,-1,-1,-1
energy_storage,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,5,5,5,5,6,4
boiler_fuel,-1,-1,-1,-1,-1,-1,-1,7,7,7,5,5,5,5,5,5,5,5,5,5,5,6,4
generator,-1,-1,-1,-1,-1,-1,-1,7,7,7,5,5,5,5,5,5,5,5,5,5,5,6,4
fuel_receipts_costs,-1,-1,-1,-1,-1,-1,-1,7,6,7,4,4,4,4,4,4,4,4,4,4,4,5,3
plant_frame,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,4,4,4,4,4,4,4,4,4,4,4,5,3
plant_frame_puerto_rico,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,4,4,4,5,3
emissions_control,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,4,4,4,4,4,4,4,4,4,4,5,-1
energy_storage,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,5,5,5,5,5,4
boiler_fuel,-1,-1,-1,-1,-1,-1,-1,7,7,7,5,5,5,5,5,5,5,5,5,5,5,5,4
generator,-1,-1,-1,-1,-1,-1,-1,7,7,7,5,5,5,5,5,5,5,5,5,5,5,5,4
fuel_receipts_costs,-1,-1,-1,-1,-1,-1,-1,7,6,7,4,4,4,4,4,4,4,4,4,4,4,4,3
plant_frame,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,4,4,4,4,4,4,4,4,4,4,4,4,3
plant_frame_puerto_rico,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,4,4,4,4,3
emissions_control,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,4,4,4,4,4,4,4,4,4,4,4,-1
17 changes: 1 addition & 16 deletions src/pudl/workspace/datastore.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,35 +166,20 @@ def get_json_string(self) -> str:
class ZenodoDoiSettings(BaseSettings):
"""Digital Object Identifiers pointing to currently used Zenodo archives."""

# Sandbox DOIs are provided for reference
censusdp1tract: ZenodoDoi = "10.5281/zenodo.4127049"
# censusdp1tract: ZenodoDoi = "10.5072/zenodo.674992"
eia860: ZenodoDoi = "10.5281/zenodo.10067566"
# eia860: ZenodoDoi = "10.5072/zenodo.1222854"
eia860m: ZenodoDoi = "10.5281/zenodo.10204686"
# eia860m: ZenodoDoi = "10.5072/zenodo.1225517"
eia861: ZenodoDoi = "10.5281/zenodo.10204708"
# eia861: ZenodoDoi = "10.5072/zenodo.1229930"
eia923: ZenodoDoi = "10.5281/zenodo.8172818"
# eia923: ZenodoDoi = "10.5072/zenodo.1217724"
eia923: ZenodoDoi = "10.5281/zenodo.10067550"
eia_bulk_elec: ZenodoDoi = "10.5281/zenodo.7067367"
# eia_bulk_elec: ZenodoDoi = "10.5072/zenodo.1103572"
epacamd_eia: ZenodoDoi = "10.5281/zenodo.7900974"
# epacamd_eia: ZenodoDoi = "10.5072/zenodo.1199170"
epacems: ZenodoDoi = "10.5281/zenodo.8235497"
# epacems: ZenodoDoi = "10.5072/zenodo.1228519"
ferc1: ZenodoDoi = "10.5281/zenodo.8326634"
# ferc1: ZenodoDoi = "10.5072/zenodo.1234455"
ferc2: ZenodoDoi = "10.5281/zenodo.8326697"
# ferc2: ZenodoDoi = "10.5072/zenodo.1236695"
ferc6: ZenodoDoi = "10.5281/zenodo.8326696"
# ferc6: ZenodoDoi = "10.5072/zenodo.1236703"
ferc60: ZenodoDoi = "10.5281/zenodo.8326695"
# ferc60: ZenodoDoi = "10.5072/zenodo.1236694"
ferc714: ZenodoDoi = "10.5281/zenodo.8326694"
# ferc714: ZenodoDoi = "10.5072/zenodo.1237565"
phmsagas: ZenodoDoi = "10.5281/zenodo.8346646"
# phmsagas: ZenodoDoi = "10.5072/zenodo.1239253"

class Config:
"""Pydantic config, reads from .env file."""
Expand Down
16 changes: 8 additions & 8 deletions test/validate/eia_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,15 @@ def test_no_null_cols_eia(pudl_out_eia, live_dbs, cols, df_name):
@pytest.mark.parametrize(
"df_name,raw_rows,monthly_rows,annual_rows",
[
("bf_eia923", 1_559_257, 1_559_257, 127_412),
("bga_eia860", 141_652, 141_652, 141_652),
("boil_eia860", 83_356, 83_356, 83_356),
("frc_eia923", 639_647, 261_583, 25_370),
("gen_eia923", None, 5_179_377, 433_332),
("gens_eia860", 556_948, 556_948, 556_948),
("gf_eia923", 2_879_884, 2_879_884, 244_795),
("bf_eia923", 1_569_568, 1_569_568, 128_252),
("bga_eia860", 142_391, 142_391, 142_391),
("boil_eia860", 83_416, 83_416, 83_416),
("frc_eia923", 646_677, 264_043, 25_443),
("gen_eia923", None, 5_179_478, 433_336),
("gens_eia860", 556_949, 556_949, 556_949),
("gf_eia923", 2_907_735, 2_907_735, 246_324),
("own_eia860", 89_741, 89_741, 89_741),
("plants_eia860", 200_514, 200_514, 200_514),
("plants_eia860", 200_511, 200_511, 200_511),
("pu_eia860", 199_635, 199_635, 199_635),
("utils_eia860", 139_883, 139_883, 139_883),
("emissions_control_equipment_eia860", 56_616, 56_616, 56_616),
Expand Down
10 changes: 5 additions & 5 deletions test/validate/mcoe_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,11 @@ def test_no_null_rows_mcoe(pudl_out_mcoe, live_dbs, df_name, thresh):
@pytest.mark.parametrize(
"df_name,monthly_rows,annual_rows",
[
("hr_by_unit", 387_670, 32_414),
("hr_by_gen", 599_496, 50_070),
("fuel_cost", 599_496, 50_070),
("capacity_factor", 5_179_377, 433_332),
("mcoe", 5_179_785, 433_366),
("hr_by_unit", 389_530, 32_569),
("hr_by_gen", 602_580, 50_327),
("fuel_cost", 602_580, 50_327),
("capacity_factor", 5_179_478, 433_336),
("mcoe", 5_179_886, 433_370),
],
)
def test_minmax_rows_mcoe(pudl_out_mcoe, live_dbs, monthly_rows, annual_rows, df_name):
Expand Down
Loading