Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate EIA 860 2022 final release data #3008

Closed
12 tasks done
Tracked by #2699
aesharpe opened this issue Nov 2, 2023 · 0 comments · Fixed by #3040
Closed
12 tasks done
Tracked by #2699

Integrate EIA 860 2022 final release data #3008

aesharpe opened this issue Nov 2, 2023 · 0 comments · Fixed by #3040
Assignees
@aesharpe
Labels
eia860 Anything having to do with EIA Form 860 new-data Requests for integration of new data.

Comments

@aesharpe
Copy link
Member

aesharpe commented Nov 2, 2023 •
edited
Loading

  • Add the new Zenodo archive DOI values to pudl/workspace/datastore.py.
  • Run the datastore script to download the new data.
  • Update the information in pudl/package_data if necessary:
    • file maps (remove _Early_Release suffix!)
    • column maps (probably the same)
    • page maps (probably the same)
    • skip footer (probably the same)
    • skip rows (remove the +1 for early release data)
  • Launch dagit and refresh the code location.
  • Materialize the raw_eia860 asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update the package_data accordingly.
  • Materialize the norm_eia asset group. You'll probably see some errors related to encoding. Take a look at which column it's talking about and look at metadata/resources/eia.py to see which encoder in CODE_METADATA to tweak.
  • Update the validation test test_minmax_rows in test/validate/eia_test.py.
@aesharpe aesharpe mentioned this issue Nov 2, 2023
Closed
@aesharpe aesharpe added eia860 Anything having to do with EIA Form 860 new-data Requests for integration of new data. labels Nov 2, 2023
@aesharpe aesharpe self-assigned this Nov 6, 2023
@aesharpe aesharpe moved this from New to In progress in Catalyst Megaproject Nov 6, 2023
@aesharpe aesharpe linked a pull request Nov 13, 2023 that will close this issue
Add EIA860 2022 final release data #3040
Merged
@github-project-automation github-project-automation bot moved this from In progress to Done in Catalyst Megaproject Nov 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aesharpe aesharpe

Labels
eia860 Anything having to do with EIA Form 860 new-data Requests for integration of new data.
Projects
Catalyst Megaproject
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add EIA860 2022 final release data catalyst-cooperative/pudl
1 participant
@aesharpe