Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parquet download link to data dictionary #3984

Merged
merged 5 commits into from
Dec 18, 2024
Merged

Add parquet download link to data dictionary #3984

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
39dce73
Add parquet download link to data dictionary
bendnorman Dec 3, 2024
ba080ea
Merge branch 'main' into add-parquet-files-data-dictionary
e-belfer Dec 11, 2024
a56b721
Add parquet file access method to data access page
bendnorman Dec 11, 2024
734073f
Merge branch 'main' into add-parquet-files-data-dictionary
bendnorman Dec 12, 2024
b207852
Make small parquet description changes and add hourly table descripti…
bendnorman Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 12 additions & 23 deletions docs/data_access.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@ PUDL data, so if you have a suggestion, please `open a GitHub issue
<https://github.com/catalyst-cooperative/pudl/issues>`__. If you have a question, you
can `create a GitHub discussion <https://github.com/orgs/catalyst-cooperative/discussions/new?category=help-me>`__.

PUDL's primary data output is the ``pudl.sqlite`` database. We recommend working with
tables with the ``out_`` prefix, as these tables contain the most complete and easiest
to work with data. For more information about the different types
of tables, read through :ref:`PUDL's naming conventions <asset-naming>`.
PUDL's primary data output is the ``pudl.sqlite`` database. All the tables are also
distributed as individual Parquet files which are more space efficient, have richer
data types and are better suited for distributed and large-scale data analysis.
We recommend working with tables with the ``out_`` prefix, as these tables contain
the most complete and easiest to work with data. For more information about the
different types of tables, read through :ref:`PUDL's naming conventions <asset-naming>`.

.. _access-modes:

Expand Down Expand Up @@ -106,32 +108,19 @@ resulting outputs pass all of the data validation tests we've defined, the outpu
automatically uploaded to the `AWS Open Data Registry
<https://registry.opendata.aws/catalyst-cooperative-pudl/>`__, and used to deploy a new
version of Datasette (see above). These nightly build outputs can be accessed using the
AWS CLI, or programmatically via the S3 API. They can also be downloaded directly over
HTTPS using the following links:
AWS CLI, or programmatically via the S3 API. If you don't want to mess with the API
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
AWS CLI, or programmatically via the S3 API. If you don't want to mess with the API
AWS CLI, or programmatically via the S3 API.
If you don't want to mess with the API

or CLI, you can also download the data directly over HTTPS. The download links for
each table's Parquet file can be found in
the :doc:`PUDL data dictionary page </data_dictionaries/pudl_db>`.

These are the download links for the PUDL and raw FERC SQLite databases:

Fully Processed SQLite Databases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* `Main PUDL Database <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/pudl.sqlite.zip>`__
* `US Census DP1 Database (2010) <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/censusdp1tract.sqlite.zip>`__

Hourly Tables as Parquet
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this section because now all tables are available as Parquet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true, but these tables aren't in SQLite and so I think shouting them out here is still helpful - if people are looking for them they won't be able to find them in the full DB.

^^^^^^^^^^^^^^^^^^^^^^^^

Hourly time series take up a lot of space in SQLite and can be slow to query in bulk,
so we have moved to publishing all our hourly tables using the compressed, columnar
`Apache Parquet <https://parquet.apache.org/docs/>`__ file format.

* `EIA-930 BA Hourly Interchange <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/core_eia930__hourly_interchange.parquet>`__
* `EIA-930 BA Hourly Net Generation by Energy Source <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/core_eia930__hourly_net_generation_by_energy_source.parquet>`__
* `EIA-930 BA Hourly Operations <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/core_eia930__hourly_operations.parquet>`__
* `EIA-930 BA Hourly Subregion Demand <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/core_eia930__hourly_subregion_demand.parquet>`__
* `EPA CEMS Hourly Emissions <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/core_epacems__hourly_emissions.parquet>`__
* `FERC-714 Hourly Estimated State Demand <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/out_ferc714__hourly_estimated_state_demand.parquet>`__
* `FERC-714 Hourly Planning Area Demand <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/out_ferc714__hourly_planning_area_demand.parquet>`__
* `GridPath RA Toolkit Hourly Available Capacity Factors <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/out_gridpathratoolkit__hourly_available_capacity_factor.parquet>`__
* `VCE Resource Adequacy Renewable Energy (RARE) Dataset <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/out_vcerare__hourly_available_capacity_factor.parquet>`__

Raw FERC DBF & XBRL data converted to SQLite
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
7 changes: 5 additions & 2 deletions docs/templates/resource.rst.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,14 @@
**This table has no primary key.**
{%- endif %}

**Access methods:**

{% if resource.create_database_schema -%}
`Browse or query this table in Datasette. <https://data.catalyst.coop/pudl/{{ resource.name }}>`__
* `Browse or query this table in Datasette. <https://data.catalyst.coop/pudl/{{ resource.name }}>`__
{% else -%}
This table is not published to Datasette.
* This table is not published to Datasette.
{%- endif %}
* `Download this table as a Parquet file. <https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/{{ resource.name }}.parquet>`__

.. list-table::
:widths: auto
Expand Down
Loading