Hide `_out` tables and restore metadata in Datasette #3226

bendnorman · 2024-01-09T19:21:33Z

Overview

This PR:

Hides _out tables from datasette
Removes some resources sorting logic that didn't work
Resolve a bug that resulted in most of our output tables not having any metadata on datasette
Adds to test to make sure all tables in the databases have metadata

This PR will not deal with sorting the tables on datasette.

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Give feedback

Make sure full ETL runs & make pytest-integration-full passes locally
Update the release notes: reference the PR and related issues.
Review the PR yourself and call out any questions or issues you have
Write a test that ensures metadata.yaml has metadata for all tables
Options

This commit also resolves a couple of bugs in our metadata management and datasette deployment. Pretty much all out_ tables were missing metadata on datasette because DatasetteMetadata.from_data_source_ids() does some filtering that didn't include the output tables. Also, the epacamd_eia Source was incorrectly renamed to core_epa__assn_eia_epacamd during the big rename.

bendnorman · 2024-01-09T19:24:22Z

src/pudl/metadata/sources.py

@@ -299,7 +299,7 @@
        "license_raw": LICENSES["us-govt"],
        "license_pudl": LICENSES["cc-by-4.0"],
    },
-    "core_epa__assn_eia_epacamd": {


This was an oopsie from #2995

src/pudl/metadata/classes.py

bendnorman · 2024-01-09T19:28:48Z

src/pudl/metadata/classes.py

-            for res in pkg.resources
-            if res.etl_group in data_source_ids + extra_etl_groups
-        ]
+        resources = Package.from_resource_ids().get_sorted_resources()


The resources are correctly sorted in the metadata.yml file but it doesn't have an effect on the final ordering in the datasette UI. Any idea how to do this @zaneselvans?

Looks like I might need to adjust the HTML templates datasette uses. I think it's currently just pulling the table names from the database directly: https://github.com/simonw/datasette/blob/2ff4d4a60a348c143f79d63c48c329ffd0c1f02f/datasette/templates/database.html#L50-L82

The layout of Datasette pages (and the set of pages which exist) can be customized extensively with custom JavaScript, CSS, and Jinja templates

It seems like maybe we want a child-template that extends the database page that lists all of the tables, and sorts / filters them.

And if we want to hide all of the _out tables, then it seems like we can add them to the list of hidden_tables in the metadata that we generate.

Seems like we need to create a directory of our own templates and create a database-pudl.html Jinja template that overrides the default database.html template, at least for the content block?

…tasette, remove resource sorting

src/pudl/metadata/classes.py

devtools/datasette/publish.py

src/pudl/metadata/classes.py

…es_have_metadata

devtools/datasette/publish.py

…in database have datasette metadata

bendnorman · 2024-01-26T16:18:35Z

devtools/datasette/publish.py

+    # Make sure we have the expected metadata for databases
+    # headed to deployment.
+    if deploy != "metadata":
+        check_tables_have_metadata(metadata_yml, databases)


I think this solves the entanglement issue you mentioned @zaneselvans . We only check if all of the databases tables have datasette metadata if we are actually deploying datatsette.

The way the Datasette metadata is generated, and the checks rely on bespoke logic for constructing paths to the databases we expect to have metadata, and the dependence on the XBRL datapackage.json sometimes and the SQLite files other times and the PUDL Package other times all feels pretty fragile and janky, but that's beyond the scope of this PR.

src/pudl/helpers.py

zaneselvans

Added suggestions for a docstring and a more restrictive table name regex for sorting.

zaneselvans · 2024-01-26T17:15:26Z

devtools/datasette/publish.py

+    # Make sure we have the expected metadata for databases
+    # headed to deployment.
+    if deploy != "metadata":
+        check_tables_have_metadata(metadata_yml, databases)


The way the Datasette metadata is generated, and the checks rely on bespoke logic for constructing paths to the databases we expect to have metadata, and the dependence on the XBRL datapackage.json sometimes and the SQLite files other times and the PUDL Package other times all feels pretty fragile and janky, but that's beyond the scope of this PR.

src/pudl/helpers.py

src/pudl/metadata/classes.py

test/integration/datasette_metadata_test.py

src/pudl/helpers.py

…ive regex for resource sorting

codecov · 2024-01-29T19:35:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (be501da) 92.72% compared to head (b77810f) 92.83%.
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3226      +/-   ##
==========================================
+ Coverage   92.72%   92.83%   +0.12%     
==========================================
  Files         145      145              
  Lines       13111    13156      +45     
==========================================
+ Hits        12156    12213      +57     
+ Misses        955      943      -12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bendnorman commented Jan 9, 2024

View reviewed changes

src/pudl/metadata/classes.py Show resolved Hide resolved

bendnorman commented Jan 9, 2024

View reviewed changes

bendnorman added output Exporting data from PUDL into other platforms or interchange formats. datasette Issues related the accessing PUDL data via Datasette. labels Jan 12, 2024

bendnorman added this to the v2024.01 milestone Jan 12, 2024

Hide _out tables from datasette, add check for missing metadata in da…

fced804

…tasette, remove resource sorting

bendnorman commented Jan 12, 2024

View reviewed changes

src/pudl/metadata/classes.py Show resolved Hide resolved

bendnorman commented Jan 12, 2024

View reviewed changes

src/pudl/metadata/classes.py Outdated Show resolved Hide resolved

bendnorman commented Jan 12, 2024

View reviewed changes

devtools/datasette/publish.py Outdated Show resolved Hide resolved

bendnorman commented Jan 12, 2024

View reviewed changes

devtools/datasette/publish.py Outdated Show resolved Hide resolved

zaneselvans requested changes Jan 12, 2024

View reviewed changes

bendnorman added 2 commits January 12, 2024 14:49

Merge branch 'main' into order-pudl-datasette-tables

f1c3a33

Add Package.get_sorted_resources() back, use sqlalchemy in check_tabl…

00c6a37

…es_have_metadata

zaneselvans changed the title ~~Sort tables on datasette deployment~~ Hide _out tables and restore metadata in Datasette Jan 17, 2024

Merge branch 'main' into order-pudl-datasette-tables

6120e80

bendnorman linked an issue Jan 18, 2024 that may be closed by this pull request

Remove _out tables from datasette and sort tables #3223

Closed

zaneselvans reviewed Jan 18, 2024

View reviewed changes

devtools/datasette/publish.py Outdated Show resolved Hide resolved

bendnorman added 2 commits January 23, 2024 13:37

Merge branch 'main' into order-pudl-datasette-tables

bbeb4bd

Add a test to publish.py and integration test that checks all tables …

e35c35b

…in database have datasette metadata

bendnorman requested a review from zaneselvans January 25, 2024 23:29

bendnorman commented Jan 26, 2024

View reviewed changes

src/pudl/helpers.py Show resolved Hide resolved

bendnorman commented Jan 26, 2024

View reviewed changes

src/pudl/helpers.py Show resolved Hide resolved

zaneselvans assigned bendnorman Jan 26, 2024

zaneselvans requested changes Jan 26, 2024

View reviewed changes

Clean up some datasette metadata doc strings and create more restrict…

e241ab5

…ive regex for resource sorting

bendnorman requested a review from zaneselvans January 27, 2024 03:24

Merge branch 'main' into order-pudl-datasette-tables

44dcc48

zaneselvans approved these changes Jan 27, 2024

View reviewed changes

zaneselvans added the metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. label Jan 27, 2024

Merge branch 'main' into order-pudl-datasette-tables

b77810f

bendnorman marked this pull request as ready for review January 29, 2024 17:59

bendnorman enabled auto-merge (squash) January 29, 2024 18:00

zaneselvans disabled auto-merge January 29, 2024 19:37

zaneselvans merged commit dd80a65 into main Jan 29, 2024
20 checks passed

zaneselvans deleted the order-pudl-datasette-tables branch January 29, 2024 19:38

bendnorman mentioned this pull request Jan 29, 2024

Sort datasette tables so out_ tables are first #3306

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hide `_out` tables and restore metadata in Datasette #3226

Hide `_out` tables and restore metadata in Datasette #3226

bendnorman commented Jan 9, 2024 •

edited

Loading

To-do list

bendnorman Jan 9, 2024

bendnorman Jan 9, 2024

bendnorman Jan 9, 2024

zaneselvans Jan 9, 2024

zaneselvans Jan 9, 2024

zaneselvans Jan 9, 2024

bendnorman Jan 26, 2024 •

edited

Loading

zaneselvans Jan 26, 2024

zaneselvans left a comment

zaneselvans Jan 26, 2024

codecov bot commented Jan 29, 2024

Hide _out tables and restore metadata in Datasette #3226

Hide _out tables and restore metadata in Datasette #3226

Conversation

bendnorman commented Jan 9, 2024 • edited Loading

Overview

Testing

To-do list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bendnorman Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zaneselvans left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 29, 2024

Codecov Report

Hide `_out` tables and restore metadata in Datasette #3226

Hide `_out` tables and restore metadata in Datasette #3226

bendnorman commented Jan 9, 2024 •

edited

Loading

bendnorman Jan 26, 2024 •

edited

Loading