Updates to splink FERC to EIA record linkage notebook

[katie-lamb]

Overview

This PR updates the devtools notebook for FERC to EIA record linkage with splink. This notebook isn't actually run in production, but it's helpful for visualizing the model weights and the decisions that go into making the match.

What problem does this address?

The notebook now runs with the latest version of splink and reads in parquet inputs from S3 instead of from dagster storage.

What did you change?

Updated the notebook to have more documentation cells and run with the latest version of splink. Also, fixed an error that was cropping up in the plant parts list creation with datetime types. I think this came up because I'm now reading in parquet inputs from S3 instead of dagster storage, and the report_date column was an object instead of a datetime.

Documentation

Make sure to update relevant aspects of the documentation.

Tasks

Preview Give feedback

1. Update the release notes: reference the PR and related issues.
2. Update relevant Data Source jinja templates (see docs/data_sources/templates).
3. Update relevant table or source description metadata (see src/metadata).
4. Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

Run the FERC to EIA match notebook from top to bottom.

To-do list

Preview Give feedback

1. If updating analyses or data processing functions: make sure to update or write data validation tests (e.g. test_minmax_rows()).
2. Run make pytest-coverage locally to ensure that the merge queue will accept your PR.
3. Review the PR yourself and call out any questions or issues you have.
4. For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
5. For bigger ETL or data changes run the full ETL locally and then run the data validations using make pytest-validate.
6. Alternatively, run the build-deploy-pudl GitHub Action manually.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[katie-lamb]

This is kind of janky but I think when inputs are read in from S3 (instead of from Dagster storage) the report date is an object not a datetime. I was getting errors, so I threw this in there, but it's a little sloppy.

[zaneselvans]

Should this be in the python module? Or a pre-processing step in the notebook? If the dataframe doesn't match the expected schema, then maybe it should fail?

When we read from Dagster storage if it's not a persisted asset it'll be a pickled dataframe, with the datetime type intact. If we're reading from a persisted asset, it'll be coming from Parquet, and IIRC our IOManager will enforce the correct PUDL dtypes when we read the data in. If you're reading a Parquet file from S3 and passing it in outside of the Dagster context, you might try using dtype_backend="pyarrow" to ensure that the datetime column gets parsed not as an object.

[katie-lamb]

Ah okay let me try using the pyarrow backend.

[katie-lamb]

The pyarrow backend introduced other type issues, so I updated to cast to a datetime within the notebook, after reading in the Parquet inputs.

[zschira]

Looks good!