1357 convert unittest to pytest

[ggurjar333]

Overview

Closes #1357.

What problem does this address?

Replacing Unittest with Pytest to standardizing the testing code.

What did you change?

I have changed Unittest based code with Pytest

Documentation

Make sure to update relevant aspects of the documentation.

Tasks

Preview Give feedback

1. Update the release notes: reference the PR and related issues.
2. Update relevant Data Source jinja templates (see docs/data_sources/templates).
3. Update relevant table or source description metadata (see src/metadata).
4. Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Preview Give feedback

1. If updating analyses or data processing functions: make sure to update or write data validation tests (e.g. test_minmax_rows()).
2. Run make pytest-coverage locally to ensure that the merge queue will accept your PR.
3. Review the PR yourself and call out any questions or issues you have.
4. For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
5. For bigger ETL or data changes run the full ETL locally and then run the data validations using make pytest-validate.
6. Alternatively, run the build-deploy-pudl GitHub Action manually.

[zaneselvans]

@ggurjar333 I think you'll want to install the git pre-commit hooks locally so that all the static code analysis and formatting checks can be run before anything gets pushed. Some pointers in our docs here

[ggurjar333]

@ggurjar333 I think you'll want to install the git pre-commit hooks locally so that all the static code analysis and formatting checks can be run before anything gets pushed. Some pointers in our docs here

@zaneselvans
I'm facing these errors.

I have installed pre-commit

I think it is messing around here somewhere.

I tried to delete cache this way as well.

How to fix this one?

[zaneselvans]

It looks like there are a bunch of genuine test failures on your branch, so I think we should try and separate the errors that are due to changes from the errors that are due to environment / setup issues.

• Were you able to successfully run the unit tests before making any changes?
• Check out the main branch and run make pytest-unit from the top-level of the repository.
• What kind of errors come up then?
• How did you create the pudl-dev conda environment?

[ggurjar333]

It looks like there are a bunch of genuine test failures on your branch, so I think we should try and separate the errors that are due to changes from the errors that are due to environment / setup issues.

• Were you able to successfully run the unit tests before making any changes?
• Check out the main branch and run make pytest-unit from the top-level of the repository.
• What kind of errors come up then?
• How did you create the pudl-dev conda environment?

Explanation of the issue for the record here:
I am using WSL on Windows. The issue was I had mounted a volume from the Linux side, the disconnect may be what those two Windows vs Linux file paths difference, even if they were trying to actually pointing at the same files on disk. I was trying to commit from within PyCharm folder from linux (as the folder was on Windows) or some shell thing that PyCharm does it might see the Windows path, while pre-commit sees the Linux path.

Solution:
Zane helped me to understand and I moved the folder from Windows path to Linux path at the root directory. It Works!

[jdangerx]

Overall this looks much nicer when we're consistently using pytest - thank you! I do think that there are some spots where we're using the legacy setup / teardown behavior - while we're here converting everything to the new style, I think we should finish the job and convert those into pytest fixtures.

[jdangerx]

Mostly looks good, I think we should revert back to the xunit-style setup for the FERC Form 1 forest test. Everything else is non-blocking. Thanks for working on this!

[jdangerx]

Pulling out just the nodes as a fixture doesn't help much, as you see you still have to unpack and add edges etc. The actual setup that has to get done for each test is "make a forest, with nodes, edges, and tags" - and each test has a slight variation on what those are. To me that means we actually want a helper function in this module that makes creating these forests a little less verbose, rather than a few fixtures.

But that involves a fair amount of restructuring of the tests, etc., and this PR has already gone on for a long time. I would just return to the xunit-style setup using the setup_method flow you use below.

[jdangerx]

@ggurjar333 I think this is the main thing I'd like to see before we merge the PR:

1. undoing the structural change where you pull the nodes into a fixture
2. converting the TestPrunedNode.setUp function into pytest using the classic xunit-style setup guide

You had mentioned wanting to spend more time learning about nx.DiGraphs - once you feel more comfortable with those you can open up another PR that just refactors ferc1_test.py to have a clearer/more flexible way of setting up graphs for testing.

[jdangerx]

I went ahead and made the small change I requested in #4014 . Thanks for doing all this work @ggurjar333 !