Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sec10k assets work with integration tests #4065

Open
Tracked by #4050
zaneselvans opened this issue Feb 14, 2025 · 0 comments
Open
Tracked by #4050

Make sec10k assets work with integration tests #4065

zaneselvans opened this issue Feb 14, 2025 · 0 comments
Labels
sec10k Issues related to SEC 10K filing data. testing Writing tests, creating test data, automating testing, etc.

Comments

@zaneselvans
Copy link
Member

zaneselvans commented Feb 14, 2025

  • The sec10k tables are only loaded when the USE_PUDL_MODELS environment variable is set.
  • Currently it is not set in the pytest workflow in our CI, so the sec10k assets aren't tested by our integration tests when a PR gets put in the merge queue.
  • If USE_PUDL_MODELS is set in our CI, the runner doesn't have permission to pull the sec10k assets, so the tests fail.
  • Because there are utility_id_eia values from a wide range of years in the out_sec10k__ tables, and all of them are loaded any time the asset is loaded, in the etl_fast integration tests, there are FK constraint violations -- since only a few years of EIA-860 data is present in the database.
  • This means that if you have USE_PUDL_MODELS set locally, you cannot get make pytest-integration to pass, and pytest-integration only passes in CI because the sec10k assets aren't being tested.
  • How should we ensure that these assets get tested and are able to pass the integration tests both locally and in CI?
@zaneselvans zaneselvans added testing Writing tests, creating test data, automating testing, etc. sec10k Issues related to SEC 10K filing data. labels Feb 14, 2025
@zaneselvans zaneselvans moved this from New to Backlog in Catalyst Megaproject Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sec10k Issues related to SEC 10K filing data. testing Writing tests, creating test data, automating testing, etc.
Projects
Status: Backlog
Development

No branches or pull requests

1 participant