Running pytest fails due to path issues: No such file or directory: 'generated_data/ed25519_metadata'

[jas4711]

Hi! I'm working on packaging of Python-TUF for Debian and ran into this self-check failure which I can reproduce in a basic git checkout of d805a81 which is master per 2024-12-06.

Shouldn't running 'pytest' work automatically? Is there some path confusion here, or why does this fail?

root@a9cac658593b:~/python-tuf# pytest
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.12.8, pytest-8.3.3, pluggy-1.5.0
rootdir: /root/python-tuf
configfile: pyproject.toml
plugins: typeguard-4.4.1
collected 184 items / 1 error                                                                                                                                           

================================================================================ ERRORS =================================================================================
__________________________________________________________ ERROR collecting tests/test_metadata_generation.py ___________________________________________________________
tests/test_metadata_generation.py:10: in <module>
    from tests.generated_data.generate_md import generate_all_files
tests/generated_data/generate_md.py:61: in <module>
    os.mkdir(OUT_DIR)
E   FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'
=========================================================================== warnings summary ============================================================================
tests/test_repository.py:42
  /root/python-tuf/tests/test_repository.py:42: PytestCollectionWarning: cannot collect test class 'TestingRepository' because it has a __init__ constructor (from: tests/test_repository.py)
    class TestingRepository(Repository):

tests/test_updater_delegation_graphs.py:31
  /root/python-tuf/tests/test_updater_delegation_graphs.py:31: PytestCollectionWarning: cannot collect test class 'TestDelegation' because it has a __init__ constructor (from: tests/test_updater_delegation_graphs.py)
    @dataclass

tests/test_updater_delegation_graphs.py:42
  /root/python-tuf/tests/test_updater_delegation_graphs.py:42: PytestCollectionWarning: cannot collect test class 'TestTarget' because it has a __init__ constructor (from: tests/test_updater_delegation_graphs.py)
    @dataclass

tests/test_updater_fetch_target.py:22
  /root/python-tuf/tests/test_updater_fetch_target.py:22: PytestCollectionWarning: cannot collect test class 'TestTarget' because it has a __init__ constructor (from: tests/test_updater_fetch_target.py)
    @dataclass

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================== short test summary info ========================================================================
ERROR tests/test_metadata_generation.py - FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================== 4 warnings, 1 error in 0.23s ======================================================================
root@a9cac658593b:~/python-tuf# 

Thanks,
Simon

[jku]

Hi, thanks for report,

We don't actually use pytest, just unittest (pytest might still run the exact same tests but there's no advantage to using it in this test suite). The four warnings you see are minor incompatibilities with pytest and our suite. In practice the test automation actually executes aggregate_tests.py both on CI and in local testing but plain python -m unittest should have the same results.

The error you hit seems to relate to our test automation always running so that working directory is tests/: apparently not doing so triggers an error. This decision almost certainly has some historical reasons but does seem like a bug to me.

A workaround for you might be to run (cd tests/; python ./aggregate_tests.py) as your test command until this issue is fixed.

The specific issues we should fix:

• make sure test_metadata_generation.py does not expect CWD to be tests
• consider running the test suite from source root dir by default (maybe switching to plain python -m unittest in CI and in tox config)

[jas4711]

Thanks for ideas! Debian build system has some magic to "guess" the self-test system to use, and it may need help. Hard-coding whatever commands you believe are the ones to use for self-testing is fine. But I can't get any sequence of commands to work.

Just using python -m unittest from the top-level directory fails like this:

make[1]: Entering directory '/build/python-tuf-5.1.0'
python3 -m unittest
.........................................../usr/lib/python3.12/unittest/case.py:690: DeprecationWarning: It is deprecated to return a value that is not None from a test case (<bound method TestFetcher.test_download_file_length_mismatch of <tests.test_fetcher_ng.TestFetcher testMethod=test_download_file_length_mismatch>>)
  return self.run(*args, **kwds)
............E..............................................................................................................................E..
======================================================================
ERROR: tests.test_metadata_generation (unittest.loader._FailedTest.tests.test_metadata_generation)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_metadata_generation
Traceback (most recent call last):
  File "/usr/lib/python3.12/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "/build/python-tuf-5.1.0/tests/test_metadata_generation.py", line 10, in <module>
    from tests.generated_data.generate_md import generate_all_files
  File "/build/python-tuf-5.1.0/tests/generated_data/generate_md.py", line 60, in <module>
    os.mkdir(OUT_DIR)
FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'


======================================================================
ERROR: test_cleanup (tests.test_utils.TestServerProcess.test_cleanup)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-tuf-5.1.0/tests/test_utils.py", line 58, in test_cleanup
    server_process_handler = utils.TestServerProcess(
                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/python-tuf-5.1.0/tests/utils.py", line 205, in __init__
    raise e
  File "/build/python-tuf-5.1.0/tests/utils.py", line 200, in __init__
    self._start_server(timeout, extra_cmd_args, popen_cwd)
  File "/build/python-tuf-5.1.0/tests/utils.py", line 219, in _start_server
    self._wait_for_port(timeout)
  File "/build/python-tuf-5.1.0/tests/utils.py", line 306, in _wait_for_port
    raise TestServerProcessError(
tests.utils.TestServerProcessError: 'simple_server.py did not print port message as first stdout line as expected!'

----------------------------------------------------------------------
Ran 185 tests in 1.978s

FAILED (errors=2)
make[1]: *** [debian/rules:10: override_dh_auto_test] Error 1

It is the same tests/ CWD problem, isn't it?

Using (cd tests/; python ./aggregate_tests.py) immediately dies:

cd tests && python3 ./aggregate_tests.py
EEEEEEEEEEEEEEEE
======================================================================
ERROR: test_api (unittest.loader._FailedTest.test_api)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_api
Traceback (most recent call last):
  File "/usr/lib/python3.12/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "/build/python-tuf-5.1.0/tests/test_api.py", line 26, in <module>
    from tests import utils
ModuleNotFoundError: No module named 'tests'

It seems to be the namespace-issue that is mentioned here:

https://github.com/theupdateframework/python-tuf/blob/develop/pyproject.toml#L73

[tool.hatch.build.targets.wheel]
# The testing phase changes the current working directory to `tests` but the test scripts import
# from `tests` so the root directory must be added to Python's path for editable installations
dev-mode-dirs = ["."]

Finally I tried to simply use tests/aggregate_tests.py which fails like this:

make[1]: Entering directory '/build/python-tuf-5.1.0'
python3 tests/aggregate_tests.py
.......................................................E..............................................................................................................................E..
======================================================================
ERROR: tests.test_metadata_generation (unittest.loader._FailedTest.tests.test_metadata_generation)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_metadata_generation
Traceback (most recent call last):
  File "/usr/lib/python3.12/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "/build/python-tuf-5.1.0/tests/test_metadata_generation.py", line 10, in <module>
    from tests.generated_data.generate_md import generate_all_files
  File "/build/python-tuf-5.1.0/tests/generated_data/generate_md.py", line 60, in <module>
    os.mkdir(OUT_DIR)
FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'


======================================================================
ERROR: test_cleanup (tests.test_utils.TestServerProcess.test_cleanup)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-tuf-5.1.0/tests/test_utils.py", line 58, in test_cleanup
    server_process_handler = utils.TestServerProcess(
                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/python-tuf-5.1.0/tests/utils.py", line 205, in __init__
    raise e
  File "/build/python-tuf-5.1.0/tests/utils.py", line 200, in __init__
    self._start_server(timeout, extra_cmd_args, popen_cwd)
  File "/build/python-tuf-5.1.0/tests/utils.py", line 219, in _start_server
    self._wait_for_port(timeout)
  File "/build/python-tuf-5.1.0/tests/utils.py", line 306, in _wait_for_port
    raise TestServerProcessError(
tests.utils.TestServerProcessError: 'simple_server.py did not print port message as first stdout line as expected!'

----------------------------------------------------------------------
Ran 185 tests in 2.150s

FAILED (errors=2)

So I'm back to the CWD issue.

If you have any more ideas, let me know!

/Simon

[jku]

First and last issue there are indeed the CWD issue.

Second failure implies the test suite expects the tests to be installed -- this is is a unusual but I guess happens when a python project is installed with pip install <source dir> (which is what our CI does and I think our dev test setup as well).

I guess in your case you have "installed" the debian package that you're testing and this of course does not and should not contain tests... I think this is a test suite bug, or at least it should work in your case as well. I'll take a look this week.

[jku]

Note to myself:

Second failure implies the test suite expects the tests to be installed

Actually this makes some sense as a side effect of the original CWD issue: if CWD was root source dir then I think tests would actually be an importable module without installing (if I remember the import rules correctly). So fixing one issue likely makes the other disappear.

[jku]

I think the issues you found will be fixed in #2749 (apart from pytest support: it might work but I'm not planning to test for that).

• internally we'll likely keep using tox -e test to run tests (as it manages the virtual environment as well)
• but if you want to manage dependencies externally then I believe python -m unittest in the top level directory is a good choice

Feel free to test the branch or wait until merge -- I'm happy to do further fixes if they're needed either way.