CLI flags/args added via plugin not behaving as expected

[dbarbuzzi]

I believe there may be an issue with extending command-line args via pytest_load_initial_conftests in a plugin, or perhaps I am lacking some understanding of how different things work and I should be adding different flags with a different approach.

I currently have a plugin our team uses which dynamically generates the flag for JUnit report files. It works by defining the pytest_load_initial_conftests hook and using the args[:] = [*args, *new_args] pattern (where new_args is a list[str] containing the dynamically generated flags. In this trivial example, the list looks something like ['--junit-xml=test-results/junit-report-1739919926.824418.xml'].

I have also added some features to this plugin to add some additional CLI flags and, with everything enabled, new_args looks like this:

[
  '--junit-xml=test-results/junit-report-1739919926.824418.xml',
  '-o',
  'junit_logging=all',
  '-o',
  'junit_log_passing_tests=True',
  '--cov=MODULE_NAME',
  '--cov-append',
  '--cov-report=html:coverage-html',
  '--cov-report=json:coverage.json'
]

However, none of the other flags result in their anticipated effects:

• the JUnit report does not have all the expected output captured
• nothing is being captured for passing tests
• no coverage is generated despite having coverage and pytest-cov installed).

I have confirmed the contents via debugging the plugin when used from an editable install in a separate project. In this separate project, I also have a test to debug which uses the request fixture and prints out request.config.option (minimal example test below). In this Namespace, I can see what appears to be the various options configured as expected, though they do not "work":

...
cov_source=['MODULE_NAME'],
cov_report={'html': 'coverage-html', 'json': 'coverage.json'},
cov_append=True,
...
override_ini=['junit_logging=all', 'junit_log_passing_tests=True'],
...

For reference, the plugin is open-source here: https://github.com/neuralmagic/pytest-nm-releng

A simple set of test files I am using locally to test it (the unit tests all work to the best of my capability with regards to testing the plugin) – will not include any of the pytest-cov flags by design:

# test file: test_example.py
# requirements: pytest, pytest-nm-releng
import pytest

def test_example(request: pytest.FixtureRequest):
    print(request.config.option)

# running with flags added via plugin - report naming works, rest doesn't
NMRE_JUNIT_BASE="test-results" NMRE_JUNIT_PREFIX="junit-report-" NMRE_JUNIT_FULL="1" pytest test_example.py

# running with flags added via PYTEST_ADDOPTS - work as expected
PYTEST_ADDOPTS='--junit-xml test-results/report-pytest_addopts.xml -o junit_logging=all -o junit_log_passing_tests=True' pytest test_example.py

# running with flags passed directly - works as expected
pytest test_example.py --junit-xml test-results/report-direct.xml -o junit_logging=all -o junit_log_passing_tests=True