No visual indication if pytest teardown fails

[dmakhno]

Type: Bug

Behaviour

Expected vs. Actual

In Test Results and everywhere test run if has issues during tear down is green.

Consider on single run the might be couple tests. And teardown e.g. module level might group some of them.
The one-size-fits all might be adding node for failed teardowns (fixtures).
Or fail run, if there is any error, according to https://docs.pytest.org/en/stable/reference/exit-codes.html.

Imho, particular test should stay green, if all its teardown passes. Thus if module teardown fail, related module tests with uses this fixture should fail... but not test in the same module if they don't use this fixture.

References:

• New test explorer and test results: no feedback when fixture throws exception #21722 (seems like this, steps are the same, I added only class level with group exception)
• Tests raising an exception are not marked as run #21625

Steps to reproduce:

import pytest

@pytest.fixture
def broken_release():
    yield
    raise Exception('cannot teardown')

@pytest.fixture(scope='class')
def broken_release_cls():
    yield
    raise Exception('cannot teardown cls')
    
def test_resource(broken_release):
    pass

class TestResource:
    def test_class_resource(self, broken_release, broken_release_cls):
        pass

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.1
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

testing
• cwd: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.22.1
VS Code version: Code 1.85.2 (Universal) (8b3775030ed1a69b13e4f4c628c612102e30a681, 2024-01-18T06:40:32.531Z)
OS version: Darwin arm64 23.2.0
Modes:
Connection to 'SSH: switch' could not be established Canceled

System Info

Item	Value
CPUs	Apple M2 Max (12 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	2, 2, 2
Memory (System)	96.00GB (26.13GB free)
Process Argv	--crash-reporter-id 0ddd1806-9bb8-4a88-8fc3-dfe586936e48
Screen Reader	no
VM	0%

Connection to 'SSH: switch' could not be established Canceled

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256859
vscod805:30301674
binariesv615:30325510
vsaa593:30376534
py29gd2263:30899288
vsclangdc:30486549
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30933248
pythongtdpath:30769146
welcomedialog:30910333
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302

[dmakhno]

@tekumara, do you see resolved #21722 ?

[eleanorjboyd]

thanks for the bug- I will investigate

[dmakhno]

@eleanorjboyd, were you able to see and repro the issue?

If it matters for the motivation part:

• I have a heavy teardown by SUT nature, and if teardown fails (environment validation)... there is no sense to start next test, since this test did not clean after itself.
• And... with existing confusion, it is easy to miss result - and starting to dig wrong (following after) test.

[tekumara]

For me exceptions before yield make the test go red but after yield it's green.

[brucepang]

also ran into this issue and would like to have it supported.

[eleanorjboyd]

investigated this and ran into some issues regarding what pytest marks them as upstream. I will keep looking into this

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!