Inherited tests show run buttons in child test class

[DScott-2]

Type: Bug

Behaviour

Expected vs. Actual

Expected:
In test classes which inherit tests from parent classes, the inherited tests only appear in the testing side-pane, and do not appear as run-test buttons in the child test class in the code window
Actual:
The inherited tests display run test buttons in the child class, confusing what button is for what test.

Steps to reproduce:

1. Write this code:
   ./main.py

from abc import ABC, abstractmethod


class MultiplierAbs(ABC):
    @abstractmethod
    def execute(self,val: float) -> float:
        pass


class MultiplierTwo(MultiplierAbs):
    def execute(self,val: float) -> float:
        return val * 2

./tst_multiplication_abs.py

import pytest
from abc import ABC, abstractmethod

from main import MultiplierAbs


class TESTMultiplicationAbs(ABC):
    @abstractmethod
    @pytest.fixture
    def multiplier(self) -> MultiplierAbs:
        pass

    def test_zero(self, multiplier: MultiplierAbs) -> None:
        result = multiplier.execute(0.0)
        assert result == 0.0

./test/test_multiplication_two.py

import pytest
from main import MultiplierAbs, MultiplierTwo
from tst_multiplication_abs import TESTMultiplicationAbs


class TestMultiplicationTwo(TESTMultiplicationAbs):
    """Class which shows abnormal behaviour:
    One test-run button on line 14 for TESTMultiplicationAbs.test_zero
    One on line 15 for TestMultiplicationTwo.test_one"""
    @pytest.fixture
    def multiplier(self) -> MultiplierAbs:
        return MultiplierTwo()
    
    
    def test_one(self, multiplier: MultiplierAbs) -> None:
        assert multiplier.execute(1.0) == 2.0

2. Discover tests
3. Issue appears as this:
   Green run button on line 14 is for TESTMultiplicationAbs.test_zero, and should not appear in this file
   

Granted, in this trivial example this is no big deal. But, in a more extensive codebase with many examples where dozens of tests are inherited from parent classes this becomes a bigger annoyance.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.2
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2024-03-14 08:58:26.644 [info] > ~\AppData\Local\Programs\Python\Python312\python.exe -m pytest -p vscode_pytest --collect-only test
2024-03-14 08:58:26.644 [info] cwd: .
2024-03-14 08:58:28.076 [info] ============================= test session starts =============================

2024-03-14 08:58:28.076 [info] platform win32 -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0

2024-03-14 08:58:28.077 [info] rootdir: c:\Users\dscott2\source\sandbox\inheritence_test_minimal_example

2024-03-14 08:58:28.112 [info] collected 2 items

<Dir inheritence_test_minimal_example>
  <Dir test>

2024-03-14 08:58:28.113 [info]     <Module test_mutiplication_two.py>
      <Class TestMultiplicationTwo>
        <Function test_zero>
        <Function test_one>

2024-03-14 08:58:28.142 [info] Test server connected to a client.
2024-03-14 08:58:28.145 [info] ResultResolver EOT received for discovery.
2024-03-14 08:58:28.146 [info] 

2024-03-14 08:58:28.147 [info] ========================= 2 tests collected in 0.06s ==========================

2024-03-14 08:58:28.219 [info] Disposing data receiver for c:\Users\dscott2\source\sandbox\inheritence_test_minimal_example and deleting UUID; pytest discovery.

User Settings

{
    "python.testing.pytestArgs": [
        "test"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true
}

Extension version: 2024.3.10660538
VS Code version: Code 1.86.1 (31c37ee8f63491495ac49e43b8544550fbae4533, 2024-02-07T09:08:20.941Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item	Value
CPUs	12th Gen Intel(R) Core(TM) i7-1270P (16 x 2496)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	15.67GB (3.91GB free)
Process Argv	--folder-uri file:///c%3A/Users/dscott2/source/sandbox/inheritence_test_minimal_example --crash-reporter-id b278ba45-f201-4142-bd80-a0212bd0019b
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:30899288
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythongtdpath:30769146
welcomedialogc:30910334
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
accentitlementsc:30887149
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:30979542
d34g3935:30971562
fegfb526:30981948
bg6jg535:30979843
dsvsc020:30976470

[eleanorjboyd]

sorry for the delay- i will look into this

[eleanorjboyd]

Where would you expect to see the inherited function? In ./tst_multiplication_abs.py or in the ./test/test_multiplication_two.py file?

[DScott-2]

Where would you expect to see the inherited function? In ./tst_multiplication_abs.py or in the ./test/test_multiplication_two.py file?

I think they should not appear at all in the files themselves. tst_multiplication_abs is abstract and not a test file (it isn't hit by test discovery and could be inherited by 10, 20 or 100 other concrete test classes -- for example, our abstract test class for API actions has >100 different concrete test class implementations, and this issue happens on every single one).

test_multiplication_two should not show it, since the test code being run by the test is in a different file, and it causes these confusing buttons shown above and below

The abstract tests should only show up in the testing side-pane view

In actual real code which uses abstract test classes, these abstract test buttons end up in completely nonsense locations, e.g. over fixtures or imports, which only makes things confusing

Or, maybe instead of hiding altogether, all abstract tests for a class could be run with a single button on the line with the class keyword for that concrete implementation. This is maybe more elegant, but I could see it being confused for a 'run all tests in this class' button for people who don't know better

[eleanorjboyd]

Hi! I revisited this one and when I run python -m pytest -v --collect-only

<Dir 20369-abstract>
  <Module test_multi_2.py>
    <Class TestMultiplicationTwo>
      Class which shows abnormal behaviour:
      One test-run button on line 14 for TESTMultiplicationAbs.test_zero
      One on line 15 for TestMultiplicationTwo.test_one
      <Function test_zero>
      <Function test_one>

You can see from the output the lines returned from pytest, without the extension, point to line 14 and 15. In this way it seems like an upstream pytest issue that it returns the incorrect line for the abstract class. Potentially I could remove the line number manually for the abstract class but would have to somehow programmatically be able to detect the abstract class. If there is a way which a pytest plugin can detect the class being abstract I could look into adding that- so let me know if you have any ideas. Thanks!