Tests discovery failing on symlink folder

[ZacCranko]

Type: Bug

1. Open a remote-ssh connection to a server, on which run
2. poetry new github_test_discovery_issue
3. code-insiders github_test_discovery_issue
4. Create a new file touch tests/test_foo.py then add to it

def test_foo():
    assert True

5. Configure vscode to use pytest with the following settings.json (the one generated by vscode by default)

{
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true
}

• After following the above steps, the test explorer shows only the folder 'tests', but does not discover test_foo:

• Pytest run from the terminal in this folder succeeds in discovering and running test_foo

➜  github_test_discovery_issue pytest tests/
====================== test session starts ======================
platform linux -- Python 3.8.10, pytest-7.2.1, pluggy-1.0.0
rootdir: /mnt/disks/persist/code/github_test_discovery_issue
plugins: typeguard-2.13.3, anyio-3.6.2
collected 1 item                                                

tests/test_foo.py .                                       [100%]

======================= 1 passed in 0.01s =======================

• Please help. Thanks!

Extension version: 2023.3.10481011
VS Code version: Code - Insiders 1.76.0-insider (Universal) (96a795cc0b1cd62f9ff66c347637f917c6dde3da, 2023-02-17T22:20:52.580Z)
OS version: Darwin arm64 22.3.0
Modes:
Sandboxed: Yes
Connection to 'ssh-remote+zac-dev' could not be established
Remote OS version: Linux x64 5.13.0-1023-gcp

System Info

Item	Value
CPUs	Apple M2 (8 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	5, 4, 3
Memory (System)	16.00GB (0.74GB free)
Process Argv	--crash-reporter-id d2f555fd-c740-461a-926d-b30c1004a468
Screen Reader	no
VM	0%

Connection to 'ssh-remote+zac-dev' could not be established

Item	Value
Remote	SSH: zac-dev
OS	Linux x64 5.13.0-1023-gcp
CPUs	AMD EPYC 7B12 (240 x 2250)
Memory (System)	400.47GB (394.47GB free)
VM	0%

A/B Experiments

vsliv695:30137379
vsins829:30139715
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30422396
pythontb:30258533
pythonptprofiler:30281269
vshan820:30294714
pythondataviewer:30285072
vscod805cf:30301675
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30404738
cppdebug:30492333
vscaac:30438845
vsclangdf:30492506
c4g48928:30535728
dsvsc012:30540252
pynewext54:30618038
pylantcb52:30590116
pyindex848:30611229
nodejswelcome1:30587009
pyind779:30611226
pythonsymbol12:30651887
2i9eh265:30646982
showlangstatbar:30659908
pythonb192:30661256

[eleanorjboyd]

Could you share your configuration for your poetry environment? Thanks

[ZacCranko]

Sure, here's the output from poetry config --list.

cache-dir = "/home/zcranko/.cache/pypoetry"
experimental.new-installer = true
experimental.system-git-client = false
installer.max-workers = null
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/zcranko/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

[eleanorjboyd]

Hi, can you also send logs from the python output? Sorry I forgot that in my first message. You can get these by going to view -> output -> and select python from the dropdown on the right. Thanks!

[ZacCranko]

No problem, it's as follows:

Experiment 'pythonSurveyNotificationcf' is active
Experiment 'pythonPromptNewToolsExt' is active
LSP Notebooks experiment is enabled
LSP Notebooks interactive window support is enabled
> conda info --json
> poetry env list --full-path
cwd: .
Python interpreter path: /bin/python
> /bin/python -I ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/get_output_via_markers.py ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/interpreterInfo.py
> /usr/bin/python -I ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/get_output_via_markers.py ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/interpreterInfo.py
> /mnt/disks/persist/code/mdj-toolkit/.venv/bin/python -I ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/get_output_via_markers.py ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/interpreterInfo.py
Starting Pylance language server.
> /bin/python ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .

[corey-dawson]

I have the same issue, where the the test tab won't discover my pytest inside the "tests" folder. I have a feeling it is because get_current_traceback changed to DebugTraceback.
using conda with python 3.9.16
Here is a section of my output;

ImportError: cannot import name 'get_current_traceback' from 'werkzeug.debug.tbtools' (C:\Users\myuser\Miniconda3\envs\analytics\lib\site-packages\werkzeug\debug\tbtools.py)
ERROR conda.cli.main_run:execute(49): `conda run python c:\Users\myuser\.vscode\extensions\ms-python.python-2022.20.2\pythonFiles\get_output_via_markers.py c:\Users\myuser\.vscode\extensions\ms-python.python-2022.20.2\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\Users\myuser\Documents\Git Repos\battery_placement_opt_py -s --cache-clear tests` failed. (See above for error)

[eleanorjboyd]

@corey-dawson can you file a separate issue, since you are using conda and not poetry. Thanks!

[ZacCranko]

@eleanorjboyd Please let me know if there's anything further you need

[eleanorjboyd]

Hi @ZacCranko, can you please run this from your command line and see if it returns the expected output discover pytest -- --rootdir . -s --cache-clear tests? If that doesn't work can you try disabling color on the output? Sometimes this interferes with our ability to parse the pytest output.

[ZacCranko]

➜  github_test_discovery_issue /bin/python ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10531009/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
[{"rootid": ".", "root": "/mnt/disks/persist/code/github_test_discovery_issue", "parents": [{"id": "./tests", "kind": "folder", "name": "tests", "parentid": ".", "relpath": "./tests"}, {"id": "./tests/test_foo.py", "kind": "file", "name": "test_foo.py", "parentid": "./tests", "relpath": "./tests/test_foo.py"}], "tests": [{"id": "./tests/test_foo.py::test_foo", "name": "test_foo", "source": "./tests/test_foo.py:1", "markers": [], "parentid": "./tests/test_foo.py"}]}]

Looks like test_foo is being discovered with that command. Viz. "tests": [{"id": "./tests/test_foo.py::test_foo"...

Can you confirm I've run it correctly (I added the run_adapter prefix). Also I'm not sure what you mean about disabling colour on the output, if you explain to me what you want me to try I'm happy to give it a go.

[corey-dawson]

@corey-dawson can you file a separate issue, since you are using conda and not poetry. Thanks!

will do

[eleanorjboyd]

@ZacCranko, no need to append the run_adapter prefix can you try it like this python -m pytest --rootdir . -s --cache-clear. Secondly I am not sure the exact setting but feel free to ask the poetry team here if you can't find it: https://github.com/python-poetry/poetry.

[ZacCranko]

I needed the run_adapter prefix for the discover command in discover pytest -- --rootdir . -s --cache-clear tests to do something.

The output of poetry run python -m pytest --rootdir . -s --cache-clear is as follows:

➜  github_test_discovery_issue poetry run python -m pytest --rootdir . -s --cache-clear

================================= test session starts =================================
platform linux -- Python 3.8.10, pytest-7.2.1, pluggy-1.0.0
rootdir: /mnt/disks/persist/code/github_test_discovery_issue
collected 1 item                                                                      

tests/test_foo.py .

================================== 1 passed in 0.01s ==================================

[ZacCranko]

I am not an expert on the python vscode extension but it appears in the python output, vscode is internally running the following command to detect the tests:

> ./.venv/bin/python ~/.vscode-server-insiders/extensions/ms-python.python-2023.3.10581011/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .

where the prefix ./.venv/bin/python vscode pointing to the poetry python 3.8.10 virtual environment in the project directory. If I run that command in an ordinary terminal I get the following output, which looks to me like JSON that VScode Python should be populating the Test Explorer UI with, but, for whatever reason, this is failing to happen.

[
    {
        "rootid": ".",
        "root": "/mnt/disks/persist/code/github_test_discovery_issue",
        "parents": [
            {
                "id": "./tests",
                "kind": "folder",
                "name": "tests",
                "parentid": ".",
                "relpath": "./tests"
            },
            {
                "id": "./tests/test_foo.py",
                "kind": "file",
                "name": "test_foo.py",
                "parentid": "./tests",
                "relpath": "./tests/test_foo.py"
            }
        ],
        "tests": [
            {
                "id": "./tests/test_foo.py::test_foo",
                "name": "test_foo",
                "source": "./tests/test_foo.py:1",
                "markers": [],
                "parentid": "./tests/test_foo.py"
            }
        ]
    }
]

[ZacCranko]

Hi there, any update on this @eleanorjboyd @karthiknadig ?

[eleanorjboyd]

Hi! We are still unable to see this on our side- could you share your current project or a minimal repro?

[ZacCranko]

Hi, no problem. Please see attached.
github_test_discovery_issue.tar.gz

[Jerry-Ma]

Hi,
I am having the exact same issue as described in this thread.

Following the steps described in this thread, I can confirm that when running the run_adapter.py command manually in a terminal, I do see that the tests are found successfully and returned in the json blob. However, the UI does not get populated.

I am not using poetry but a nominal pyproject.toml setup. The project lives inside WSL and I am connecting to it via the WSL remote functionality.

The vscode version that I am using:

Version: 1.77.0-insider (user setup)
Commit: bbe8e872dc7c8899eb80bc4b650a6f090f4bc952
Date: 2023-03-14T05:23:49.668Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: 10.2.154.26-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: Yes

[ZacCranko]

@Jerry-Ma 👊

[ZacCranko]

@Jerry-Ma are you able to reproduce the bug with the example project I provided above?

[eleanorjboyd]

Hi @ZacCranko, thank you for the minimal repro! I gave it a try outside of poetry and it worked for me. Does it work if you are not in your poetry environment? I will now give it a try with poetry but wanted to check in the meantime.

Thanks

[eleanorjboyd]

Hello again! Here are the steps I took and I could not repro the issue.

1. cloned your repo
2. deleted venv folder (to make my own vnenv)
3. create venv with python 3.10
4. ran poetry install
5. deleted all pytest_cache (from root and from test folder)
6. reloaded vscode window
7. opened up the test explorer to see output shown in image below

Do any of these steps sound incorrect to your process or did I forget something about your setup? Thanks

[Azureblade3808]

I was having a similar issue when the workspace folder was a symbolic link, and I worked around it by opening the folder directly using its original path. Hope this info helps.

[davidsben]

I was having a similar issue when the workspace folder was a symbolic link, and I worked around it by opening the folder directly using its original path. Hope this info helps.

Just to add another data point. I was hitting an identical issue as mentioned here (no tests discovered but running the discover command from the plugin manually appeared to be working).

In my case also, my workspace had been opened through a path that included a symbolic link, and if I instead opened the underlying path directly it worked correctly.

@eleanorjboyd
I was able to reproduce with the provided code by adding a step between 5 and 6 of your repro steps above:

ln -s github_test_discovery_issue symlink_test 

Then you can easily switch between the two folders to reproduce:

code ./github_test_discovery_issue # Test Discovery Works
code ./symlink_test # No Tests Discovered

(Note: In my case the symbolic link was not the workspace folder itself, but a folder a few levels up in the hierarchy that contained the workspace folder)

[eleanorjboyd]

@davidsben is this the folder structure you are suggesting? Not sure where the symlink_test folder should be located in relation to the github_test_discovery_issue folder. Thanks

[Jerry-Ma]

I can confirm that in my case (failed discovering the tests), I do have my workspace placed in an external drive whose mnt point is sym-linked to my home directory:

Real path to my repo (the workspace):  /mnt/my_drive/foo/my_repo
Symlinked mnt point: /home/me/foo  -> /mnt/my_drive/foo 

And I am opening my repo in VSCode by navigating to /home/me/foo/my_repo

[Jerry-Ma]

And I can further confirm that when I open my repo via the full real path by navigating to /mnt/my_drive/foo/my_repo, I can discover the tests normally.

Thanks a lot Azureblade3808 and davidsben !

[Jerry-Ma]

@davidsben is this the folder structure you are suggesting? Not sure where the symlink_test folder should be located in relation to the github_test_discovery_issue folder. Thanks

davidsben could chime in on this.

@eleanorjboyd I believe what you want to do is to make a symlink of the entire directory, or put the entire directory inside a directory whose path contains a node that is a symlink.

In your screenshot above, you only made symlink of the python module.

An example sequence of commands to setup the test structure

cd /path/to/some/path/
tar -zxvf github_test_discovery_issue.tag.gz
ln -s github_test_discovery_issue ./symlink_test

[ZacCranko]

Oh my god this was totally it. Thank you @Jerry-Ma @Azureblade3808 @davidsben

[davidsben]

For completeness I created a new version of the test case that includes a symlinked directory in the right place. updated_github_test_discovery_issue.tar.gz

You still need to create a venv and install pytest into the github_test_discovery_issue folder first

code ./github_test_discovery_issue # Test Discovery Works
code ./symlink_test # No Tests Discovered

[eleanorjboyd]

Thank you everyone for your responses and help in figuring out this issue! Closing as it seems the issue is resolved but please let me know if more conversation is needed. Thanks!

[ZacCranko]

Hi @eleanorjboyd, in what sense is the issue resolved. Does vscode Python now support properly resolving symlinks?

[davidsben]

Just to echo @ZacCranko 's sentiment. Has a fix been implemented in the extension? The discussion above was, at best, a workaround to a bug in the extension (and / or vscode). Outside of vscode there is no problem using pytest in the symlink case, it only fails to work when used within the vscode setup.

[karthiknadig]

This should not be closed, yet. But the fix for this is that we created a pytest plugin to detect pytest tests via this PR here #20631 . Which should be able to detect these, but we still need to verify that this case is handled by it.

[eleanorjboyd]

Consolidating all issues on testing and symlinks to issue #9347. Let me know if you do not think this other issue is representative of the bug you are facing. Thanks!