Pytest discovery env incorrect in multiroot workspace since ver 2023.10.0

[lukaspiatkowski]

Type: Bug

Behaviour

Expected vs. Actual

Tests to show up in "Testing" tab, no errors showing up in Output > Python Test Log, pytest running under the project's venv environment.

Steps to reproduce:

1. Open up a workspace with multiroot setup where each root has its own venv
2. Open "Testing" tab, notice the discovery is failing if pytest --collect-only requires some of the packages from venv installed

Some more context:
We have 2 root directories libs and server, both have their own .vscode/settings.json files. The content of the settings file:

{
  "python.defaultInterpreterPath": ".server-venv/bin/python",
  "python.testing.pytestArgs": ["-s", "--disable-warnings"],
  "python.testing.unittestEnabled": false,
  "python.testing.pytestEnabled": true
}

I have bisected the problem to be the Python extension version 2023.10.0, on 2023.8.0 the discovery works as expected - pytest is being invoked with the proper environemnt (inside venv), since version 2023.10.0 it doesn't work (also tested with latest 2023.18.0 and the pre-release version). Interestingly the issue only happens when I open the project through our multiroot workspace, but it works fine if I open only the server folder. Our workspace file is also short:

{
  "folders": [
    {
      "path": "../server"
    },
    {
      "path": "../libs"
    }
  ]
}

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.5
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

Multiroot scenario, following user settings may not apply:

languageServer: "Pylance"

formatting
• provider: "none"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.18.0
VS Code version: Code 1.83.1 (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:57:32.750Z)
OS version: Darwin arm64 22.6.0
Modes:

System Info

Item	Value
CPUs	Apple M1 Pro (10 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	5, 6, 7
Memory (System)	16.00GB (0.07GB free)
Process Argv	monorepo.code-workspace --crash-reporter-id da6a607f-f75f-4468-94ba-7fa2d6b2e8e8
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263:30856252
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyonecf:30548226
2e4cg342:30602488
f6dab269:30613381
showlangstatbar:30737416
962ge761:30841074
03d35959:30757346
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
copilotsettingc:30859502
dsvsc013:30795093
dsvsc014:30804076
diffeditorv2:30821572
dsvsc015:30845448
pythonregdiagcf:30859692
pythoncet00cf:30859750

[eleanorjboyd]

Hello! Investigating this now- thanks for submitting an issue

[eleanorjboyd]

Hi! I. think this is related to #22218. I have put in a fix which is now available on the pre-release version of the vscode-python extension. Is there any way you could try it and see if it fixes your problem? Thanks

[lukaspiatkowski]

I've just tried v2023.19.12911010 and the issue remains there.

[eleanorjboyd]

interesting, could you include your logs from "python" and "python test logs" in the output channel? First if you could set the log level to trace you can do so via the command palette. From there I should be able to see if it is trying it on the wrong cwd or something like that. Thanks

[lukaspiatkowski]

I have sent you a link to the gist to your @microsoft.com e-mail, I didn't want to share it publicly here.

Some things that I've noticed:

• multiroot workspace was a red herring. I said that "it works fine if I open only the server folder", but it doesn't, not sure why it did the first time I've tried.
• From logs it looks like the command that is being run looks like this: ./.server-venv/bin/python -m pytest -p vscode_pytest --collect-only -s --disable-warnings, so I've tried to run it (but without the -p vscode_pytest part) and it failed with the same error for me. So I've run it again with PYTHONPATH like this: PYTHONPATH=".:./src:../libs" ./.server-venv/bin/python -m pytest --collect-only -s --disable-warnings and it worked.
• I also tried to set the "python.envFile": "./.vscode/env", option and provide PYTHONPATH=".:./src:../libs" in the env file, but it didn't change anything, the Python extension didn't pick it up for pytest discovery

[eleanorjboyd]

Hello! Thank you so much for sending your logs and describing these steps you tried! I was able to repro you bug and then narrow down that we were appending some previously set PYTHONPATH values but not the ones that we would have picked up from the envFile. The PR should be the fix for your issue as it fixed my repro and the PYTHONPATH was correct afterwords. I will let you know once this change is merged so you can try it yourself to see if it works!

[eleanorjboyd]

@lukaspiatkowski were you able to verify that this issue is fixed? No worries if not since I can also have a team member do so. Thanks!

[lukaspiatkowski]

It still isn't fixed, I've tried it on v2023.19.12961014. I've sent you the Gist via e-mail again.
One notable thing is that PYTHONPATH is printed once: "PYTHONPATH":"/Users/lukas/.vscode/extensions/ms-python.python-2023.19.12961014/pythonFiles". And it doesn't contain the things I've set via python.envFile.

Additionally when I was testing it out I once caught a debug log saying something like this:

Shell exec . /User/lukas/repo/server/.server-venv/bin/activate python -m /Users/lukas/.vscode/extensions/ms-python.python-2023.19.12961014/pythonFiles/printEnvVariables.py
<and then printent the Env, but the PYTHONPATH was missing in this one>

(Sorry, I haven't saved this log and I am writing it down from memory). What is interesting is that I have export PYTHONPATH=/Users/lukas/server/src:/Users/lukas/libs inside the /User/lukas/repo/server/.server-venv/bin/activate script and yet PYTHONPATH was missing there.

[eleanorjboyd]

hm ok thank you for checking, will discuss with my teammate!

[eleanorjboyd]

Can you try switching "python.envFile": "./.vscode/env" to "python.envFile": "${workspaceFolder}/.vscode/.env"? I am trying that for myself and it is working for me. Here is what I am setting / seeing maybe my setup is different from yours as well.

contents of .env located inside .vscode

PYTHONPATH="./hello"

logging:
All environment variables set for pytest discovery: {"PYTHONPATH":"/Users/eleanorboyd/vscode-python/pythonFiles:./hello".....

[lukaspiatkowski]

That was it! Thanks so much @eleanorjboyd for finding the solution!
One day I hope the Python VSCode extension setup will be more foolproof, but for now I am super glad we got it to work :)

[eleanorjboyd]

Thank you for confirming @lukaspiatkowski. We hope that the setup will be foolproof and are constantly working towards that goal. With that in mind we really appreciate any user feedback, ideas, or bug reports because as you can see with our discussion here we really do listen and want to make the extension work as desired. Thank you for submitting an issue and please submit more in the future as they are extremely important to our extension improvement!