Error while collecting test results (Socket communication)

[zenodallavalle]

Type: Bug

Behaviour

When running tests using the Python extension, the test count displayed (passed/total) shows numbers lower than the actual results, as if the execution results of some tests are lost between the test run and communication with the extension. Specifically, the number of executed tests appears lower, sometimes significantly so (e.g., 151/179, but the total tests run were 1524).

This issue does not occur with version 2024.20.0. The behavior is random and unpredictable. I am unable to reproduce it outside the project I am currently working on.

I am using pytest, but the tests are contained in methods of classes inheriting from django.test.TestCase (which has never been an issue before).

Steps to reproduce:

I am unable to reproduce it outside the project I am currently working on.

Pytest plugins used: cov-6.0.0, env-1.1.5, django-4.9.0, xdist-3.6.1

Diagnostic data

Output for Python in the Output panel

2025-01-15 12:23:12.054 [info] arg: --rootdir already exists in args, not adding.
2025-01-15 12:23:12.054 [info] Attempting to use temp directory for test ids file, file name: test-ids-732037f67fb792c59a96.txt
2025-01-15 12:23:12.056 [info] All environment variables set for pytest execution: {"COMMAND_MODE":"unix2003","CONDA_DEFAULT_ENV":"base","CONDA_EXE":"/Users/<user>/miniconda3/bin/conda","CONDA_PREFIX":"/Users/<user>/miniconda3","CONDA_PROMPT_MODIFIER":"(base) ","CONDA_PYTHON_EXE":"/Users/<user>/miniconda3/bin/python","CONDA_SHLVL":"1","DISPLAY":"/private/tmp/com.apple.launchd.3AfN5T5XOJ/org.xquartz:0","HOME":"/Users/<user>","HOMEBREW_CELLAR":"/opt/homebrew/Cellar","HOMEBREW_PREFIX":"/opt/homebrew","HOMEBREW_REPOSITORY":"/opt/homebrew","INFOPATH":"/opt/homebrew/share/info:/usr/local/share/info:/opt/homebrew/share/info:","LOGNAME":"<user>","MallocNanoZone":"0","OLDPWD":"/","ORIGINAL_XDG_CURRENT_DESKTOP":"undefined","PWD":"/","SHELL":"/bin/zsh","SHLVL":"0","SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.FhApkOHkGo/Listeners","TMPDIR":"/var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/","USER":"<user>","VSCODE_CODE_CACHE_PATH":"/Users/<user>/Library/Application Support/Code/CachedData/fabdb6a30b49f79a7aba0f2ad9df9b399473380f","VSCODE_CRASH_REPORTER_PROCESS_TYPE":"extensionHost","VSCODE_CWD":"/","VSCODE_ESM_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_IPC_HOOK":"/Users/<user>/Library/Application Support/Code/1.96-main.sock","VSCODE_NLS_CONFIG":"{\"userLocale\":\"it\",\"osLocale\":\"it-it\",\"resolvedLanguage\":\"en\",\"defaultMessagesFile\":\"/Applications/Visual Studio Code.app/Contents/Resources/app/out/nls.messages.json\",\"locale\":\"it\",\"availableLanguages\":{}}","VSCODE_PID":"18580","XPC_FLAGS":"0x0","XPC_SERVICE_NAME":"application.com.microsoft.VSCode.122208696.122208702","_":"/Applications/Visual Studio Code.app/Contents/MacOS/Electron","__CFBundleIdentifier":"com.microsoft.VSCode","__CF_USER_TEXT_ENCODING":"0x1F5:0x0:0x4","ELECTRON_RUN_AS_NODE":"1","VSCODE_L10N_BUNDLE_LOCATION":"","PATH":"/Users/<user>/.local/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/<user>/miniconda3/bin:/Users/<user>/miniconda3/condabin:/usr/local/bin:/usr/local/sbin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/Applications/VMware Fusion.app/Contents/Public","PYTHONPATH":"/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/python_files","TEST_RUN_PIPE":"/var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/python-test-results-86431cb8369df911ae16","COVERAGE_ENABLED":"True","RUN_TEST_IDS_PIPE":"/var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/test-ids-732037f67fb792c59a96.txt"}
2025-01-15 12:23:12.056 [info] Running pytest with arguments: /Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=/Users/<user>/<path_to_project> for workspace /Users/<user>/<path_to_project> 

2025-01-15 12:23:12.056 [info] > ./env/bin/python ~/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=.
2025-01-15 12:23:12.056 [info] cwd: .
2025-01-15 12:23:29.206 [error] Test Results named pipe /var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/python-test-results-86431cb8369df911ae16 error: SyntaxError: Bad control character in string literal in JSON at position 982 (line 1 column 983)
    at JSON.parse (<anonymous>)
    at Object.decode (/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/out/client/extension.js:2:2073569)
    at Object.thunk (/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/out/client/extension.js:2:2055845)
    at t.Semaphore.doRunNext (/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/out/client/extension.js:2:2065052)
    at Immediate.<anonymous> (/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/out/client/extension.js:2:2064829)
    at processImmediate (node:internal/timers:483:21)

[…]

2025-01-15 12:23:36.949 [error] Test Results named pipe /var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/python-test-results-86431cb8369df911ae16 error: Error: Header must provide a Content-Length property.
{"{\"jsonrpc\"":"\"2.0\", \"params\": {\"cwd\": \"/Users/<user>/<path_to_project>\", \"status\": \"success\", \"result\": {\"/Users/<user>/<path_to_project>::Test_User_MakeUserStaff_Mutation::test_make_user_staff_fails_if_no_authorization_is_provided\": {\"test\": \"/Users/<user>/<path_to_project>::Test_User_MakeUserStaff_Mutation::test_make_user_staff_fails_if_no_authorization_is_provided\", \"outcome\": \"success\", \"message\": null, \"traceback\": null, \"subtest\": null}}, \"not_found\": null, \"error\": null}}content-length: 605","content-type":"application/json"}
    at b.onData (/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/out/client/extension.js:2:2055260)
    at Socket.<anonymous> (/Users/<user>/.vscode/extensions/ms-python.python-2024.22.2-darwin-arm64/out/client/extension.js:2:2054954)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Socket.Readable.push (node:internal/streams/readable:392:5)
    at Pipe.onStreamRead (node:internal/stream_base_commons:191:23)
2025-01-15 12:23:37.443 [info] Test Result named pipe /var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/python-test-results-86431cb8369df911ae16  cancelled

Extension version: 2024.22.2
VS Code version: Code 1.96.2 (Universal) (fabdb6a30b49f79a7aba0f2ad9df9b399473380f, 2024-12-19T10:22:47.216Z)
OS version: Darwin arm64 24.2.0
Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.10.9
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

User Settings

languageServer: "Pylance"

testing
• debugPort: 4500
• pytestArgs: "<placeholder>"
• pytestEnabled: true

experiments
• optInto: ["pythonTestAdapter"]

Installed Extensions

Extension Name	Extension Id	Version
Black Formatter	ms-python.black-formatter	2024.4.0
CSV to Markdown Table	phoihos.csv-to-md-table	0.3.1
Data Wrangler	ms-toolsai.datawrangler	1.16.0
GitHub Copilot	GitHub.copilot	1.257.0
GitHub Copilot Chat	GitHub.copilot-chat	0.23.2
GitHub Pull Requests	GitHub.vscode-pull-request-github	0.102.0
isort	ms-python.isort	2023.10.1
JavaScript Debugger	ms-vscode.js-debug	1.96.0
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.3
Jupyter	ms-toolsai.jupyter	2024.11.0
Jupyter Cell Tags	ms-toolsai.vscode-jupyter-cell-tags	0.1.9
Jupyter Keymap	ms-toolsai.jupyter-keymap	1.1.2
Jupyter Notebook Renderers	ms-toolsai.jupyter-renderers	1.0.21
Jupyter Slide Show	ms-toolsai.vscode-jupyter-slideshow	0.1.6
Live Server	ritwickdey.LiveServer	5.7.9
Prettier - Code formatter	esbenp.prettier-vscode	11.0.0
Pylance	ms-python.vscode-pylance	2024.12.1
Python	ms-python.python	2024.22.2
Python Debugger	ms-python.debugpy	2024.14.0
SQLite	alexcvzz.vscode-sqlite	0.14.1
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
vscode-pdf	tomoki1207.pdf	1.2.2

System Info

Item	Value
CPUs	Apple M1 Pro (10 x 2400)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	3, 3, 3
Memory (System)	16.00GB (0.89GB free)
Process Argv	--crash-reporter-id 2d7568f2-6fd6-431a-8379-c4e4f40ce524
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
pythonnoceb:30805159
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
dvdeprecation:31068756
dwnewjupyter:31046869
newcmakeconfigv2:31071590
nativerepl1:31139838
pythonrstrctxt:31112756
nativeloc2:31192216
cf971741:31144450
iacca1:31171482
notype1:31157159
5fd0e150:31155592
dwcopilot:31170013
stablechunks:31184530
6074i472:31201624
dwoutputs:31217127

[eleanorjboyd]

Hi! Can you send a minimal repro? It seems like something that is being collected in your tests is not able to be parsed 2025-01-15 12:23:29.206 [error] Test Results named pipe /var/folders/s9/khln37bn4qv7j6frpznhlm1r0000gn/T/python-test-results-86431cb8369df911ae16 error: SyntaxError: Bad control character in string literal in JSON at position 982 (line 1 column 983)

[zenodallavalle]

Unfortunately, I’m unable to provide a minimal reproducible example from my project, nor was I successful in setting up other examples to reproduce the issue. However, I observed that disabling pytest-xdist resolved the problem. Also, using pytest-xdist with -n1 (a single worker) avoids the issue, which seems to support this hypothesis.

Could it be that parallel execution with multiple workers is interfering with how pytest communicates with the VS Code extension, leading to data corruption?

[eleanorjboyd]

very interesting- @karthiknadig any ideas here?

[karthiknadig]

We can try replicating this with pytest-xdist and see if it repros. I als assuming somehow the child processes are communicating when they should not.