Clang Language Server Crashing on Startup

[george-tyler-315-incline]

I have a Mac where I have vscode installed (versioning at the bottom). I have the Remote - SSH extension and ssh to my Ubuntu workstation, where the git repo for all of the code belongs and where all the actual code stuff (building, running, etc) gets done.
A few days ago clangd extension in vscode stopped working. The goto does not work nor does any feature really, and I get notifications "Clang Language Server client: couldn't create connection to server." and "Restarting server failed". I get the same errors each time when looking at the Output tab in vscode for clangd. Note in the settings for clangd both remote and workspace I set --log=verbose yet I have not seen any verbose logs. I think the error happens very quickly since when I do watch -n0.1 "ps aux | grep clangd" on my ubuntu machine and restart vscode on my mac I see clangd appear and disappear so fast I can barely read it. I have tried reinstalling the remote clangd extension from Mac, deleting all of ~/.vscode-server on my Ubuntu machine, disabling and reenabling clangd, restarting my ubuntu workstation many times, and more, but nothing has worked at all. Here are the logs I see:

Logs
"[Error - 12:48:20 PM] Client Clang Language Server: connection to server is erroring. Shutting down server.
[Error - 12:48:20 PM] Stopping server failed
Error: Client is not running and can't be stopped. It's current state is: starting
at oc.shutdown (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:8633)
at oc.stop (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:8212)
at oc.stop (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:51156)
at oc.handleConnectionError (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:13298)
at errorHandler (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:11963)
at /home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:21662
at y1.invoke (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:15350)
at Gn.fire (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:16144)
at writeErrorHandler (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:7088)
at y1.invoke (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:15350)
at Gn.fire (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:16144)
at Os.fireError (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:430)
at Socket. (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:1263)
at Socket.emit (node:events:518:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
[Info - 12:48:20 PM] Connection to server got closed. Server will restart.
[Error - 12:48:20 PM] Server initialization failed.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Clang Language Server client: couldn't create connection to server.
Message: Pending response rejected since connection got disposed
Code: -32097
[Info - 12:48:20 PM] Connection to server got closed. Server will restart.
[Error - 12:48:20 PM] Server initialization failed.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Clang Language Server client: couldn't create connection to server.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Client Clang Language Server: connection to server is erroring. Shutting down server.
[Error - 12:48:20 PM] Stopping server failed
Error: Client is not running and can't be stopped. It's current state is: startFailed
at oc.shutdown (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:8633)
at oc.stop (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:8212)
at oc.stop (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:51156)
at oc.handleConnectionError (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:13298)
at errorHandler (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:11963)
at /home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:21662
at y1.invoke (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:15350)
at Gn.fire (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:16144)
at writeErrorHandler (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:7088)
at y1.invoke (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:15350)
at Gn.fire (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:16144)
at Os.fireError (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:430)
at Socket. (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:1263)
at Socket.emit (node:events:518:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
[Info - 12:48:20 PM] Connection to server got closed. Server will restart.
[Error - 12:48:20 PM] Server initialization failed.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Clang Language Server client: couldn't create connection to server.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Restarting server failed
Message: Pending response rejected since connection got disposed
Code: -32097
[Info - 12:48:20 PM] Connection to server got closed. Server will restart.
[Error - 12:48:20 PM] Server initialization failed.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Clang Language Server client: couldn't create connection to server.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:20 PM] Restarting server failed
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:21 PM] Client Clang Language Server: connection to server is erroring. Shutting down server.
[Error - 12:48:21 PM] Stopping server failed
Error: Client is not running and can't be stopped. It's current state is: startFailed
at oc.shutdown (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:8633)
at oc.stop (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:8212)
at oc.stop (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:51156)
at oc.handleConnectionError (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:13298)
at errorHandler (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:11963)
at /home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:39:21662
at y1.invoke (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:15350)
at Gn.fire (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:16144)
at writeErrorHandler (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:7088)
at y1.invoke (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:15350)
at Gn.fire (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:2:16144)
at Os.fireError (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:430)
at Socket. (/home/matcha/.vscode-server/extensions/llvm-vs-code-extensions.vscode-clangd-0.1.29/out/bundle.js:3:1263)
at Socket.emit (node:events:518:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
[Error - 12:48:21 PM] The Clang Language Server server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.
[Error - 12:48:21 PM] Server initialization failed.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:21 PM] Clang Language Server client: couldn't create connection to server.
Message: Pending response rejected since connection got disposed
Code: -32097
[Error - 12:48:21 PM] Restarting server failed
Message: Pending response rejected since connection got disposed
Code: -32097
"

System information
Ubuntu Workstation (where clangd is running):

• vscode-server binary version (~/.vscode-server/code-f1e16e1e6214d7c44d078b1f0607b2388f29d729 --version)
  • code 1.91.1 (commit f1e16e1e6214d7c44d078b1f0607b2388f29d729)
• system clangd binary version (/usr/bin/clangd)
  • Ubuntu clangd version 14.0.0-1ubuntu1.1
    Features: linux+grpc
    Platform: x86_64-pc-linux-gnu
• vscode-server's clangd binary version (~/.vscode-server/data/User/globalStorage/llvm-vs-code-extensions.vscode-clangd/install/18.1.3/clangd_18.1.3/bin/clangd --version)
  • clangd version 18.1.3 (https://github.com/llvm/llvm-project c13b7485b87909fcf739f62cfa382b55407433c0)
    Features: linux+grpc
    Platform: x86_64-unknown-linux-gnu
• Operating System
  • LSB Version: core-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
    Distributor ID: Ubuntu
    Description: Ubuntu 22.04.4 LTS
    Release: 22.04
    Codename: jammy

Mac (where I have the code IDE open and am sshing into my ubuntu workstation):

• vscode version:
  • Version: 1.91.1 (Universal)
    Commit: f1e16e1e6214d7c44d078b1f0607b2388f29d729
    Date: 2024-07-09T22:07:54.982Z (6 days ago)
    Electron: 29.4.0
    ElectronBuildId: 9728852
    Chromium: 122.0.6261.156
    Node.js: 20.9.0
    V8: 12.2.281.27-electron.0
    OS: Darwin arm64 23.5.0
• clangd extension version: v0.1.29
• system clangd version (I assume this is irrelevant but providing just in case):
  • Apple clangd version 15.0.0 (clang-1500.3.9.4)
    Features: mac+xpc
    Platform: x86_64-apple-darwin23.5.0; target=arm64-apple-darwin23.5.0

Thank you very much for the help.

[HighCommander4]

Can you try ssh-ing into the server, going into the workspace directory and running ~/.vscode-server/data/User/globalStorage/llvm-vs-code-extensions.vscode-clangd/install/18.1.3/clangd_18.1.3/bin/clangd --check=some_file.cpp (where some_file.cpp is a path to a source file in your workspace), and see if that exhibits any sort of crash or other error?

[george-tyler-315-incline]

I fixed it and think I kinda found the bug...maybe? I ran that on one of my C files and had no issues. It exited without error. I wanted to find the exact arguments it was invoked with to start it up that way and see if I could reproduce it. strace was not working for various reasons with not being able to attach to the process invoking clangd fast enough. But auditd seemed to show the issue maybe.
I created the rule: sudo auditctl -an exit,always -F arch=b64 -S execve and restarted code on my Mac repeatedly. It was interesting the only arguments clangd was invoked with were the two additional ones I added in the Preferences. Here is a line from the audit log:

type=EXECVE msg=audit(1721234432.528:185): argc=3 a0="/home/matcha/.vscode-server/data/User/globalStorage/llvm-vs-code-extensions.vscode-clangd/install/18.1.3/clangd_18.1.3/bin/clangd" a1="--header_insertion=never" a2="--log=verbose"

I had added these arguments in the VSCode Preferences: Open Workspace Settings Remote [SSH] tab. I figured they would be additional arguments but it seems maybe they override what would have otherwise been passed in.
I removed the arguments and restarted VSCode and the output showed it worked, as did the go to definition feature.
In the audit logs this is what I now see:

type=EXECVE msg=audit(1721235826.269:2447): argc=1 a0="/home/matcha/.vscode-server/data/User/globalStorage/llvm-vs-code-extensions.vscode-clangd/install/18.1.3/clangd_18.1.3/bin/clangd"

Sometimes I see it with --version appended, but for whatever reason it seems maybe adding those arguments broke it.

[HighCommander4]

It was interesting the only arguments clangd was invoked with were the two additional ones I added in the Preferences.

This part is expected. The language client does not pass any command-line arguments of its own to the server; all of its communication is done by sending messages to the server's stdin after launch.

As for why removing the arguments --header_insertion=never and --log=verbose fixes the issue... I don't have a theory about that.