Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

missing Windows on Arm64 natives in CDT #969

Closed
chirontt opened this issue Dec 13, 2024 · 14 comments · Fixed by #983
Closed

missing Windows on Arm64 natives in CDT #969

chirontt opened this issue Dec 13, 2024 · 14 comments · Fixed by #983

Comments

@chirontt
Copy link
Contributor

Description

Eclipse IDE for C/C++ Developers, version 2024-12, is now available for Windows on Arm64 (WoA):

image

But running it on a WoA box shows some deficiencies:

  • Opening a Terminal view and choosing the Local Terminal seems OK (i.e. the CMD window is showing properly in a view), but any command entered to that view would show nothing, due to exceptions thrown in the Eclipse console:
!ENTRY org.eclipse.cdt.core.native 4 0 2024-12-12 19:47:13.627
!MESSAGE Failed to load the PTY library. This may indicate a configuration problem, but can be ignored if no further problems are observed.
!STACK 0
java.lang.UnsatisfiedLinkError: no winpty in java.library.path: C:\apps\eclipse\cpp-2024-12;C:\WINDOWS\Sun\Java\bin;...
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2458)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
        at java.base/java.lang.System.loadLibrary(System.java:2063)
        at org.eclipse.cdt.utils.pty.PTY.<clinit>(PTY.java:307)
        at org.eclipse.tm.terminal.connector.local.launcher.LocalLauncherDelegate.createTerminalConnector(LocalLauncherDelegate.java:317)
        at org.eclipse.tm.terminal.view.ui.services.TerminalService.createTerminalConnector(TerminalService.java:246)
        at org.eclipse.tm.terminal.view.ui.services.TerminalService.executeServiceOperation(TerminalService.java:168)
        at org.eclipse.tm.terminal.view.ui.services.TerminalService.openConsole(TerminalService.java:258)
        at org.eclipse.tm.terminal.connector.local.launcher.LocalLauncherDelegate.execute(LocalLauncherDelegate.java:224)
        at org.eclipse.tm.terminal.view.ui.internal.handler.LaunchTerminalCommandHandler.execute(LaunchTerminalCommandHandler.java:85)
  • Building a simple project seems (surprisingly) working; here's its console output:

image

but the Eclipse console shows some exceptions:

!ENTRY org.eclipse.cdt.core.native 4 0 2024-12-12 20:14:58.627
!MESSAGE no spawner in java.library.path: C:\apps\eclipse\cpp-2024-12;C:\WINDOWS\Sun\Java\bin;...

From the above exceptions, I guess those winpty and spawner binaries (for WoA) are not available in the Eclipse IDE package, causing e.g. the Terminal not working.

Version Information

  • OS and OS Version/extra details: Windows 11 on Arm64
  • Eclipse Version: 2024-12 (4.34)
  • CDT Version from Help -> About -> Installation Details -> Features tab: 11.6.1

Additional context
Browsing the CDT source code, I notice that there is no WoA native fragment equivalent to the existing one for x84_64 (org.eclipse.cdt.core.win32.x86_64/os/win32/x86_64) so I guess those pty.dll, spawner.dll, starter.exe, winpty.dll, winpty-agent.exe binaries need be recompiled for WoA, for a new org.eclipse.cdt.core.win32.aarch64 fragment.

I will submit a PR for this new fragment for WoA. But the sticky point is: how will the CDT team re-compile those binaries for WoA? (I can't submit the WoA binaries built by me, for obvious security reason.) Does the team have access to a WoA box to build them?

@jonahgraham
Copy link
Member

Thanks @chirontt - that would be great.

We use Docker and cross-compile Windows for x86-64 at the moment. See for example here where we install gcc-mingw-w64-x86-64 and the building documentation for natives

How it works is that as part of a PR you provide the DLLs as built with the docker images and explained in the above documentation. Our build verifies, by rebuilding, that the DLLs in your commit match the sources with this script as part of the this GHA workflow

There are some challenges we need to overcome to update this workflow for WoA, mostly that we are based on Ubuntu 18.04 LTS at the moment, which I assume doesn't have WoA cross compilers. The timing is good as CDT 12.0 is the next release, so updating this build infrastructure to a new version of Ubuntu is well timed.

If you can contribute the sources and identify if such a cross-compiler exists in the Ubuntu ecosystem, I can handle the infra changes. If you want to even make the Dockerfile changes, it would be welcome too.

Please let me know how else I can support you on this task.

@jonahgraham
Copy link
Member

Does the team have access to a WoA box to build them?

To be explicit, at the moment we do not within the project. The Eclipse Foundation does, and if there is no cross-compile option in the same way we are doing it now, we can look at some manual builds on that infrastructure (and automate it if it becomes useful)

@chirontt
Copy link
Contributor Author

We use Docker and cross-compile Windows for x86-64

This wouldn't work for WoA, as the current/latest GCC 14 compiler itself doesn't support output of WoA binaries; it will be supported in GCC 15 though in next year's release.

In the meantime we have to use a different compiler in WoA, and in my PR #971 I describe in details the MSYS2/CLANGARM64 shell which provides a unixy environment, plus a LLVM-based MinGW compiler package, to successfully re-compile the CDT binaries in WoA.

Alternatively, to do cross-compiling from a Windows x64 host, this llvm-mingw compiler may work. And it also provides a Ubuntu tarball, so it might be able to cross-compile to WoA from Linux! I need more research on this compiler.

@jonahgraham
Copy link
Member

Does GitHub have any runners that would be suitable? e.g. can we have a GitHub action that builds these files?

@chirontt
Copy link
Contributor Author

If you meant the Windows Arm64 runner then it's only available for "Team and Enterprise Cloud plans", not for the free tiers.

Here's the relevant issue about it: actions/runner-images#10820

@jonahgraham
Copy link
Member

Based on (from #971 (comment))

and there won't be any need for a WoA box to do tons of manual stuff mentioned in the description above.

it sounds like it isn't relevant anyways. If it does become relevant, we have access to some non-free resources within the project.

jonahgraham added a commit to jonahgraham/cdt that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

The initialization code has

This is needed so that WoA can run without needing to port the
unneeded WinPTY library

Part of eclipse-cdt#969
jonahgraham added a commit to jonahgraham/cdt that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

The initialization code has

This is needed so that WoA can run without needing to port the
unneeded WinPTY library

Part of eclipse-cdt#969
jonahgraham pushed a commit to jonahgraham/cdt that referenced this issue Dec 18, 2024
- add a new fragment 'org.eclipse.cdt.core.win32.aarch64' containing
re-compiled Arm64 natives suitable for the Windows on Arm64 platform.

- add the 'serial.dll' binary re-compiled for Windows on Arm64.

- add the win32/win32/aarch64 environment triplet to various Maven
build scripts to support building CDT software for that environment.

The updated dlls/exes are not included in this commit and will be provided
in subsequent commits with the corresponding update to build infrastructure

Fixes eclipse-cdt#969
jonahgraham added a commit to jonahgraham/cdt that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of eclipse-cdt#969
jonahgraham pushed a commit that referenced this issue Dec 18, 2024
- add a new fragment 'org.eclipse.cdt.core.win32.aarch64' containing
re-compiled Arm64 natives suitable for the Windows on Arm64 platform.

- add the 'serial.dll' binary re-compiled for Windows on Arm64.

- add the win32/win32/aarch64 environment triplet to various Maven
build scripts to support building CDT software for that environment.

The updated dlls/exes are not included in this commit and will be provided
in subsequent commits with the corresponding update to build infrastructure

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
The docker images all have new, simpler names and use Ubuntu 20.04
(instead of 18.04) as their base.

A few new tools have been added, specifically what is needed for:

- Linux on RISC-V - see #980
- Winodows on ARM - see #969

Fixes #976
jonahgraham pushed a commit that referenced this issue Dec 18, 2024
- add a new fragment 'org.eclipse.cdt.core.win32.aarch64' containing
re-compiled Arm64 natives suitable for the Windows on Arm64 platform.

- add the 'serial.dll' binary re-compiled for Windows on Arm64.

- add the win32/win32/aarch64 environment triplet to various Maven
build scripts to support building CDT software for that environment.

The updated dlls/exes are not included in this commit and will be provided
in subsequent commits with the corresponding update to build infrastructure

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
The docker images all have new, simpler names and use Ubuntu 20.04
(instead of 18.04) as their base.

A few new tools have been added, specifically what is needed for:

- Linux on RISC-V - see #980
- Winodows on ARM - see #969

Fixes #976
jonahgraham pushed a commit that referenced this issue Dec 18, 2024
- add a new fragment 'org.eclipse.cdt.core.win32.aarch64' containing
re-compiled Arm64 natives suitable for the Windows on Arm64 platform.

- add the 'serial.dll' binary re-compiled for Windows on Arm64.

- add the win32/win32/aarch64 environment triplet to various Maven
build scripts to support building CDT software for that environment.

The updated dlls/exes are not included in this commit and will be provided
in subsequent commits with the corresponding update to build infrastructure

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
The docker images all have new, simpler names and use Ubuntu 20.04
(instead of 18.04) as their base.

A few new tools have been added, specifically what is needed for:

- Linux on RISC-V - see #980
- Winodows on ARM - see #969

Fixes #976
jonahgraham pushed a commit that referenced this issue Dec 18, 2024
- add a new fragment 'org.eclipse.cdt.core.win32.aarch64' containing
re-compiled Arm64 natives suitable for the Windows on Arm64 platform.

- add the 'serial.dll' binary re-compiled for Windows on Arm64.

- add the win32/win32/aarch64 environment triplet to various Maven
build scripts to support building CDT software for that environment.

The updated dlls/exes are not included in this commit and will be provided
in subsequent commits with the corresponding update to build infrastructure

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
The docker images all have new, simpler names and use Ubuntu 20.04
(instead of 18.04) as their base.

A few new tools have been added, specifically what is needed for:

- Linux on RISC-V - see #980
- Winodows on ARM - see #969

Fixes #976
jonahgraham added a commit that referenced this issue Dec 18, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of #969
jonahgraham added a commit that referenced this issue Dec 18, 2024
The docker images all have new, simpler names and use Ubuntu 20.04
(instead of 18.04) as their base.

A few new tools have been added, specifically what is needed for:

- Linux on RISC-V - see #980
- Winodows on ARM - see #969

Fixes #976
@chirontt
Copy link
Contributor Author

chirontt commented Dec 19, 2024

@jonahgraham I'm checking out the CDT's support_windows_on_arm64 branch, building it with mvn clean verify -DskipTests=true and the build is fine. Will test the build with x64 platform first, then with the WoA platform.

But I've seen some disturbing behaviours of winpty in the x64 platform, for the current Eclipse IDE. Here are the steps to see them:

  • take the current Eclipse IDE for C/C++ Developers (version 2024-12) which contains the CDT version 11.6.1
  • modify its eclipse.ini file by adding the following property:
-Dorg.eclipse.cdt.core.conpty_enabled=true
  • start up the Eclipse IDE, show the Terminal view and open the Local Terminal in it. Type some command in the local terminal, say dir, to see that everything is working fine with it. Then close the local terminal and its Terminal view
  • shut down the IDE, then change its eclipse.ini file with:
-Dorg.eclipse.cdt.core.conpty_enabled=false

i.e. force the disabling of ConPTY

  • now start up the Eclipse IDE again, show the Terminal view and open the Local Terminal again, what will you see on your x64 machine?
  • on my x64 machine, I get an external Windows Terminal opened at the same time with the local terminal view in Eclipse, i.e.

image

and the two terminals share the same I/O for all commands executed in either of them!

image

Can you reproduce the problem with your Windows x64 box? I hope it's not unique to my Windows 11 (x64) box!

@jonahgraham
Copy link
Member

Can you reproduce the problem with your Windows x64 box? I hope it's not unique to my Windows 11 (x64) box!

Yes. The version on winpty is ancient in CDT and when I last tested it it worked fine on a Windows 7 box. The issue is that Windows Console team have been very active in the last few years and it seems the way we (winpty really) is launching the hidden console is not working anymore.

If I was a little braver I would rip out winpty implementation entirely. -Dorg.eclipse.cdt.core.conpty_enabled=false really only existed to ensure that when I first wrote and enabled conpty if I had regressed any user flows we could back out of it. I don't intend to support -Dorg.eclipse.cdt.core.conpty_enabled=false.

@jonahgraham
Copy link
Member

jonahgraham commented Dec 19, 2024

and the two terminals share the same I/O for all commands executed in either of them!

BTW and FWIW that is exactly how winpty is supposed to work, just that the window should be hidden. What winpty does is shuffle data back and forth from the hidden window to the integrated window.

@jonahgraham
Copy link
Member

@jonahgraham I'm checking out the CDT's support_windows_on_arm64 branch, building it with mvn clean verify -DskipTests=true and the build is fine. Will test the build with x64 platform first, then with the WoA platform.

That is fine that you are doing it locally - I am working on ensuring a fully signed/etc version works too. The build is running now after a few false starts and hopefully this one will run to conclusion and there aren't any other small updates that I need to do.

I will let you know when it is ready, but you can have a look on https://download.eclipse.org/tools/cdt/builds/cdt/ - there shoudl be a new support_windows_on_arm64 directory when the build succeeds which will have the p2 site you can install from into your 2024-12 SDK.

@jonahgraham
Copy link
Member

The version on winpty is ancient in CDT and when I last tested it it worked fine on a Windows 7 box.

I spun it up on a Windows 11 box and if you use conhost instead of Windows Terminal the window is correctly hidden. WinPTY is only realistically useful if you also use conhost, so that makes sense.

This is the Windows setting you have to change to reproduce this:

image

(And with that I hope that is the last time I have to deal with WinPTY until we decide to delete it from the sources!)

@jonahgraham
Copy link
Member

The build was successful. @chirontt can you please try out this p2 site and let me know if it seems ok https://download.eclipse.org/tools/cdt/builds/cdt/support_windows_on_arm64/

@chirontt
Copy link
Contributor Author

Yes, that p2 site works fine. I use a fresh Eclipse SDK 4.34 package for WoA, and update it with that p2 site to install the needed CDT features/plugins. The Terminal view testing is good without any issues. With the winpty/pty libraries (for WoA) removed from the CDT distribution, the ConPTY class works fine in WoA.

jonahgraham pushed a commit that referenced this issue Dec 28, 2024
- add a new fragment 'org.eclipse.cdt.core.win32.aarch64' containing
re-compiled Arm64 natives suitable for the Windows on Arm64 platform.

- add the 'serial.dll' binary re-compiled for Windows on Arm64.

- add the win32/win32/aarch64 environment triplet to various Maven
build scripts to support building CDT software for that environment.

The updated dlls/exes are not included in this commit and will be provided
in subsequent commits with the corresponding update to build infrastructure

Part of #969
jonahgraham added a commit that referenced this issue Dec 28, 2024
Refactors the initialization code of PTY to allow code to avoid
loading WinPTY if it isn't needed. The previous code would
load WinPTY even if ConPTY was in use.

To achieve this the code was refactored to have cleaner state
with a new field ptyType that replaces isConPTY, isWinPTY and
hasPTY.

This is needed so that WoA can run without needing to port the
unneeded WinPTY library. In particular, the winpty.dll and
pty.dll are no longer used on WoA.

Part of #969
jonahgraham added a commit that referenced this issue Dec 28, 2024
The docker images all have new, simpler names and use Ubuntu 20.04
(instead of 18.04) as their base.

A few new tools have been added, specifically what is needed for:

- Linux on RISC-V - see #980
- Winodows on ARM - see #969

Fixes #976
@jonahgraham
Copy link
Member

Thanks @chirontt for the contribution and testing the result. I have now merged the fix into main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants