Wayland: Implement native sub-windows

[Riteo]

psss... the text of this PR got completely rewritten now that the PR is ready. If you want to see the original text, use GH's history feature

Everything below will be extremely boring. So, here's a pretty picture of the editor running on sway:

I mean, it's still probably somewhat boring but that's the nature of what I do :P

---

The backend is now mature enough to not explode with multiple windows but the DisplayServer API still cannot meet some guarantees required by the various Wayland protocols we use. To meet those guarantees this patch adds three new elements to the DisplayServer API, with relative handling logic for Window and Popup nodes:

• WINDOW_EVENT_FORCE_CLOSE, which tells a window to forcefully close itself and ensure a proper cleanup of its references, as Wayland enforces this behavior;
• WINDOW_FLAG_POPUP_WM_HINT, which explicitly declares a window as a "popup", as Wayland enforces this distinction and heuristics are not reliable enough;
• FEATURE_SELF_FITTING_WINDOWS, which signals that the compositor can fit windows to the screen automatically and that nodes should not do that themselves.

Given the size of this feature, this patch also includes various WaylandThread reworks and fixes including:

• Improvements to frame wait logic, with fixes to various stalls and a configurable (through a #define) timeout amount; Edit: stalls fixed in Wayland: Fix engine stalls while waiting frames #102674.
• A proper implementation of window_can_draw;
• Complete overhaul of pointer and tablet handling. Now everything is always accumulated and handled only on each respective frame event. This makes their logic simpler and more robust.
• Better handling of pointer leaving and pointer enter/exit event sending;
• Keyboard focus tracking;
• More solid window references using IDs instead of raw pointers as windows can be deleted at any time;
• More aggressive messaging to window nodes to enforce rects imposed by the compositor.

---

The new APIs that I added will be useful for other DisplayServer implementations too, perhaps with the exception of FEATURE_SELF_FITTING_WINDOWS.

As you can see, this PR includes also a lot of core improvements and fixes to stuff that became more evident while testing multiple windows. I'm definitely going to backport/separate some of those.

It does not even include the FIFO additions (#101454) or fixes for some leftover issues that @arlo-phoenix found in the original PR (mainly I think only missing keyboard composition) so once everything is done the Wayland experience should become even more solid.

Known issues

• There's some error spam while starting the editor. It's an engine bug triggered by the fact that now we have a proper implementation of window_can_draw: At startup ERROR: editor/progress_dialog.cpp:169 - Do not use progress dialog (task) while flushing the message queue or using call_deferred()! #100900

[scgm0]

When a subwindow appears, it will cause the main window to exit maximized mode.

2025-01-19.23-10-21.mp4

[Riteo]

Hi @scgm0, thank you for testing! I think I know why. I'll push a fix soon.

[bruvzg]

WINDOW_EVENT_FORCE_CLOSE, which is a surefire way of cleaning up a window. It basically makes the window call hide on itself.

This will be useful for Windows/macOS/X11 display server popup auto close logic, currently these are using WINDOW_EVENT_CLOSE_REQUEST, but expect window to be closed unconditionally, if it's not, it can stay opened in a partially broken state.

[Riteo]

This will be useful for Windows/macOS/X11 display server popup auto close logic, currently these are using WINDOW_EVENT_CLOSE_REQUEST, but expect window to be closed unconditionally, if it's not, it can stay opened in a partially broken state.

Hey that's exactly what I was trying to avoid! I'm really glad this will be useful for other backends too! :D

[Riteo]

@scgm0 I pushed quite a few changes, including a fix for the issue you mentioned.

[scgm0]

@scgm0 I pushed quite a few changes, including a fix for the issue you mentioned.

New issue, the borderless setting of the main window is not working

[Riteo]

New issue, the borderless setting of the main window is not working

Does this happen only on the main window? Because unfortunately that's a known issue with the decoration library we're using, see #93472. I had it disabled until this last push to make debugging simpler.

I think I might need to find a workaround since the upstream bug report went nowhere unfortunately.

[scgm0]

New issue, the borderless setting of the main window is not working

Does this happen only on the main window? Because unfortunately that's a known issue with the decoration library we're using, see #93472. I had it disabled until this last push to make debugging simpler.

I think I might need to find a workaround since the upstream bug report went nowhere unfortunately.

But before you fix the maximized window, the main window can be borderless, which is a bit strange.

[Riteo]

But before you fix the maximized window, the main window can be borderless, which is a bit strange.

It stopped working because I also pushed other things along the fix, including "Re-add libdecor support". I usually work in bursts and push everything at once.

[scgm0]

But before you fix the maximized window, the main window can be borderless, which is a bit strange.

It stopped working because I also pushed other things along the fix, including "Re-add libdecor support". I usually work in bursts and push everything at once.

I see, so would libdecor support be of any benefit?

[Riteo]

I see, so would libdecor support be of any benefit?

Ye it is, as some compositors do not have their own decorations, like GNOME. We could implement our own decorations instead of using a library but it's not exactly simple. I'd like to look into that eventually though as it allows fancier features.

[Riteo]

Heads up to whoever might be testing this PR: expect instability and crashes with popups in the latest batch of commits, I'm doing some experiments

[Riteo]

update: things should have come back to a decent state, but popups are a bit iffy when they get pushed back by the compositor, as godot tries to do some fitting itself. I'll need to find a good way to stop popups from trying to be smart.

[Riteo]

Folks, I think I have a plan!

It's quite simple, we add a feature flag for "self-fitting windows" (not a great name I know) and if that's available we avoid "adjusting" popup windows. Now things should work great!

Edit (I don't want to spam too much) here's what it allows:

Not bad for not having screen-space window handling :P

[Riteo]

Heads up: I fixed an issue with child toplevels being shrunk to their minimum size. I thought it was my project metadata being funky but no, it was an issue in the window adjusting code I special cased for the new feature flag 😅

[Riteo]

Welp, I think I implemented everything that there is to implement. Some missing things were already missing "upstream" and we'll need to integrate other stuff along the road anyways so I'm marking this as ready to review.

I did my usual funny of making a comically large commit description and updated the PR text. I mean, it's a huge commit so... Yea. Gotta make those git blames worth it :P

Please test kthxbye

[YeldhamDev]

Opening a window spams this while it's open:

ERROR: Invalid window ID received from Wayland thread.

[Riteo]

Hi @YeldhamDev, thanks for testing! I can't reproduce this unfortunately. Could you send me a complete debug log?

WAYLAND_DEBUG=1 insert-path-to-godot-here --verbose > wayland.log 2>&1

[YeldhamDev]

@Riteo wayland.log

[Riteo]

Thank you @YeldhamDev. I think I found the issue! Those errors coincided with IME done events, which sent malformed messages to the main thread. Please tell me if the latest force push fixed your issue :D

[YeldhamDev]

Still happens: wayland.log

[Riteo]

All right, I think I fixed this one for good. Please tell me if the latest commit fixes your issue.

[YeldhamDev]

That did it! 🎉