Virtual/screen keyboard support

[Beuc]

Hi,

AFAICS there is no on-screen keyboard support in the Emscripten SDL2 port, which is a problem on mobile platforms.

There doesn't seem to be a browser API to control the virtual keyboard (or maybe I missed it?), so we could:

• re-implement a basic keyboard (directly in SDL, or in HTML overlay - if HTML it needs to be included in the full-screen mode DOM element) or
• trick the browser into displaying the virtual keyboard by focusing on a <input type="text"> element somehow
• other ideas?

[Beuc]

(clarified a mistake about fullscreen)

[Beuc]

Several "javascript virtual keyboards" exist, notably https://github.com/hodgef/simple-keyboard which has no dependencies (though none of them describe why they were developed in the first place).

I'm not sure what stack should provide the virtual keyboard. I'm wondering what's the SDL team's stance on shipping additional javascript+css code in SDL2 to handle an IME?

[Beuc]

Alternatively, I made a test with an off-screen HTML text input, to leverage the browser's built-in support. This properly show/hides the virtual keyboard on focus/unfocus, and JS events allowed tracking what was typed. Keycodes are caught and forwarded to the application as well, for consistency with SDL2/Android.

https://www.beuc.net/tmp/ti1.html

Limitations:

• Browsers seem to only pass invalid 229 keycodes in keydown/keyup events, except for Enter, or they just don't generate the keyboard event. This notably prevents the SDL2 app from supporting text deletion (backspace key). This looks like a general issue with virtual keyboards in Android browsers https://bugs.chromium.org/p/chromium/issues/detail?id=118639

• The handling of insertText (~SDL_TEXTINPUT) vs. insertCompositionText (~SDL_TEXTEDITING) events looks tricky. In particular, the browser doesn't generate a final insertText (unlike SDL2), may generate events for past words if the user uses Backspace. If I delete the text as the user types, Firefox issues duplicate insertCompositionText events. In short, browsers don't seem to map well with SDL2.

• There's no HTML5 API to detect whether there's a physical keyboard. We'd probably need to have this always enabled, or follow an heuristic on browser agent. It works fine on desktop though.

With these limitations it sounds like this isn't the right approach, but I'd be happy to discuss this further.
ti1.zip

[Beuc]

Also here's a sample SDL2 app for comparing the behavior between desktop/web/android:
https://github.com/renpy/renpyweb/blob/master/tests/testtextinput.c

[Beuc]

Referencing emscripten-discuss thread for info:
https://groups.google.com/forum/#!topic/emscripten-discuss/FN-8Tdxi1D8

Also given there was 0 reaction here after 1 year, it is unlikely that any solution will be implemented at the SDL2 layer :/

[feliwir]

@Beuc I'd find a solution for that problem useful aswell

[Beuc]

@feliwir In the end we had to re-implement a keyboard in-game:
renpy/renpy#2336
Browsers' behavior was too inconsistent, or maybe I just missed a better solution.

[ghost]

I too could use this functionality. The off-screen text field thing is the type of workaround that would be consistent with Emscripten's approach. Myself, I'll try superimposing a visible text field first. A special soft keyboard like you're apparently doing for Ren'Py would be a bit heavy and non-native, but for a VN it's probably not a problem.

[ghost]

HTML form superimposed over the canvas activates the soft keyboard but does not accept input (input being received by SDL with preventDefault). This means the form submit event is not recognized if you press the virtual enter key. Instead, there seems to be some weird keyboard issue where Enter isn't received at all or differs from the US QWERTY key code.

Any ideas?

EDIT: The Return scancode is the same on Android's soft keyboard (the "Go" button) as on a physical keyboard (the "Enter" key). Even though Emscripten input events are blocking, user code can detect Return and its ilk via soft keyboard, as long as other issues are resolved like lag.

[uyjulian]

Related: https://twitter.com/danlucraft/status/1225763866765905922

[ghost]

they create an invisible contenteditable div that they update to always contain the text of the line you are working on, and that has the focus and is where text input is sent.

Could work. I use a text field with 0 opacity, but don't tell Google because of "security" or whatever.