String encodings continue to be one of the hardest things #103
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we get a string for an item name from the game, it might be in one of several encodings: - ascii - Latin-1 1252, aka ISO-8859 most likely the 9 variation - UCS2, which is almost like UTF-16 LE except it's fixed width We must take this varied input and turn it into valid UTF-8 strings for Rust usage & logging, and for display in the HUD via Imgui. We must also not disturb any already-valid strings by mangling them while attempting to decode them into a modern encoding. So, we do the following. First, use `chardet::detect()` to guess at an encoding. If it's utf-8 already or one of the ISO-8859s, we can decode immediately and return. If it is not one of those, we make the _assumption_ that it is UCS2. The string might be almost-correctly detected as UTF-16le if it uses the second byte for any characters, but if the characters it is holding are plain old ascii, it'll be reported as ASCII and that is wrong. So we guess, and then decode the UCS2. Broke the string manglers out into their own file and wrote some basic tests. Changed the helper wrappers to use cstr null-terminated version for item name handling, since those come from the game with null termination. If I discover the null termination is also double-width I guess I'll have a bug to fix. Put mcm-meta-helper to work on the task for which it was invented. Time to test on Windows now.
ceejbot
added a commit
that referenced
this pull request
Jan 8, 2024
When we get a string for an item name from the game, it might be in one of several encodings: - ascii - Latin-1 1252, aka ISO-8859 most likely the 9 variation - UCS2, which is almost like UTF-16 LE except it's fixed width We must take this varied input and turn it into valid UTF-8 strings for Rust usage & logging, and for display in the HUD via Imgui. We must also not disturb any already-valid strings by mangling them while attempting to decode them into a modern encoding. So, we do the following. First, use `chardet::detect()` to guess at an encoding. If it's utf-8 already or one of the ISO-8859s, we can decode immediately and return. If it is not one of those, we make the _assumption_ that it is UCS2. The string might be almost-correctly detected as UTF-16le if it uses the second byte for any characters, but if the characters it is holding are plain old ascii, it'll be reported as ASCII and that is wrong. So we guess, and then decode the UCS2. Broke the string manglers out into their own file and wrote some basic tests. Changed the helper wrappers to use cstr null-terminated version for item name handling, since those come from the game with null termination. If I discover the null termination is also double-width I guess I'll have a bug to fix. Put mcm-meta-helper to work on the task for which it was invented. Added better logging for when fmtstr fails on huditems.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
String encodings continue to be one of the hardest things
When we get a string for an item name from the game, it might be
in one of several encodings:
We must take this varied input and turn it into valid UTF-8 strings
for Rust usage & logging, and for display in the HUD via Imgui. We
must also not disturb any already-valid strings by mangling them while
attempting to decode them into a modern encoding.
So, we do the following. First, use
chardet::detect()to guessat an encoding. If it's utf-8 already or one of the ISO-8859s,
we can decode immediately and return. If it is not one of those,
we make the assumption that it is UCS2. The string might be
almost-correctly detected as UTF-16le if it uses the second byte
for any characters, but if the characters it is holding are plain
old ascii, it'll be reported as ASCII and that is wrong. So we guess,
and then decode the UCS2.
Broke the string manglers out into their own file and wrote some
basic tests. Changed the helper wrappers to use cstr null-terminated
version for item name handling, since those come from the game with
null termination. If I discover the null termination is also
double-width I guess I'll have a bug to fix.
Put mcm-meta-helper to work on the task for which it was invented.
Added better logging for when fmtstr fails on huditems.