Linking does not work for games that contain certain characters

[rzumer]

The linking window cannot reproduce some characters in game titles, as in the rest of the application (as expected). What is unexpected is that when trying to link, a second entry is created with a different title, based on the incorrectly-decoded name of the first one.

This example is what happens following an attempt at linking. The first entry is the entry that was created upon attempting to link with the second one. The character that should be decoded as an apostrophe (’) is in fact decoded (or at least encoded on the website) as â.

Similarly, / in an existing game title gets converted to -, and a new entry is created based on that substitution.

[Jamiras]

Which version of the toolkit were you using? The 0.74 (official release) is known to not be escaping URL parameters correctly.

Using the test string
Tést ' / © ½ Test
I get this URL parameter in 0.74:
i=Tést+'+/+©+½+Test
In current master, I get this:
i=T%E9st+%27+%2F+%A9+%BD+Test

Note that %E9, %A9, and %BD are Latin-1 sequences and should be sent to the server as UTF-8 sequences, so even properly escaped, I wouldn't expect them to behave perfectly. Unicode support is currently known to be very buggy and minimally supported.

But that doesn't explain why "/" would be getting converted to "_" - that must be happening on the server. Same with the apostrophe getting converted into a double apostrophe.

As for the unicode characters - the dropdown is populated using single-byte Windows APIs. Similarly, the text box is read using single-byte APIs. I wouldn't expect either to support Unicode characters at this time. I am unable to paste Japanese into the test field (it gets converts to ???? on paste). If you're getting them in there some other way, I'm not surprised the code trying to read it out is confused.

[rzumer]

I'm quite sure I was using a recent master version compiled this week. Reproduction should be straightforward, but it is possible that it's a back-end issue. I'm just surprised that linking relies on any component being able to decode a string correctly.

Japanese characters show up on my end because my system locale is Japanese. You should see a different combination of characters on your end if you attempt to reproduce by inserting the same apostrophe character as in the opening post.

[Jamiras]

The API inputs are console ID, game title, and MD5. The response is the game ID. The association to an existing game ID is purely reliant on the title matching: https://github.com/RetroAchievements/RAWeb/blob/1881a9be7aaa2df503f35282fd8e0c3364998b80/lib/database/game.php#L1097

[Jamiras]

And you can see the replacing of "/" with "-" here: https://github.com/RetroAchievements/RAWeb/blob/1881a9be7aaa2df503f35282fd8e0c3364998b80/lib/database/game.php#L1102

[rzumer]

Then regarding the title decoding/encoding issue resulting in entry duplication, either the characters being sent back to the server when submitting from the list are not reproduced correctly, or it is an error in the back-end.

[televandalist]

For what it's worth, I did go through and replace all instances of "/" with "|" in non-arcade titles.