Support utf-8 encoding guessing

[sunbohong]

utf-8(['ascii', 'utf-8', 'utf-16', 'utf-32']) encoding guessing is disabled by this line. Ignore encodings that cannot guess correctly
But this link didn't mention it.
And in my test,the follow text will be guessing as utf-8 correctly(result='utf-8').

jschardet result:

In the other hand,with the follow setting, it will be detect as GBK

[vscodebot]

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Wrongly guess encoding as Windows 1252 or ISO-8859-2 (#27419)
• Git: "Stage selected ranges" command changes encoding to UTF-8 (#36219)

[sunbohong]

Not as simple as you see...
See these issues:
https://github.com/microsoft/vscode/labels/file-guess-encoding
aadsm/jschardet#48
aadsm/jschardet#49

These issues will be solve by #84503

[sunbohong]

Not as simple as you see...
See these issues:
https://github.com/microsoft/vscode/labels/file-guess-encoding
aadsm/jschardet#48
aadsm/jschardet#49

In particular, aadsm/jschardet#48 will be treat as utf-8

[bpasero]

I like this change because it makes jschardet more deterministic by giving it full control over the detection. I still think more work is needed to increase the confidence of the detection, but that can continue in other issues.

[bpasero]

@sunbohong I noticed a bad regression though that made me push ddfca30 to ignore the guessed encoding "ascii" for a simple reason:

• user opens a text file with just ascii characters
• we guess the encoding to be "ascii"
• user types special characters (like german umlaut)
• user saves and closes the file
• user reopens the file

=> the file is still guessed as "ascii" because it was not saved with a proper encoding

[bpasero]

Verification:

• configure "files.autoGuessEncoding": true
• configure "files.encoding": "windows1252" (simply to a non-UTF8 encoding)
• save a file with special characters contents (e.g. 私は和食が好きです。) as UTF-8
• close all files
• open it in VSCode

=> the encoding should be UTF-8 from the status bar.

[bpasero]

Given scary issues such as #85821 I am putting UTF-16 and 32 back to the list of ignored encodings for guessing. Still, UTF-8 can be guessed.

[sunbohong]

@bpasero From this issue #85821, we really need to replace jschardet with a new library