Allow to configure a list of encoding-confidences to use when guessing

[sunbohong]

Since there were so many encodings issues,I plan to upgrade the code guessing workflow to V2.

• First, jschardet will be upgraded and multiple results will be returned. For example, [{encoding: 'UTF-8', confidence: 0.95}, {encoding: 'GBK', confidence: 0.95}].

• Then, we will support the configuration of multiple sets of coding confidence in the settings.files.encodingInitConfidences。
  [{encoding:'utf-8',confidence:0.01},{encoding:'GBK',confidence:0.03}]。

• Finally,add the two together to get the final result, because GBK has the highest confidence, so the file is recognized as GBK
[{encoding:'utf-8',confidence:0.06},{encoding:'GBK',confidence:0.08}]。

[vscodebot]

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Issues auto guessing file with GBK encoding (#24195)

[bpasero]

First, jschardet will be upgraded and multiple results will be returned

@sunbohong are you planning to contribute to JSChardet?

[sunbohong]

First, jschardet will be upgraded and multiple results will be returned

@sunbohong are you planning to contribute to JSChardet?

I found that https://github.com/runk/node-chardet can meet my requirements.

[bpasero]

I am not sure VSCode would move off jschardet to another module, seems rather risky. I would suggest you try to become a contributor on jschardet to improve it if possible.

[sunbohong]

I tried to add some logs to jschardet, but it failed the following test. aadsm/jschardet#48

node-chardet works well with this test.
The best is utf-8

jschardet cann't pass this test.

windows-1252 confidence 0.95

UTF-8 confidence = 0.505

[sunbohong]

@bpasero

Since jschardet can't provide the expected results.
We can provide node-chardet as an optional option. If there are enough people to choose the new switch in the future, we can remove the jschardet in the future

[byyxx128]

I like the mode provided by node-chardet. Just a friendly reminder that please always use GB18030 to replace GB2312 or GBK regardless of the confidence value.

Reasons:

• New characters in a file encoded in GB18030 will lose if the file is edited in lower encoding.
• New characters cannot be displayed normally with lower encoding (nothing will lose if it is just opened, but if it is saved with lower encoding, information will be destroyed)
• A file encoded in GB2312 or GBK can be decoded as GB18030. Even though new characters would lose if it is decoded by lower encoding again, nothing is wrong at least.

All in all, the safest way is always using GB18030 instead of GB2312 or GBK.

Here is a file encoded by GB18030 for your test.
test_GB18030.txt

[vscodebot]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[vscodebot]

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.

Happy Coding

[vscodebot]

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.

Happy Coding

[vscodebot]

🙂 This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[bpasero]

@sunbohong please merge this with #36951, I feel the 2 suggestions are very similar.

[sunbohong]

I am not sure VSCode would move off jschardet to another module, seems rather risky. I would suggest you try to become a contributor on jschardet to improve it if possible.

When we use the new version of jschardet , we are also taking risks.
So, why not try the node-chardet?

[bpasero]

Some requirements to go with another library such as node-chardet imho are:

• it runs in web too (that is a requirement that is relatively new)
• tests of jschardet work in node-chardet (this would give some confidence)

[bpasero]

/duplicate #36951