-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Excuse me, how to support other language search, such as Chinese search, thank you #201
Comments
Hello @sinianzhiren , Unfortunately I do not know enough about Chinese language to guide you here, but other users have successfully used MiniSearch for Chinese (look for example at this comment or at this issue). Did you encounter a specific problem supporting Chinese or other languages? If so, you can describe it, and I would be happy to help if I can. |
You should do Chinese word segmentation with library like nodejieba before indexing documents. |
If you don't care about supporting Firefox, |
Firefox Nightly support Use const segmenter =
Intl.Segmenter && new Intl.Segmenter("zh", { granularity: "word" });
const miniSearch = new MiniSearch({
fields: ["text"],
processTerm: (term) => {
if (!segmenter) return term;
const tokens = [];
for (const seg of segmenter.segment(term)) {
tokens.push(seg.segment);
}
return tokens;
},
});
const documents = [
{ id: 1, text: "为字段添加 required 属性,并在提交时进行表单验证" },
{
id: 2,
text: "By default, the same processing is applied to search queries. In order to apply a different processing to search queries, supply a processTerm search option:",
},
];
miniSearch.addAll(documents);
console.log("===");
console.log(miniSearch.search("添加")); This is online example: https://duoyun-ui.gemjs.org/zh/ |
I also encountered the problem of searching Chinese, for example, when searching for "预置", due to the problem of word segmentation, the content cannot be searched due to the word segmentation of "预" and "置", my project uses vitepress, indirectly uses minisearch, and finally I configured it like this to support search: ...
export default defineConfig({
...
themeConfig: {
search: {
options: {
miniSearch: {
options: {
tokenize: (term) => {
if (typeof term === 'string') term = term.toLowerCase();
// @ts-ignore
const segmenter = Intl.Segmenter && new Intl.Segmenter("zh", { granularity: "word" });
if (!segmenter) return [term];
const tokens = [];
for (const seg of segmenter.segment(term)) {
// @ts-ignore
tokens.push(seg.segment);
}
return tokens;
},
},
searchOptions: {
combineWith: 'AND', // important for search chinese
processTerm: (term) => {
if (typeof term === 'string') term = term.toLowerCase();
// @ts-ignore
const segmenter = Intl.Segmenter && new Intl.Segmenter("zh", { granularity: "word" });
if (!segmenter) return term;
const tokens = [];
for (const seg of segmenter.segment(term)) {
// @ts-ignore
tokens.push(seg.segment);
}
return tokens;
},
},
},
},
},
},
...
}); Thanks to @mantou132 |
Excuse me, how to support other language search, such as Chinese search, thank you .
The text was updated successfully, but these errors were encountered: