Preprocessing : Japanese support

[noctetemp]

Hello Experts

I was checking PrePorcessing module and found out there is no Japanese support.
Does this means it is not recommended to use Haystack RAG with Japanese documents ? is it planned for the future ?
is it something easy to add ?

thank you in advance

[ZanSara]

Hello! right now our preprocessing capabilities are quite limited for non alphabetic scripts and even though it's not impossible, I'm afraid they would perform poorly on Japanese text. However, if you find a good library for that, you can very easily write a custom component for Haystack 2.0 (here a guide: https://docs.haystack.deepset.ai/v2.0/docs/custom-components) and we would be really happy to integrate it with Haystack, so let us know!