Support non-space separated language?

[gembin]

How to support non-space separated language?
I tried non-space separated language, it's not working.

[adrienball]

Hi @gembin , apologize for the very late answer, your issue somehow got lost.
This rust library works on atomic elements, named tokens. It is indeed targeting inputs which are space separated, and tokens on this assumption. This means that for non-space separated languages, the tokenization must be done beforehand in order to provide a space-separated input to the main API of this lib.

I think that the API could be improved in order to abstract this tokenization, and allow the caller to provide a custom tokenization. This is a bit of work though, not on our priority at the moment.
Best