New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tokenizer] 希望将tokenizer_type等写入yaml，从而可以在yaml中配置相关字段 #2204

Closed

xingchensong opened this issue Dec 8, 2023 · 1 comment

Assignees

Member

xingchensong commented Dec 8, 2023

现在的tokenizer初始化方式是通过判断bpe_model是否为空来执行的，未来引入更多类型的tokenizer时已显得不适用

xingchensong assigned Mddct and xingchensong

Collaborator

Mddct commented Dec 13, 2023

#2205 (comment) 这个pr已经写入了conf

Screenshot 2023-12-13 at 14 24 12

Mddct closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment