Skip to content

Commit

Permalink
The testing model from HF doesn't have the tokenizer max length set c…
Browse files Browse the repository at this point in the history
…orrectly
  • Loading branch information
AngledLuffa committed Nov 22, 2024
1 parent f3dbc36 commit c150269
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion stanza/models/common/bert_embedding.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ def __init__(self, length, max_len, line_num, text):


def update_max_length(model_name, tokenizer):
if model_name in ('google/muril-base-cased',
if model_name in ('hf-internal-testing/tiny-bert',
'google/muril-base-cased',
'google/muril-large-cased',
'airesearch/wangchanberta-base-att-spm-uncased',
'camembert/camembert-large',
Expand Down

0 comments on commit c150269

Please sign in to comment.