Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[text] refine tokenizer #2165

Merged
merged 20 commits into from
Nov 28, 2023
Merged

[text] refine tokenizer #2165

merged 20 commits into from
Nov 28, 2023

Commits on Nov 25, 2023

  1. [text] refine tokenizer

    @Mddct
    Mddct committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    8144a2d View commit details
    Browse the repository at this point in the history

  2. [text] fix flake8

    @Mddct
    Mddct committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    99cf7d7 View commit details
    Browse the repository at this point in the history

  3. [text] fix lint

    @Mddct
    Mddct committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    5694565 View commit details
    Browse the repository at this point in the history

  4. [text] fix unit

    @Mddct
    Mddct committed Nov 25, 2023
    Configuration menu
    Copy the full SHA
    2418d79 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2023

  1. [text] add bpe tokenizer and char tokenizer

    @Mddct
    Mddct committed Nov 26, 2023
    Configuration menu
    Copy the full SHA
    3552b94 View commit details
    Browse the repository at this point in the history

  2. [text] add char tokenizer unit test

    @Mddct
    Mddct committed Nov 26, 2023
    Configuration menu
    Copy the full SHA
    9912df9 View commit details
    Browse the repository at this point in the history

  3. [text] add bpe tokenizer unit test

    @Mddct
    Mddct committed Nov 26, 2023
    Configuration menu
    Copy the full SHA
    266a4fa View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. [text] add WhisperTokenizer for test_whisper.py

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    c2ecc7c View commit details
    Browse the repository at this point in the history

  2. [text] revert wenet/utils/file_utils.py

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    55da48a View commit details
    Browse the repository at this point in the history

  3. [text] add consistency for char and bpe unit

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    50422fc View commit details
    Browse the repository at this point in the history

  4. [text] merge main

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    a479de7 View commit details
    Browse the repository at this point in the history

  5. [text] merge main

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    cf754ff View commit details
    Browse the repository at this point in the history

  6. [text] add symbol table

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    bd24277 View commit details
    Browse the repository at this point in the history

  7. [text] add init_tokenizer unit test

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    49994bf View commit details
    Browse the repository at this point in the history

  8. [text] uncomment

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    301af9e View commit details
    Browse the repository at this point in the history

  9. [text] fix bpe model in multiprocess env

    @Mddct
    Mddct committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    a13dedf View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. [text] fix whisper tokenzier in multiprocess env

    @Mddct
    Mddct committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    ec2d838 View commit details
    Browse the repository at this point in the history

  2. [text] add test unit parallel for bpe and whisper

    @Mddct
    Mddct committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    51a10fa View commit details
    Browse the repository at this point in the history

  3. [text] fix none type in test_whisper.py

    @Mddct
    Mddct committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    1dc2d79 View commit details
    Browse the repository at this point in the history

  4. [text] all work

    @Mddct
    Mddct committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    f1099a8 View commit details
    Browse the repository at this point in the history