Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer #314

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer #314

wants to merge 12 commits into from

Commits on Jun 18, 2024

  1. Add TUI environment for testing encoding in tiktoken

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    f26d68f View commit details
    Browse the repository at this point in the history

  2. Fix typo in environment method name in tiktoken/core.py

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    ccab1bb View commit details
    Browse the repository at this point in the history

  3. Refactor environment method in tiktoken/core.py to accept allowed_spe… 

    …cial and disallowed_special parameters
    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    a9e4047 View commit details
    Browse the repository at this point in the history

  4. update README.md to mention tiktoken environment

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    1800e3e View commit details
    Browse the repository at this point in the history

  5. update README.md to mention tiktoken environment

    @LVivona
    LVivona authored Jun 18, 2024
    Configuration menu
    Copy the full SHA
    39c6069 View commit details
    Browse the repository at this point in the history

  6. Add support for PNG files in MANIFEST.in

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    c674c91 View commit details
    Browse the repository at this point in the history

  7. clean up unused dependencies src/lib.rs

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    def13cc View commit details
    Browse the repository at this point in the history

  8. Remove unused bstr::ByteSlice import in src/lib.rs

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    9a35100 View commit details
    Browse the repository at this point in the history

  9. Revert "clean up unused dependencies src/lib.rs" 

    This reverts commit def13cc.
    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    afc66ed View commit details
    Browse the repository at this point in the history

  10. Refactor imports and clean up code in src/lib.rs

    @LVivona
    LVivona committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    a075d45 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. update layout, and scrollview of tokenizer

    @LVivona
    LVivona committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    04f1f8d View commit details
    Browse the repository at this point in the history

  2. Update menu layout and add scroll functionality in src/lib.rs

    @LVivona
    LVivona committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    2c6af5c View commit details
    Browse the repository at this point in the history