Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(feat): Novelai tokenizer re-implement || New LLM #75

Merged
merged 13 commits into from
Sep 26, 2024
Merged

(feat): Novelai tokenizer re-implement || New LLM #75

merged 13 commits into from
Sep 26, 2024

Commits on Sep 24, 2024

  1. ♻️ refactor(enum): add type aliases and reintroduce functions

    ♻️ refactor(cost): use ModelTypeAlias for model parameter
    
    ♻️ refactor(generate_image): use type aliases for fields
    
    ♻️ refactor(login): update debug log message
    sudoskys committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    9828ee8 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. ♻️ refactor(enum): add type aliases and reintroduce functions

    ♻️ refactor(cost): use ModelTypeAlias for model parameter
    
    ♻️ refactor(generate_image): use type aliases for fields
    
    ♻️ refactor(login): update debug log message
    sudoskys committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    bd426a0 View commit details
    Browse the repository at this point in the history
  2. ♻️ refactor(enum): add type aliases and reintroduce functions

    ♻️ refactor(cost): use ModelTypeAlias for model parameter
    
    ♻️ refactor(generate_image): use type aliases for fields
    
    ♻️ refactor(login): update debug log message
    sudoskys committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    f94dfc6 View commit details
    Browse the repository at this point in the history
  3. ♻️ refactor(enum): add type aliases and reintroduce functions

    ♻️ refactor(cost): use ModelTypeAlias for model parameter
    
    ♻️ refactor(generate_image): use type aliases for fields
    
    ♻️ refactor(login): update debug log message
    sudoskys committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    fbc9fa1 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. 🔄 refactor(playground): update model and improve prompt handling

    - Switch model from KRAKE_V2 to ERATO in `generate.py`
    - Adjust `print` statement formatting in `generate.py`
    - Provide explicit prompt in `generate.py` for `chat` function
    - Remove unnecessary comments and update variable names in `generate_stream.py`
    - Update tokenizer model and add additional functionality in `usage.py`
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    24d40a0 View commit details
    Browse the repository at this point in the history
  2. 🚀 feat: add frontend directory to .gitignore and update README

    - Added `/frontend/` to `.gitignore`
    - Updated README with new `tokenizer` task
    - Enhanced `chat` function with default preset parameters
    - Added tokenizer example in README
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    34a0660 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3a98ed9 View commit details
    Browse the repository at this point in the history
  4. ```plaintext

    🐛 fix: ensure correct dtype for token decoding in generate methods
    ```
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    61cbcf7 View commit details
    Browse the repository at this point in the history
  5. ✨ feat(generate): add new theme and schema for LLM generation

    - Added `themes.json` to support new AI generation themes.
    - Introduced `_schema.py` with data classes and models for AI generation.
    - Updated `_const.py` with new repetition penalty and ban words lists.
    - Enhanced `_enum.py` to manage themes and presets with caching.
    - Improved `__init__.py` to support advanced LLM settings and parameters.
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    5064555 View commit details
    Browse the repository at this point in the history
  6. ✨ feat(generate): add new theme and schema for LLM generation

    - Added `themes.json` to support new AI generation themes.
    - Introduced `_schema.py` with data classes and models for AI generation.
    - Updated `_const.py` with new repetition penalty and ban words lists.
    - Enhanced `_enum.py` to manage themes and presets with caching.
    - Improved `__init__.py` to support advanced LLM settings and parameters.
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    ba8d904 View commit details
    Browse the repository at this point in the history
  7. ♻️ refactor(ai): simplify LLMStream build and request logic

    Refactored the build method and request logic in LLMStream class.
    Simplified parameters handling and improved error messages.
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    52a806a View commit details
    Browse the repository at this point in the history
  8. ♻️ refactor(utils): update encoding and json functions

    - Refactored `b64_to_tokens` and `tokens_to_b64` to support different dtypes
    - Added `default_when_error` parameter to `try_jsonfy` function
    sudoskys committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    c2d266e View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f4af2e6 View commit details
    Browse the repository at this point in the history