Add huggingface llava #524

jp1924 · 2025-01-16T08:11:31Z

Summary

#514
transformer

pip install git+https://github.com/huggingface/transformers.git@5fa35344755d8d9c29610b57d175efd03776ae9e

Testing Done

huggingface-env

transformers: 4.49.0.dev0
huggingface_hub version: 0.26.2
Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
Python version: 3.10.12
Running in iPython ?: No
Running in notebook ?: No
Running in Google Colab ?: No
Running in Google Colab Enterprise ?: No
Token path ?: /root/.cache/huggingface/token
Has saved token ?: True
Who am I ?: jp1924
Configured git credential helpers:
FastAI: N/A
Tensorflow: N/A
Torch: 2.5.1+cu121
Jinja2: 3.1.4
Graphviz: N/A
keras: N/A
Pydot: N/A
Pillow: 11.0.0
hf_transfer: N/A
gradio: N/A
tensorboard: N/A
numpy: 2.1.3
pydantic: 2.9.2
aiohttp: 3.11.6
ENDPOINT: https://huggingface.co
HF_HUB_CACHE: /root/.cache/huggingface/hub
HF_ASSETS_CACHE: /root/.cache/huggingface/assets
HF_TOKEN_PATH: /root/.cache/huggingface/token
HF_STORED_TOKENS_PATH: /root/.cache/huggingface/stored_tokens
HF_HUB_OFFLINE: False
HF_HUB_DISABLE_TELEMETRY: False
HF_HUB_DISABLE_PROGRESS_BARS: None
HF_HUB_DISABLE_SYMLINKS_WARNING: False
HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
HF_HUB_DISABLE_IMPLICIT_TOKEN: False
HF_HUB_ENABLE_HF_TRANSFER: False
HF_HUB_ETAG_TIMEOUT: 10
HF_HUB_DOWNLOAD_TIMEOUT: 10

torch&hw-env

Collecting environment information...
PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 560.35.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
CPU family: 6
Model: 106
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 2
Stepping: 6
CPU max MHz: 3300.0000
CPU min MHz: 800.0000
BogoMIPS: 4200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 1.1 MiB (24 instances)
L1i cache: 768 KiB (24 instances)
L2 cache: 30 MiB (24 instances)
L3 cache: 36 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-11,24-35
NUMA node1 CPU(s): 12-23,36-47
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==2.1.3
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] torch==2.5.1+cu121
[pip3] torchvision==0.20.1+cu121
[pip3] triton==3.1.0
[conda] Could not collect

run make test to ensure correctness
run make checkstyle to ensure code style
run make test-convergence to ensure convergence

…ommented code and simplifying model type checks

…_kernel_to_llava function

…r bfloat16 support

… outdated configurations

…_to_llava for model type handling

…iguration parameters

jp1924 · 2025-01-16T08:22:10Z

make test-convergence log

HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/test_mini_models.py
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /root/Liger-Kernel
configfile: pyproject.toml
collecting ...
--------------------------------------------------------------------------------------------------------------------- live log collection ----------------------------------------------------------------------------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 19 items

test/convergence/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED                                                                                                                          [  5%]
test/convergence/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                 [ 10%]
test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                            [ 15%]
test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                  [ 21%]
test/convergence/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                           [ 26%]
test/convergence/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                 [ 31%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                            [ 36%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                  [ 42%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype8-1e-05-0.1-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                           [ 47%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype9-0.001-0.05-0.1-0.01-0.01-0.01] PASSED                                                                                                                               [ 52%]
test/convergence/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype10-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                            [ 57%]
test/convergence/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                  [ 63%]
test/convergence/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype12-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [ 68%]
test/convergence/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype13-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                               [ 73%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype14-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [ 78%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype15-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                [ 84%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype16-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                       [ 89%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype17-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                              [ 94%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma2-32-0.0001-dtype18-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [100%]

========================================================================================================== 19 passed, 2 warnings in 282.00s (0:04:41) ==========================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/test_mini_models_multimodal.py
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /root/Liger-Kernel
configfile: pyproject.toml
collecting ...
--------------------------------------------------------------------------------------------------------------------- live log collection ----------------------------------------------------------------------------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 6 items

test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                   [ 16%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                         [ 33%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                     [ 50%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                           [ 66%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [ 83%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [100%]

========================================================================================================== 6 passed, 15 warnings in 216.30s (0:03:36) ==========================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/test_mini_models_with_logits.py
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /root/Liger-Kernel
configfile: pyproject.toml
collecting ...
--------------------------------------------------------------------------------------------------------------------- live log collection ----------------------------------------------------------------------------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 19 items

test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED                                                                                                              [  5%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                     [ 10%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [ 15%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [ 21%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                               [ 26%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                     [ 31%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                [ 36%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                      [ 42%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype8-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 47%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype9-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [ 52%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype10-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                [ 57%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                      [ 63%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype12-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 68%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype13-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [ 73%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype14-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 78%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype15-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                    [ 84%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype16-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                           [ 89%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype17-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                  [ 94%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma2-32-0.0001-dtype18-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [100%]

========================================================================================================== 19 passed, 2 warnings in 258.24s (0:04:18) ==========================================================================================================

jp1924 · 2025-01-16T08:23:29Z

cc @Tcc0403 @ByronHsu

Tcc0403 · 2025-01-16T10:12:26Z

Good work! I'll take a look in few days.

Tcc0403 · 2025-01-18T03:30:46Z

There are multiple breaking changes in transformers recently. Convergence test couldn't pass with newer transformers version.

    def test_mini_model(
        model_name,
        num_steps,
        lr,
        dtype,
        loss_atol,
        loss_rtol,
        logits_atol,
        logits_rtol,
        param_atol,
        param_rtol,
    ):
        # Non-liger models should be initialized and tested first to avoid the module being overridden

>       expected_output = run_mini_model(model_name=model_name, num_steps=num_steps, dtype=dtype, lr=lr)

test/convergence/test_mini_models.py:773:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test/convergence/test_mini_models.py:519: in run_mini_model
    output = model(**batch)
.venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1736: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
.venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1747: in _call_impl
    return forward_call(*args, **kwargs)
.venv/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.py:492: in forward
    inputs_embeds, attention_mask, labels, position_ids = self._merge_input_ids_with_image_features(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = LlavaForConditionalGeneration(
  (vision_tower): CLIPVisionModel(
    (vision_model): CLIPVisionTransformer(
      (em...rotary_emb): LlamaRotaryEmbedding()
    )
    (lm_head): Linear(in_features=1024, out_features=32064, bias=False)
  )
)
image_features = None
inputs_embeds = tensor([[[ 1.1444e-05, -1.3367e-02,  4.3640e-03,  ...,  1.9043e-02,
           3.6377e-02, -8.7891e-03],
         [ 1....-02,
           5.7373e-03, -9.4604e-04]]], device='cuda:0', dtype=torch.bfloat16,
       grad_fn=<EmbeddingBackward0>)
input_ids = tensor([[    2,     2,     2,  ..., 16334, 20084, 28747],
        [    2,     2,     2,  ...,   528,  4085, 28723],
  ...,     2,     2,  ...,     2,     2,     1],
        [    2,     2,     2,  ...,     1,  1682, 28747]], device='cuda:0')
attention_mask = tensor([[0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 0, 0, 1],
        ...,
        [0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 0, 0, 1],
        [0, 0, 0,  ..., 1, 1, 1]], device='cuda:0')
labels = tensor([[    2,     2,     2,  ..., 16334, 20084, 28747],
        [    2,     2,     2,  ...,   528,  4085, 28723],
  ...,     2,     2,  ...,     2,     2,     1],
        [    2,     2,     2,  ...,     1,  1682, 28747]], device='cuda:0')

    def _merge_input_ids_with_image_features(self, image_features, inputs_embeds, input_ids, attention_mask, labels):
>       num_images, num_image_patches, embed_dim = image_features.shape
E       AttributeError: 'NoneType' object has no attribute 'shape'

.venv/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.py:303: AttributeError
------------------------------------------------- Captured stdout call -------------------------------------------------
Liger kernel patches have been reverted.
=============================================== short test summary info ================================================
FAILED test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] - AttributeError: 'NoneType' object has no attribute 'shape'
FAILED test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] - AttributeError: 'NoneType' object has no attribute 'shape'

Environment:

❯ python -m liger_kernel.env_report
Environment Report:
-------------------
Operating System: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Python version: 3.10.12
Liger Kernel version: 0.5.2
PyTorch version: 2.5.1+cu124
CUDA version: 12.4
HIP(ROCm) version: Not available
Triton version: 3.1.0
Transformers version: 4.47.1
XPU version: XPU Not Available

Let's make a condition to handle different function signatures for older and newer transformers version, since both are still being used by users.

something like

if transformer_version < ...:
  ...
else:
  ...

test/convergence/test_mini_models.py

jp1924 · 2025-01-20T07:21:36Z

@Tcc0403
Ah, this is an error from transformers
You can see the exact cause in this link: huggingface/transformers#35526
However, this issue has been fixed in the latest version of transformers on GitHub, so to run it correctly, you will need to install from the source using pip in the transformers repo.

pip install git+https://github.com/huggingface/transformers.git@5fa35344755d8d9c29610b57d175efd03776ae9e

jp1924 · 2025-01-20T07:24:37Z

That's why if you look at the hw&sw specification, my transformer is 4.49.0.dev instead of 4.48.0 due to this issue.

Tcc0403 · 2025-02-21T10:22:42Z

Yes, see #571.

Push the make checkstyle, and it should be able to fast forward merge into the main branch.

jp1924 · 2025-03-03T23:49:33Z

Hey @Tcc0403, is the test problem solved now?
I think we should PR this too.

Tcc0403 · 2025-03-07T15:24:21Z

I found why ruff doesn't format files correctly. Uninstall ruff outside of your virtual environtment, so that ruff in venv can fetch the correct configuration.

Overall the code LGTM. All tests passed. We can merge it after formatting.

minor: address the issue mentioned in #573, matching the latest transformers APIs (num_logits_to_keep -> logits_to_keep) like what #596 does. If you don't have time to take a look for it, it's all fine. I can handle it later after this PR merge.

Tcc0403

Format files to match main branch

jp1924 · 2025-03-08T03:35:12Z

@Tcc0403, all test is pass. could you please review this?

test convergence log

HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/fp32/test_mini_models.py
======================================================================================================================= test session starts =======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/jp/Liger-Kernel
configfile: pyproject.toml
plugins: jaxtyping-0.2.36, anyio-4.6.2.post1
collecting ...
----------------------------------------------------------------------------------------------------------------------- live log collection -----------------------------------------------------------------------------------------------------------------------
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO     numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads.
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 13 items

test/convergence/fp32/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED                                                                                                                        [  7%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype1-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                          [ 15%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [ 23%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                          [ 30%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype4-1e-05-0.1-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [ 38%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype5-1e-05-0.1-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                       [ 46%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_olmo2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                          [ 53%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype7-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                           [ 61%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype8-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                        [ 69%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype9-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                        [ 76%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype10-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                     [ 84%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_gemma2-32-0.0001-dtype11-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                       [ 92%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_granite3-32-0.0001-dtype12-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                     [100%]

============================================================================================================ 13 passed, 1 warning in 115.47s (0:01:55) ============================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/fp32/test_mini_models_multimodal.py
======================================================================================================================= test session starts =======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/jp/Liger-Kernel
configfile: pyproject.toml
plugins: jaxtyping-0.2.36, anyio-4.6.2.post1
collecting ...
----------------------------------------------------------------------------------------------------------------------- live log collection -----------------------------------------------------------------------------------------------------------------------
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO     numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads.
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 4 items

test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                 [ 25%]
test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_5_vl-32-0.0001-dtype1-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                               [ 50%]
test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                   [ 75%]
test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                    [100%]

============================================================================================================ 4 passed, 7 warnings in 81.11s (0:01:21) =============================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/fp32/test_mini_models_with_logits.py
======================================================================================================================= test session starts =======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/jp/Liger-Kernel
configfile: pyproject.toml
plugins: jaxtyping-0.2.36, anyio-4.6.2.post1
collecting ...
----------------------------------------------------------------------------------------------------------------------- live log collection -----------------------------------------------------------------------------------------------------------------------
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO     numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads.
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 13 items

test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED                                                                                                            [  7%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype1-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                              [ 15%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 23%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                              [ 30%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                           [ 38%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype5-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                         [ 46%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_olmo2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                              [ 53%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype7-1e-08-1e-05-0.005-1e-05-0.005-1e-05] make test-convergencePASSED                                                                                                               [ 61%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype8-1e-08-1e-05-0.005-1e-05-0.005-1e-05PASSED                                                                                                            [ 69%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype9-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                            [ 76%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype10-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                         [ 84%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_gemma2-32-0.0001-dtype11-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                           [ 92%]
test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_granite3-32-0.0001-dtype12-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                         [100%]

============================================================================================================ 13 passed, 1 warning in 112.86s (0:01:52) ============================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/bf16/test_mini_models.py
======================================================================================================================= test session starts =======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/jp/Liger-Kernel
configfile: pyproject.toml
plugins: jaxtyping-0.2.36, anyio-4.6.2.post1
collecting ...
----------------------------------------------------------------------------------------------------------------------- live log collection -----------------------------------------------------------------------------------------------------------------------
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO     numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads.
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 12 items

test/convergence/bf16/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                               [  8%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                [ 16%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_granite3-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                             [ 25%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                               [ 33%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype4-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                [ 41%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype5-0.001-0.05-0.1-0.01-0.01-0.01] PASSED                                                                                                                             [ 50%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype6-0.001-0.05-0.1-0.01-0.01-0.01] PASSED                                                                                                                           [ 58%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                 [ 66%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype8-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                              [ 75%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_olmo2-32-0.0001-dtype9-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                [ 83%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype10-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                              [ 91%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                            [100%]

============================================================================================================ 12 passed, 1 warning in 67.10s (0:01:07) =============================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/bf16/test_mini_models_multimodal.py
======================================================================================================================= test session starts =======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/jp/Liger-Kernel
configfile: pyproject.toml
plugins: jaxtyping-0.2.36, anyio-4.6.2.post1
collecting ...
----------------------------------------------------------------------------------------------------------------------- live log collection -----------------------------------------------------------------------------------------------------------------------
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO     numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads.
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 4 items

test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                       [ 25%]
test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_5_vl-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                     [ 50%]
test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                         [ 75%]
test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                          [100%]

============================================================================================================ 4 passed, 7 warnings in 60.99s (0:01:00) =============================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/bf16/test_mini_models_with_logits.py
======================================================================================================================= test session starts =======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/jp/Liger-Kernel
configfile: pyproject.toml
plugins: jaxtyping-0.2.36, anyio-4.6.2.post1
collecting ...
----------------------------------------------------------------------------------------------------------------------- live log collection -----------------------------------------------------------------------------------------------------------------------
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO     numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads.
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 12 items

test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [  8%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                    [ 16%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_granite3-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                 [ 25%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [ 33%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype4-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                    [ 41%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                 [ 50%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype6-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                               [ 58%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                     [ 66%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype8-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                  [ 75%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype9-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [ 83%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype10-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                [ 91%]
test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_olmo2-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [100%]

============================================================================================================ 12 passed, 1 warning in 70.05s (0:01:10) =============================================================================================================

Tcc0403 · 2025-03-08T04:09:38Z

test/convergence/bf16/test_mini_models_with_logits.py

@@ -496,7 +507,7 @@
            tie_word_embeddings=False,
            use_cache=True,
            vocab_size=32000,  # 128256,
-            logits_scaling=8.0,


Does it fail the test?

yes, there was nothing that failed regarding mini_granite3.

Even after changing the values and running it again, it still passed the test.

convergence-test

HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/fp32/test_mini_models.py ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/jp/Liger-Kernel configfile: pyproject.toml plugins: jaxtyping-0.2.36, anyio-4.6.2.post1 ----------------------------- live log collection ------------------------------ INFO numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads. INFO datasets:config.py:54 PyTorch version 2.5.1+cu121 available. collected 13 items test/convergence/fp32/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED [ 7%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype1-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 15%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 23%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 30%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype4-1e-05-0.1-0.005-1e-05-0.005-1e-05] PASSED [ 38%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype5-1e-05-0.1-0.005-1e-05-0.005-1e-05] PASSED [ 46%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_olmo2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 53%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype7-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 61%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype8-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 69%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype9-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [ 76%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype10-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [ 84%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_gemma2-32-0.0001-dtype11-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [ 92%] test/convergence/fp32/test_mini_models.py::test_mini_model[mini_granite3-32-0.0001-dtype12-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [100%] ================== 13 passed, 1 warning in 112.41s (0:01:52) =================== HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/fp32/test_mini_models_multimodal.py ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/jp/Liger-Kernel configfile: pyproject.toml plugins: jaxtyping-0.2.36, anyio-4.6.2.post1 ----------------------------- live log collection ------------------------------ INFO numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads. INFO datasets:config.py:54 PyTorch version 2.5.1+cu121 available. collected 4 items test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 25%] test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_5_vl-32-0.0001-dtype1-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 50%] test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 75%] test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [100%] =================== 4 passed, 5 warnings in 81.22s (0:01:21) =================== HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/fp32/test_mini_models_with_logits.py ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/jp/Liger-Kernel configfile: pyproject.toml plugins: jaxtyping-0.2.36, anyio-4.6.2.post1 ----------------------------- live log collection ------------------------------ INFO numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads. INFO datasets:config.py:54 PyTorch version 2.5.1+cu121 available. collected 13 items test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED [ 7%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype1-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 15%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 23%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 30%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 38%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype5-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 46%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_olmo2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 53%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype7-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 61%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype8-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 69%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype9-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [ 76%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype10-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [ 84%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_gemma2-32-0.0001-dtype11-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [ 92%] test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_granite3-32-0.0001-dtype12-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED [100%] ================== 13 passed, 1 warning in 112.43s (0:01:52) =================== HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/bf16/test_mini_models.py ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/jp/Liger-Kernel configfile: pyproject.toml plugins: jaxtyping-0.2.36, anyio-4.6.2.post1 ----------------------------- live log collection ------------------------------ INFO numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads. INFO datasets:config.py:54 PyTorch version 2.5.1+cu121 available. collected 12 items test/convergence/bf16/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 8%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 16%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_granite3-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 25%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 33%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype4-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 41%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype5-0.001-0.05-0.1-0.01-0.01-0.01] PASSED [ 50%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype6-0.001-0.05-0.1-0.01-0.01-0.01] PASSED [ 58%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 66%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype8-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 75%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_olmo2-32-0.0001-dtype9-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 83%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype10-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 91%] test/convergence/bf16/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [100%] =================== 12 passed, 1 warning in 67.35s (0:01:07) =================== HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/bf16/test_mini_models_multimodal.py ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/jp/Liger-Kernel configfile: pyproject.toml plugins: jaxtyping-0.2.36, anyio-4.6.2.post1 ----------------------------- live log collection ------------------------------ INFO numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads. INFO datasets:config.py:54 PyTorch version 2.5.1+cu121 available. collected 4 items test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 25%] test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_5_vl-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 50%] test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 75%] test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [100%] =================== 4 passed, 5 warnings in 61.02s (0:01:01) =================== HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/bf16/test_mini_models_with_logits.py ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/jp/Liger-Kernel configfile: pyproject.toml plugins: jaxtyping-0.2.36, anyio-4.6.2.post1 ----------------------------- live log collection ------------------------------ INFO numexpr.utils:utils.py:148 Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO numexpr.utils:utils.py:161 NumExpr defaulting to 16 threads. INFO datasets:config.py:54 PyTorch version 2.5.1+cu121 available. collected 12 items test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 8%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 16%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_granite3-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 25%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 33%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype4-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 41%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 50%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype6-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 58%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 66%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype8-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 75%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype9-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 83%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype10-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 91%] test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_olmo2-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [100%] =================== 12 passed, 1 warning in 66.39s (0:01:06) ===================

src/liger_kernel/transformers/model/llava.py

src/liger_kernel/transformers/monkey_patch.py

…ecated

Tcc0403 · 2025-03-08T07:22:18Z

src/liger_kernel/transformers/model/llava.py

+
+
+@add_start_docstrings_to_model_forward(LLAVA_INPUTS_DOCSTRING)
+@deprecate_kwarg("num_logits_to_keep", new_name="logits_to_keep", version="4.50")


As far as I know, deprecate_kwarg decorator should be on the latest function instead of the deprecated one, since it is meant to replace num_logits_to_keep with logits_to_keep to match the latest function signature.

Tcc0403 · 2025-03-08T07:33:02Z

test/convergence/bf16/test_mini_models_multimodal.py

@@ -407,7 +515,6 @@ def run_mini_model_multimodal(
        print(f"Step {i}, Loss: {output.loss.item()}")
        loss_list.append(output.loss.item())

-    MINI_MODEL_SETUPS[model_name].liger_kernel_patch_revert_func(**revert_kwargs)


any reason why it is removed?

jp1924 added 19 commits January 8, 2025 01:17

Add: LLaVa lce_loss

f84c03f

Add: LLaVa monkey_patched

4aaf1ef

Add: llava config

8a6f8cd

Add: load_image_processing_config

4477479

Add: llava test

f77d647

Add: llava

53efec3

Add: llava test

1fadb0e

Add: llava test

2f14cf6

Refactor: update llava model forward method and enhance loss handling

a3db21c

debugging

79e2102

Refactor: simplify lce_forward method and improve loss calculation

80cfc08

Refactor: clean up apply_liger_kernel_to_llava function by removing c…

24ca457

…ommented code and simplifying model type checks

rm debugging

8c181a6

Refactor: remove redundant import of llava_lce_forward in apply_liger…

24bc56c

…_kernel_to_llava function

Refactor: update mini_llava model configuration and add test cases fo…

42dbe36

…r bfloat16 support

Fix: loss value error

f6ba33f

Refactor: add new processor and tokenizer configuration files, remove…

9223959

… outdated configurations

Refactor: clean up lce_forward function and update apply_liger_kernel…

61476e8

…_to_llava for model type handling

Add: processor configuration loading function and clean up model conf…

22cafbf

…iguration parameters

Merge branch 'main' into add_llava

7f61b13

Tcc0403 reviewed Jan 18, 2025

View reviewed changes

test/convergence/test_mini_models.py Outdated Show resolved Hide resolved

Tcc0403 reviewed Jan 18, 2025

View reviewed changes

test/convergence/test_mini_models.py Outdated Show resolved Hide resolved

jp1924 added 2 commits January 20, 2025 07:19

Fix: typo Qwen2-VL -> LLaVa

21bdc0a

Refactor: remove unused input list from run_mini_model function

37857ef

jp1924 added 4 commits February 25, 2025 16:15

Merge branch 'main' into add_llava

23b8b75

apply checkstyle

70921a5

Merge branch 'main' into add_llava

d32c208

Merge branch 'main' into add_llava

969923b

Merge branch 'main' into add_llava

9e44857

Tcc0403 requested changes Mar 7, 2025

View reviewed changes

jp1924 and others added 5 commits March 8, 2025 01:32

Merge branch 'main' into add_llava

a2aabd9

Apply: linkedin#596

fafeccb

Update: llava

a67c612

apply lint

7bd027c

Add: llava test

fd3fc8e

jp1924 requested a review from Tcc0403 March 8, 2025 03:37

jp1924 added 2 commits March 8, 2025 12:47

rm: legacy

c312a3c

revert indent

22a34c3

Tcc0403 reviewed Mar 8, 2025

View reviewed changes

src/liger_kernel/transformers/model/llava.py Outdated Show resolved Hide resolved

Tcc0403 reviewed Mar 8, 2025

View reviewed changes

src/liger_kernel/transformers/model/llava.py Show resolved Hide resolved

fix: logit scale

eb495f5

Tcc0403 reviewed Mar 8, 2025

View reviewed changes

src/liger_kernel/transformers/monkey_patch.py Outdated Show resolved Hide resolved

jp1924 added 4 commits March 8, 2025 14:21

Add: deprecate_kwarg

8e416f7

Fix: nn missing

a651cb0

Fix: correct variable name for num_logits_to_keep in lce_forward_depr…

4358b82

…ecated

fix: deprecate version

89cee9e

jp1924 requested a review from Tcc0403 March 8, 2025 05:53

Tcc0403 reviewed Mar 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add huggingface llava #524

Add huggingface llava #524

jp1924 commented Jan 16, 2025 •

edited

Loading

jp1924 commented Jan 16, 2025 •

edited by Tcc0403

Loading

jp1924 commented Jan 16, 2025

Tcc0403 commented Jan 16, 2025

Tcc0403 commented Jan 18, 2025

jp1924 commented Jan 20, 2025 •

edited

Loading

jp1924 commented Jan 20, 2025

Tcc0403 commented Feb 21, 2025

jp1924 commented Mar 3, 2025

Tcc0403 commented Mar 7, 2025 •

edited

Loading

Tcc0403 left a comment

jp1924 commented Mar 8, 2025

Tcc0403 Mar 8, 2025

jp1924 Mar 8, 2025

jp1924 Mar 8, 2025

Tcc0403 Mar 8, 2025

Tcc0403 Mar 8, 2025



		@add_start_docstrings_to_model_forward(LLAVA_INPUTS_DOCSTRING)
		@deprecate_kwarg("num_logits_to_keep", new_name="logits_to_keep", version="4.50")

Add huggingface llava #524

Are you sure you want to change the base?

Add huggingface llava #524

Conversation

jp1924 commented Jan 16, 2025 • edited Loading

Summary

Testing Done

jp1924 commented Jan 16, 2025 • edited by Tcc0403 Loading

jp1924 commented Jan 16, 2025

Tcc0403 commented Jan 16, 2025

Tcc0403 commented Jan 18, 2025

jp1924 commented Jan 20, 2025 • edited Loading

jp1924 commented Jan 20, 2025

Tcc0403 commented Feb 21, 2025

jp1924 commented Mar 3, 2025

Tcc0403 commented Mar 7, 2025 • edited Loading

Tcc0403 left a comment

Choose a reason for hiding this comment

jp1924 commented Mar 8, 2025

Tcc0403 Mar 8, 2025

Choose a reason for hiding this comment

jp1924 Mar 8, 2025

Choose a reason for hiding this comment

jp1924 Mar 8, 2025

Choose a reason for hiding this comment

Tcc0403 Mar 8, 2025

Choose a reason for hiding this comment

Tcc0403 Mar 8, 2025

Choose a reason for hiding this comment

jp1924 commented Jan 16, 2025 •

edited

Loading

jp1924 commented Jan 16, 2025 •

edited by Tcc0403

Loading

jp1924 commented Jan 20, 2025 •

edited

Loading

Tcc0403 commented Mar 7, 2025 •

edited

Loading