About added Token '<point_patch>': 32000, '<point_start>': 32001, '<point_end>': 32002

[junbao-zhou]

I run your demo locally by command

python pointllm/eval/eval_modelnet_cls.py --model_name ./PointLLM_7B_v1.1_init --prompt_index 1

But there is an error:

/home/conda/feedstock_root/build_artifacts/libtorch_1715185017593/work/aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [152,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
...
...
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

After carefully debubgging, I find that the problem is in pointllm/model/pointllm.py line 106

        if inputs_embeds is None:
            inputs_embeds = self.embed_tokens(input_ids)

The embedding has shape torch.Size([32000, 4096]), but the max value in input_ids is 32002.

After more carefully debugging, I find that in pointllm/eval/eval_modelnet_cls.py, the tokenizer load 3 added token, I print them as:

tokenizer.get_added_vocab() = {'<point_patch>': 32000, '<point_start>': 32001, '<point_end>': 32002}

But the embedding seems do not have these tokens. How to solve it?

[RunsenXu]

Hi,

To test the model, please try to load another fine-tuned model rather than the init_model. That is, use --model_name RunsenXu/PointLLM_7B_v1.2