Running on mac computer

[nogaeps]

Hey,
Im trying to run LLMLingua on my mac computer.
I tried both using llamaindex and llmlingua package.
Both raised the same error: AssertionError: Torch not compiled with CUDA enabled
It seems like mac doesn't have CUDA since the GPU is not made by nvidia.
How can I make it work?
Thanks!

[iofu728]

Hi @nogaeps, since macOS does not have CUDA,
you can use mps, please refer to the following code:

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(device_map="mps")
# or using other models
llm_lingua = PromptCompressor("lgaalves/gpt2-dolly", device_map="mps")

compressed_prompt = llm_lingua.compress_prompt(prompt, instruction="", question="", target_token=200)

[gitkaz]

Hi @iofu728, Thank you for developing and maintain this program.

I'm not familiar with programming, but as far as I tested my M1 pro Mac environment,
I just modified following line, then Mac uses own GPU and improved speed for output the compressed prompt.
(cpu: 80.48s, mps: 3.77s).

prompt_compressor.py

---

39 self.device = (
40 device_map if any(key in device_map for key in ["cuda", "cpu"]) else "cuda"

---

add "mps" to line 40.

---

39 self.device = (
40 device_map if any(key in device_map for key in ["cuda", "cpu", "mps"]) else "cuda"

---

Then specify 'device_map="mps"' in own code.

[iofu728]

Hi @gitkaz, thank you for your assistance; it has been very helpful. I will apply the updates to the code.

[iofu728]

Hi @gitkaz, the modifications have been merged into the main branch under the #27. Thanks for your help.