doc_embedder raised 'AssertionError: Torch not compiled with CUDA enabled'

[keebeegee]

Hi team,
started playing around w/ haystack and got myself a EC2 instance without GPU support as I read some place that haystack also runs on CPU.

I tried to deploy:
https://haystack.deepset.ai/blog/guide-to-using-zephyr-with-haystack2 / https://colab.research.google.com/drive/1gvfDSWyx2uJQokxO2DG1EXBZIFI-2D0N#scrollTo=2xpzuCipXonU

on a t4g.xlarge and got "doc_embedder raised 'AssertionError: Torch not compiled with CUDA enabled'" when executing:

indexing.run({"cleaner":{"documents":raw_docs}})

Checked google, discord and this forum with no success.

Consider now upgrading to a p2.xlarge - any thoughts? Are there any recommendation to get started with haystack using an AMAZON EC2 instance?

Thanks in advance for support and advise.

[julian-risch]

Hi @keebeegee the example code that you shared uses 4bit quantization, which is only available with CUDA installed. That's why you are seeing this error. Almost all other parts of Haystack do not require CUDA and while they benefit from running on a GPU you can run them just on a CPU too.
If you want to try the open source LLM from the example, I suggest you use the 4bit quantization and switch to an EC2 instance with a GPU and a torch version with CUDA enabled, yes.