Responses are slow and also weird/awkward

[jbarker7]

I've got Oobabooga up and running no problem and just finished a custom server build for our AI lab using the following hardware:
Threadripper 3960 (24 cores)
256 GB RAM
4x 3090s (24 GB RAM each)
Windows 11 Pro

We also tried using TheBloke_Llama-2-70B-AWQ as the model with AutoAWQ as the loader

We are having two problems:

1. It seems to be incredibly slow - 3 tokens/second. I would have expected much much faster on 4 3090s.
2. When using the API using the --public-api switch, it gives crazy results that don't make any sense that are also very short. First time I've had a model actually swear like a sailor :P

Any ideas?

[pyphan1]

im getting very weird answers with gpt2, did anyone find a fix