Coral USB Accelerator

[Zuzia-Sweetheart]

Hello friends, could Koboldcpp use Coral USB Accelerator TPU? I don't know much about said accelerator on a technical level as i am still learning, but it does sound interesting, what do you think?

[LostRuins]

TPU is not supported for KoboldCpp, only GPUs

[Zuzia-Sweetheart]

but Koboldcpp could technically do it in the future or is it a dumb idea?

[LostRuins]

Not very likely

[Zuzia-Sweetheart]

I hope you don't mind me asking but, its simply not worth or too hard to implement?

[LostRuins]

I lack the familiarity or technical expertise to work with that device.

[Zuzia-Sweetheart]

fair

[MathapMathonwy]

fair

For the most part, TPUs/NPUs are marketing fluff. They aren't connected to enough high speed memory to matter. These were designed years ago, pre-LLMs to run tiny models of only a few MBs, for things like simple object detection in images or low res video, noise reduction for audio, background blur, etc. They really aren't relevant for LLMs, despite all the attempts to market them as for "AI".
You're asking about a USB connected device, which has almost no memory of its own, for models that can saturate the entire memory of bus of a computer.