Foundation Model Inference

All

3 repositories

FlexLLMGen
Public
Running large language models on a single GPU for throughput-oriented scenarios.
machine-learning offloading high-throughput opt gpt-3 large-language-models deep-learning
Python
•
Apache License 2.0
•549•9.2k•51•6•Updated Oct 28, 2024Oct 28, 2024
H2O
Public
[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
sparsity high-throughput heavy-hitters kv-cache gpt-3 large-language-models
Python
•39•387•30•1•Updated Aug 1, 2024Aug 1, 2024
DejaVu
Public
Python
•35•283•27•1•Updated Apr 2, 2024Apr 2, 2024