Skip to content

@Infini-AI-Lab Infini-AI-Lab

Change the repository type filter

All

    Repositories list

    16 repositories

    • UMbreLLa

      Public
      LLM Inference on consumer devices
      offloadingllm-inferencespeculative-decoding
      Python
      •
      Apache License 2.0
      •7•83•9•3•Updated Feb 2, 2025Feb 2, 2025

    • APE

      Public
      Python
      •0•0•0•0•Updated Jan 30, 2025Jan 30, 2025

    • RULER

      Public
      This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
      Python
      •
      Apache License 2.0
      •64•0•0•0•Updated Jan 30, 2025Jan 30, 2025

    • Sequoia

      Public
      scalable and robust tree-based speculative decoding algorithm
      efficiencyinferencellmspeculative-decoding
      Python
      •38•332•7•3•Updated Jan 28, 2025Jan 28, 2025

    • lm-evaluation-harness

      Public
      A framework for few-shot evaluation of language models.
      Python
      •
      MIT License
      •2.1k•0•0•0•Updated Jan 11, 2025Jan 11, 2025

    • S2FT

      Public
      Python
      •2•12•0•0•Updated Jan 3, 2025Jan 3, 2025

    • S2FT-Page

      Public
      JavaScript
      •0•0•0•0•Updated Dec 30, 2024Dec 30, 2024

    • MagicPIG

      Public
      [ICLR2025] MagicPIG: LSH Sampling for Efficient LLM Generation
      decodinglsh-algorithmllm-inferencegpu-cpu
      Python
      •
      Apache License 2.0
      •14•184•9•1•Updated Dec 16, 2024Dec 16, 2024

    • MagicDec

      Public
      [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
      Python
      •
      Apache License 2.0
      •6•106•6•0•Updated Dec 4, 2024Dec 4, 2024

    • MagicPIG-Page

      Public
      JavaScript
      •1•0•0•0•Updated Dec 2, 2024Dec 2, 2024

    • Factor

      Public
      0•1•0•0•Updated Nov 7, 2024Nov 7, 2024

    • MagicDec-part1

      Public
      Speculative decoding for high-throughput long-context inference
      JavaScript
      •
      Apache License 2.0
      •0•0•0•0•Updated Sep 10, 2024Sep 10, 2024

    • Sirius

      Public
      Sirius, an efficient correction mechanism, which significantly boosts Contextual Sparsity models on reasoning tasks while maintaining its efficiency gain.
      Python
      •4•21•0•0•Updated Sep 10, 2024Sep 10, 2024

    • MagicDec-part2

      Public
      MagicDec: Breaking the Latency-Throughput Tradeoff for Long Contexts with Speculative Decoding
      JavaScript
      •
      Apache License 2.0
      •0•0•0•0•Updated Sep 5, 2024Sep 5, 2024

    • TriForce

      Public
      [COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
      accelerationefficiencyinferencellmlong-contextllm-inferencespeculative-decoding
      Python
      •15•238•7•0•Updated Aug 31, 2024Aug 31, 2024

    • Sequoia-Page

      Public
      JavaScript
      •1•0•0•0•Updated May 21, 2024May 21, 2024