Replies: 2 comments 4 replies
-
|
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Thank you for your answers. Regarding question 2: Is there anything specific I need to do to enable asynchronous-concurrent generation? I couldn’t find any details in the wiki. Also, if I disable it, what happens when a second request comes through—will it time out, or will it wait in a queue? For question 3: Could you elaborate on how significant the negative impact might be? Would a slight increase have a minimal effect, whereas a larger increase, say 50%, cause a more substantial degradation in quality? |
Beta Was this translation helpful? Give feedback.
-
|
Hi. The API server uses the AsyncAphrodite class, which is an async wrapper over the main engine. The scheduler, worker, and async engine perform automatic, continuous batching. The user only needs to set a maximum batch size, and the engine will try and accommodate that; it will handle queuing, scheduling, swapping, etc. You can set the max batch size with
Up to 2x the original context length, there's practically no degradation - although it depends on the model. Some models, like Llama-3, can extend up to 32k with little issues. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding question 1: Although everything appears to be loaded correctly, it still doesn't work. It threw an error. There seem to be a timeout issue. What am I missing please? |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I can't seem to reproduce this on 4x A40s. What GPUs were you using? |
Beta Was this translation helpful? Give feedback.
-
It’s strange. I was using 4 x RTX A4000 on RunPod, which should have enough memory (64 GB vRAM) for turboderp/Llama-3-70B-Instruct-exl2 on 6.0bpw. Using 4 x A40 would be excessive. Based on your earlier suggestion, should I switch to a 4 x RTX A4000 with this PyTorch-based instance (runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04) instead? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We have conducted initial tests on the Aphrodite-engine and are impressed with the results. We are now considering replacing vLLM with Aphrodite-engine for production. However, I have a few questions:
We plan to run it on RunPod using this template. To utilise four GPUs, is it sufficient to set NUM_GPUS to 4? We were planning to use turboderp/Llama-3-70B-Instruct-exl2 with 6.0bpw quantisation and were hoping to deploy it on 4 x 16 GB VRAM GPUs, totalling 64 GB. Is this configuration possible, and is exl2 supported across 4 GPUs?
Regarding how Aphrodite-engine manages concurrent API requests: Does it batch and process them sequentially, or does it handle them concurrently in parallel? I read on Reddit that it supports concurrent processing but at a reduced quality of 30%-40% when asynchronous-concurrent generation is enabled. In our case, maintaining high response quality is critical. Is there an option to queue the requests to run sequentially or with reduced concurrency to avoid quality degradation?
Is it possible to increase the context length on Llama-3 from 8192 to 9728 by setting CONTEXT_LENGTH to 9728? Aphrodite apparently supports automatic RoPE scaling. Would this adjustment negatively impact the quality of responses?
Many thanks.
Beta Was this translation helpful? Give feedback.
All reactions