Skip to content

Commit

Permalink
[Bugfix] fix output parsing error for trtllm backend (vllm-project#4137)
Browse files Browse the repository at this point in the history
Co-authored-by: Roger Wang <[email protected]>
  • Loading branch information
elinx and ywang96 authored Apr 17, 2024
1 parent 8438e05 commit fe3b5bb
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion benchmarks/backend_request_func.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ async def async_request_trt_llm(
"data:")

data = json.loads(chunk)
output.generated_text += data["text_output"]
timestamp = time.perf_counter()
# First token
if ttft == 0.0:
Expand All @@ -149,7 +150,6 @@ async def async_request_trt_llm(
most_recent_timestamp = timestamp

output.latency = most_recent_timestamp - st
output.generated_text = json.loads(data)["text_output"]
output.success = True

else:
Expand Down

0 comments on commit fe3b5bb

Please sign in to comment.