[Bug]: zero usage returned for streaming from text completions api of litellm proxy server #8349

minwhoo · 2025-02-07T07:57:47Z

What happened?

Token usage information is empty when streaming from completions api of litellm proxy server. Observed on openai/openai-compatible endpoints.
As the screenshot below shows, only the streaming+completions api combination has this issue, while streaming+chat completions work correctly.

Using litellm proxy server v1.60.6

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.60.6

Twitter / LinkedIn details

No response

minwhoo · 2025-02-07T09:04:57Z

Possible source of bug: the stream_options arg is missing when instantiating TextCompletionStreamWrapper class below.
Adding stream_options=kwargs.get('stream_options'), seems to fix it for my use case.

litellm/litellm/main.py

Lines 3941 to 3950 in 7739be3

    
           return TextCompletionStreamWrapper( 
        
               completion_stream=_async_streaming( 
        
                   response=response, 
        
                   model=model, 
        
                   custom_llm_provider=custom_llm_provider, 
        
                   args=args, 
        
               ), 
        
               model=model, 
        
               custom_llm_provider=custom_llm_provider, 
        
           )

minwhoo added the bug Something isn't working label Feb 7, 2025

ishaan-jaff added logging feb 2025 openai labels Feb 7, 2025

krrishdholakia changed the title ~~[Bug]: zero usage returned for streaming from completions api of litellm proxy server~~ [Bug]: zero usage returned for streaming from text completions api of litellm proxy server Feb 8, 2025

krrishdholakia added the high priority label Feb 8, 2025

Kaushikdkrikhanu linked a pull request Feb 9, 2025 that will close this issue

Pass stream options #8419

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: zero usage returned for streaming from text completions api of litellm proxy server #8349

[Bug]: zero usage returned for streaming from text completions api of litellm proxy server #8349

minwhoo commented Feb 7, 2025

minwhoo commented Feb 7, 2025

[Bug]: zero usage returned for streaming from text completions api of litellm proxy server #8349

[Bug]: zero usage returned for streaming from text completions api of litellm proxy server #8349

Comments

minwhoo commented Feb 7, 2025

What happened?

Relevant log output

Are you a ML Ops Team?

What LiteLLM version are you on ?

Twitter / LinkedIn details

minwhoo commented Feb 7, 2025