Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Inconsistent stream output between OpenAI and LiteLLM clients during tool calling #8012

Open
VicentGJ opened this issue Jan 26, 2025 · 3 comments · May be fixed by #8429
Open

[Bug]: Inconsistent stream output between OpenAI and LiteLLM clients during tool calling #8012

VicentGJ opened this issue Jan 26, 2025 · 3 comments · May be fixed by #8429
Labels
bug Something isn't working feb 2025 mlops user request openai

Comments

@VicentGJ
Copy link

What happened?

I have been working on modifying the Swarm agents library to integrate with LiteLLM, allowing for the use of multiple other LLMs instead of being limited to OpenAI's. The modification required was minimal; I only needed to change line 70 in core.py from return self.client.chat.completions.create(**create_params) to return completion(**create_params).

However, I encountered an issue when using streaming to run the agents. The error is as follows:

Traceback (most recent call last):
  File "/Users/vicente/Developer/agents-test/swarm_test/main.py", line 53, in <module>
    for chunk in stream:
                 ^^^^^^
  File "/Users/vicente/Developer/agents-test/swarm_test/swarm/core.py", line 215, in run_and_stream
    tool_call_object = ChatCompletionMessageToolCall(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vicente/Developer/agents-test/.venv/lib/python3.12/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatCompletionMessageToolCall
type
  Input should be 'function' [type=literal_error, input_value='functionfunctionfunction...unctionfunctionfunction', input_type=str]

The issue appears to be related to the fact that the OpenAI client outputs the tool's type only once in the stream. Consequently, only the first chunk of the tool call contains that property, as illustrated below:

{'content': None, 'function_call': None, 'refusal': None, 'role': None, 'tool_calls': [{'index': 1, 'id': 'call_hFCNIDFOnRANhrZsXH8V2fhy', 'function': {'arguments': '', 'name': 'web_search'}, 'type': 'function'}]}
{'content': None, 'function_call': None, 'refusal': None, 'role': None, 'tool_calls': [{'index': 1, 'id': None, 'function': {'arguments': '{"qu', 'name': None}, 'type': None}]}

In contrast, LiteLLM seems to output the tool type in every chunk of the stream:

{'id': 'call_Yy84QjsA2lMv66k9R0tEcCDW', 'function': {'arguments': '', 'name': 'web_search'}, 'type': 'function', 'index': 2}], 'audio': None, 'sender': 'Researcher'}
{'refusal': None, 'content': None, 'role': 'assistant', 'function_call': None, 'tool_calls': [{'id': None, 'function': {'arguments': '{"qu', 'name': None}, 'type': 'function', 'index': 2}], 'audio': None, 'sender': 'Researcher'}

This discrepancy was straightforward to address on my end. However, considering that LiteLLM aims to provide an OpenAI-compatible client, resolving this minor compatibility issue could be advantageous.
Interestingly, the 'assistant' property also appears only once in the first chunk of the OpenAI stream, whereas it is present in every chunk of the LiteLLM stream. While this behavior did not cause issues in my specific use case, it may be worth investigating further.

Relevant log output



Are you a ML Ops Team?


Yes


What LiteLLM version are you on ?


v1.59.8


Twitter / LinkedIn details


No response

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @VicentGJ
VicentGJ




          added
  the 

  bug

  Something isn't working
 label


      Jan 26, 2025

    

  



    







      

  
      



  

  @ishaan-jaff





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

  





      


        


  
    

      
          
hi @VicentGJ can you give me a litellm.completion request to repro this problem ?

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            







      

  
      



  

  @ishaan-jaff





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

  





      


        


  
    

      
          
this is being picked up by @Kaushikdkrikhanu

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
          

  

    
  


  

      

    @Kaushikdkrikhanu
Kaushikdkrikhanu



    mentioned this issue
    
      Feb 9, 2025
    





  


  

      
   Closed

  







  









      

  
      



  

  @Kaushikdkrikhanu





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

  





      


        


  
    

      
          
I think the observation here is that, if function id is none then type is also none.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
          

  

    
  


  

      

    @Kaushikdkrikhanu
Kaushikdkrikhanu



    linked a pull request
    
      Feb 10, 2025
    

    that will
    close
    this issue





  


  

      
   Open

  







  













  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    bug

  Something isn't working
  

    feb 2025

  

    mlops user request

  

    openai









      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  



      

    

  
      

  

    

      3 participants
    

    

        
          @ishaan-jaff 
        
          @VicentGJ 
        
          @Kaushikdkrikhanu