[Feature]: aiohttp migration - 10-100x Higher RPS Master ticket

[ishaan-jaff]

The Feature

Creating this ticket to breakdown the larger problem

• Ensure litellm proxy can hit 2k RPS, at 140ms median latency with 4 instances
• Use provider config request / response transformations in BaseLLMAIOHTTPHandler
• Add streaming support for aiohttp_openai/
• Add testing for using aiohttp_openai with litellm router
• Migrate BaseLLMHTTPHandler to use BaseLLMAIOHTTPHandler
• Migrate openai_like to BaseLLMAIOHTTPHandler
• Migrate openai, azure to BaseLLMAIOHTTPHandler
• Migrate huggingface/ to BaseLLMAIOHTTPHandler
• Migrate anthropic/ to BaseLLMAIOHTTPHandler

Motivation, pitch

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[mirodrr2]

Does this need to be done for bedrock? Or is bedrock support already implemented in an async way?

[ishaan-jaff]

bedrock is already async @mirodrr2 - but all providers are going to be migrated to aiohttp. So yes needs to be done for bedrock too. Can you help @mirodrr2 ?

It's a bit involed so happy to get on a call and discuss too https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat