Bearer Authorization with Open AIlike

[mackenziedott]

My local llm is now behind a litellm proxy with a Bearer Authorization in the form of 'Authorization':'Bearer SECRETKEY'} in headers. How would i add this to OpenAIlike?

[dosubot[bot]]

To add Bearer Authorization to OpenAIlike for a local LLM behind a litellm proxy, you can modify the _get_credential_kwargs method to include the Bearer token in the headers. Here is an example of how you can do it:

def _get_credential_kwargs(self, **kwargs: Any) -> Dict[str, Any]:
    if self.use_azure_ad:
        self._azure_ad_token = refresh_openai_azuread_token(self._azure_ad_token)
        self.api_key = self._azure_ad_token.token

    headers = {
        "Authorization": f"Bearer {self.api_key}",
        **self.default_headers,
    }

    return {
        "api_key": self.api_key,
        "max_retries": self.max_retries,
        "timeout": self.timeout,
        "azure_endpoint": self.azure_endpoint,
        "azure_deployment": self.azure_deployment,
        "api_version": self.api_version,
        "default_headers": headers,
        "http_client": self._http_client,
        **kwargs,
    }

Additionally, if you are using the LlamaIndex framework and need to pass headers while initializing an embedding model, you can do so by specifying the headers in the model initialization. Here is an example:

from llama_index.embeddings.openai import OpenAIEmbedding

self.llm = OpenAIEmbedding(
    model=self.model_name,
    temperature=self.temperature,
    max_tokens=self.max_tokens,
    frequency_penalty=self.frequency_penalty,
    top_p=self.top_p,
    headers={
        "Authorization": f"Bearer {self.api_key}",
        **self.default_headers,
    }
)

# LLM Predictor
self.llm_predictor = LLMPredictor(llm=self.llm)

This modification ensures that the Bearer token is included in the headers for authorization when making requests to the local LLM behind the litellm proxy, and it also shows how to pass headers when initializing an embedding model in LlamaIndex ^[1][2].

To continue talking to Dosu, mention @dosu.

[mackenziedott]

after updating a CustomOpenAILike class with that function, running llm.complete gets me this error
`File ~/miniconda3/envs/testRequests/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py:260, in Dispatcher.span..wrapper(func, instance, args, kwargs)
252 self.span_enter(
253 id_=id_,
254 bound_args=bound_args,
(...)
257 tags=tags,
258 )
259 try:
--> 260 result = func(*args, **kwargs)
261 except BaseException as e:
262 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

File ~/miniconda3/envs/testRequests/lib/python3.12/site-packages/llama_index/llms/openai_like/base.py:88, in OpenAILike.complete(self, prompt, formatted, **kwargs)
85 if not formatted:
86 prompt = self.completion_to_prompt(prompt)
---> 88 return super().complete(prompt, **kwargs)

File ~/miniconda3/envs/testRequests/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py:260, in Dispatcher.span..wrapper(func, instance, args, kwargs)
252 self.span_enter(
253 id_=id_,
...
(...)
19 **kwargs,
20 }

TypeError: 'NoneType' object is not a mapping
`

[dosubot[bot]]

The "certificate verify failed: self-signed certificate in certificate chain" issue occurs because the SSL certificate verification fails when the server's certificate is self-signed or not trusted by the client. This can happen if LlamaIndex is not configured to use the default certificate or if the default certificate is not trusted.

To resolve this issue, you can configure the requests session to ignore SSL certificate verification by setting verify=False in the post request. Here is an example of how you can modify the _send_sambaverse_request method to disable SSL verification:

def _send_sambaverse_request(
    self, endpoint: str, data: Dict[str, Any], stream: bool = False
) -> requests.Response:
    url = self._get_full_url(endpoint)
    headers = {
        "key": self.sambaverse_api_key,
        "Content-Type": "application/json",
        "modelName": self.sambaverse_model_name,
    }
    try:
        return self._client.post(url, headers=headers, json=data, stream=stream, verify=False)
    except Exception as e:
        raise ValueError(f"Error sending request to Sambaverse: {e}")

By setting verify=False, the SSL certificate verification is disabled, which should resolve the "certificate verify failed" issue. However, please note that this approach is not recommended for production environments due to security risks ^[1].

To continue talking to Dosu, mention @dosu.

[mackenziedott]

@dosu how can i configure llamaindex to use the default cert?

[dosubot[bot]]

To configure LlamaIndex to use the default certificate provided by certifi for SSL verification, you can modify the get_client and get_aclient functions to include the certifi certificate bundle. Here is how you can do it:

import os
import certifi
from typing import Optional, TYPE_CHECKING

from llama_index.core.constants import (
    DEFAULT_APP_URL,
    DEFAULT_BASE_URL,
)

if TYPE_CHECKING:
    from llama_cloud.client import AsyncLlamaCloud, LlamaCloud


def get_client(
    api_key: Optional[str] = None,
    base_url: Optional[str] = None,
    app_url: Optional[str] = None,
    timeout: int = 60,
) -> "LlamaCloud":
    """Get the sync platform API client."""
    from llama_cloud.client import LlamaCloud

    base_url = base_url or os.environ.get("LLAMA_CLOUD_BASE_URL", DEFAULT_BASE_URL)
    app_url = app_url or os.environ.get("LLAMA_CLOUD_APP_URL", DEFAULT_APP_URL)
    api_key = api_key or os.environ.get("LLAMA_CLOUD_API_KEY", None)

    return LlamaCloud(base_url=base_url, token=api_key, timeout=timeout, verify=certifi.where())


def get_aclient(
    api_key: Optional[str] = None,
    base_url: Optional[str] = None,
    app_url: Optional[str] = None,
    timeout: int = 60,
) -> "AsyncLlamaCloud":
    """Get the async platform API client."""
    from llama_cloud.client import AsyncLlamaCloud

    base_url = base_url or os.environ.get("LLAMA_CLOUD_BASE_URL", DEFAULT_BASE_URL)
    app_url = app_url or os.environ.get("LLAMA_CLOUD_APP_URL", DEFAULT_APP_URL)
    api_key = api_key or os.environ.get("LLAMA_CLOUD_API_KEY", None)

    return AsyncLlamaCloud(base_url=base_url, token=api_key, timeout=timeout, verify=certifi.where())

This modification ensures that the LlamaCloud and AsyncLlamaCloud clients use the certificate bundle provided by certifi for SSL verification, which should resolve the "certificate verify failed: self-signed certificate in certificate chain" issue ^[1].

To continue talking to Dosu, mention @dosu.

[mackenziedott]

@dosu how do i do this for OpenAILike

[dosubot[bot]]

To configure the OpenAILike class in LlamaIndex to use the default certificate provided by certifi for SSL verification, you can follow these steps:

1. Import httpx and certifi.
2. Create httpx clients with SSL verification using certifi.
3. Pass these clients to the OpenAILike class.

Here is an example code snippet:

import httpx
import certifi
from llama_index.embeddings.openai import OpenAIEmbedding

# Create httpx clients with SSL verification using certifi
http_client = httpx.Client(verify=certifi.where())
async_http_client = httpx.AsyncClient(verify=certifi.where())

# Configure the OpenAILike class
openai_embedding = OpenAIEmbedding(
    api_key="your_openai_api_key",
    http_client=http_client,
    async_http_client=async_http_client,
    # other parameters as needed
)

This will ensure that the OpenAILike class uses the default certificate provided by certifi for SSL verification ^[1].

To continue talking to Dosu, mention @dosu.

[ishaan-jaff]

Hi @mackenziedott , thanks for using LiteLLM. Any chance we can hop on a call to learn how we can improve LiteLLM Proxy for you ?

We’re planning roadmap and I’d love to get your feedback

• my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
• my linkedin if you prefer DMs: https://www.linkedin.com/in/reffajnaahsi/

[tan-yong-sheng]

I am planning to replicate other people's llama-index research agent by switching llamaindex OpenAI into llamaindex OpenAILike via LiteLLM proxy. The original idea is from pyquantnews blog: https://www.pyquantnews.com/the-pyquant-newsletter/build-powerful-ai-agent-makes-research-reports.

For llama-index OpenAI, it did work successfully as below:

However, for llama-index OpenAILike, although I could verify the LiteLLM connection is working, as shown below:

the agent is not working as below, as it didn't output anything (or error code). I'm surprised because I just switch from llama-index OpenAI to llama-index OpenAILike only....

Here are the examples of the jupyter notebook files I've mentioned above. (Note: need to change their file format from .txt to .ipynb for these two files)

• Using llama-index OpenAI: 133_ReportAgent.txt
• using llama-index OpenAILike with LiteLLM proxy: litellm-failed-133_ReportAgent.txt

Hope for help. Thanks a lot for the tools.