[Question] Use a locally deployed LLM whose service interfaces follow openai's standards.

[chenk-gd]

Description

Use a locally deployed LLM whose service interfaces follow openai's standards.

llm_config = LLM(
  model='openai/qwen72b',
  api_key="None",
  base_url="http://xxx.xxx.xxx.xxx:8000/v1",
)
...
crew = Crew(
    agents=[researcher, writer],
    tasks=[task1, task2],
    verbose=True,
    memory=True,
    planning=True,  # Enable planning feature for the crew
    embedder=embedder_config,
    planning_llm=llm_config,
    function_calling_llm=llm_config,
)

if crew's parameter 'planning' is set to true, got the following error：
instructor.exceptions.InstructorRetryException: litellm.AuthenticationError: AuthenticationError: OpenAIException - Error code: 401 - {'error': {'message': 'Incorrect API key provided: None. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

Steps to Reproduce

The complete code is as follows

Expected behavior

Ability to use locally deployed models

Screenshots/Code snippets

import os

from crewai import LLM, Agent, Task, Crew
from crewai_tools import SerperDevTool

os.environ["SERPER_API_KEY"] = "xxxx"  # serper.dev API key

llm_config = LLM(
  model='openai/qwen72b',
  api_key="None",
  base_url="http://xxx.xxx.xxx.xxx:8000/v1",
)

embedder_config = {
    "provider": "azure",
    "config": {
        "api_key": "xxxx",
        "api_version": "2024-02-15-preview",
        "api_base": "https://xxx.openai.azure.com/",
        "api_type": 'azure',
        "model": "text-embedding-3-small",
    },
}

# Loading Tools
search_tool = SerperDevTool()

# Define your agents with roles, goals, tools, and additional attributes
researcher = Agent(
    role='Senior Research Analyst',
    goal='Uncover cutting-edge developments in AI and data science',
    backstory=(
        "You are a Senior Research Analyst at a leading tech think tank. "
        "Your expertise lies in identifying emerging trends and technologies in AI and data science. "
        "You have a knack for dissecting complex data and presenting actionable insights."
    ),
    verbose=True,
    allow_delegation=False,
    tools=[search_tool],
    llm=llm_config,  # Use custom LLM for this agent
)
writer = Agent(
    role='Tech Content Strategist',
    goal='Craft compelling content on tech advancements',
    backstory=(
        "You are a renowned Tech Content Strategist, known for your insightful and engaging articles on technology and innovation. "
        "With a deep understanding of the tech industry, you transform complex concepts into compelling narratives."
    ),
    verbose=True,
    allow_delegation=True,
    tools=[search_tool],
    cache=False,  # Disable cache for this agent
    llm=llm_config,  # Use custom LLM for this agent
)

# Create tasks for your agents
task1 = Task(
    description=(
        "Conduct a comprehensive analysis of the latest advancements in AI in 2024. "
        "Identify key trends, breakthrough technologies, and potential industry impacts. "
        "Compile your findings in a detailed report. "
        "Make sure to check with a human if the draft is good before finalizing your answer."
    ),
    expected_output='A comprehensive full report on the latest AI advancements in 2024, leave nothing out',
    agent=researcher,
    human_input=True
)

task2 = Task(
    description=(
        "Using the insights from the researcher\'s report, develop an engaging blog post that highlights the most significant AI advancements. "
        "Your post should be informative yet accessible, catering to a tech-savvy audience. "
        "Aim for a narrative that captures the essence of these breakthroughs and their implications for the future."
    ),
    expected_output='A compelling 3 paragraphs blog post formatted as markdown about the latest AI advancements in 2024',
    agent=writer,
    human_input=True
)

# Instantiate your crew with a sequential process
crew = Crew(
    agents=[researcher, writer],
    tasks=[task1, task2],
    verbose=True,
    memory=True,
    planning=True,  # Enable planning feature for the crew
    embedder=embedder_config,
    planning_llm=llm_config,
    function_calling_llm=llm_config,
)

# Get your crew to work!
result = crew.kickoff()

print("######################")
print(result)

Operating System

Windows 11

Python Version

3.11

crewAI Version

0.98.0

crewAI Tools Version

N/A

Virtual Environment

Venv

Evidence

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\workspace\PythonProjects\fe-crewai\.venv\Lib\site-packages\crewai\utilities\converter.py", line 27, in to_pydantic
    return self._create_instructor().to_pydantic()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\workspace\PythonProjects\fe-crewai\.venv\Lib\site-packages\crewai\utilities\internal_instructor.py", line 40, in to_pydantic
    model = self._client.chat.completions.create(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\workspace\PythonProjects\fe-crewai\.venv\Lib\site-packages\instructor\client.py", line 176, in create
    return self.create_fn(
           ^^^^^^^^^^^^^^^
  File "C:\workspace\PythonProjects\fe-crewai\.venv\Lib\site-packages\instructor\patch.py", line 193, in new_create_sync
    response = retry_sync(
               ^^^^^^^^^^^
  File "C:\workspace\PythonProjects\fe-crewai\.venv\Lib\site-packages\instructor\retry.py", line 181, in retry_sync
    raise InstructorRetryException(
instructor.exceptions.InstructorRetryException: litellm.AuthenticationError: AuthenticationError: OpenAIException - Error code: 401 - {'error': {'message': 'Incorrect API key provided: None. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

Possible Solution

None

Additional context

None

[Jean-Diniz]

I think is the same of #2033.
The PR is already accepted, but I didn't see a new version.