OpenAI function calling support

[slundberg]

As a first step towards better tool-use support in Guidance version 0.0.64 includes function calling support. Initially it is tested just for OpenAI models, but is designed to work with all open models going forward as well. Here is an example in action:

import guidance

# define the chat model we want to use (must be a recent model supporting function calls)
guidance.llm = guidance.llms.OpenAI("gpt-3.5-turbo-0613", caching=False)

# define a guidance program that uses tools
program = guidance("""
{{~#system~}}
You are a helpful assistant.
{{>tool_def functions=functions}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#each range(10)~}}
    {{~#assistant~}}
    {{gen 'answer' max_tokens=50 function_call="auto"}}
    {{~/assistant~}}

    {{#if not callable(answer)}}{{break}}{{/if}}
    
    {{~#function name=answer.__name__~}}
    {{answer()}}
    {{~/function~}}
{{~/each~}}""")

# call the program, passing in the function definition we want to use as JSON
executed_program = program(functions=[
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        }
    }
], get_current_weather=get_current_weather)

For more details see the following work-in-progress notebook: (feedback is welcome!)
https://github.com/microsoft/guidance/blob/main/notebooks/art_of_prompt_design/tool_use.ipynb

[parisetflorian]

Amazing @slundberg ! Thanks for working on that; I can't wait to use it.

Am I missing something? I've copy-pasted the example from the notebook and got this:

ValueError                                Traceback (most recent call last)
Cell In[17], line 52
     28 program = guidance("""
     29 {{~#system~}}
     30 You are a helpful assistant.
   (...)
     47     {{[~/function](https://file+.vscode-resource.vscode-cdn.net/Users/parisetflorian/Documents/GitHub/Test-Scripts/Test%20Open%20AI%20Function/~/function)~}}
     48 {{[~/each](https://file+.vscode-resource.vscode-cdn.net/Users/parisetflorian/Documents/GitHub/Test-Scripts/Test%20Open%20AI%20Function/~/each)~}}""")
     51 # call the program, passing in the function definition we want to use as JSON
---> 52 executed_program = program(functions=[
     53     {
     54         "name": "get_current_weather",
     55         "description": "Get the current weather in a given location",
     56         "parameters": {
     57             "type": "object",
     58             "properties": {
     59                 "location": {
     60                     "type": "string",
     61                     "description": "The city and state, e.g. San Francisco, CA",
     62                 },
     63                 "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
     64             },
     65             "required": ["location"],
     66         }
     67     }
     68 ], get_current_weather=get_current_weather)

File [/usr/local/lib/python3.11/site-packages/guidance/_program.py:210](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/guidance/_program.py:210), in Program.__call__(self, **kwargs)
    201 new_program = Program(
    202     text=self.marked_text,
    203 
   (...)
    206     **{**{k: v if callable(v) else copy.deepcopy(v) for k,v in self._variables.items()}, **kwargs}
    207 )
    209 # create an executor for the new program (this also marks the program as executing)
--> 210 new_program._executor = ProgramExecutor(new_program)
    212 # if we are in async mode, schedule the program in the current event loop
    213 if new_program.async_mode:

File [/usr/local/lib/python3.11/site-packages/guidance/_program_executor.py:51](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/guidance/_program_executor.py:51), in ProgramExecutor.__init__(self, program)
     49     program._variables.update(update_variables)
     50     return out
---> 51 text = re.sub(r"{{>(.*?)}}", replace_partial, program._text)
     53 # parse the program text
     54 try:

File [/usr/local/Cellar/python](https://file+.vscode-resource.vscode-cdn.net/usr/local/Cellar/python)@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/re/__init__.py:185, in sub(pattern, repl, string, count, flags)
    178 def sub(pattern, repl, string, count=0, flags=0):
    179     """Return the string obtained by replacing the leftmost
    180     non-overlapping occurrences of the pattern in string by the
    181     replacement repl.  repl can be either a string or a callable;
    182     if a string, backslash escapes in it are processed.  If it is
    183     a callable, it's passed the Match object and must return
    184     a replacement string to be used."""
--> 185     return _compile(pattern, flags).sub(repl, string, count)

File [/usr/local/lib/python3.11/site-packages/guidance/_program_executor.py:37](https://file+.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/guidance/_program_executor.py:37), in ProgramExecutor.__init__..replace_partial(match)
     35 # ,args_string = match.group(1).split(" ", 1)
     36 if partial_name not in program._variables:
---> 37     raise ValueError("Partial '%s' not given in the keyword args:" % partial_name)
     38 out = "{{#block '"+partial_name+"'"
     39 if len(parts) > 1:

ValueError: Partial 'tool_def' not given in the keyword args:

[slundberg]

Are you on version 0.0.64 and using the OpenAI LLM?

[abhinavdayal]

I am using 0.0.64 with OpenAI LlM and see following error running this example

(venv) user:~/app$python functest.py            
Traceback (most recent call last):
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 109, in run
    await self.visit(self.parse_tree, VariableStack([self.program._variables], self))
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/library/_each.py", line 93, in each
    new_content += await parser.visit(
                   ^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/library/_assistant.py", line 13, in assistant
    return await role(role_name="assistant", hidden=hidden, _parser_context=_parser_context, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/library/_role.py", line 17, in role
    new_content += await parser.visit(
                   ^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 266, in visit
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 266, in <listcomp>
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/_program_executor.py", line 379, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "app/venv/lib/python3.11/site-packages/guidance/library/_gen.py", line 170, in gen
    generated_value += new_text
TypeError: can only concatenate str (not "NoneType") to str

Error in program:  can only concatenate str (not "NoneType") to str

[cb-ms]

Hi @slundberg ,

I'm on Guidance 0.0.64 + OpenAI LLM and seeing the same error as abhinavdayal when running the code from here in a single Python file (up to [4]). I see someone else has reported getting the same error: #272. Is there anything we can do to get past this error?

Thanks for your help and amazing work!

[parisetflorian]

@slundberg Yes, I updated to 0.0.64 before running your example.

Here's the code I copy/pasted from this thread:

# define a tool we would like the model to use
import json

def get_current_weather(location, unit="fahrenheit"):
    """ Get the current weather in a given location.
    
    Parameters
    ----------
    location : string
        The city and state, e.g. San Francisco, CA
    unit : "celsius" or "fahrenheit"
    """
    weather_info = {
        "location": location,
        "temperature": "71",
        "unit": unit,
        "forecast": ["sunny", "windy"],
    }
    return json.dumps(weather_info)


import guidance

# define the chat model we want to use (must be a recent model supporting function calls)
guidance.llm = guidance.llms.OpenAI("gpt-3.5-turbo-0613", caching=False)

# define a guidance program that uses tools
program = guidance("""
{{~#system~}}
You are a helpful assistant.
{{>tool_def functions=functions}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#each range(10)~}}
    {{~#assistant~}}
    {{gen 'answer' max_tokens=50 function_call="auto"}}
    {{~/assistant~}}

    {{#if not callable(answer)}}{{break}}{{/if}}
    
    {{~#function name=answer.__name__~}}
    {{answer()}}
    {{~/function~}}
{{~/each~}}""")


# call the program, passing in the function definition we want to use as JSON
executed_program = program(functions=[
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        }
    }
], get_current_weather=get_current_weather)

[xnohat]

Hi @slundberg,
Does we have any fix for this feature ?
I got same error

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 109, in run
    await self.visit(self.parse_tree, VariableStack([self.program._variables], self))
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "/opt/conda/lib/python3.10/site-packages/guidance/library/_assistant.py", line 13, in assistant
    return await role(role_name="assistant", hidden=hidden, _parser_context=_parser_context, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/guidance/library/_role.py", line 17, in role
    new_content += await parser.visit(
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 266, in visit
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 266, in <listcomp>
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
  File "/opt/conda/lib/python3.10/site-packages/guidance/_program_executor.py", line 379, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "/opt/conda/lib/python3.10/site-packages/guidance/library/_gen.py", line 170, in gen
    generated_value += new_text
TypeError: can only concatenate str (not "NoneType") to str

Error in program:  can only concatenate str (not "NoneType") to str

[xnohat]

I found workaround for this error

set guidance silent=False to make function call working. I still debug why this feature not working in silent mode

[xnohat]

I propose a fix for this error here #315