[feature request] Add support for schema definition with tools/function calling

[nobilelucifero]

Is your feature request related to a problem? Please describe.
I was playing with Scrapegraph, and I wanted to define my structured output using tools both in the SmartScraperGraph() function and the graph_config configuration object.

Describe the solution you'd like
Is there a way to do something like this currently?

chat_response = chat_completion_request(
    messages, tools=tools, tool_choice=...
)
...

Describe alternatives you've considered
Besides re-prompting the initial output, I've tried:

graph_config = {
    "llm": {
        "api_key": OPENAI_API_KEY,
        "model": "gpt-3.5-turbo",
        "temperature": 0,
        "model_kwargs": {
          "tools": tools,
          "tool_choice" : {"type": "function", "function": {"name": "get_tov"}}
        }
    },
}

and then

...
result = smart_scraper_graph.run()

Which returns the following error:

JSONDecodeError                           Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/langchain_core/output_parsers/json.py](https://localhost:8080/#) in parse_result(self, result, partial)
     65             try:
---> 66                 return parse_json_markdown(text)
     67             except JSONDecodeError as e:

15 frames
[/usr/local/lib/python3.10/dist-packages/langchain_core/utils/json.py](https://localhost:8080/#) in parse_json_markdown(json_string, parser)
    146             json_str = match.group(2)
--> 147     return _parse_json(json_str, parser=parser)
    148 

[/usr/local/lib/python3.10/dist-packages/langchain_core/utils/json.py](https://localhost:8080/#) in _parse_json(json_str, parser)
    159     # Parse the JSON string into a Python dictionary
--> 160     return parser(json_str)
    161 

[/usr/local/lib/python3.10/dist-packages/langchain_core/utils/json.py](https://localhost:8080/#) in parse_partial_json(s, strict)
    119     # for the original string.
--> 120     return json.loads(s, strict=strict)
    121 

[/usr/lib/python3.10/json/__init__.py](https://localhost:8080/#) in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    358         kw['parse_constant'] = parse_constant
--> 359     return cls(**kw).decode(s)

[/usr/lib/python3.10/json/decoder.py](https://localhost:8080/#) in decode(self, s, _w)
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()

[/usr/lib/python3.10/json/decoder.py](https://localhost:8080/#) in raw_decode(self, s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

OutputParserException                     Traceback (most recent call last)
[<ipython-input-22-2af99396d10b>](https://localhost:8080/#) in <cell line: 1>()
----> 1 result = smart_scraper_graph.run()

[/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/smart_scraper_graph.py](https://localhost:8080/#) in run(self)
    107 
    108         inputs = {"user_prompt": self.prompt, self.input_key: self.source}
--> 109         self.final_state, self.execution_info = self.graph.execute(inputs)
    110 
    111         return self.final_state.get("answer", "No answer found.")

[/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py](https://localhost:8080/#) in execute(self, initial_state)
    104 
    105             with get_openai_callback() as cb:
--> 106                 result = current_node.execute(state)
    107                 node_exec_time = time.time() - curr_time
    108                 total_exec_time += node_exec_time

[/usr/local/lib/python3.10/dist-packages/scrapegraphai/nodes/generate_answer_node.py](https://localhost:8080/#) in execute(self, state)
    144             # Chain
    145             single_chain = list(chains_dict.values())[0]
--> 146             answer = single_chain.invoke({"question": user_prompt})
    147 
    148         # Update the state with the generated answer

[/usr/local/lib/python3.10/dist-packages/langchain_core/runnables/base.py](https://localhost:8080/#) in invoke(self, input, config)
   2497         try:
   2498             for i, step in enumerate(self.steps):
-> 2499                 input = step.invoke(
   2500                     input,
   2501                     # mark each step as a child run

[/usr/local/lib/python3.10/dist-packages/langchain_core/output_parsers/base.py](https://localhost:8080/#) in invoke(self, input, config)
    167     ) -> T:
    168         if isinstance(input, BaseMessage):
--> 169             return self._call_with_config(
    170                 lambda inner_input: self.parse_result(
    171                     [ChatGeneration(message=inner_input)]

[/usr/local/lib/python3.10/dist-packages/langchain_core/runnables/base.py](https://localhost:8080/#) in _call_with_config(self, func, input, config, run_type, **kwargs)
   1624             output = cast(
   1625                 Output,
-> 1626                 context.run(
   1627                     call_func_with_variable_args,  # type: ignore[arg-type]
   1628                     func,  # type: ignore[arg-type]

[/usr/local/lib/python3.10/dist-packages/langchain_core/runnables/config.py](https://localhost:8080/#) in call_func_with_variable_args(func, input, config, run_manager, **kwargs)
    345     if run_manager is not None and accepts_run_manager(func):
    346         kwargs["run_manager"] = run_manager
--> 347     return func(input, **kwargs)  # type: ignore[call-arg]
    348 
    349 

[/usr/local/lib/python3.10/dist-packages/langchain_core/output_parsers/base.py](https://localhost:8080/#) in <lambda>(inner_input)
    168         if isinstance(input, BaseMessage):
    169             return self._call_with_config(
--> 170                 lambda inner_input: self.parse_result(
    171                     [ChatGeneration(message=inner_input)]
    172                 ),

[/usr/local/lib/python3.10/dist-packages/langchain_core/output_parsers/json.py](https://localhost:8080/#) in parse_result(self, result, partial)
     67             except JSONDecodeError as e:
     68                 msg = f"Invalid json output: {text}"
---> 69                 raise OutputParserException(msg, llm_output=text) from e
     70 
     71     def parse(self, text: str) -> Any:

OutputParserException: Invalid json output:

Additional context
This guide explains it better than I could:
https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models

[DiTo97]

@nobilelucifero could you share the full relevant code excerpt which you tried running and how should it look like to achieve tools customizability?

will think how we might improve that area.

[VinciGit00]

Wi, we are defining something similar, stay tuned!

[nobilelucifero]

@nobilelucifero could you share the full relevant code excerpt which you tried running and how should it look like to achieve tools customizability?

will think how we might improve that area.

Amazing! This is the code (more or less because I've been playing with it) without using Scrapegraph:

# %%capture
# !pip install openai

import requests
import json

from google.colab import userdata
from openai import OpenAI

llm_client = OpenAI(
    api_key="YOUR_OPENAI_KEY"
)

"""Set up tools"""

def get_tov(audience, purpose):
  result = {
      "audience": audience,
      "purpose": purpose
  }

  return json.dumps(result)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_tov",
            "description": "Get the Tone of Voice of a content source piece written by the author themselves.",
            "parameters": {
                "type": "object",
                "properties": {
                    "audience": {
                        "type": "string",
                        "description": "Defines who are the primary audiences or target demographics of the input. Each item will be a 2-3 word description.",
                    },
                    "purpose": {
                        "type": "string",
                        "description": "What's the main purpose or goal of the text?",
                    },
                },
                "required": ["audience", "purpose"],
            },
        }
    }
]

tool_choice = {"type": "function", "function": {"name": "get_tov"}}

def converse(messages):
    response = llm_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=messages,
        tools=tools,
        tool_choice="auto"
        # tool_choice=tool_choice
    )

    response_message = response.choices[0].message
    tool_calls = response_message.tool_calls

    if tool_calls:
      messages.append(response_message)

      available_functions = {
          "get_tov": get_tov
      }

      for tool_call in tool_calls:
        print(f"Function: {tool_call.function.name}")
        print(f"Params: {tool_call.function.arguments}")

        function_name = tool_call.function.name
        function_to_call = available_functions[function_name]
        function_args = json.loads(tool_call.function.arguments)
        function_response = function_to_call(
            audience = function_args.get("audience"),
            purpose = function_args.get("purpose"),
        )
        print(f"Tool reponse: {function_response}")

        messages.append({
            "tool_call_id": tool_call.id,
            "role": "tool",
            "name": function_name,
            "content": function_response
        })

        second_reponse = llm_client.chat.completions.create(
            model = "gpt-3.5-turbo",
            messages = messages,
        )

        return second_reponse.choices[0].message.content


    # return response.choices[0].message;

input = """
A poor Woodman was cutting down a tree near the edge of a deep pool in the forest. It was late in the day and the Woodman was tired. He had been working since sunrise and his strokes were not so sure as they had been early that morning. Thus it happened that the axe slipped and flew out of his hands into the pool.

The Woodman was in despair. The axe was all he possessed with which to make a living, and he had not money enough to buy a new one. As he stood wringing his hands and weeping, the god Mercury suddenly appeared and asked what the trouble was. The Woodman told what had happened, and straightway the kind Mercury dived into the pool. When he came up again he held a wonderful golden axe.

"Is this your axe?" Mercury asked the Woodman.

"No," answered the honest Woodman, "that is not my axe."

Mercury laid the golden axe on the bank and sprang back into the pool. This time he brought up an axe of silver, but the Woodman declared again that his axe was just an ordinary one with a wooden handle.

Mercury dived down for the third time, and when he came up again he had the very axe that had been lost.

The poor Woodman was very glad that his axe had been found and could not thank the kind god enough. Mercury was greatly pleased with the Woodman's honesty.

"I admire your honesty," he said, "and as a reward you may have all three axes, the gold and the silver as well as your own."

The happy Woodman returned to his home with his treasures, and soon the story of his good fortune was known to everybody in the village. Now there were several Woodmen in the village who believed that they could easily win the same good fortune. They hurried out into the woods, one here, one there, and hiding their axes in the bushes, pretended they had lost them. Then they wept and wailed and called on Mercury to help them.

And indeed, Mercury did appear, first to this one, then to that. To each one he showed an axe of gold, and each one eagerly claimed it to be the one he had lost. But Mercury did not give them the golden axe. Oh no! Instead he gave them each a hard whack over the head with it and sent them home. And when they returned next day to look for their own axes, they were nowhere to be found.

Honesty is the best policy.
"""

result = converse(messages=[
      {
        "role": "system",
        "content": "You are a savvy copywriter for SEO, Social Media, and Blogs."
      }, {
         "role": "user",
         "content": "What's the tone of voice of this text?"
      }, {
          "role": "user",
          "content": input,
      },
])

print(result)