Improving Ollama nested model behavior

[andrewdmalone]

This is just a first attempt at improving how nested object models are handled via Ollama. Very unsure about whether or not this is the best approach, but I wanted to at least take a stab at it before asking for feedback.

Addresses #607, but nested model performance with Ollama could still be improved a lot beyond what this PR adds. This fixes issues where the Ollama response follows the specified data model but provides oddly-formatted JSON responses. It does not, however, address the fact that Ollama just doesn't seem to like adhering to the model to begin with.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[samuelcolvin]

Please can you explain what this is doing? Ideally with an example of what Ollama returns.

[andrewdmalone]

I outlined it in #607. I can put together a more robust example set tomorrow but in light of #242 I don't know if this is even worth merging at all. If you're considering using the Ollama Python library I imagine that would be a far more complete solution for both issues.

[samuelcolvin]

I'm no fan of the Ollama Python library, it's not properly tested and it's indented with 2 spaces I'd like to keep using the OpenAI SDK if we can.

[Ynn]

Thank you for taking the time to review this. Great work. However, even with this modification, I am unable to get the provided example to work. It seems, as mentioned in issue #242, that Ollama AI struggles to return something reliable.

I’ve used the Ollama API extensively for structured requests without encountering issues, but I understand that code quality issues are critical in this context.

Here is the full trace for the example I provided (It’s clear that on the second attempt, Ollama loses its way and produces random output.)

---

Full Trace

────────────────────────────────────────────────────────────────────────── Request 1 ──────────────────────────────────────────────────────────────────────────
Client 1 / http://localhost:11434/v1/chat/completions
{
  "messages": [
    {
      "role": "user",
      "content": "The windy city in the US of A."
    }
  ],
  "model": "llama3.1",
  "n": 1,
  "parallel_tool_calls": true,
  "stream": false,
  "tool_choice": "required",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "final_result",
        "description": "The final response which ends this conversation",
        "parameters": {
          "$defs": {
            "Country": {
              "properties": {
                "name": {
                  "title": "Name",
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "title": "Country",
              "type": "object"
            }
          },
          "properties": {
            "city": {
              "title": "City",
              "type": "string"
            },
            "country": {
              "$ref": "#/$defs/Country"
            }
          },
          "required": [
            "city",
            "country"
          ],
          "title": "MyModel",
          "type": "object"
        }
      }
    }
  ]
}
────────────────────────────────────────────────────────────────────────── Response 1 ──────────────────────────────────────────────────────────────────────────
Server 1 / http://localhost:11434/v1/chat/completions
{
  "id": "chatcmpl-268",
  "object": "chat.completion",
  "created": 1736272720,
  "model": "llama3.1",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_c1d0dsvd",
            "index": 0,
            "type": "function",
            "function": {
              "name": "final_result",
              "arguments": "{\"city\":\"Chicago\",\"country\":\"USA\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": {
    "prompt_tokens": 173,
    "completion_tokens": 23,
    "total_tokens": 196
  }
}
────────────────────────────────────────────────────────────────────────── Request 2 ──────────────────────────────────────────────────────────────────────────
Client 2 / http://localhost:11434/v1/chat/completions
{
  "messages": [
    {
      "role": "user",
      "content": "The windy city in the US of A."
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "call_c1d0dsvd",
          "type": "function",
          "function": {
            "name": "final_result",
            "arguments": "{\"city\":\"Chicago\",\"country\":\"USA\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_c1d0dsvd",
      "content": "1 validation errors: [\n  {\n    \"type\": \"model_type\",\n    \"loc\": [\n      \"country\"\n    ],\n    \"msg\": \"Input should be an object\",\n  }\n]"
    }
  ],
  "model": "llama3.1",
  "n": 1,
  "parallel_tool_calls": true,
  "stream": false,
  "tool_choice": "required",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "final_result",
        "description": "The final response which ends this conversation",
        "parameters": {
          "$defs": {
            "Country": {
              "properties": {
                "name": {
                  "title": "Name",
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "title": "Country",
              "type": "object"
            }
          },
          "properties": {
            "city": {
              "title": "City",
              "type": "string"
            },
            "country": {
              "$ref": "#/$defs/Country"
            }
          },
          "required": [
            "city",
            "country"
          ],
          "title": "MyModel",
          "type": "object"
        }
      }
    }
  ]
}
────────────────────────────────────────────────────────────────────────── Response 2 ──────────────────────────────────────────────────────────────────────────
Server 2 / http://localhost:11434/v1/chat/completions
{
  "id": "chatcmpl-430",
  "object": "chat.completion",
  "created": 1736272720,
  "model": "llama3.1",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<|python_tag|>import geopy.geocoders\n\ngeolocator = geopy.geocoders.Nominatim(user_agent=\"myGeocoder\")\n\nlocation = geolocator.geocode(\"The windy city\")"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 131,
    "completion_tokens": 49,
    "total_tokens": 180
  }
}

[andrewdmalone]

Closing this PR. After testing it more thoroughly I'm afraid it's just a bandaid fix at best and doesn't actually address the larger issue of Ollama ignoring tools (see #242).