Structured Outputs vs Function Calling - Performance & Accuracy Analysis

[shreyashankar]

Context

We currently implement extraction tasks using function calling, but some APIs support structured outputs. We need data to make an informed decision between these approaches.

Proposed Methodology

1. Create Synthetic Test Dataset

• Take Wikipedia articles as base text (e.g., articles about cities, history)
• Randomly insert known car names throughout the text (from a predefined list of 100+ cars)
• Generate 50+ documents with varying:
  • Document length (short/medium/long)
  • Density of car mentions
  • Complexity of surrounding context
• Store ground truth: exact positions and names of inserted cars

2. Benchmark Implementation

Test both approaches.
Measure:

• Latency (p50, p90, p99)
• Token usage
• Accuracy (precision/recall against ground truth)
• Error rates/edge cases
• Cost implications

3. Model Support Documentation

Create compatibility matrix, like as follows (this might not be accurate:

Model	Function Calling	Structured Outputs
GPT-4o	✓	✓
GPT-4o-mini	✓	✓
Claude	✓	✓
Llama-2	?	?
Mistral	?	?
[Add others]

Action Items

1. Create data generation script
2. Implement both extraction methods
3. Build benchmarking harness
4. Run tests across different models
5. Document findings & recommendations
6. Update codebase to use optimal method with fallback

Expected Output

• Public benchmark results & methodology
• Clear recommendation with data backing
• PR to update implementation based on findings

[shreyashankar]

Closed via #291