Json output parser for Mixtral 8

[trnq-eu]

Hi.
I want to reply this tutoria using Mixtral 8 instead of GPT 3.5: https://haystack.deepset.ai/tutorials/28_structured_output_with_loop

Problem is that I keep getting an error because it seems I get an invalid json from the language model.

Errors are always like: Error from OutputValidator: Unterminated string starting at: line 5 column 15 (char 37)
OutputValidator at Iteration 14: Invalid JSON from LLM - Let's try again.
Output from LLM:

"cities": [
{
"name": "Berl
Error from OutputValidator: Unterminated string starting at: line 5 column 15 (char 37)
OutputValidator at Iteration 15: Invalid JSON from LLM - Let's try again.

Looks like Mixtral is unable to correctly produce a json output from this very simple string of text, but it seems improbable to me since I already managed to get a json output from Mixtral (using a retriever, for example), even though I never manage to validate the output against a json schema.

Here's a chunk of my code. The HuggingFaceTGIGenerator seems to be working fine

from haystack.components.generators import HuggingFaceTGIGenerator

from getpass import getpass

HF_TOKEN = getpass('HuggingFace Token:')

generator = HuggingFaceTGIGenerator(model="mistralai/Mixtral-8x7B-Instruct-v0.1", token=HF_TOKEN)

from haystack import Pipeline

pipeline = Pipeline(max_loops_allowed=5)

pipeline.add_component(instance=prompt_builder, name="prompt_builder")
pipeline.add_component(instance=generator, name="llm")
pipeline.add_component(instance=output_validator, name="output_validator")

pipeline.connect("prompt_builder", "llm")
pipeline.connect("llm", "output_validator")

pipeline.connect("output_validator.invalid_replies", "prompt_builder.invalid_replies")
pipeline.connect("output_validator.error_message", "prompt_builder.error_message")

[BadrinathMJ]

@trnq-eu I facing same issue. Did you get any solution ?

[trnq-eu]

Unfortunately no.