Need help improving the answer result of haystack

[felixriehm]

Hey everyone

i got two questions about the results of my chatbot application that i have built with haystack. My haystack setup follows the tutorials and does not have any special customization to it (once i described it another GitHub discussion)).

I have 40 plain text documents that are of type explanatory or descriptive. When i query haystack with questions that are definitely answerable and can be found in the text i have two problems:

• in most cases the score of the answers are extremely low (in range 0.1 to 0.6) even if the answer is correct
• the answers are in most cases too short

Example: 
   question: "Does the brown fox jump over the fence and climb a tree?"
   answer: "The fox jumps over the fence"
   score: 0.4
   context: "... (more text) ... The brown fox jumps over the fence and climbs a tree. ... (more text) ..."

Expectation: "The brown fox jumps over the fence and climbs a tree."

How do i get higher scores for the answers (without the annotation tool)? And how can i improve the length of the answers (without a GenerativeQA pipeline, or would that in general incorporate more information into the answer, aside from the boilerplate text?)?

Best,
Felix

[bglearning]

Hi @felixriehm ,

Just to confirm (based on the previous discussions), you now have two pipelines? one for general docs, one for FAQ.
Guessing these questions are for the document pipeline. Which retriever (model) and reader (model) are you using currently?

[bglearning]

Hi @felixriehm
I'm afraid getting longer answers without fine-tuning on your data might be tricky as many of the models (as above with roberta-base-squad2) are trained on SQuAD which usually doesn't have long answers.

There is LFQA as a possible option but it's a GenerativeQA pipeline.

Maybe another option is to consider some UX possibility of mitigating the answers being shortened. E.g. highlight the answer but also show context up to a certain length around the answer. But I realize it might be a bit tricky for a chat application.

As an aside, the example question above is a Yes/No one but expects an answer for a wh-question. So it might throw off the model. Of course I understand it's more for demonstration and not exactly representative of the QAs you are working with.

[felixriehm]

I'm afraid getting longer answers without fine-tuning on your data might be tricky as many of the models (as above with roberta-base-squad2) are trained on SQuAD which usually doesn't have long answers.

What other datasets are more suitable for longer answers? Can you refer one on huggingface.co? Roberta seems state of the art, I would just grab roberta with a different dataset and try, if there are more suitable datasets.

[bglearning]

I'm not exactly sure of such datasets and models. This post might present a good start. @brandenchan Any recommendation on the above discussion? i.e. finding or adapting models that generate somewhat longer answers.

Further thoughts:

Doesn't answer the question directly. But just some thoughts:

I think you are correct in not going for GenerativeQA if it's closed-domain and the answers are to be from the 40 text documents.

The approach to take might also depend on the specifics of your use-case. E.g. Is it that (as in the example above), the answer is say 6-10 words long and the model gives 2-4 word answers? Or are the answers usually multiple sentences long (~20 words say)?
An analysis of this might be useful. E.g. fine-tuning the model might get it to go from 2-4 words on average to 6-10 words on average answers. But it may still not consistently output multiple sentence answers. Also I think this analysis would reveal how much of "model tending to generate short answers" is an issue.

In the latter case (answers usually multi-sentence) maybe the problem can even be reformulated to be retrieval of passages (this would probably increase recall at the cost of precision of the answer). But again, I realize it might not be suitable in the context of a chatbot. Instead of directly replying with an answer, it'd have to perhaps say "Here is a passage which may contain what you are looking for: ".

[brandenchan]

Hi @felixriehm, sorry for the late reply, but I have a couple thoughts for you. Haystack and FARM offer the possibility to calibrate confidence scores. You can learn more about it in the Reader documentation.

Ensuring a certain sentence length is tricky. SQUAD generally features very short answer spans. The Natural Questions dataset features a mixture of both short and long answers. There may be models out there that train on this dataset, but I also know that often they will train on just the short answers.

Off the top of my head, I agree with @bglearning that UX is an important lever. Providing context can be useful. Perhaps you can give your user the option to ask for the context by saying something like "Show me the context of this answer".

I think that treating this as a retrieval task can also be a good way to return more substantial answers. You could split by paragraphs, by a certain chosen word count, or by sentence. Or even all of these if you're ok with indexing your data multiple times.

[felixriehm]

Thanks for your feedback. I will give the things you mentioned a try!