Warning message from LFQA and QA Chatbot

[dkbs12]

Hi,
I'm trying to create the QA system for Long Form QA and QA Chatbot on my own using Haystack.
Finally I made prototypes of both since I started it a few month ago.
But I faced some warning messages while I run them on Colab and I'd like to ask you about them as below.

1. QA Chatbot with Agent

   • retriever : BM25Retriever, pipeline PromptNode model : text-davinci-003, Agent PromptNode model : gpt-3.5-turbo
   • document store : Elastic Search, documents language : English
   • 1st warning message
     : warnings.warn(
     /usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py:418: UserWarning: num_beams is set to 1. However, length_penalty is set to 2.0 -- this flag is only used in beam-based generation modes. You should set num_beams>1 or unset length_penalty
   • 2nd warning message
     : /usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1090: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
   • github : Phase02_Agent_test_02_03.ipynb
     

2. Long Form QA

   • retriever : BM25Retrieve & embedding_retriever(text-embedding-ada-002), pipeline PromptNode model : text-davinci-003
   • document store : Elastic Search, documents language : English & Korean
   • 1st warning message
     : WARNING:haystack.nodes.prompt.invocation_layer.open_ai:The prompt has been truncated from 6285 tokens to 3897 tokens so that the prompt length and answer length (200 tokens) fit within the max token limit (4097 tokens). Reduce the length of the prompt to prevent it from being cut off.
   • 2nd warning message
     : WARNING:haystack.utils.openai_utils:1 out of the 1 completions have been truncated before reaching a natural stopping point. Increase the max_tokens parameter to allow for longer completions.
   • github : Phase02_LFQA_test_06.ipynb

I would Appreciate it if you let me know the reason of above warning meaages and how to avoid them.
Thanks.

[julian-risch]

Hi @dkbs12 let me try to help you by giving some context about the warnings.
The first warning is that the length_penalty parameter only has an effect if num_beams is larger than 1. It's triggered here: https://github.com/huggingface/transformers/blob/fc63914399b6f60512c720959f9182b02ae4a45c/src/transformers/generation/configuration_utils.py#L396 You could increase num_beams and the warning would disappear but it's not a concern.
The second warning warning is triggered here: https://github.com/huggingface/transformers/blob/fc63914399b6f60512c720959f9182b02ae4a45c/src/transformers/pipelines/base.py#L1102 Just like the first warning, this one is triggered by transformers and not by Haystack. For your prototyping this is not a concern either because it's just about efficiency.
The third warning (1s in Long Form QA) is about the text input you provide to the PromptNode, which is too long for the LLM that you are using. You should shorten the input text. An alternative is to use a model that works with longer input texts but I wouldn't recommend it. gpt-4-32k would be such a model.
The fourth warning is related to the one before. The text generated by the LLM is truncated/cut off because it becomes to long together with the input text. Shortening the input text is your best option here.
Good luck with your project! 🚀 🙂

[dkbs12]

Hi, @julian-risch
Thank you for your kind explanation and advice.
I had many tests of above cases after read your reply and I found some other points to ask you of as below.

1. As for Long Form QA warning message

   • As you can see, I stored both of English and Korean languge documents all together. So I guessed htis kind of truncated/cut off error happened due to different langugae ducuments. (Generally, Korean language needs much more tokens than English in the same length of sentence) Do you agree with my opinion?
   • Could EmbbedingRetriever be the reason for this kind of truncated/cut off?
   • I had another test only with the BM25Retriever in the same condition again and then I found there is no truncated/cut off error when I made a query in Korean with top_k is less than 3. The only solution for this case is just shortening the input text?
   • When I made a query in Korean, of which correct answer was in the document in English, the response come out mixed in Korean and English. Moreover the response was completely wrong accompanied with truncated/cut off error. Is there better solution to get the answer from the documents made in different languages?
   • github : Phase02_LFQA_test_06_02.ipynb

2. Discrepancy between the number top_k from the results and the number of top_k of inputs.

   • I used both of BM25Retriever and EmbeddingRetriever to bring the documents to Reader through JoinDocuments node
   • I input '5' for the top_k of both retriever respectively and expected it to come out 10 documents in total as a result
   • However, I found 7 documents in total as a result of res["documents"].
   • I have found this kind of dsicrepancy in every single case using both of BM25 Retriever and Embedding Retriever together. Why is it? Is there any logic to get rid of the overlapped document in the JoinDocuments node?
   • github : Phase02_LFQA_test_01_01.ipynb

Thank you for your help in advance.