JSON context differ from CSV context after exporting from Haystack Annotation Tool

[tarangMittal]

Hi,

When I export my annotations to a JSON, the context field, is always the entire document text. However, when exported as a CSV, the context field is shorter covering the answer annotated for the respective question. This is what I would also expect a context would be, rather than the whole document text.

Was wondering why this discrepancy exists?

Thanks!
Tarang

[CTRFL5]

I believe you need to upload the document in multiple parts, so you get multiple contexts. So if you uploaded your document as one file, you get one large context, but if you divided it up into many pieces then you get many smaller contexts.

[tarangMittal]

Sorry may not have clearly explained it.
So, when I upload my document as one file, I do get one large context for the JSON file exported. However for the CSV exported, the context is shown based on where the annotation is done in the document.

For example here in the JSON exported, the context field continues on and it is the string as the entire documents text:

However here in the CSV exported, as you can see the context field is just a portion where the answer has been annotated which seems more "useful" as a context

So was just wondering why this difference exists between the exported files.

[bogdankostic]

Hi @tarangMittal, if you want to use the labels inside Haystack, I'd recommend using the JSON file in SQuaD format, just as we do it in the Evaluation tutorial and fine-tuning of reader tutorial. During training, Haystack takes care of splitting large Documents into smaller chunks.