How to Add Metadata to Document Objects Using Pipeline

[ekohrt]

I'm modifying the Haystack tutorial notebook in Colab to make a basic extractive QA system with my own text files. Everything works great, but now I want to link each answer back to its original .txt file.

In the Preparing Documents section, the TextIndexingPipeline processes a list of file paths into a DocumentStore, splitting the files into smaller chunks, but the resulting Document objects don't have any metadata. Can I modify the pipeline to add the original file path to each document's metadata?

[anthropoy]

To do that, you need to add the meta data manually. If you are completing this tutorial https://haystack.deepset.ai/tutorials/03_scalable_qa_system. Modify indexing_pipeline.run_batch(file_paths=files_to_index) to
indexing_pipeline.run_batch(file_paths=files_to_index, meta=meta_files). meta_files has to be a list with dictionary in it, the same length as your files_to_index list, e.g. in the form of List[Dict[str, Any]]

[demongolem-biz2]

Couldn't PreProcessor handle this? I have a similar indexing pipeline except that PDFConverter is the first node followed by PreProcessor and DocumentStore. Because to do this manually, you would have to write the Document objects created by the PreProcessor to disk, manually scan through all of them (which is over 4000 in my case) and then link to that manual file. This process seems not to scale too well.

Although it looks like maybe the PdfToTextConverter node could be modified for this purpose once #5963 is complete