Node to add extra context to documents generated by Preprocessor Node

[Radu1999]

Is there any implementation of a node that supports enriching documents after the preprocessor splits the main documents or do we need to implement it from scratch?

[julian-risch]

Hello @Radu1999 There are some example nodes that enrich documents after the preprocessor splits them. I would suggest that you implement your own node based on these examples. For example, there is the EntityExtractor and the DocumentClassifier:

• https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/extractor/entity.py#L76
• https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/document_classifier/transformers.py#L54

The DocumentClassifier has some example code of how it's used in an indexing pipeline and this is how I imagine your custom node to work as well:

    ...
    converter = TextConverter()
    preprocessor = Preprocessor()
    document_store = ElasticsearchDocumentStore()
    document_classifier = TransformersDocumentClassifier(model_name_or_path="bhadresh-savani/distilbert-base-uncased-emotion",
                                                         batch_size=16)
    p = Pipeline()
    p.add_node(component=converter, name="TextConverter", inputs=["File"])
    p.add_node(component=preprocessor, name="Preprocessor", inputs=["TextConverter"])
    p.add_node(component=document_classifier, name="DocumentClassifier", inputs=["Preprocessor"])
    p.add_node(component=document_store, name="DocumentStore", inputs=["DocumentClassifier"])
    p.run(file_paths=file_paths)

The result is then stored in doc.meta["classification"].