You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to use the OpenAIDocumentEmbedder with this, I get this error I am forced to exhaust the python generator and load all documents into memory.
Many components in the haystack library require List when an Iterable would do just fine. For example, OpenAIDocumentEmbedder could be updated to use pyhthon generators instead of raw lists, making the process more memory effecitent.
The text was updated successfully, but these errors were encountered:
bendavis78
changed the title
Components expecting List[...] input should use Iterable
Components expecting List[...] input should use Iterable[...] instead
Oct 26, 2024
For example, let's say I have a generator that yields my documents:
In order to use the
OpenAIDocumentEmbedder
with this, I get this error I am forced to exhaust the python generator and load all documents into memory.Many components in the haystack library require
List
when anIterable
would do just fine. For example,OpenAIDocumentEmbedder
could be updated to use pyhthon generators instead of raw lists, making the process more memory effecitent.The text was updated successfully, but these errors were encountered: