Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Components expecting List[...] input should use Iterable[...] instead #8494

Open
bendavis78 opened this issue Oct 26, 2024 · 0 comments
Open

Comments

@bendavis78
Copy link

bendavis78 commented Oct 26, 2024

For example, let's say I have a generator that yields my documents:

def get_documents():
    for some_thing in some_other_generator():
        yield some_thing

In order to use the OpenAIDocumentEmbedder with this, I get this error I am forced to exhaust the python generator and load all documents into memory.

Many components in the haystack library require List when an Iterable would do just fine. For example, OpenAIDocumentEmbedder could be updated to use pyhthon generators instead of raw lists, making the process more memory effecitent.

@bendavis78 bendavis78 changed the title Components expecting List[...] input should use Iterable Components expecting List[...] input should use Iterable[...] instead Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant