Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make PineconeHandler more efficient (fetch only fresher markets) #528

Open
gabrielfior opened this issue Oct 21, 2024 · 0 comments
Open

Comments

@gabrielfior
Copy link
Contributor

224036c#diff-1b9d8df9eb7cb55918e642ebc0f26902d5211d4946a8c4d0454e9889dd05c371R124-R128

Instead of fetching all markets, fetch most recent market (based on close_time) and fetch only markets whose close_time >= max_close_time_from_pinecone).
Note that Pinecone Python's client does not have allow metadata-querying, hence some magic is required - see below

start_item_ids = handler.list_paginated(top_k=10) # fetch IDs
start_items = handler.query(ids=ids, include_metadata=True) # fetch top 10 items
max_market_close_time = max(start_item.close_time for start_item in start_item['matches'])
while True:
    # fetch new markets with filter close_time $gt max_market_close_time
    #. if items delivered = [], break
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant