Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update harvard ordering command #4388

Merged
merged 7 commits into from
Sep 4, 2024
Merged

Update harvard ordering command #4388

merged 7 commits into from
Sep 4, 2024

Conversation

quevon24
Copy link
Member

@quevon24 quevon24 commented Aug 30, 2024

I added a small change to update the way we get the harvard queryset, which has over ~350k objects.

The original approach could cause memory issues because we get the ~350k clusters and use prefetch_related to get their opinions, if we still wanted to use prefetch_related, we will have to use an iterator with chunk_size param to avoid getting everything.

The new approach is similar to the one bill implemented for columbia ordering, we get the cluster_ids of the clusters that we need to update their opinions, and then use the id to get the opinions and order them.

@quevon24 quevon24 requested a review from flooie August 30, 2024 20:45
Copy link

sentry-io bot commented Aug 30, 2024

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: cl/corpus_importer/management/commands/update_opinions_order.py

Function Unhandled Issue
sort_columbia_opinions NameError: name 'x' is not defined cl.corpus_impo...
Event Count: 1
sort_columbia_opinions FileNotFoundError: [Errno 2] No such file or directory: '/opt/courtlistener/_columbia/washington/supreme_court_opini... ...
Event Count: 1
sort_harvard_opinions ValueError: chunk_size must be provided when using QuerySet.iterator() after prefetch_related(). cl.corpus_importer.management.command...
Event Count: 1

Did you find this useful? React with a 👍 or 👎

@quevon24 quevon24 marked this pull request as draft August 30, 2024 23:37
@quevon24 quevon24 changed the title Update ordering command to use iterator() with harvard and columbia querysets Update harvard ordering command Sep 2, 2024
@quevon24 quevon24 marked this pull request as ready for review September 2, 2024 18:40
@flooie flooie enabled auto-merge September 4, 2024 01:38
@flooie flooie merged commit 68318f9 into main Sep 4, 2024
13 checks passed
@flooie flooie deleted the update_ordering_command branch September 4, 2024 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants