4826 Replicate RECAP PDF uploads to subdockets

[albertisfu]

This PR adds support for replicating PDF uploads to sub-dockets, following a similar approach to attachment pages. A step was added in find_subdocket_pdf_rds before process_recap_pdf to look for subdockets where documents should be merged.

The process flow is:

• The process aborts if the PDF upload belongs to an appellate court, since doppel cases cannot exist in appellate courts.
• Similar to attachment pages, a RECAPDocument queryset finds unique RDs with the same pacer_doc_id in the same court. This query was moved to a helper method since it's common to both find_subdocket_pdf_rds and find_subdocket_att_page_rds.
• Additional ProcessingQueue entries are created for each additional RECAPDocument where the PDF needs replication.
• If the original PQ lacks a pacer_case_id (optional in PDF uploads), one is assigned during the first iteration so the PQ can succeed when processed by process_recap_pdf. Otherwise, the lookup will fail with RECAPDocument.MultipleObjectsReturned.
• PQ creation is wrapped in transaction.atomic to roll back any objects if errors occur. This change was also applied to find_subdocket_att_page_rds.
• Removed redundant code block in process_recap_attachment.

When working on this I noticed that within process_recap_pdf there is the fallback query:

courtlistener/cl/recap/tasks.py

Line 264 in cb012e8

rd = await RECAPDocument.objects.aget(pacer_doc_id=pq.pacer_doc_id)

Is it correct that this query only use pacer_doc_id ? Not sure if pacer_doc_ids are unique across all courts in PACER. If they're not, I think it would be safer to change it to:

rd = await RECAPDocument.objects.aget(pacer_doc_id=pq.pacer_doc_id, court_id=pq.court_id)?

Let me know what do you think.

[mlissner]

Is it correct that this query only use pacer_doc_id ? Not sure if pacer_doc_ids are unique across all courts in PACER. If they're not, I think it would be safer to change it to:

Yeah, that's definitely a bug, and your fix of adding the court to it should help a lot, thank you.

[johnhawkinson]

Is it correct that this query only use pacer_doc_id ? Not sure if pacer_doc_ids are unique across all courts in PACER.

They are unique! The first 3 digits identify the court, see the doc1 URLs section of https://github.com/freelawproject/juriscraper/blob/main/juriscraper/pacer/notes.md

Yeah, that's definitely a bug, and your fix of adding the court to it should help a lot, thank you.

Why "definitely"?

[mlissner]

I forgot about that, you're right, John!

[mlissner]

LGTM thanks! Onward to a proper review.

[ERosendo]

I believe we can avoid creating the full list of appellate courts. Consider the following approach:

Suggested change

	appellate_court_ids = [
	court_pk
	async for court_pk in (
	Court.federal_courts.appellate_pacer_courts().values_list(
	"pk", flat=True
	)
	)
	]
	if pq.court_id in appellate_court_ids:
	# Abort the process for appellate documents. Subdockets cannot be found
	# in appellate cases.
	return pqs_to_process_pks
	appellate_court_ids = Court.federal_courts.appellate_pacer_courts()
	if appellate_court_ids.filter(pk=pq.court_id).exists():
	# Abort the process for appellate documents. Subdockets cannot be found
	# in appellate cases.
	return pqs_to_process_pks

[albertisfu]

Great! Yeah this approach is simpler. I've applied it.

[ERosendo]

LGTM. Let's merge after addressing my comment.

[albertisfu]

@ERosendo Thanks! I've applied your suggestion regarding the exists Court query for validating the court type. I've also replicated the same approach in other similar queries in cl.recap.api_serializers to optimize them.

Let me know what do you think.

[ERosendo]

Thanks @albertisfu. LGTM

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ WriteTimeout /api/rest/{version}/recap/ View Issue
• ‼️ ReadTimeout /api/rest/{version}/recap/ View Issue
• ‼️ Multiple RECAPDocuments returned when processing pdf upload /api/rest/{version}/recap/ View Issue

_{Did you find this useful? React with a 👍 or 👎}