Verbatim PFB handover fails when using a common prefix

[nadove-ucsc]

This was observed when testing #6815 due to a situation like this:

Activity 65f821a0-c4c2-4f1b-83f1-292dfe24f719 is found in a primary bundle 1f47f146-6d32-adbd-bd87-93002479d031 (A) and also in a replica bundle (B). Its generated file (e47f0944-0bdb-422f-a251-3fab0c19a029) is also found in (A), and additionally in a separate replica bundle (C). Only (B) was indexed due to the use of common prefix 65/1. The resulting manifest contained a dangling reference to the generated file.

[edit @hannes-ucsc] … causing Terra to reject the manifest

[nadove-ucsc]

Here is an example of an import that failed for this reason

[nadove-ucsc]

Assignee to consider whether there's any risk that this error might occur in anvilprod.

[nadove-ucsc]

This will break anvilprod when filtering by non-dataset fields. As an example, consider a manifest with a filter to select files smaller than 1 gigabyte, and a primary bundle with the following structure:

The manifest will include file 1 and every entity upstream from it, but will omit file 2. The sequencing activity contains foreign key references to both files via its generated_file_id column. File 2 will then become a dangling relation that will cause Terra to reject the manifest.

[nadove-ucsc]

Other observations:

The foreign key anvil_assayactivity.antibody_id references the anvil_antibody table, but since we don't index contributions from that table, its replicas only occur as orphans (from replica bundles). If a manifest is created with filters that exclude orphans, dangling reference errors will occur for each anvil_assayactivity replica with a non-null antibody_id column. However, the anvil_assayactivity table is empty for all snapshots currently indexed on anvilprod, so this should be unobservable.

anvil_project presents a similar edge case (being a table described by the schema that we only index as orphans), but there are no foreign keys that reference that table, so it can't cause the dangling reference error.

[hannes-ucsc]

I moved these latter two reproduction to a separate issue (#6898).

[hannes-ucsc]

I hope that this can be solved by making the inclusion of relations conditional upon whether files from a source with a common prefix are included in the manifest (in addition to any other conditions that #6898 might add). In order to demonstrate a handover with relations in a lower deployment, one would have to select only small datasets, under the assumption that there is a 1:1 relationship between datasets and sources and that the sources for those datasets will have been indexed without a common prefix.

[achave11-ucsc]

Blocker landed.