Globus paginated index by default will only process 10000 entries

[nocollier]

Results may be incomplete. We need to think about how to address this.

• I am not inclined to simply raise this limit as large number of results can take a long time to process. Chances are, you made that query not sure for what you were searching and should be more specific.
• If the number of results is beyond a certain threshold, we may opt to provide shallow search results where we populate the cat.unique() using facets from the search.
• Then if the large search is what was intended we would need to detect that the catalog is shallow and query for all information before getting the file information.