Improve error handling for document missing response from OpenSearch

[jordanpadams]

💡 Description

When a user tries to execute a harvest load, they are getting this error messages like this:

[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt003a::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt003b::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt003c::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt006a::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt006b::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt41bl::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt41gr::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt42a::1.0]: document missing
[ERROR] [_doc][urn:nasa:pds:relab:data_reflectance:offmt42b::1.0]: document missing

This is not really helpful or useful for a user. Let's catch this error and provide a more useful error message that can help a user debug the issue.

⚔️ Parent Epic / Related Tickets

Parent: #154

[alexdunnjpl]

@jordanpadams I'll happily speculate about what's relevant and how to usefully present it, but if @plawton-umd or another user can chime in on the following, I can probably deliver something better

• My workflow looks like X
• Common errors I encounter are X, Y, Z
• My resolution workflow for these errors looks like X, Y, Z

From that, I can infer what information to display and how to display it (for example, it might be useful to supply a report in JSON format if it's desirable to do something to problematic elements with a script)

[plawton-umd]

@alexdunnjpl This specific error is not one I recall encountering. If I saw this, on my side I would check
the paths and permissions for were the product is expected to be loaded from. However, it is not clear
to me if this error message is referring to not finding the product in my source files or somewhere on the
EN side of the interface.

My workflow looks like

• select usually a bundle, sometimes a collection, to harvest
• determine how many products I expect (overall number and numbers of bundle, collection, document, data, etc. products)
• determine how many products may already be in the registry with the query parameters I will use to check at the end of the harvest process - being aware of any duplicates, versions, etc.
• run harvest
• review logs - looking for errors, warnings, number of products uploads, skipped, failed
• determine how many products (using the same query parameters) are now in the registry
• try sort out what does not make sense and when that fails, submit a GitHub issue

Common errors I have encountered (I expect new ones with the move to MCP)

• skipped products
• failed to load products (why? checksum fail? something else fail?)
• query shows more products than log says loaded
• retries
• missing dictionary

My resolution to these errors

• nag about dictionary
• determining which products had issues (a summary list at the end would be nice)
• a lot of queries - looking at product, load time, archive status, (Is there a message in there?) - trying to confirm each supposedly missing/failed file
• lack of feeling confident that everything is there as it should be

[alexdunnjpl]

@plawton-umd fantastic, thanks for this

[jordanpadams]

@alexdunnjpl this was an issue other users have encountered and pinged me with on numerous occasions.