Problem: DO regen task not consistently working at scale

[fiver-watson]

Current Behavior

Putting the context up here first, because this may be a Heisenbug - we do not have the capability to recreate the client conditions easily elsewhere, so I am not 100% certain if other factors (such as network connectivity, etc) might be impacting this.

During a client migration project, running the DO regeneration task repeatedly failed to generate all derivatives, for multiple different media types, and for both local and external URL originals. Repeated runs using the --no-overwrite option (to ensure that initial progress was not lost) would often generate a few more but leave many originals without any derivatives. When found via the AtoM user interface, it was possible to manually follow the link to the original (where it was a URI and not a local upload), AND it was possible to set AtoM to auto-generate new derivatives in the DO metadata area and have those work - so in individual cases, there seemed to be no problem. However, running the task again would still produce only partial results.

Steps to reproduce the behavior

1. Have a large site with many descriptions and digital objects. The client site in question had approximately:

   • 162 top-level descriptions
   • ~201K total descriptions
   • ~46K digital objects, of which ~21.5K are images, ~18.5K are text, and the remainder are audio and video
   • Nearly all of the images were external URI links, while most of the text DO were local PDF uploads

2. Run the digital object regen task

3. Inspect the results

4. If there are derivatives missing, try running it again using the --no-overwrite option, and then inspect again

Error encountered

• Many derivatives were still missing, despite repeated runs
• In most cases, it was possible to a) follow the links to the original, and b) manually regenerate working derivatives via the UI

Expected Behavior

• The task should work as expected, and generate derivatives for all digital objects.
• If it is not possible to generate a specific DO's derivatives, this should be captured in the verbose outupt of the tool, ideally with a reason (e.g. could not find; parsing error; timeout waiting for response, etc)
• The task options should work consistently as documented (or the documentation should be fixed)

Possible Solution

No idea... but possible enhancements that would make using the tool easier would be an output option to generate a CSV report of digital objects that the task could not create derivatives for. The task error logs are do not make it easy to troubleshoot problem DOs that are skipped because they mention a filename, but no related IO information.

Our developers were able to use these logs to create a CSV that included the slug and title of the related information object, as well as the path and filename of the DO, for cases where there were actual errors (e.g. 404s or parsing errors due to special characters in the name). It would be GREAT to have something similar built into the task itself.

Context and Notes

It is worth noting that this particular client's site is the internal edit site of what will be (at project's conclusion) part of a 2-site deployment. The internal site does require VPN access - though the external DOs were on a publicly accessible server, and the DO task was run from within the VPN. Nevertheless, it is possible this added some latency that affected the task.

Version used

AtoM 2.8.2

Operating System and version

Ubuntu 20.04

Default installation culture

en

PHP version

PHP 7.4

Contact details

No response