WIP: Add fileGrp USE="Derivative" handling to fetch

[ross-spencer]

Fetch jobs were not successfully picking up the derivative type which
is used by Dataverse transfers. We add that here.

Connected to #52

[ross-spencer]

@peterVG @tw4l what do you think about this fileGrp handling in AIPscan. The derivative type expresses a relationship between a tabular format in Dataverse and the concept of a Bundle which is described in the Wiki. (Tessa, you probably have more institutional knowledge of this!)

We noticed the original-derivatives report wasn't being written out correctly. While I happened to use derivative as the term in that report, the other parts of AIPscan are focused on either originals or preservation copies:

So, that leads me to ask, what is the full extent of the change I might make for AIPscan to handle derivatives properly. Largely, they are in the camp of:

1. Treat them as unique, and update the screens (and maybe add a derivative type to the report).
2. Treat them as originals, and leave the screens the same, and don't talk about them separately.
3. Treat them as preservation copies (As ever putting all options out there, this one doesn't feel correct).
4. Something else?

[ross-spencer]

Ah! Just going to read this Tessa! #55 (which seems timely!)

[tw4l]

@ross-spencer It's an interesting question!

Are these Dataverse derivatives unique in the respect of being derivatives that are not necessarily preservation derivatives and were created in advance of ingest into Archivematica? Theoretically, I could see other integrations extending use of this concept. Practically, I know in the EPrints-Archivematica integration work, derivatives generated by EPrints are treated as an original file by Archivematica, but that was largely out of convenience.

It seems that the choice to use fileGrp="derivative" in the METS signifies an intent not to think of them in Archivematica as original files or preservation copes (I agree with you that the latter seems particularly not right).

I'm wavering between 1 and 2, and might suggest a slightly hybrid approach:

• Store them in the database as a distinct type of file (maybe type="dataverse_derivative" or type="derivative", to tie it to my suggestions in Originals and Copies database tables could be collapsed into single Files table #55) to retain granularity of the data.
• Decide whether to treat them as originals or as a distinct file type on a per-report basis. This might be where some user stories would really help us out. If as an administrator looking for reports my intention is to separate out the files Archivematica received from the files Archivematica created, I could see a use case for considering the Dataverse derivatives as an original file in some cases. In some other cases, it might make sense to view them separately.

[tw4l]

Closing due to inactivity and because this branch has drifted from main.