Problem: There isn't a lot of information on post-store callback in Archivematica, especially the original <source_id> post-store AIP call-back

[ross-spencer]

Version of the documentation

Archivematica Storage Service: 0.15.

Page (and section, if applicable) where the issue occurs

Storage service API wiki and docs themselves.

• Wiki.
• New-style callback in the docs.
• SWORD2 info in the docs
• User interface:

Description of the issue

• As a user, I want to know what capabilities the storage service has to support my integrations.
• As a developer, I want to be able to find information about those same capabilities, including what is supported and why (and how).
• As an administrator I want to know what information is being communicated outwards from my system.

The storage service supports two types of call-back. A callback is a communication to another server, following a specific trigger (the successful storage of an AIP, or explicit call to the server to repeat the message so it can be consumed if missed previously.).

The documentation around the old-style format is a little limited, and difficult to trigger. The old-style call-back is specifically Post-store AIP (source files) or post_store rather than post_store_<package_type> e.g. post_store_aip, post_store_dip, post_store_aic. This API endpoint is entirely for the old-style callback.

Eliciting a response from the endpoint to see what we can expect...

If I create an artificial state that will help us to mimic the interaction, i.e. manually add files to the Files model for a specific AIP, then I can make an API call as follows:

http --pretty=format \
    GET "http://127.0.0.1:62081/api/v2/file/420216b1-692e-40ec-9cfa-e94b3271a2d2/send_callback/post_store/" \
    Authorization:"ApiKey test:test"

If I set up a server to receive arbitrary requests, I can read the request as per below:

2019/10/15 11:38:51 POST / HTTP/1.1
Host: 127.0.0.1:8080
connection: keep-alive
accept-encoding: gzip, deflate
accept: */*
user-agent: python-requests/2.21.0
content-type: application/json
custom-header: Kind regards
content-length: 57

2019/10/15 11:38:51 Headers: map[Accept-Encoding:[gzip, deflate] Accept:[*/*] User-Agent:[python-requests/2.21.0] Content-Type:[application/json] Custom-Header:[Kind regards] Content-Length:[57] Connection:[keep-alive]]
2019/10/15 11:38:51 Content: map[File UUID:bc01b7b7-9339-424a-a469-42bc8698661c]

In this instance, the Content translates to JSON:

{
   "File UUID": "bc01b7b7-9339-424a-a469-42bc8698661c"
}

If there were three files in this AIP, this would repeat three times with different file UUIDs.

The response to my original API call is:

HTTP/1.1 204 NO CONTENT
Connection: keep-alive
Content-Language: en
Content-Length: 0
Date: Tue, 15 Oct 2019 11:54:33 GMT
Server: nginx/1.16.0
Vary: Accept, Accept-Language, Cookie
X-Frame-Options: SAMEORIGIN

HttpNoContent is on the happy path in the code and so we should update the docs to say that an expected response from the server for a GET request is a 204.

If the message cannot be delivered, then the response indicates that it could not be sent.

HTTP/1.1 500 INTERNAL SERVER ERROR
Connection: keep-alive
Content-Language: en
Content-Type: application/json
Date: Tue, 15 Oct 2019 11:56:14 GMT
Server: nginx/1.16.0
Transfer-Encoding: chunked
Vary: Accept, Accept-Language, Cookie
X-Frame-Options: SAMEORIGIN

{
    "callback_uris": [
        "http://127.0.0.1:8080"
    ],
    "failure_count": 3,
    "message": "Failed to POST 3 responses to callback URI"
}

The callback is configured as follows:

If the AIP is stored correctly then this callback is triggered. Again artificially populating the File model makes this response work for us.

Source ID translates to a UUID assigned to a specific file in an AIP per the File model.

This model is never normally populated per #342.

Suggested fix

1. Incorporate specifics about the callback into the various documentation, (wiki, apid-docs).
2. Specifically, it seems this call is limited to SWORD2 workflows around Archidora/Islandora.
3. Other parts of the configuration work the same as the new-style callback, in that headers are configurable.
4. Document that this API endpoint doesn't cover the new-style callbacks at all, and for that, a user would need functionality like Allow users to trigger test calls of post store callbacks #1164 or Feature: show post store callback results and allow retries #881 suggests.

Other things that we may consider include some sort of automated test (AMAUAT) to trigger this call for Fedora style SWORD2 ingests to make sure this works consistently with the integration. The feature file and code would provide useful reference for developers in future.

Additional information

Most of the information above is included to (hopefully) make it easier for someone looking at this in future to understand the capability of the endpoint. Some additional context is included below.

• PR from original development.
• New-style callback in storage service.
• Tangentially related to Problem: Get file information for package (API) endpoint only works for transfers (those sent to backlog) #342, the model involved in this interaction models.File is the same that is engaged via the /v2/file/<uuid>/contents/ endpoint. The Files table is not updated for any other AIP type, which is what restricts this feature to SWORD2 (I believe).

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• Documentation regarding this issue has been written and merged
• Details about this issue have been added to the release notes (if applicable)

[replaceafill]

I investigated this endpoint today and found an additional issue: it assumes that the checksum algorithm is always SHA-512 and looks for a manifest-sha512.txt file in the bag layout of the AIP.

However the default value for the checksum algorithm setting is sha256 which produces a manifest-sha256.txt file instead, and the user can also modify it from the dashboard.

[ross-spencer]

@replaceafill you might find the linked issue helpful with regard to sha-512. I'll add the link again here: artefactual/archivematica-storage-service#225.

[replaceafill]

Ah, thanks @ross-spencer! I definitely missed that link 😁