Problem: can't store encrypted uncompressed AIPs

[sevein]

Expected behaviour
It should be possible to store encrypted uncompressed AIPS.

Current behaviour
Store the AIP fails:

STDOUT >>>
"/api/v2/location/2585ce71-a747-466b-b0ae-dffb4ebfd54f/" "/var/archivematica/sharedDirectory/currentlyProcessing/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a" "6b570ab1-072f-48cf-bed7-33320c5a273a" "BagTransfer_1540057000" "SIP"

STDERR >>>
storeaip_v0.0: WARNING   2018-10-20 17:38:55,612  archivematica.mcp.client.storeAIP.store_aip:215  SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_file'.
SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_file'. See logs for more details.Traceback (most recent call last):
  File "/src/MCPClient/lib/job.py", line 102, in JobContext
    yield
  File "/src/MCPClient/lib/clientScripts/store_aip.py", line 313, in call
    args.sip_uuid, args.sip_name, args.sip_type))
  File "/src/MCPClient/lib/clientScripts/store_aip.py", line 216, in store_aip
    raise Exception(errmsg + " See logs for more details.")
Exception: SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_file'. See logs for more details.

In Storage Service:

INFO      2018-10-20 10:38:53  locations.models.gpg:gpg:_create_tar:308:  creating archive of BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a at /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a.tar, relative to /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a
INFO      2018-10-20 10:38:53  locations.models.gpg:gpg:_gpg_encrypt:239:  Successfully encrypted /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a at /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a.gpg
ERROR     2018-10-20 10:38:53  locations.models.async_manager:async_manager:wrapper:122:  Task threw an error: 'NoneType' object has no attribute 'get_file'
Traceback (most recent call last):
  File "/src/storage_service/locations/models/async_manager.py", line 119, in wrapper
    value = task_fn(*args, **kwargs)
  File "/src/storage_service/locations/api/resources.py", line 701, in task
    self._store_bundle(bundle)
  File "/src/storage_service/locations/api/resources.py", line 675, in _store_bundle
    premis_agents=agents, aip_subtype=aip_subtype)
  File "/src/storage_service/locations/models/package.py", line 613, in store_aip
    pointer_file, storage_effects))
  File "/src/storage_service/locations/models/package.py", line 994, in create_new_pointer_file_given_storage_effects
    old_fsentry = old_pointer_file.get_file(file_uuid=self.uuid)
AttributeError: 'NoneType' object has no attribute 'get_file'

Steps to reproduce
From aip-encryption.feature:

  @uncompressed
  Scenario: Richard wants to ensure that he can encrypt uncompressed AIPs.
    Given there is a standard GPG-encrypted space in the storage service
    And there is a standard GPG-encrypted AIP Storage location in the storage service
    And the default processing config is in its default state
    And the processing config decision "Select compression algorithm" is set to "Uncompressed"
    When a transfer is initiated on directory ~/archivematica-sampledata/SampleTransfers/BagTransfer
    And standard AIP-creation decisions are made
    And the user waits for the "Store AIP location" decision point to appear and chooses "Store AIP Encrypted in standard Archivematica Directory" during ingest
    And the user waits for the AIP to appear in archival storage
    And the user queries the API until the AIP has been stored
    Then the uncompressed AIP on disk at /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/ is encrypted
    When the user downloads the AIP
    Then the downloaded uncompressed AIP is an unencrypted tarfile

~/archivematica-sampledata/SampleTransfers/BagTransfer can be found at the sampledata repository.

Your environment (version of Archivematica, OS version, etc)
AM18 qa/1.x 89dfc80d

---

For Artefactual use:
Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

• All PRs related to this issue are properly linked 👍
• All PRs related to this issue have been merged 👍
• Test plan for this issue has been implemented and passed 👍
• Documentation regarding this issue has been written and it has been added to the release notes, if needed 👍

[sevein]

Could this issue be related?
artefactual/archivematica-storage-service#324

[ross-spencer]

@sevein @jraddaoui my first instinct was pointer files. This was my last commit related to them, I wonder if this would pass before that? artefactual/archivematica-storage-service@c27fb48

[jraddaoui]

Hi @sevein, @ross-spencer, @joel-simpson,

artefactual/archivematica-storage-service#424 fixes the issue for now, just avoiding to access pointer files that may not exist, like in uncompressed AIPs. It should also fix similar issues in the replication and re-ingest processes.

Please, let me know if you think this is good enough or if you want me to tackle artefactual/archivematica-storage-service#324 for the 1.8 release.

[sevein]

@jraddaoui, why this was not a problem when we tested 1.7 releases? I'm assuming we've made a change that is causing this? Was it introduced in SS v0.12? SS v0.13?

[ross-spencer]

@jraddaoui @sevein knowing the regression would be useful I agree. For your other question though @jraddaoui (artefactual/archivematica-storage-service#324) @sallain or @sromkey might have a better idea why it didn't have a 1.8 milestone, but I assume, potential complexity, and sponsorship are two likely factors. Your fix seems like a nice in-between at this stage in the release for 1.8.

[jraddaoui]

@sevein, @ross-spencer,

Found this artefactual/archivematica-storage-service#350 linked in artefactual/archivematica-storage-service#324. I guess we never cherry-picked the fix to the qa branch.

[jraddaoui]

Fixed in SS repository, qa/0.x and stable/0.12.x branches.

[sallain]

I am maybe 75% confident that this is resolved. @jraddaoui, can you check the settings on am18bionic to see if it's set up properly to be receiving encrypted AIPs? If everything looks good to you, then we know that it's working.

[jraddaoui]

Hi @sallain, lol-3 and lol-4 seem to be stored encrypted and uncompressed.

[sallain]

Thank you @jraddaoui!!!

[sallain]

Added to release notes; no docs needed.