[Active Drive] Max fragment size must be under 1MB when using ADs

[JeffreyDevloo]

Problem description

According to @toolslive, the max fragment size on a preset has to be toned down when using adding an active drive

Todos for the fwk:

• display a warning for every preset that would be adjusted due to this in the add osd wizard
• after adding the osd (before claiming) the presets fragmnet size should be changed

[wimpers]

Can we combine this with #204 ?

• On the "Add preset" wizard (under advanced settings), add under compression a field to specify the fragment size. Default size is 4MB. In case an AD is added already, default is set to under 1MB and the value is not allowed to be changed.
• In case an AD disk is added the fragment size of all preset is set to under 1MB. (clearly mark this on the add wizard)
• On the backend detail page, add the fragment size (after encryption). Under edit preset, allow to change the fragment size, unless an AD is added (in that case editing the fragment size isn't allowed.)

Note that the API should have the same checks.

[wimpers]

This becomes obsolete if the AD supports higher fragment sizes. Let's keep on hold for now hence assigning to me.

[toolslive]

mind, compression can increase the size of a fragment.

[wimpers]

Sure but to me that isn't a framework problem but more of an ALBA problem as ALBA knows when it goes above the fragment size limit and should act accordingly.

[domsj]

This could indeed be solved inside alba too.
E.g.:

• split each large value into multiple large values when writing to an AD, using some naming scheme for the keys (append _0001, _0002, ...). this will require some changes to the download path too.
• or during upload cap the fragment size to the maximum allowed by ADs (when any AD is involved for the backend)

[toolslive]

or calculate a fragment size depending on the preset. We know the maximum fixed cost of a compressor, so if a compressor is used, the fragment_size = preset.fragment_size - compressor.fixed_cost.

[wimpers]

To me it looks like the solution of @toolslive would be the easiest to implement?

Also this boils down to 2 scenarios:

• the fragment can be compressed relatively good. In every case you end up with small objects on the backend (smaller than max fragment size).
• the fragments can't be compressed really good. In case we use @toolslive solution the stored object will be almost max fragment size. If we use the split-up method we will have 2 objects which are half of max fragment size.

[domsj]

who does the calculation / when does the calculation happen in @toolslive suggestion?
Is it done by the framework when configuring the preset, or later during upload as in my second suggestion?

[wimpers]

I don't see where the framework would be involved in this. ALBA get a preset including max fragment size and whether or not compression is set. It is up to ALBA to take the necessary cost into account.

[toolslive]

It should be done during upload. Basically, my suggestion is to redefine/reinterpret the fragment size in
the preset to become the maximum packed fragment size.

[wimpers]

Back to BAM: Anything to do for framework team?

[wimpers]

After discussion with @toolslive , no need for the framework team to take action as this will be covered by the ALBA team during upload (integration part of the AD).