Improve speed and RAM consumption of buffered slice writer

[philippotto]

Description:

• Improves RAM consumption by avoiding to concate all data sections at once. Instead, the given sections are traversed in chunks (using the shard size) and each chunk is filled so that it can be written to disk.
• As a byproduct, the speed also increased a bit. However, I'm not too sure whether this is a general win. Either way, np.pad is not necessary anymore, because a sufficiently sized buffer is created first into which the sections are written.
• Adds more tests for BufferedSliceWriter
• Adds warnings when the BufferedSliceWriter is used without aligning it to the dataset's chunk size.

As a performance benchmark I created a WKW dataset with chunk_shape=(32, 32, 32) and chunks_per_shard=(32, 32, 32). Then, I wrote random data with shape (4096, 4096, 32) to it (multiple times).

Runs	Avg. Time [s]	Avg. RAM [MB]
New	4.76	1188
Old	5.94	1713

So, the new approach only uses ~70% of the RAM for this benchmark. It should be close to 50%, but I measured the RAM with time which also measures the overall ram consumption of the entire python script and pytest environment in which I executed the benchmark.

Issues:

• related to https://github.com/scalableminds/voxelytics/issues/3318

Todos:

Make sure to delete unnecessary points or to check all before merging:

• Updated Changelog
• Updated Documentation
• Added / Updated Tests
• Considered adding this to the Examples

[normanrz]

Runs Avg. Time [s] Avg. RAM [MB]
Old 4.76 1188
New 5.94 1713

Did you mix up old <> new?

[philippotto]

Runs Avg. Time [s] Avg. RAM [MB]
Old 4.76 1188
New 5.94 1713

Did you mix up old <> new?

Oops, yes, I did. Corrected it now.

[normanrz]

Looks good. I think the next iteration could include to read the input images in a chunked manner instead of having to read entire images at once.

[normanrz]

So, in test_buffered_slice_writer_along_different_axis you aligned the offset and here are testing that unaligned screams?

[philippotto]

Yes, exactly.

[philippotto]

Looks good. I think the next iteration could include to read the input images in a chunked manner instead of having to read entire images at once.

I'm not sure if I understand correctly. The BufferedSliceWriter isn't responsible for reading images. Do you mean its interface should support sending chunks to it so that users of the Writer can read images in chunks?

[normanrz]

Looks good. I think the next iteration could include to read the input images in a chunked manner instead of having to read entire images at once.

I'm not sure if I understand correctly. The BufferedSliceWriter isn't responsible for reading images. Do you mean its interface should support sending chunks to it so that users of the Writer can read images in chunks?

I know this isn't the responsibility of the BufferedSliceWriter. But when looking holistically at Dataset.from_images (and similar APIs), that would be my next angle of attack for optimization.

[philippotto]

Ah ok, sorry for the misunderstanding. Sounds good 👍