Non-deterministic header and free context

[cgohlke]

There are two potential issues in the blosc2 Cython extension:

1. The header written by the compress2 function is not deterministic because the default filters and filters_meta fields are defined as unordered sets, not ordered sequences.

2. The compress2 and decompress2 functions do not free the created contexts.

diff --git a/blosc2/blosc2_ext.pyx b/blosc2/blosc2_ext.pyx
index c0e1d04..4580f4c 100644
--- a/blosc2/blosc2_ext.pyx
+++ b/blosc2/blosc2_ext.pyx
@@ -358,6 +358,8 @@ cdef extern from "blosc2.h":

     int blosc2_remove_dir(const char *path)

+    void blosc2_free_ctx(blosc2_context* context) nogil
+

 MAX_TYPESIZE = BLOSC_MAX_TYPESIZE
 MAX_BUFFERSIZE = BLOSC2_MAX_BUFFERSIZE
@@ -522,8 +524,8 @@ cparams_dflts = {
         'blocksize': 0,
         'splitmode': BLOSC_FORWARD_COMPAT_SPLIT,
         'schunk': None,
-        'filters': {0, 0, 0, 0, 0, BLOSC_SHUFFLE},
-        'filters_meta': {0, 0, 0, 0, 0, 0},
+        'filters': [0, 0, 0, 0, 0, BLOSC_SHUFFLE],
+        'filters_meta': [0, 0, 0, 0, 0, 0],
         'prefilter': None,
         'preparams': None,
         'udbtune': None,
@@ -593,6 +595,7 @@ def compress2(src, **kwargs):
     with nogil:
         cctx = blosc2_create_cctx(cparams)
         size = blosc2_compress_ctx(cctx, buf.buf, <int32_t> buf.len, _dest, len_dest)
+        blosc2_free_ctx(cctx)
     PyBuffer_Release(buf)
     free(buf)
     if size < 0:
@@ -633,11 +636,13 @@ def decompress2(src, dst=None, **kwargs):
             raise ValueError("The dst length must be greater than 0")
         size = blosc2_decompress_ctx(dctx, <void*>&typed_view_src[0], typed_view_src.nbytes,
                                      <void*>&typed_view_dst[0], typed_view_dst.nbytes)
+        blosc2_free_ctx(dctx)
     else:
         dst = PyBytes_FromStringAndSize(NULL, nbytes)
         if dst is None:
             raise RuntimeError("Could not get a bytes object")
         size = blosc2_decompress_ctx(dctx, <void*>&typed_view_src[0], typed_view_src.nbytes, <void*> <char *> dst, nbytes)
+        blosc2_free_ctx(dctx)
         if size >= 0:
             return dst
     if size < 0:

[FrancescAlted]

Thanks Christoph! I added this, but still having problems on the wheel with aarch64 and musl. Something else should be happening...

[cgohlke]

Re leaks: there are still many potential memory leaks in the c-blosc library. PVS-Studio reports these potential issues, among others, when run on the c-blosc2-2.3.0 code:

V773 The function was exited without releasing the 'shape' pointer. A memory leak is possible. ndlz8x8.c 74
V773 The function was exited without releasing the 'chunkshape' pointer. A memory leak is possible. ndlz8x8.c 74
V773 The function was exited without releasing the 'blockshape' pointer. A memory leak is possible. ndlz8x8.c 74
V773 The function was exited without releasing the 'shape' pointer. A memory leak is possible. ndlz4x4.c 72
V773 The function was exited without releasing the 'chunkshape' pointer. A memory leak is possible. ndlz4x4.c 72
V773 The function was exited without releasing the 'blockshape' pointer. A memory leak is possible. ndlz4x4.c 72
V773 The function was exited without releasing the 'samples_sizes' pointer. A memory leak is possible. blosc2.c 2429
V773 The function was exited without releasing the 'filters' pointer. A memory leak is possible. blosc2.c 2605
V773 The function was exited without releasing the 'index_path' pointer. A memory leak is possible. sframe.c 33
V773 The function was exited without releasing the 'chunk_path' pointer. A memory leak is possible. sframe.c 50
V773 The function was exited without releasing the 'vlmetalayer' pointer. A memory leak is possible. schunk.c 1445
V773 The function was exited without releasing the 'offtooff' pointer. A memory leak is possible. frame.c 310
V773 The function was exited without releasing the 'offtodata' pointer. A memory leak is possible. frame.c 583
V773 The function was exited without releasing the 'data_tmp' pointer. A memory leak is possible. frame.c 924
V773 The function was exited without releasing the 'header' pointer. A memory leak is possible. frame.c 1405
V773 The function was exited without releasing the 'trailer' pointer. A memory leak is possible. frame.c 1585
V773 The function was exited without releasing the 'off_chunk' pointer. A memory leak is possible. frame.c 2384
V773 The function was exited without releasing the 'offsets' pointer. A memory leak is possible. frame.c 2733
V773 The function was exited without releasing the 'offsets' pointer. A memory leak is possible. frame.c 2940
V773 The function was exited without releasing the 'offsets' pointer. A memory leak is possible. frame.c 3167

V774 The 'fname' pointer was used after the memory was released. directories.c 54

V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'chunk' is lost. Consider assigning realloc() to a temporary pointer. schunk.c 729
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'schunk->data' is lost. Consider assigning realloc() to a temporary pointer. schunk.c 736
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'chunk' is lost. Consider assigning realloc() to a temporary pointer. schunk.c 819
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'schunk->data' is lost. Consider assigning realloc() to a temporary pointer. schunk.c 826
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'chunk' is lost. Consider assigning realloc() to a temporary pointer. schunk.c 936
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'h2' is lost. Consider assigning realloc() to a temporary pointer. frame.c 266
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'h2' is lost. Consider assigning realloc() to a temporary pointer. frame.c 290
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'h2' is lost. Consider assigning realloc() to a temporary pointer. frame.c 322
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'h2' is lost. Consider assigning realloc() to a temporary pointer. frame.c 336
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'trailer' is lost. Consider assigning realloc() to a temporary pointer. frame.c 561
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'trailer' is lost. Consider assigning realloc() to a temporary pointer. frame.c 587
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'trailer' is lost. Consider assigning realloc() to a temporary pointer. frame.c 619
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'trailer' is lost. Consider assigning realloc() to a temporary pointer. frame.c 633
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'trailer' is lost. Consider assigning realloc() to a temporary pointer. frame.c 652
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'frame->cframe' is lost. Consider assigning realloc() to a temporary pointer. frame.c 711
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'frame->cframe' is lost. Consider assigning realloc() to a temporary pointer. frame.c 1258
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'data_chunk' is lost. Consider assigning realloc() to a temporary pointer. frame.c 1812
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'framep' is lost. Consider assigning realloc() to a temporary pointer. frame.c 2422
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'framep' is lost. Consider assigning realloc() to a temporary pointer. frame.c 2627
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'framep' is lost. Consider assigning realloc() to a temporary pointer. frame.c 2829
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'framep' is lost. Consider assigning realloc() to a temporary pointer. frame.c 3070
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'framep' is lost. Consider assigning realloc() to a temporary pointer. frame.c 3225
V701 realloc() possible leak: when realloc() fails in allocating memory, original pointer 'framep' is lost. Consider assigning realloc() to a temporary pointer. frame.c 3375

V1004 The 'chunksize' pointer was used unsafely after it was verified against nullptr. Check lines: 426, 469. frame.c 469
V1004 The 'frame' pointer was used unsafely after it was verified against nullptr. Check lines: 538, 689. frame.c 689
V1020 The function exited without calling the 'LeaveCriticalSection' function. Check lines: 2605, 2599. blosc2.c 2605
V1020 The function exited without calling the 'LeaveCriticalSection' function. Check lines: 2603, 2599. blosc2.c 2603

V506 Pointer to local variable 'cparams' is stored outside the scope of this variable. Such a pointer will become invalid. schunk.c 211

V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndlz8x8.c 68
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. ndlz8x8.c 67
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndlz8x8.c 69
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndlz4x4.c 66
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. ndlz4x4.c 65
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndlz4x4.c 67
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 20
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 22
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 21
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 137
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 139
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 138
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 227
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 229
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 228
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 369
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 368
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 367
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 485
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 484
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 486
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 617
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 619
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. blosc2-zfp.c 618
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. ndmean.c 16
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndmean.c 18
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndmean.c 17
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. ndmean.c 194
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndmean.c 196
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndmean.c 195
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndcell.c 19
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. ndcell.c 18
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndcell.c 20
V829 Lifetime of the heap-allocated variable 'shape' is limited to the current function's scope. Consider allocating it on the stack instead. ndcell.c 140
V829 Lifetime of the heap-allocated variable 'blockshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndcell.c 142
V829 Lifetime of the heap-allocated variable 'chunkshape' is limited to the current function's scope. Consider allocating it on the stack instead. ndcell.c 141

[FrancescAlted]

Yeah, we need to put way more love on fixing these. Fortunately, we have recently got a grant from NumFOCUS for enhancing python-blosc2, so we should be addressing these soon. Thanks for reporting!

[FrancescAlted]

I think it is still useful to keep this open so to keep track of the PVS-Studio suggestions.