From 2b4970bbdd0346c01a363aa69ff220727a024deb Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 16 Apr 2024 09:27:13 +0200 Subject: [PATCH 1/3] Small fices --- ANNOUNCE.md | 26 +++++++++++++------------- README-DEVELOPERS.md | 2 +- README.md | 2 +- 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/ANNOUNCE.md b/ANNOUNCE.md index 6b769fe..f2e19b2 100644 --- a/ANNOUNCE.md +++ b/ANNOUNCE.md @@ -8,11 +8,11 @@ This mode is only supported for integer datasets (mainly RGB and gray-scale images). To get an idea of how this works, see: -https://github.com/Blosc/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode. +https://github.com/ironArray/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode For more info, please see the release notes in: -https://github.com/Blosc/blosc2_btune/blob/main/RELEASE_NOTES.md +https://github.com/ironArray/blosc2_btune/blob/main/RELEASE_NOTES.md ## What is it? @@ -22,24 +22,24 @@ Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combina ## Download sources -The github repository is over here: +The github repository is here: -https://github.com/Blosc/blosc2_btune +https://github.com/ironArray/blosc2_btune -Blosc is distributed using the GNU Affero General Public License, +Btune is distributed using the GNU Affero General Public License, see LICENSE.txt for details. -## Mailing list -There is an official Blosc mailing list at: +## Follow us -blosc@googlegroups.com -https://groups.google.com/g/blosc +You can follow us on Mastodon: https://mastodon.social/@ironArray +or on Twitter: https://twitter.com/ironArray -## Tweeter feed -Please follow @Blosc2 to get informed about the latest developments. +## Contact +Please contact ironArray at: https://ironarray.io -Enjoy Data! -- The Blosc Development Team + +We make compression better +- The ironArray Team diff --git a/README-DEVELOPERS.md b/README-DEVELOPERS.md index 342341b..258dc5b 100644 --- a/README-DEVELOPERS.md +++ b/README-DEVELOPERS.md @@ -1,4 +1,4 @@ -# Blosc2 Btune +# Btune for Blosc2 For using Btune you will first have to create and install its wheel. diff --git a/README.md b/README.md index 433f55b..31352bc 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Blosc2 Btune +# Btune for Blosc2 Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combination of compression parameters. It works by training a neural network on your most representative datasets. From 34c0a1c3cb1a873cdb41474684d9bb83e33e9b49 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 16 Apr 2024 10:36:46 +0200 Subject: [PATCH 2/3] Add ironArray SLU to the copyright --- ANNOUNCE.md | 4 ++-- CMakeLists.txt | 4 +++- LICENSE.txt | 4 +++- README.md | 10 +++------- blosc2_btune/__init__.py | 4 +++- examples/btune_config.py | 4 +++- examples/btune_example.c | 11 +++++++++++ examples/create_ndarray.py | 4 +++- examples/lossy.py | 11 +++++++---- examples/reuse_models.py | 16 ++++++++++------ pyproject.toml | 6 ++++-- setup.py | 4 +++- src/CMakeLists.txt | 4 +++- src/btune-private.h | 4 +++- src/btune.c | 4 +++- src/btune.h | 4 +++- src/btune_info_public.h | 13 ++++++++++--- src/btune_model.cpp | 4 +++- src/btune_model.h | 4 +++- src/context.h | 4 +++- src/entropy_probe.c | 4 +++- src/entropy_probe.h | 4 +++- 22 files changed, 92 insertions(+), 39 deletions(-) diff --git a/ANNOUNCE.md b/ANNOUNCE.md index f2e19b2..6aa95b7 100644 --- a/ANNOUNCE.md +++ b/ANNOUNCE.md @@ -12,7 +12,7 @@ https://github.com/ironArray/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode For more info, please see the release notes in: -https://github.com/ironArray/blosc2_btune/blob/main/RELEASE_NOTES.md +https://github.com/ironArray/Blosc2-Btune/blob/main/RELEASE_NOTES.md ## What is it? @@ -24,7 +24,7 @@ Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combina The github repository is here: -https://github.com/ironArray/blosc2_btune +https://github.com/ironArray/Blosc2-Btune Btune is distributed using the GNU Affero General Public License, see LICENSE.txt for details. diff --git a/CMakeLists.txt b/CMakeLists.txt index 104ca05..cdebc9b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## diff --git a/LICENSE.txt b/LICENSE.txt index c6ced6a..031d659 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -8,7 +8,9 @@ For Blosc - A blocking, shuffling and lossless compression library Copyright (c) 2023-present, The Blosc Development Team - All rights reserved. + All rights reserved. + Copyright (c) 2023-present, ironArray SLU + All rights reserved. Preamble diff --git a/README.md b/README.md index 31352bc..8654d84 100644 --- a/README.md +++ b/README.md @@ -6,13 +6,9 @@ By default, this software uses a genetic algorithm to test different combination The process of finding optimal compression parameters in Blosc2 can be slow because of the large number of combinations of compression parameters (codec, compression level, filter, split mode, number of threads, etc.). This can require a significant amount of trial and error to find the best combinations. However, you can significantly accelerate this process by training a neural network on your own datasets. -To begin the training process, provide your datasets to the Blosc Development Team. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2. In exchange, we request financial contributions to the project. +To begin the training process, you should buy a license from [ironArray SLU](https://ironarray.io) and provide us with a representative sample of your datasets. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2. -Furthermore, we added another mode for supporting lossy compression for datasets made out of -images which datatypes are integers. This works as a combination of neural networks -and heuristic results. - -If interested, please contact us at contact@blosc.org. +If interested, please mail us at contact@ironarray.io ## Install the Btune wheel @@ -89,7 +85,7 @@ You can see in the column `Winner` if the combination is a winner (`W`), it does ## Btune Models -The Blosc Development Team offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the Blosc Development Team has completed training. Btune will then automatically use the trained model; keep reading for how this works. +ironArray SLU offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the ironArray team has completed the training. Btune will then automatically use the trained model; keep reading for how this works. To determine the number of chunks for performing inference, use `BTUNE_USE_INFERENCE`. If set to -1, it performs inference on all chunks. If set to a number greater than 0, it performs inference on this number of chunks and then tweaks parameters for the rest of the chunks. If set to 0, it does not perform inference at all. The default is -1. diff --git a/blosc2_btune/__init__.py b/blosc2_btune/__init__.py index f8bc3d5..e173afd 100644 --- a/blosc2_btune/__init__.py +++ b/blosc2_btune/__init__.py @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## diff --git a/examples/btune_config.py b/examples/btune_config.py index 69baf65..502c8e3 100644 --- a/examples/btune_config.py +++ b/examples/btune_config.py @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## diff --git a/examples/btune_example.c b/examples/btune_example.c index ec8974a..535c2c6 100644 --- a/examples/btune_example.c +++ b/examples/btune_example.c @@ -1,3 +1,14 @@ +/********************************************************************* + Btune for Blosc2 - Automatically choose the best codec/filter for your data + + Copyright (c) 2023-present Blosc Development Team + https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io + License: GNU Affero General Public License v3.0 + See LICENSE.txt for details about copyright and rights to use. +**********************************************************************/ + #include #include #include "blosc2.h" diff --git a/examples/create_ndarray.py b/examples/create_ndarray.py index 41c93b3..e8cc18b 100644 --- a/examples/create_ndarray.py +++ b/examples/create_ndarray.py @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## diff --git a/examples/lossy.py b/examples/lossy.py index 75b4c98..9740ad9 100644 --- a/examples/lossy.py +++ b/examples/lossy.py @@ -1,9 +1,12 @@ ############################################################################## -# blosc2_grok: Grok (JPEG2000 codec) plugin for Blosc2 +# Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Development Team -# https://blosc.org -# License: GNU Affero General Public License v3.0 (see LICENSE.txt) +# Copyright (c) 2023-present Blosc Development Team +# https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io +# License: GNU Affero General Public License v3.0 +# See LICENSE.txt for details about copyright and rights to use. ############################################################################## import blosc2 diff --git a/examples/reuse_models.py b/examples/reuse_models.py index 7affc86..430eac7 100644 --- a/examples/reuse_models.py +++ b/examples/reuse_models.py @@ -1,10 +1,14 @@ -####################################################################### -# Copyright (c) 2019-present, Blosc Development Team -# All rights reserved. +############################################################################## +# Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# This source code is licensed under a BSD-style license (found in the -# LICENSE file in the root directory of this source tree) -####################################################################### +# Copyright (c) 2023-present Blosc Development Team +# https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io +# License: GNU Affero General Public License v3.0 +# See LICENSE.txt for details about copyright and rights to use. +############################################################################## + import blosc2 from time import time import blosc2_btune diff --git a/pyproject.toml b/pyproject.toml index c23fe66..ef6297b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## @@ -16,7 +18,7 @@ name = "blosc2_btune" version = "1.2.1.dev0" readme = "README.md" authors = [ - {name = "Blosc Development Team", email = "contact@blosc.org"}, + {name = "ironArray SLU", email = "contact@ironarray.io"}, ] description = "Btune plugin for Blosc2. Automatically choose the best codec/filter for your data." keywords = ["plugin", "blosc2"] diff --git a/setup.py b/setup.py index 9ebbad4..7ea9b14 100644 --- a/setup.py +++ b/setup.py @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index a399014..b9e70f1 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -1,8 +1,10 @@ ############################################################################## # Btune for Blosc2 - Automatically choose the best codec/filter for your data # -# Copyright (c) 2023 The Blosc Developers +# Copyright (c) 2023-present Blosc Development Team # https://btune.blosc.org +# Copyright (c) 2023-present ironArray SLU +# https://ironarray.io # License: GNU Affero General Public License v3.0 # See LICENSE.txt for details about copyright and rights to use. ############################################################################## diff --git a/src/btune-private.h b/src/btune-private.h index 72323ea..805c90d 100644 --- a/src/btune-private.h +++ b/src/btune-private.h @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/btune.c b/src/btune.c index 22948de..1406df3 100644 --- a/src/btune.c +++ b/src/btune.c @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/btune.h b/src/btune.h index eccbaf2..3b1e6e1 100644 --- a/src/btune.h +++ b/src/btune.h @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/btune_info_public.h b/src/btune_info_public.h index ee53295..c0e7703 100644 --- a/src/btune_info_public.h +++ b/src/btune_info_public.h @@ -1,6 +1,13 @@ -// -// Created by Francesc Alted on 19/10/23. -// +/********************************************************************* + Btune for Blosc2 - Automatically choose the best codec/filter for your data + + Copyright (c) 2023-present Blosc Development Team + https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io + License: GNU Affero General Public License v3.0 + See LICENSE.txt for details about copyright and rights to use. +**********************************************************************/ #ifndef BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H #define BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H diff --git a/src/btune_model.cpp b/src/btune_model.cpp index aba7a9b..35729d3 100644 --- a/src/btune_model.cpp +++ b/src/btune_model.cpp @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/btune_model.h b/src/btune_model.h index 723be5d..3d46f31 100644 --- a/src/btune_model.h +++ b/src/btune_model.h @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/context.h b/src/context.h index 2eeeb6c..de91298 100644 --- a/src/context.h +++ b/src/context.h @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/entropy_probe.c b/src/entropy_probe.c index 750778a..e0ab818 100644 --- a/src/entropy_probe.c +++ b/src/entropy_probe.c @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ diff --git a/src/entropy_probe.h b/src/entropy_probe.h index 84e76bd..a7d22c3 100644 --- a/src/entropy_probe.h +++ b/src/entropy_probe.h @@ -1,8 +1,10 @@ /********************************************************************* Btune for Blosc2 - Automatically choose the best codec/filter for your data - Copyright (c) 2023 The Blosc Developers + Copyright (c) 2023-present Blosc Development Team https://btune.blosc.org + Copyright (c) 2023-present ironArray SLU + https://ironarray.io License: GNU Affero General Public License v3.0 See LICENSE.txt for details about copyright and rights to use. **********************************************************************/ From 486c305e63e52f3c31f7ea1c2cc729d6f357dae4 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 16 Apr 2024 11:57:23 +0200 Subject: [PATCH 3/3] Force the dtype to be equal in assign --- examples/lossy.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/examples/lossy.py b/examples/lossy.py index 9740ad9..af9ba17 100644 --- a/examples/lossy.py +++ b/examples/lossy.py @@ -45,7 +45,8 @@ # Create a ndarray made out of the same image `nchunks` times chunks = blocks = [1] + list(np_array.shape) bl_array = blosc2.uninit( - shape = [nchunks] + list(np_array.shape), + shape=[nchunks] + list(np_array.shape), + dtype=np_array.dtype, chunks=chunks, blocks=blocks, mode="w",