-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #14 from ironArray/new-copyright
New copyright including ironArray SLU
- Loading branch information
Showing
23 changed files
with
107 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,11 +8,11 @@ This mode is only supported for integer datasets (mainly RGB and gray-scale | |
images). | ||
|
||
To get an idea of how this works, see: | ||
https://github.com/Blosc/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode. | ||
https://github.com/ironArray/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode | ||
|
||
For more info, please see the release notes in: | ||
|
||
https://github.com/Blosc/blosc2_btune/blob/main/RELEASE_NOTES.md | ||
https://github.com/ironArray/Blosc2-Btune/blob/main/RELEASE_NOTES.md | ||
|
||
|
||
## What is it? | ||
|
@@ -22,24 +22,24 @@ Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combina | |
|
||
## Download sources | ||
|
||
The github repository is over here: | ||
The github repository is here: | ||
|
||
https://github.com/Blosc/blosc2_btune | ||
https://github.com/ironArray/Blosc2-Btune | ||
|
||
Blosc is distributed using the GNU Affero General Public License, | ||
Btune is distributed using the GNU Affero General Public License, | ||
see LICENSE.txt for details. | ||
|
||
## Mailing list | ||
|
||
There is an official Blosc mailing list at: | ||
## Follow us | ||
|
||
[email protected] | ||
https://groups.google.com/g/blosc | ||
You can follow us on Mastodon: https://mastodon.social/@ironArray | ||
or on Twitter: https://twitter.com/ironArray | ||
|
||
## Tweeter feed | ||
|
||
Please follow @Blosc2 to get informed about the latest developments. | ||
## Contact | ||
|
||
Please contact ironArray at: https://ironarray.io | ||
|
||
Enjoy Data! | ||
- The Blosc Development Team | ||
|
||
We make compression better | ||
- The ironArray Team |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,7 +8,9 @@ | |
For Blosc - A blocking, shuffling and lossless compression library | ||
|
||
Copyright (c) 2023-present, The Blosc Development Team <[email protected]> | ||
All rights reserved. | ||
All rights reserved. | ||
Copyright (c) 2023-present, ironArray SLU <[email protected]> | ||
All rights reserved. | ||
|
||
Preamble | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
# Blosc2 Btune | ||
# Btune for Blosc2 | ||
|
||
For using Btune you will first have to create and install its wheel. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,18 +1,14 @@ | ||
# Blosc2 Btune | ||
# Btune for Blosc2 | ||
|
||
Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combination of compression parameters. It works by training a neural network on your most representative datasets. | ||
|
||
By default, this software uses a genetic algorithm to test different combinations of compression parameters that meet your requirements for both compression ratio and speed for every chunk in the dataset. It assigns a score to each combination and, after a number of iterations, the software stops and uses the best score (minimal value) found for the rest of the dataset. For more info and a graphical visualization, visit https://btune.blosc.org. | ||
|
||
The process of finding optimal compression parameters in Blosc2 can be slow because of the large number of combinations of compression parameters (codec, compression level, filter, split mode, number of threads, etc.). This can require a significant amount of trial and error to find the best combinations. However, you can significantly accelerate this process by training a neural network on your own datasets. | ||
|
||
To begin the training process, provide your datasets to the Blosc Development Team. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2. In exchange, we request financial contributions to the project. | ||
To begin the training process, you should buy a license from [ironArray SLU](https://ironarray.io) and provide us with a representative sample of your datasets. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2. | ||
|
||
Furthermore, we added another mode for supporting lossy compression for datasets made out of | ||
images which datatypes are integers. This works as a combination of neural networks | ||
and heuristic results. | ||
|
||
If interested, please contact us at [email protected]. | ||
If interested, please mail us at [email protected] | ||
|
||
## Install the Btune wheel | ||
|
||
|
@@ -89,7 +85,7 @@ You can see in the column `Winner` if the combination is a winner (`W`), it does | |
|
||
## Btune Models | ||
|
||
The Blosc Development Team offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the Blosc Development Team has completed training. Btune will then automatically use the trained model; keep reading for how this works. | ||
ironArray SLU offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the ironArray team has completed the training. Btune will then automatically use the trained model; keep reading for how this works. | ||
|
||
To determine the number of chunks for performing inference, use `BTUNE_USE_INFERENCE`. If set to -1, it performs inference on all chunks. If set to a number greater than 0, it performs inference on this number of chunks and then tweaks parameters for the rest of the chunks. If set to 0, it does not perform inference at all. The default is -1. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,14 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
||
#include <btune.h> | ||
#include <blosc2/tuners-registry.h> | ||
#include "blosc2.h" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,12 @@ | ||
############################################################################## | ||
# blosc2_grok: Grok (JPEG2000 codec) plugin for Blosc2 | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Development Team <[email protected]> | ||
# https://blosc.org | ||
# License: GNU Affero General Public License v3.0 (see LICENSE.txt) | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
||
import blosc2 | ||
|
@@ -42,7 +45,8 @@ | |
# Create a ndarray made out of the same image `nchunks` times | ||
chunks = blocks = [1] + list(np_array.shape) | ||
bl_array = blosc2.uninit( | ||
shape = [nchunks] + list(np_array.shape), | ||
shape=[nchunks] + list(np_array.shape), | ||
dtype=np_array.dtype, | ||
chunks=chunks, | ||
blocks=blocks, | ||
mode="w", | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,14 @@ | ||
####################################################################### | ||
# Copyright (c) 2019-present, Blosc Development Team <[email protected]> | ||
# All rights reserved. | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# This source code is licensed under a BSD-style license (found in the | ||
# LICENSE file in the root directory of this source tree) | ||
####################################################################### | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
||
import blosc2 | ||
from time import time | ||
import blosc2_btune | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
@@ -16,7 +18,7 @@ name = "blosc2_btune" | |
version = "1.2.1.dev0" | ||
readme = "README.md" | ||
authors = [ | ||
{name = "Blosc Development Team", email = "contact@blosc.org"}, | ||
{name = "ironArray SLU", email = "contact@ironarray.io"}, | ||
] | ||
description = "Btune plugin for Blosc2. Automatically choose the best codec/filter for your data." | ||
keywords = ["plugin", "blosc2"] | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
############################################################################## | ||
# Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
# | ||
# Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
# Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
# https://btune.blosc.org | ||
# Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
# https://ironarray.io | ||
# License: GNU Affero General Public License v3.0 | ||
# See LICENSE.txt for details about copyright and rights to use. | ||
############################################################################## | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,13 @@ | ||
// | ||
// Created by Francesc Alted on 19/10/23. | ||
// | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
||
#ifndef BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H | ||
#define BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,10 @@ | ||
/********************************************************************* | ||
Btune for Blosc2 - Automatically choose the best codec/filter for your data | ||
Copyright (c) 2023 The Blosc Developers <[email protected]> | ||
Copyright (c) 2023-present Blosc Development Team <[email protected]> | ||
https://btune.blosc.org | ||
Copyright (c) 2023-present ironArray SLU <[email protected]> | ||
https://ironarray.io | ||
License: GNU Affero General Public License v3.0 | ||
See LICENSE.txt for details about copyright and rights to use. | ||
**********************************************************************/ | ||
|