Skip to content

Commit

Permalink
Merge pull request #14 from ironArray/new-copyright
Browse files Browse the repository at this point in the history
New copyright including ironArray SLU
  • Loading branch information
FrancescAlted authored Apr 16, 2024
2 parents 892ae9d + 486c305 commit 538b7b1
Show file tree
Hide file tree
Showing 23 changed files with 107 additions and 53 deletions.
26 changes: 13 additions & 13 deletions ANNOUNCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ This mode is only supported for integer datasets (mainly RGB and gray-scale
images).

To get an idea of how this works, see:
https://github.com/Blosc/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode.
https://github.com/ironArray/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode

For more info, please see the release notes in:

https://github.com/Blosc/blosc2_btune/blob/main/RELEASE_NOTES.md
https://github.com/ironArray/Blosc2-Btune/blob/main/RELEASE_NOTES.md


## What is it?
Expand All @@ -22,24 +22,24 @@ Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combina

## Download sources

The github repository is over here:
The github repository is here:

https://github.com/Blosc/blosc2_btune
https://github.com/ironArray/Blosc2-Btune

Blosc is distributed using the GNU Affero General Public License,
Btune is distributed using the GNU Affero General Public License,
see LICENSE.txt for details.

## Mailing list

There is an official Blosc mailing list at:
## Follow us

[email protected]
https://groups.google.com/g/blosc
You can follow us on Mastodon: https://mastodon.social/@ironArray
or on Twitter: https://twitter.com/ironArray

## Tweeter feed

Please follow @Blosc2 to get informed about the latest developments.
## Contact

Please contact ironArray at: https://ironarray.io

Enjoy Data!
- The Blosc Development Team

We make compression better
- The ironArray Team
4 changes: 3 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion LICENSE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
For Blosc - A blocking, shuffling and lossless compression library

Copyright (c) 2023-present, The Blosc Development Team <[email protected]>
All rights reserved.
All rights reserved.
Copyright (c) 2023-present, ironArray SLU <[email protected]>
All rights reserved.

Preamble

Expand Down
2 changes: 1 addition & 1 deletion README-DEVELOPERS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Blosc2 Btune
# Btune for Blosc2

For using Btune you will first have to create and install its wheel.

Expand Down
12 changes: 4 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,14 @@
# Blosc2 Btune
# Btune for Blosc2

Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combination of compression parameters. It works by training a neural network on your most representative datasets.

By default, this software uses a genetic algorithm to test different combinations of compression parameters that meet your requirements for both compression ratio and speed for every chunk in the dataset. It assigns a score to each combination and, after a number of iterations, the software stops and uses the best score (minimal value) found for the rest of the dataset. For more info and a graphical visualization, visit https://btune.blosc.org.

The process of finding optimal compression parameters in Blosc2 can be slow because of the large number of combinations of compression parameters (codec, compression level, filter, split mode, number of threads, etc.). This can require a significant amount of trial and error to find the best combinations. However, you can significantly accelerate this process by training a neural network on your own datasets.

To begin the training process, provide your datasets to the Blosc Development Team. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2. In exchange, we request financial contributions to the project.
To begin the training process, you should buy a license from [ironArray SLU](https://ironarray.io) and provide us with a representative sample of your datasets. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2.

Furthermore, we added another mode for supporting lossy compression for datasets made out of
images which datatypes are integers. This works as a combination of neural networks
and heuristic results.

If interested, please contact us at [email protected].
If interested, please mail us at [email protected]

## Install the Btune wheel

Expand Down Expand Up @@ -89,7 +85,7 @@ You can see in the column `Winner` if the combination is a winner (`W`), it does

## Btune Models

The Blosc Development Team offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the Blosc Development Team has completed training. Btune will then automatically use the trained model; keep reading for how this works.
ironArray SLU offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the ironArray team has completed the training. Btune will then automatically use the trained model; keep reading for how this works.

To determine the number of chunks for performing inference, use `BTUNE_USE_INFERENCE`. If set to -1, it performs inference on all chunks. If set to a number greater than 0, it performs inference on this number of chunks and then tweaks parameters for the rest of the chunks. If set to 0, it does not perform inference at all. The default is -1.

Expand Down
4 changes: 3 additions & 1 deletion blosc2_btune/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion examples/btune_config.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
11 changes: 11 additions & 0 deletions examples/btune_example.c
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/

#include <btune.h>
#include <blosc2/tuners-registry.h>
#include "blosc2.h"
Expand Down
4 changes: 3 additions & 1 deletion examples/create_ndarray.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
14 changes: 9 additions & 5 deletions examples/lossy.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
##############################################################################
# blosc2_grok: Grok (JPEG2000 codec) plugin for Blosc2
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Development Team <[email protected]>
# https://blosc.org
# License: GNU Affero General Public License v3.0 (see LICENSE.txt)
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################

import blosc2
Expand Down Expand Up @@ -42,7 +45,8 @@
# Create a ndarray made out of the same image `nchunks` times
chunks = blocks = [1] + list(np_array.shape)
bl_array = blosc2.uninit(
shape = [nchunks] + list(np_array.shape),
shape=[nchunks] + list(np_array.shape),
dtype=np_array.dtype,
chunks=chunks,
blocks=blocks,
mode="w",
Expand Down
16 changes: 10 additions & 6 deletions examples/reuse_models.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
#######################################################################
# Copyright (c) 2019-present, Blosc Development Team <[email protected]>
# All rights reserved.
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# This source code is licensed under a BSD-style license (found in the
# LICENSE file in the root directory of this source tree)
#######################################################################
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################

import blosc2
from time import time
import blosc2_btune
Expand Down
6 changes: 4 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand All @@ -16,7 +18,7 @@ name = "blosc2_btune"
version = "1.2.1.dev0"
readme = "README.md"
authors = [
{name = "Blosc Development Team", email = "contact@blosc.org"},
{name = "ironArray SLU", email = "contact@ironarray.io"},
]
description = "Btune plugin for Blosc2. Automatically choose the best codec/filter for your data."
keywords = ["plugin", "blosc2"]
Expand Down
4 changes: 3 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion src/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion src/btune-private.h
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/btune.c
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/btune.h
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
13 changes: 10 additions & 3 deletions src/btune_info_public.h
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
//
// Created by Francesc Alted on 19/10/23.
//
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/

#ifndef BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H
#define BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H
Expand Down
4 changes: 3 additions & 1 deletion src/btune_model.cpp
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/btune_model.h
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/context.h
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/entropy_probe.c
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/entropy_probe.h
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down

0 comments on commit 538b7b1

Please sign in to comment.