Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New copyright including ironArray SLU #14

Merged
merged 3 commits into from
Apr 16, 2024
Merged

New copyright including ironArray SLU #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2b4970b
Small fices
FrancescAlted Apr 16, 2024
34c0a1c
Add ironArray SLU to the copyright
FrancescAlted Apr 16, 2024
486c305
Force the dtype to be equal in assign
FrancescAlted Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions ANNOUNCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ This mode is only supported for integer datasets (mainly RGB and gray-scale
images).

To get an idea of how this works, see:
https://github.com/Blosc/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode.
https://github.com/ironArray/Blosc2-Btune?tab=readme-ov-file#btune-quality-mode

For more info, please see the release notes in:

https://github.com/Blosc/blosc2_btune/blob/main/RELEASE_NOTES.md
https://github.com/ironArray/Blosc2-Btune/blob/main/RELEASE_NOTES.md


## What is it?
Expand All @@ -22,24 +22,24 @@ Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combina

## Download sources

The github repository is over here:
The github repository is here:

https://github.com/Blosc/blosc2_btune
https://github.com/ironArray/Blosc2-Btune

Blosc is distributed using the GNU Affero General Public License,
Btune is distributed using the GNU Affero General Public License,
see LICENSE.txt for details.

## Mailing list

There is an official Blosc mailing list at:
## Follow us

[email protected]
https://groups.google.com/g/blosc
You can follow us on Mastodon: https://mastodon.social/@ironArray
or on Twitter: https://twitter.com/ironArray

## Tweeter feed

Please follow @Blosc2 to get informed about the latest developments.
## Contact

Please contact ironArray at: https://ironarray.io

Enjoy Data!
- The Blosc Development Team

We make compression better
- The ironArray Team
4 changes: 3 additions & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion LICENSE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
For Blosc - A blocking, shuffling and lossless compression library

Copyright (c) 2023-present, The Blosc Development Team <[email protected]>
All rights reserved.
All rights reserved.
Copyright (c) 2023-present, ironArray SLU <[email protected]>
All rights reserved.

Preamble

Expand Down
2 changes: 1 addition & 1 deletion README-DEVELOPERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Blosc2 Btune
# Btune for Blosc2

For using Btune you will first have to create and install its wheel.

Expand Down
12 changes: 4 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,14 @@
# Blosc2 Btune
# Btune for Blosc2

Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combination of compression parameters. It works by training a neural network on your most representative datasets.

By default, this software uses a genetic algorithm to test different combinations of compression parameters that meet your requirements for both compression ratio and speed for every chunk in the dataset. It assigns a score to each combination and, after a number of iterations, the software stops and uses the best score (minimal value) found for the rest of the dataset. For more info and a graphical visualization, visit https://btune.blosc.org.

The process of finding optimal compression parameters in Blosc2 can be slow because of the large number of combinations of compression parameters (codec, compression level, filter, split mode, number of threads, etc.). This can require a significant amount of trial and error to find the best combinations. However, you can significantly accelerate this process by training a neural network on your own datasets.

To begin the training process, provide your datasets to the Blosc Development Team. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2. In exchange, we request financial contributions to the project.
To begin the training process, you should buy a license from [ironArray SLU](https://ironarray.io) and provide us with a representative sample of your datasets. We will then perform the training and provide neural network models tailored to your needs, along with general tuning advice for Blosc2.

Furthermore, we added another mode for supporting lossy compression for datasets made out of
images which datatypes are integers. This works as a combination of neural networks
and heuristic results.

If interested, please contact us at [email protected].
If interested, please mail us at [email protected]

## Install the Btune wheel

Expand Down Expand Up @@ -89,7 +85,7 @@ You can see in the column `Winner` if the combination is a winner (`W`), it does

## Btune Models

The Blosc Development Team offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the Blosc Development Team has completed training. Btune will then automatically use the trained model; keep reading for how this works.
ironArray SLU offers **Btune Models**, a service in which we provide neural network models trained specifically for your data to determine the optimal combination of codecs and filters. To use these models, set `BTUNE_MODELS_DIR` to the directory containing the models files after the ironArray team has completed the training. Btune will then automatically use the trained model; keep reading for how this works.

To determine the number of chunks for performing inference, use `BTUNE_USE_INFERENCE`. If set to -1, it performs inference on all chunks. If set to a number greater than 0, it performs inference on this number of chunks and then tweaks parameters for the rest of the chunks. If set to 0, it does not perform inference at all. The default is -1.

Expand Down
4 changes: 3 additions & 1 deletion blosc2_btune/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion examples/btune_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
11 changes: 11 additions & 0 deletions examples/btune_example.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/

#include <btune.h>
#include <blosc2/tuners-registry.h>
#include "blosc2.h"
Expand Down
4 changes: 3 additions & 1 deletion examples/create_ndarray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
14 changes: 9 additions & 5 deletions examples/lossy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
##############################################################################
# blosc2_grok: Grok (JPEG2000 codec) plugin for Blosc2
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Development Team <[email protected]>
# https://blosc.org
# License: GNU Affero General Public License v3.0 (see LICENSE.txt)
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################

import blosc2
Expand Down Expand Up @@ -42,7 +45,8 @@
# Create a ndarray made out of the same image `nchunks` times
chunks = blocks = [1] + list(np_array.shape)
bl_array = blosc2.uninit(
shape = [nchunks] + list(np_array.shape),
shape=[nchunks] + list(np_array.shape),
dtype=np_array.dtype,
chunks=chunks,
blocks=blocks,
mode="w",
Expand Down
16 changes: 10 additions & 6 deletions examples/reuse_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
#######################################################################
# Copyright (c) 2019-present, Blosc Development Team <[email protected]>
# All rights reserved.
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# This source code is licensed under a BSD-style license (found in the
# LICENSE file in the root directory of this source tree)
#######################################################################
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################

import blosc2
from time import time
import blosc2_btune
Expand Down
6 changes: 4 additions & 2 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand All @@ -16,7 +18,7 @@ name = "blosc2_btune"
version = "1.2.1.dev0"
readme = "README.md"
authors = [
{name = "Blosc Development Team", email = "contact@blosc.org"},
{name = "ironArray SLU", email = "contact@ironarray.io"},
]
description = "Btune plugin for Blosc2. Automatically choose the best codec/filter for your data."
keywords = ["plugin", "blosc2"]
Expand Down
4 changes: 3 additions & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion src/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
##############################################################################
# Btune for Blosc2 - Automatically choose the best codec/filter for your data
#
# Copyright (c) 2023 The Blosc Developers <[email protected]>
# Copyright (c) 2023-present Blosc Development Team <[email protected]>
# https://btune.blosc.org
# Copyright (c) 2023-present ironArray SLU <[email protected]>
# https://ironarray.io
# License: GNU Affero General Public License v3.0
# See LICENSE.txt for details about copyright and rights to use.
##############################################################################
Expand Down
4 changes: 3 additions & 1 deletion src/btune-private.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data

Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/btune.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/btune.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
13 changes: 10 additions & 3 deletions src/btune_info_public.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
//
// Created by Francesc Alted on 19/10/23.
//
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/

#ifndef BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H
#define BLOSC2_BTUNE_BTUNE_INFO_PUBLIC_H
Expand Down
4 changes: 3 additions & 1 deletion src/btune_model.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/btune_model.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/context.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data

Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/entropy_probe.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
4 changes: 3 additions & 1 deletion src/entropy_probe.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/*********************************************************************
Btune for Blosc2 - Automatically choose the best codec/filter for your data
Copyright (c) 2023 The Blosc Developers <[email protected]>
Copyright (c) 2023-present Blosc Development Team <[email protected]>
https://btune.blosc.org
Copyright (c) 2023-present ironArray SLU <[email protected]>
https://ironarray.io
License: GNU Affero General Public License v3.0
See LICENSE.txt for details about copyright and rights to use.
**********************************************************************/
Expand Down
Loading