Code

[DRAFT] Pyfive Masterpiece#189

Code

Draft

valeriupredoi wants to merge 104 commits intomainNCAS-CMS/PyActiveStorage:mainfrom pyfiveNCAS-CMS/PyActiveStorage:pyfiveCopy head branch name to clipboard

+1,686-1,334

Commits

Commits on Feb 29, 2024

• A few wrinkles to sort out, some changes to pyfive needed, but this is a pre-train interim commit

  

  Bryan Lawrence

  committed
  e9f9ee6

  Copy full SHA for e9f9ee6

Commits on Mar 3, 2024

• bnl_test works with pyfive. The tests are borked because they're riddled with zarr dependencies

  

  Bryan Lawrence

  committed
  eb9e853

  Copy full SHA for eb9e853

Commits on Mar 4, 2024

• Gzip decompression working. Will document issues remaining in #188

  

  Bryan Lawrence

  committed
  0d7bf55

  Copy full SHA for 0d7bf55

• Split compression and filters

  

  Bryan Lawrence

  committed
  d73fcd8

  Copy full SHA for d73fcd8

• Slight doc improvement in hdf2numcodec

  

  Bryan Lawrence

  committed
  0c4272f

  Copy full SHA for 0c4272f

• Need to turn off decoding filter pipeline if there is none.

  

  Bryan Lawrence

  committed
  c20b259

  Copy full SHA for c20b259

Commits on Mar 5, 2024

• Support for contiguous data. Passing lots of tests now.

  

  Bryan Lawrence

  committed
  68e49d0

  Copy full SHA for 68e49d0

• Remove the file context manager for now

  

  Bryan Lawrence

  committed
  b745d5e

  Copy full SHA for b745d5e

• A couple more h5py lurkers and one more bug squashed.

  

  Bryan Lawrence

  committed
  74f8662

  Copy full SHA for 74f8662

• removed redundant test file

  valeriupredoi

  committed
  b628a68

  Copy full SHA for b628a68

• removed redundant test cases

  valeriupredoi

  committed
  1803e7f

  Copy full SHA for 1803e7f

• making sure we know Pyfive is run haha

  valeriupredoi

  committed
  f5a375c

  Copy full SHA for f5a375c

• fix bigger data test with native pyfive exception

  valeriupredoi

  committed
  d1c9282

  Copy full SHA for d1c9282

• fix package test

  valeriupredoi

  committed
  69c3dca

  Copy full SHA for 69c3dca

• mark lock test as xfail

  valeriupredoi

  committed
  c50c3d2

  Copy full SHA for c50c3d2

• toy for testing missing test

  

  Bryan Lawrence

  committed
  09b4b3f

  Copy full SHA for 09b4b3f

• Merge branches 'pyfive' and 'pyfive' of github.com:valeriupredoi/PyActiveStorage into pyfive

  

  Bryan Lawrence

  committed
  03254dc

  Copy full SHA for 03254dc

• turn off some printing for now

  valeriupredoi

  committed
  8a7e35b

  Copy full SHA for 8a7e35b

• starting to understand the test flakiness

  valeriupredoi

  committed
  cacb94e

  Copy full SHA for cacb94e

• Addresses issues with missing data and with thread safety (basically we need to read the chunk cache before sending things off into the weeds).

  

  Bryan Lawrence

  committed
  ce0479c

  Copy full SHA for ce0479c

• Merge branch 'pyfive' of github.com:valeriupredoi/PyActiveStorage into pyfive

  

  Bryan Lawrence

  committed
  39884bc

  Copy full SHA for 39884bc

Commits on Mar 6, 2024

• Fixed the regression wrt files with no chunking introduced by thread safety.

  

  Bryan Lawrence

  committed
  02d6120

  Copy full SHA for 02d6120

• Made some changes to Active just so we can use masked data as a standalone method which I don't like. We would be better to pull that out and do this differently. I also fixed the disgusting test ...

  

  Bryan Lawrence

  committed
  2ffd644

  Copy full SHA for 2ffd644

• fix test compression

  valeriupredoi

  committed
  408558f

  Copy full SHA for 408558f

• fix the other compression test

  valeriupredoi

  committed
  3294052

  Copy full SHA for 3294052

• fix mocker tests

  valeriupredoi

  committed
  ca30bb2

  Copy full SHA for ca30bb2

• minor fix to active

  valeriupredoi

  committed
  c1b2823

  Copy full SHA for c1b2823

• ass test todo

  valeriupredoi

  committed
  3c1b420

  Copy full SHA for 3c1b420

• fix last test

  valeriupredoi

  committed
  589c772

  Copy full SHA for 589c772

• checkout and install pyfive in dev mode

  valeriupredoi

  committed
  c022040

  Copy full SHA for c022040

• fix Bryans test

  valeriupredoi

  committed
  016242e

  Copy full SHA for 016242e

• install Pyfive in OSX GHA too

  valeriupredoi

  committed
  9bfb099

  Copy full SHA for 9bfb099

• run the s3 tests too with Minio

  valeriupredoi

  committed
  68827eb

  Copy full SHA for 68827eb

• run the s3 tests too with remote reductionist

  valeriupredoi

  committed
  1d925dc

  Copy full SHA for 1d925dc

• Fixed incorrect handling of missing data in reductionist encode_missing

  

  Bryan Lawrence

  committed
  400e230

  Copy full SHA for 400e230

• it's not a test!

  

  Bryan Lawrence

  committed
  afe3546

  Copy full SHA for afe3546

• Fixed HDF5 NetCDF attribute issue

  

  Bryan Lawrence

  committed
  44980b7

  Copy full SHA for 44980b7

• A little more care was needed ...

  

  Bryan Lawrence

  committed
  98a74c5

  Copy full SHA for 98a74c5

• new pyfive File API

  valeriupredoi

  committed
  4bea4bb

  Copy full SHA for 4bea4bb

• print out the attrs

  valeriupredoi

  committed
  cf1ff2d

  Copy full SHA for cf1ff2d

• print out file attrs

  valeriupredoi

  committed
  2808dc9

  Copy full SHA for 2808dc9

Commits on Mar 7, 2024

• V made me do it

  

  Bryan Lawrence

  committed
  38add11

  Copy full SHA for 38add11

• use correct attributes compression and shuffle

  valeriupredoi

  committed
  277762d

  Copy full SHA for 277762d

• use issue60 branch

  valeriupredoi

  committed
  5bd0fe1

  Copy full SHA for 5bd0fe1

Commits on Mar 8, 2024

• Unit test for createding reductionist json without s3

  

  Bryan Lawrence

  committed
  74beccd

  Copy full SHA for 74beccd

• Reductionist was incorrectly encoding offset and size, and the test for it had defaults of None (which it can never be). Also modified tests to make easier to extract and debug standalone.

  

  Bryan Lawrence

  committed
  e91d02c

  Copy full SHA for e91d02c

• Will this break actions?

  

  Bryan Lawrence

  committed
  3bb0b8b

  Copy full SHA for 3bb0b8b

• Merge branch 'pyfive' of github.com:valeriupredoi/PyActiveStorage into pyfive

  

  Bryan Lawrence

  committed
  12d0f0b

  Copy full SHA for 12d0f0b

• ignore bnl for testing

  valeriupredoi

  committed
  7a03fa4

  Copy full SHA for 7a03fa4

• correct imports

  valeriupredoi

  committed
  aa1a481

  Copy full SHA for aa1a481

• fix test with correct loaders and imports

  valeriupredoi

  committed
  e5430fd

  Copy full SHA for e5430fd

• same for this one

  valeriupredoi

  committed
  dd57664

  Copy full SHA for dd57664

• clean test module

  valeriupredoi

  committed
  f317149

  Copy full SHA for f317149

• reduce test activity

  valeriupredoi

  committed
  4dafa3e

  Copy full SHA for 4dafa3e

• plop Pyfive in the PR test wkflow too

  valeriupredoi

  committed
  e9aff71

  Copy full SHA for e9aff71

Commits on Mar 11, 2024

• Adding the ability for Active to optionally self report metrics

  

  Bryan Lawrence

  committed
  08c8b87

  Copy full SHA for 08c8b87

• Merge branch 'pyfive' of github.com:valeriupredoi/PyActiveStorage into pyfive

  

  Bryan Lawrence

  committed
  2475b18

  Copy full SHA for 2475b18

Commits on Mar 12, 2024

• Adding some experimental data

  

  Bryan Lawrence

  committed
  20acfdd

  Copy full SHA for 20acfdd

• Plotting and data code.

  

  Bryan Lawrence

  committed
  54967d2

  Copy full SHA for 54967d2

Commits on Mar 14, 2024

• Added more organised metrics to Active, available after a getitem from .metric_data attribute

  

  Bryan Lawrence

  committed
  20b36d1

  Copy full SHA for 20b36d1

• Active _version1 and _version2 differ again

  

  Bryan Lawrence

  committed
  ddcff5c

  Copy full SHA for ddcff5c

Commits on Mar 19, 2024

• Active v1 on S3 now works. Stopping default verbosity from reductionist

  

  Bryan Lawrence

  committed
  df6b5bb

  Copy full SHA for df6b5bb

Commits on Mar 23, 2024

• test runner

  

  Bryan Lawrence

  committed
  ef001ee

  Copy full SHA for ef001ee

Commits on Mar 25, 2024

• all-zero chunks

  davidhassell

  committed
  7c1e704

  Copy full SHA for 7c1e704

• new test file

  davidhassell

  committed
  209d3fa

  Copy full SHA for 209d3fa

• fix previously failing Anon bkt test

  valeriupredoi

  committed
  620b403

  Copy full SHA for 620b403

• timeOut for v2, v1 does the pancakes now

  valeriupredoi

  committed
  b49ffa0

  Copy full SHA for b49ffa0

• timeOut for v2, v1 does the pancakes now

  valeriupredoi

  committed
  e99a509

  Copy full SHA for e99a509

• and again v2

  valeriupredoi

  committed
  d1dc658

  Copy full SHA for d1dc658

• another pancake

  valeriupredoi

  committed
  b43dd21

  Copy full SHA for b43dd21

• Merge pull request #196 from valeriupredoi/zero-chunk

  Show description for 4816902
  valeriupredoi

  authored
  4816902

  Copy full SHA for 4816902

Commits on Mar 26, 2024

• zero count

  davidhassell

  committed
  b5d5925

  Copy full SHA for b5d5925

Commits on Mar 27, 2024

• fix test

  valeriupredoi

  committed
  8b46b4b

  Copy full SHA for 8b46b4b

Commits on Apr 25, 2024

• Merge pull request #197 from valeriupredoi/zero-chunk

  Show description for e4f2067
  valeriupredoi

  authored
  e4f2067

  Copy full SHA for e4f2067

Commits on Jul 22, 2024

• Merge branch 'main' into pyfive

  valeriupredoi

  committed
  648684a

  Copy full SHA for 648684a

• add a conda list cmd

  valeriupredoi

  committed
  2152ac5

  Copy full SHA for 2152ac5

Commits on Aug 5, 2024

• Merge branch 'main' into pyfive

  valeriupredoi

  committed
  0a4297a

  Copy full SHA for 0a4297a

Commits on Aug 12, 2024

• Merge branch 'main' into pyfive

  valeriupredoi

  authored
  74868b7

  Copy full SHA for 74868b7

Commits on Oct 24, 2024

• Merge branch 'main' into pyfive

  valeriupredoi

  authored
  4b18a17

  Copy full SHA for 4b18a17

Commits on Jan 20, 2025

• Merge branch 'main' into pyfive

  valeriupredoi

  authored
  5f67387

  Copy full SHA for 5f67387

• start work to port new Pyfive

  valeriupredoi

  committed
  8b08743

  Copy full SHA for 8b08743

• correct test for new Pyive API

  valeriupredoi

  committed
  ab64817

  Copy full SHA for ab64817

Commits on Jan 21, 2025

• Conforming to current index structure a bit better

  

  Bryan Lawrence

  committed
  5c8095e

  Copy full SHA for 5c8095e

• new GH repo and branch for Pyfive in GAs

  valeriupredoi

  committed
  fe88026

  Copy full SHA for fe88026

• Merge branch 'fix_pyfive_branch' of https://github.com/NCAS-CMS/PyActiveStorage into fix_pyfive_branch

  valeriupredoi

  committed
  d3a8f92

  Copy full SHA for d3a8f92

• Use the new pyfive method to get chunk position in file. Includes bad hack for contiguous data.

  

  Bryan Lawrence

  committed
  e68a0fe

  Copy full SHA for e68a0fe

• Merge remote-tracking branch 'refs/remotes/origin/fix_pyfive_branch' into fix_pyfive_branch

  

  Bryan Lawrence

  committed
  ca5022e

  Copy full SHA for ca5022e

• fix the mocker json test

  valeriupredoi

  committed
  a728ef0

  Copy full SHA for a728ef0

• new fh method

  valeriupredoi

  committed
  3df5b36

  Copy full SHA for 3df5b36

• small typo

  valeriupredoi

  committed
  762d85a

  Copy full SHA for 762d85a

• Cleared up the compression problem

  

  Bryan Lawrence

  committed
  f3c35e0

  Copy full SHA for f3c35e0

• Merge remote-tracking branch 'refs/remotes/origin/fix_pyfive_branch' into fix_pyfive_branch

  

  Bryan Lawrence

  committed
  41c9ce9

  Copy full SHA for 41c9ce9

• Merge pull request #234 from NCAS-CMS/fix_pyfive_branch

  Show description for bd1f06f
  valeriupredoi

  authored
  bd1f06f

  Copy full SHA for bd1f06f

• reinstate tests

  valeriupredoi

  committed
  2fabbe3

  Copy full SHA for 2fabbe3

Commits on Jan 22, 2025

• remove bnl references

  valeriupredoi

  committed
  68851e4

  Copy full SHA for 68851e4

• remove workflow that runs bnl bucket test

  valeriupredoi

  committed
  1d13c6a

  Copy full SHA for 1d13c6a

• rplace bnl with relevant info

  valeriupredoi

  committed
  29fd648

  Copy full SHA for 29fd648

• remove commented out Bryan server

  valeriupredoi

  committed
  4792342

  Copy full SHA for 4792342

• rm ref Bryan

  valeriupredoi

  committed
  43ea853

  Copy full SHA for 43ea853

Commits on Feb 11, 2025

• Merge branch 'main' into pyfive

  valeriupredoi

  committed
  549711c

  Copy full SHA for 549711c

• use wacasoft branch of Pyfive fork

  valeriupredoi

  committed
  e73712a

  Copy full SHA for e73712a

Commits on Feb 24, 2025

• Merge branch 'main' into pyfive

  valeriupredoi

  authored
  a39e4ea

  Copy full SHA for a39e4ea

Commits on Feb 25, 2025

• Merge branch 'main' into pyfive

  valeriupredoi

  committed
  0cadeb3

  Copy full SHA for 0cadeb3

• Merge branch 'pyfive' of https://github.com/NCAS-CMS/PyActiveStorage into pyfive

  valeriupredoi

  committed
  bd47be8

  Copy full SHA for bd47be8