Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kt deskew ngff #58

Merged
merged 117 commits into from
Jan 18, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
04d776b
added ims_to_zarr function
kttakasaki Oct 11, 2023
55a5b8d
debugging
kttakasaki Oct 11, 2023
0a9eab6
debugging
kttakasaki Oct 11, 2023
0dc3971
debugging
kttakasaki Oct 11, 2023
c4ae796
debugging
kttakasaki Oct 11, 2023
062a8d0
debugging
kttakasaki Oct 11, 2023
c21c6ee
adding support for permuting stage axes coordinates in group position…
kttakasaki Feb 16, 2024
94d746f
having trouble with argschema implementation of axes swapping
kttakasaki Feb 16, 2024
ca92b21
module for visor-zarr conversion
kttakasaki Jun 4, 2024
09aa81e
cleaning up module
kttakasaki Jun 4, 2024
98fd1b4
streamlining
kttakasaki Jun 4, 2024
2fcf56b
write to same zarr
kttakasaki Jun 4, 2024
6a5f3ba
fixing mip groups
kttakasaki Jun 4, 2024
5d61599
fixing mip groups
kttakasaki Jun 4, 2024
2cb0ef9
refactoring shared methods
kttakasaki Jul 9, 2024
225b4cf
input parameter fix
kttakasaki Jul 9, 2024
0d7b75f
adding transpose for exaspim deskew
kttakasaki Jul 9, 2024
206cceb
fixing
kttakasaki Jul 9, 2024
9afaca7
redundant methods
kttakasaki Jul 16, 2024
c84c760
clean up ims conversion module old code
kttakasaki Jul 30, 2024
720b391
attempting block conversion
kttakasaki Jul 31, 2024
5995d63
fixing block conversion
kttakasaki Jul 31, 2024
e93b426
fixing block conversion
kttakasaki Jul 31, 2024
1568b62
fixing block conversion
kttakasaki Jul 31, 2024
24b56cd
fixing block conversion
kttakasaki Aug 1, 2024
075520e
fixing block conversion
kttakasaki Aug 1, 2024
9707af8
fixing block conversion
kttakasaki Aug 1, 2024
6ebddce
fixing bugs
kttakasaki Aug 2, 2024
52cbae2
testing
kttakasaki Aug 2, 2024
adc5dce
threading multilevel block write
kttakasaki Aug 5, 2024
291f333
pixel shifting block deskew
kttakasaki Aug 6, 2024
dde31f0
bug fix
kttakasaki Aug 6, 2024
34c7537
new ps deskew approach
kttakasaki Aug 6, 2024
1ebff06
bug fixing
kttakasaki Aug 6, 2024
aa243a5
bug fixing
kttakasaki Aug 6, 2024
d7026ea
bug fixing
kttakasaki Aug 6, 2024
d7663ee
bug fixing
kttakasaki Aug 6, 2024
98ff6b7
bug fixing subchunk conversion
kttakasaki Aug 6, 2024
c329197
bug fixing subchunk conversion
kttakasaki Aug 6, 2024
8296319
bug fixing subchunk conversion
kttakasaki Aug 6, 2024
e9f4913
bug fixing subchunk conversion
kttakasaki Aug 6, 2024
6477939
bug fixing subchunk conversion
kttakasaki Aug 6, 2024
13cc660
bug fixing subchunk conversion
kttakasaki Aug 6, 2024
7322a73
bug fixing subchunk conversion
kttakasaki Aug 7, 2024
83bb803
bug fixing subchunk conversion
kttakasaki Aug 7, 2024
1e4e626
double block size
kttakasaki Aug 7, 2024
7f8a99f
debugging
kttakasaki Aug 7, 2024
709d3de
debugging
kttakasaki Aug 8, 2024
ad9484f
debugging transposed ims data
kttakasaki Aug 8, 2024
17ff0a8
debugging
kttakasaki Aug 8, 2024
88def9c
debugging
kttakasaki Aug 8, 2024
fc82864
vectorized resampling
kttakasaki Aug 13, 2024
ae8b6af
smaller block for vectorized
kttakasaki Aug 13, 2024
17e5f66
bug fixing vectorized
kttakasaki Aug 13, 2024
7d236e9
bug fixing vectorized
kttakasaki Aug 13, 2024
eadc6ce
debugging
kttakasaki Aug 13, 2024
82923cc
debugging
kttakasaki Aug 13, 2024
52ef7ee
debugging vectorization
kttakasaki Aug 14, 2024
f4c4b82
debugging
kttakasaki Aug 14, 2024
e81a2a6
debugging
kttakasaki Aug 14, 2024
60b09cb
commenting old code
kttakasaki Aug 14, 2024
6056d9d
debugging
kttakasaki Aug 16, 2024
dc648c4
temp debugging
kttakasaki Aug 16, 2024
38e03c9
test condition
kttakasaki Aug 16, 2024
756542c
test condition
kttakasaki Aug 16, 2024
a481cc9
test condition
kttakasaki Aug 16, 2024
646fb23
debugging
kttakasaki Aug 16, 2024
c28b53c
Merge branch 'develop' into kt_deskew_ngff
kttakasaki Aug 16, 2024
f7917f4
implementing zarr to ngff
kttakasaki Aug 16, 2024
eba27ee
testing block size change
kttakasaki Sep 19, 2024
feec173
reviving pyramid downsample
kttakasaki Sep 23, 2024
cb885bd
debugging
kttakasaki Sep 23, 2024
f8e7a8d
debugging
kttakasaki Sep 23, 2024
7666e20
debugging
kttakasaki Sep 23, 2024
ca9399a
debugging
kttakasaki Sep 23, 2024
c01fd11
hot fix for smart conversion
kttakasaki Sep 24, 2024
cc474cf
reverting deskew to loop
kttakasaki Sep 25, 2024
4136d95
cleaning code
kttakasaki Sep 25, 2024
d664791
block size testing
kttakasaki Sep 26, 2024
b7813a7
cleaning up code
kttakasaki Sep 26, 2024
e25de26
print chunks
kttakasaki Sep 27, 2024
279fd3f
block size from ims chunk size
kttakasaki Sep 27, 2024
fe56df6
trying increasing cache
kttakasaki Sep 27, 2024
ff255f7
adding ims zarr wrapper functionality
kttakasaki Sep 27, 2024
c60227d
5d dataset as zarr
kttakasaki Sep 28, 2024
facf32a
debugging
kttakasaki Sep 28, 2024
dd0fb61
debugging
kttakasaki Sep 29, 2024
e018a7b
major debug
kttakasaki Sep 30, 2024
1c75a71
reverting chunking scheme
kttakasaki Sep 30, 2024
88fa5ca
fixing deletion
kttakasaki Sep 30, 2024
0d00d1d
debugging
kttakasaki Sep 30, 2024
3b3da5b
debugging
kttakasaki Sep 30, 2024
4b7f84b
major bug fix
kttakasaki Sep 30, 2024
a5a6ff2
readding test tuple
kttakasaki Sep 30, 2024
527ea31
trying parallel writing
kttakasaki Oct 1, 2024
9f8da5f
debugging
kttakasaki Oct 1, 2024
92356ca
debugging
kttakasaki Oct 1, 2024
10cd038
trying to parallelize slices
kttakasaki Oct 1, 2024
ab584f5
trying parallel
kttakasaki Oct 1, 2024
2bbf6f0
debugging
kttakasaki Oct 1, 2024
186131b
debugging
kttakasaki Oct 1, 2024
4dd1f43
debugging
kttakasaki Oct 1, 2024
03f6eda
debugging
kttakasaki Oct 1, 2024
7c4ce1c
block size
kttakasaki Oct 1, 2024
4f470f1
block size
kttakasaki Oct 1, 2024
cad4b95
single slice by single block
kttakasaki Oct 1, 2024
53bf569
dirty masking
kttakasaki Oct 19, 2024
e1ec5f5
hotfix
kttakasaki Oct 19, 2024
bf1a034
hotfix
kttakasaki Oct 19, 2024
75c628e
chunk by slice
kttakasaki Oct 19, 2024
b658b52
fixing coordinates
kttakasaki Nov 27, 2024
4f12a68
fixing coordinates
kttakasaki Nov 27, 2024
7e6429e
bug fixing
kttakasaki Nov 27, 2024
97a7287
process concurrency
kttakasaki Nov 27, 2024
ba040c3
bug fixing
kttakasaki Nov 27, 2024
1a2d42b
Merge pull request #56 from AllenInstitute/kt_deskew_exaspim_ts
RussTorres Jan 14, 2025
8e7ad85
housecleaning and comments for merging
kttakasaki Jan 16, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,13 @@ def get_number_interleaved_channels_from_rootdir(
return interleaved_channels


def acquisition_to_ngff(acquisition_dir, output, out_dir, concurrency=5,
def acquisition_to_ngff(acquisition_dir, output, out_dir, concurrency=5, acq_parameters=None,
ngff_generation_kwargs=None, copy_top_level_files=True):
"""
"""
acq_parameters = (
{} if acq_parameters is None
else acq_parameters)
ngff_generation_kwargs = (
{} if ngff_generation_kwargs is None
else ngff_generation_kwargs)
Expand All @@ -87,6 +90,13 @@ def acquisition_to_ngff(acquisition_dir, output, out_dir, concurrency=5,
interleaved_channels = get_number_interleaved_channels_from_rootdir(
acquisition_path)
positionList = get_strip_positions_from_rootdir(acquisition_path)
ori = (1,1,1)
axes = (0,1,2)
if acq_parameters and "stage_axes" in acq_parameters:
axesStr = acq_parameters["stage_axes"]
if axesStr=="yxz":
axes = (1,0,2)
ori = (-1,1,1)

try:
setup_group_attributes = [{
Expand All @@ -95,7 +105,7 @@ def acquisition_to_ngff(acquisition_dir, output, out_dir, concurrency=5,
acquisition_path),
"unit": "um"
},
"position": p
"position": tuple(p[i]*o for i,o in zip(axes,ori))
} for p in positionList]
except (KeyError, FileNotFoundError):
setup_group_attributes = {}
Expand Down Expand Up @@ -138,6 +148,10 @@ def acquisition_to_ngff(acquisition_dir, output, out_dir, concurrency=5,
for tlf_path in top_level_files_paths:
out_tlf_path = out_path / tlf_path.name
shutil.copy(str(tlf_path), str(out_tlf_path))


class AcquisitionParameters(argschema.schemas.DefaultSchema):
stage_axes = argschema.fields.Str(required=False,default='')


class AcquisitionDirToNGFFParameters(
Expand All @@ -147,6 +161,8 @@ class AcquisitionDirToNGFFParameters(
# output_dir = argschema.fields.Str(required=True)
copy_top_level_files = argschema.fields.Bool(required=False, default=True)
position_concurrency = argschema.fields.Int(required=False, default=5)
acq_parameters = argschema.fields.Nested(
AcquisitionParameters, required=False, default=None)


class AcquisitionDirToNGFF(argschema.ArgSchemaParser):
Expand All @@ -164,6 +180,7 @@ def run(self):
acquisition_to_ngff(
self.args["input_dir"], self.args["output_format"], self.args["output_file"],
concurrency=self.args["position_concurrency"],
acq_parameters=self.args["acq_parameters"],
ngff_generation_kwargs=ngff_kwargs,
copy_top_level_files=self.args["copy_top_level_files"]
)
Expand Down
Loading
Loading