convert only writing first image...?

[tbenst]

Seems like convert is only saving the first tiff instead of 100k+ frames..?

On Oak, this folder is at users/tyler/b115/2021-06-08_rsChRmine_h2b6s/fish2/users/tyler/b115/2021-06-08_rsChRmine_h2b6s/fish2. It contains the output after manually running the ripping software.

❯ ls TSeries-lrhab-titration-123 | wc -l
240393
❯ mkdir raw && ln -s $(pwd)/TSeries-lrhab-titration-123 raw/ # code reads .csv file from raw folder 
❯ mkdir tiff && ln -s $(pwd)/TSeries-lrhab-titration-123 tiff/
TSeries-lrhab-titration-123
❯ 2p --base-path . --acquisition TSeries-lrhab-titration-123 convert --channel 3 --no-fix-tiff
2021-08-02 21:41:22.034 convert:49 INFO Reading voltage recordings from: raw/TSeries-lrhab-titration-123/TSeries-lrhab-titration-123_Cycle00001_VoltageRecording_001.csv
/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/numpy/lib/arraysetops.py:580: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask |= (ar1 == a)
2021-08-02 21:41:36.385 convert:51 INFO Voltage recordings head:
          frame starts  secondary   winfluo      Blue  VR timestamps     green       LED    respir
Time(ms)                                                                                          
0.0           0.097046   0.012512  0.341797  0.339966       0.019913  0.020142  0.000305 -0.029297
0.2           4.894409   0.013123  0.341492  0.340576       0.019226  0.020752  0.000916 -0.028381
0.4           4.893799   0.013123  0.342102  0.339966       0.020676  0.019836  0.000610 -0.029602
0.6           4.895935   0.012817  0.342407  0.340271       0.018616  0.019836  0.000916 -0.029602
0.8           4.899902   0.013428  0.342102  0.340881       0.019455  0.020142  0.000916 -0.028381
2021-08-02 21:41:36.398 convert:53 INFO Writing volatage data to hdf5: convert/TSeries-lrhab-titration-123/voltage.h5
2021-08-02 21:41:36.405 utils:145 INFO Note: NumExpr detected 36 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2021-08-02 21:41:36.405 utils:157 INFO NumExpr defaulting to 8 threads.
2021-08-02 21:41:37.612 convert:58 INFO Done writing voltage data to hdf5
2021-08-02 21:41:37.825 convert:79 INFO Reading TIFF metadata
/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/tifffile/tifffile.py:10709: RuntimeWarning: invalid value encountered in true_divide
  'ZDistance': values[:, 0] / values[:, 1],
2021-08-02 21:41:37.829 tifffile:13497 WARNING OME series: not an ome-tiff master file
2021-08-02 21:41:37.830 convert:81 INFO Found TIFF data with shape (512, 512) and type uint16
2021-08-02 21:41:37.830 convert:86 INFO Writing image data to hdf5: convert/TSeries-lrhab-titration-123/orig.h5
2021-08-02 21:41:37.831 convert:90 INFO Done writing image data hdf5
2021-08-02 21:41:37.831 convert:92 INFO Done
❯ ll convert/TSeries-lrhab-titration-123/orig.h5
-rw-rw-r-- 1 tyler tyler 514K Aug  2 21:41 convert/TSeries-lrhab-titration-123/orig.h5
❯ du -sh TSeries-lrhab-titration-123
228G	TSeries-lrhab-titration-123

[chrisroat]

I know there were some problems with the metadata (which is in the first tiff file of a tiff stack) that the Bruker ripper writes out. That is why I added the "fix-tiff" flag to rewrite the first tiff, and it is on by default. Is there a reason you turned that off?

[tbenst]

ah I hadn't appreciated that...I turned off due to this error:

2021-08-02 23:21:01.945 convert:73 WARNING Deleting previously corrected Burker tiff: tiff/TSeries-lrhab-titration-123/TSeries-lrhab-titration-123_Cycle00001_Ch3_000001.ome.tif
Traceback (most recent call last):
  File "/home/tyler/opt/anaconda3/envs/two-photon/bin/2p", line 33, in <module>
    sys.exit(load_entry_point('two-photon', 'console_scripts', '2p')())
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/click/core.py", line 1289, in invoke
    rv.append(sub_ctx.command.invoke(sub_ctx))
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/tyler/code/two-photon/two_photon/convert.py", line 76, in convert
    correct_omexml.correct_tiff(tiff_init_fixed)
  File "/home/tyler/code/two-photon/two_photon/correct_omexml.py", line 14, in correct_tiff
    with tifffile.TiffFile(fname, mode="r+b") as tif:
  File "/home/tyler/opt/anaconda3/envs/two-photon/lib/python3.8/site-packages/tifffile/tifffile.py", line 2405, in __init__
    raise TypeError(f'unexpected keyword argument: {key}')
TypeError: unexpected keyword argument: mode

[tbenst]

okay, that error was easy to fix by pip install -U tifffile, but I still get only one frame written with fix-tiff enabled

[chrisroat]

I do not see the tiff files in the OAK. I can help debug if you can give me access in some way.

For debugging, the code is simply opening the initial tiff file (which has all the metadata for the entire stack) using tifffile. If --fix-tiff is enabled, the library will write a new initial tiff file with corrected metadata and use that one.

two-photon/two_photon/convert.py

Lines 70 to 81 in 39afef0

	if fix_tiff:
	tiff_init_fixed = tiff_init.with_suffix(".fixed" + tiff_init.suffix)
	if tiff_init_fixed.exists():
	logger.warning("Deleting previously corrected Burker tiff: %s", str(tiff_init))
	tiff_init_fixed.unlink()
	shutil.copy(tiff_init, tiff_init_fixed)
	correct_omexml.correct_tiff(tiff_init_fixed)
	tiff_init = tiff_init_fixed
	logger.info("Reading TIFF metadata")
	data = tifffile.imread(tiff_init)
	logger.info("Found TIFF data with shape %s and type %s", data.shape, data.dtype)

I'm curious if your dataset and/or Bruker version has created an even more problematic metadata issue than I've seen in the past. (I made Bruker aware of their incompatibility with the OME standard. It should get fixed in later versions.)