broken svs on Bio-Formats update #691

will-moore · 2024-03-25T09:55:00Z

When testing ome/omero-web#536 on idr-testing, I noticed this issue which I think Seb mentioned was due to a Bio-Formats update (see below)

Deployed on idr-testing from branch at #537, this is returning different zoomLevelScaling from previously!?
On https://idr.openmicroscopy.org/webclient/img_detail/9840218/ that image has:

    "tiles": true,
    "tile_size": {
        "width": 240,
        "height": 240
    },
    "levels": 3,
    "zoomLevelScaling": {
        "0": 1.0,
        "1": 0.2499840367792606,
        "2": 0.029308473277568484
    },

But on idr-testing, a different zoomLevelScaling is causing problems:

From slack idr, 28th Feb
Seb: "SVS in particular is a format which went through several changes in recent versions"

Will: Ah - so that bug may not be due to my PR?

Seb:
14:30
where that bug == the resolution levels are different between deployments or that bug == the resolution levels on idr-testing are incorrect?
14:32
actually the bug is clearly in production IDR (and has been addressed in recent Bio-Formats versions)
14:33
https://idr.openmicroscopy.org/webclient/img_detail/9840219/?dataset=10450 is absolutely not a label image, it's a resolution level of the whole slide image
14:34
so this dataset will partly break as part of the OMERO.server upgrade and we might need to consider reimporting these files unfortunately

The text was updated successfully, but these errors were encountered:

will-moore · 2024-03-25T10:33:22Z

As discussed in IDR meeting:

need to fix by re-importing (will get 2 images instead of 3) and re-annotate.
need to check for other images with this issue - find other svs images and check manually

will-moore · 2024-03-27T10:20:12Z

Find 'SVS' format and count images...

idr=> select * from format where value='SVS';
 id  | permissions | value | external_id 
-----+-------------+-------+-------------
 360 |         -52 | SVS   |            
(1 row)

idr=> select count(*) from image where format=360 and series=1;
 count 
-------
   556
(1 row)

Find Datasets...

idr=> select DISTINCT parent from DatasetImageLink where child in (select id from image where format=360 and series=1);
 parent 
--------
  10348
  10349
  10352
  10354
  10356
  10358
  10388
  10413
  10414
  10415
  10425
  10426
  10427
  10428
  10429
  10431
  10436
  10450
  10461
  10474
  10475
  10476
  10478
  10479
  10480
  10481
  10484
  10485
  10492
  10517
  10530
  10550
  10557
  10558
  10559
  10560
  15274
  15280
  16651
  16652
  16653
  16654
(42 rows)

will-moore · 2024-03-27T11:22:00Z

Found 397 Broken images:

idr0070

Dataset: 10388, 3 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9838978|image-9838981|image-9838984

Dataset: 10413, 2 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839274|image-9839277

Dataset: 10414, 3 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839283|image-9839286|image-9839289

Dataset: 10415, 1 Image
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839294

Dataset: 10425, 1 Image
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839412

Dataset: 10426, 2 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839420|image-9839423

Dataset: 10427, 2 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839426|image-9839429

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840219

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840355|image-9840358|image-9840361

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840579|image-9840582|image-9840585|image-9840588

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841222|image-9841225|image-9841228

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841448|image-9841451|image-9841454

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841528

10557 - None!

10558
https://idr-testing.openmicroscopy.org/webclient/?show=image-9841778|image-9841781|image-9841784|image-9841787

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841815|image-9841818

idr0114

15274, 15280 - No svs!

idr0135

16651, 16652, 16653, 16654 - All labels OK

will-moore · 2024-03-27T12:42:30Z

All the svs above are from idr0070. The https://github.com/IDR/idr0070-kerwin-hdbr/blob/master/experimentA/idr0070-experimentA-filePaths.tsv has 400 svs images, so let's just re-import all of them.

Steps:

Delete all existing svs images from idr0070
Re-import svs images
Update https://github.com/IDR/idr0070-kerwin-hdbr/blob/master/experimentA/idr0070-experimentA-annotation.csv to remove the 400 'label' images
Re-annotate svs images (maybe re-annotate ALL images will be easier)

will-moore · 2024-03-27T14:58:24Z

To focus on idr0070, max Image ID is 9841818, so we can count etc...

$ psql -U omero -d idr -h $DBHOST -F "," -c "select count(id) from image where format=360 and series=1 and (id < 9841819)"
 count 
-------
   400
(1 row)

400 svs Image IDs in idr0070


9838978,9838981,9838984,9839274,9839277,9839283,9839286,9839289,9839293,9839412,9839420,9839423,9839426,9839429,9839432,9839435,9839438,9839441,9839444,9839447,9839450,9839453,9839456,9839459,9839462,9839465,9839468,9839471,9839474,9839477,9839480,9839483,9839486,9839489,9839492,9839495,9839498,9839501,9839504,9839507,9839510,9839513,9839567,9839570,9839573,9839576,9839579,9839582,9839585,9839588,9839591,9839594,9839597,9839600,9839603,9839606,9839609,9839612,9839615,9839618,9839621,9839624,9839628,9839633,9839637,9839640,9839671,9839674,9839677,9839680,9839683,9839686,9839689,9839692,9839773,9839776,9839779,9839782,9839785,9839788,9839791,9839794,9839797,9839800,9839803,9839806,9839809,9839812,9839815,9839825,9839828,9839831,9839836,9839839,9839842,9839845,9839848,9839851,9839854,9839857,9839860,9839863,9839866,9839873,9839876,9839879,9839882,9839885,9839888,9839891,9839894,9839897,9839900,9839903,9839906,9839914,9839917,9839920,9839923,9839926,9839929,9839932,9839935,9839938,9839941,9839947,9839950,9839953,9839956,9839959,9839962,9839965,9839970,9839973,9839976,9839979,9839982,9839985,9839988,9839991,9839994,9839997,9840000,9840005,9840017,9840020,9840023,9840026,9840029,9840032,9840035,9840038,9840041,9840066,9840069,9840072,9840075,9840078,9840081,9840084,9840087,9840090,9840093,9840096,9840099,9840102,9840105,9840108,9840111,9840114,9840117,9840219,9840355,9840358,9840361,9840478,9840481,9840484,9840487,9840490,9840493,9840496,9840499,9840502,9840505,9840508,9840511,9840514,9840517,9840523,9840526,9840536,9840539,9840542,9840545,9840550,9840553,9840556,9840559,9840562,9840565,9840568,9840571,9840574,9840579,9840582,9840585,9840588,9840625,9840628,9840631,9840634,9840637,9840640,9840643,9840646,9840649,9840652,9840655,9840658,9840661,9840664,9840667,9840670,9840673,9840676,9840679,9840682,9840685,9840688,9840691,9840694,9840697,9840700,9840703,9840706,9840709,9840712,9840715,9840718,9840721,9840724,9840727,9840730,9840733,9840736,9840740,9840743,9840746,9840749,9840752,9840755,9840758,9840761,9840764,9840767,9840770,9840773,9840776,9840779,9840782,9840785,9840788,9840791,9840794,9840797,9840800,9840803,9840806,9840809,9840812,9840815,9840818,9840821,9840824,9840827,9840830,9840833,9840845,9840848,9840851,9840854,9840857,9840860,9840863,9840866,9840869,9840872,9840875,9840878,9840881,9840884,9840887,9840890,9840893,9840896,9840899,9840902,9840905,9840909,9840912,9840915,9840918,9840921,9840924,9840927,9840930,9840933,9840936,9840939,9840942,9840945,9840948,9840951,9840954,9840957,9840960,9840963,9840966,9840969,9840972,9840975,9840978,9840981,9840984,9840987,9840990,9840993,9840996,9840999,9841002,9841005,9841008,9841011,9841014,9841017,9841070,9841073,9841076,9841079,9841082,9841085,9841088,9841091,9841094,9841097,9841100,9841103,9841106,9841109,9841112,9841115,9841118,9841121,9841124,9841127,9841130,9841133,9841136,9841139,9841142,9841145,9841148,9841151,9841154,9841157,9841163,9841166,9841169,9841172,9841175,9841222,9841225,9841228,9841448,9841451,9841454,9841528,9841712,9841715,9841718,9841721,9841724,9841727,9841772,9841775,9841778,9841781,9841784,9841787,9841791,9841794,9841797,9841800,9841803,9841806,9841809,9841812,9841815,9841818

will-moore · 2024-03-27T16:17:02Z

Test deleting even a single Image fails (hangs) due to Annotations.
First remove all annotations...

$  omero metadata populate --context deletemap --report --wait 300 --batch 100 --localcfg '{"ns":["openmicroscopy.org/mapr/organism", "openmicroscopy.org/mapr/antibody", "openmicroscopy.org/mapr/gene", "openmicroscopy.org/mapr/cell_line", "openmicroscopy.org/mapr/phenotype", "openmicroscopy.org/mapr/sirna", "openmicroscopy.org/mapr/compound", "openmicroscopy.org/mapr/protein"], "typesToIgnore":["Annotation"]}' --cfg experimentA/idr0070-experimentA-bulkmap-config.yml Project:1104

$ omero metadata populate --context deletemap --report --wait 300 --batch 100 --cfg experimentA/idr0070-experimentA-bulkmap-config.yml Project:1104

$ python /uod/idr/metadata/idr-utils/scripts/annotate/clean_orphaned_maps.py
INFO:omero.util.Resources:Starting
INFO:omero.util.Resources:Starting
INFO:omero.util.Resources:Halted
INFO:root:Found 0 orphaned Organism maps
INFO:root:Found 74 orphaned Antibody maps
INFO:root:Deleting 74 maps
INFO:root:Found 2 orphaned Gene maps
INFO:root:Deleting 2 maps
INFO:root:Found 0 orphaned Cell Line maps
INFO:root:Found 0 orphaned Phenotype maps
INFO:root:Found 0 orphaned siRNA maps
INFO:root:Found 0 orphaned Compound maps
INFO:root:Found 0 orphaned Protein maps
INFO:root:Found 0 orphaned Notebook maps
INFO:root:Found 0 orphaned Study Info maps
INFO:root:Found 0 orphaned Study Components maps
INFO:omero.util.Resources:Halted

Can't delete using a single Image:ID:

$ omero delete --dry-run --report Image:9838978
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Image:9838978 failed: 'graph-fail'
failed: within Fileset[3550638] may not delete Image[9838978] while Image[9838979] remains
Steps: 4
Elapsed time: 0.844 secs.
Flags: [FAILURE, CANCELLED]

Need to use Fileset ID:

$ omero delete --dry-run --report Fileset:3550638
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:3550638 Dry run performed
ok
Steps: 4
Elapsed time: 5.382 secs.
Flags: []
Deleted objects
  Instrument:93241
  Objective:90855
  ObjectiveSettings:91280-91282
  CommentAnnotation:25123922
  FilesetAnnotationLink:3550438
  DatasetImageLink:4540977-4540979
  Channel:26299350-26299358
  Image:9838977-9838979
  LogicalChannel:10609300-10609308
  OriginalFile:29671051,29671052
  Pixels:9838977-9838979
  ChannelBinding:25422660-25422668
  QuantumDef:9523332-9523334
  RenderingDef:9523332-9523334
  Thumbnail:11390582-11390584
  Fileset:3550638
  FilesetEntry:22140458
  FilesetJobLink:14306936-14306940
  IndexingJob:14317104
  JobOriginalFileLink:3568052
  MetadataImportJob:14317101
  PixelDataJob:14317102
  ThumbnailGenerationJob:14317103
  UploadJob:14317100

Max Fileset ID for idr0070 is 3552155

psql -U omero -d idr -h $DBHOST -c "select count(fileset) from image where format=360 and series=1 and (fileset < 3552156)"
 count 
-------
   400
(1 row)

Get Fileset IDs....

psql -U omero -d idr -h $DBHOST -c "select fileset from image where format=360 and series=1 and (fileset < 3552156)" >> idr0070_svs_filesetIds.txt

Need to remove whitespace from each line:

$ for r in $(cat idr0070_svs_filesetIds.txt); do echo $r | tr -d '[:space:]' >> idr0070_svs_filesetIds.csv && echo "" >> idr0070_svs_filesetIds.csv; done

Do the delete....

for r in $(cat idr0070_svs_filesetIds.csv); do omero delete --report Fileset:$r; done

will-moore · 2024-03-27T17:12:34Z

Now re-import....

[wmoore@test120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr0070-kerwin-hdbr/experimentA/

sudo cat idr0070-experimentA-filePaths.tsv | grep svs >> /tmp/idr0070-experimentA-filePaths_svs.tsv
sudo mv /tmp/idr0070-experimentA-filePaths_svs.tsv ./
# update bulk.yml to point at the ...svs.tsv
sudo vi idr0070-experimentA-bulk.yml

wc idr0070-experimentA-filePaths_svs.tsv
  402  1400 95150 idr0070-experimentA-filePaths_svs.tsv

As omero-server...

omero import --bulk experimentA/idr0070-experimentA-bulk.yml --file /tmp/idr0070svs.log  --errs /tmp/idr0070svs.err

On completion...
2 svs not imported because original wasn't deleted...

$ grep ClientPathExclusion /tmp/idr0070svs.err
2024-03-27 16:44:56,417 6461       [2-thread-1] INFO   .importer.exclusions.ClientPathExclusion - ClientPath match for filename: uod/idr/filesets/idr0070-kerwin-hdbr/20200414-Batch3-ftp/HDBR_SYP_IHC/1605,2,Rt hemi,10pcw,24_SYP.svs
2024-03-27 16:46:04,488 6406       [2-thread-1] INFO   .importer.exclusions.ClientPathExclusion - ClientPath match for filename: uod/idr/filesets/idr0070-kerwin-hdbr/20200422-Batch5/HDBR_MKI67_IHC/1605,2,Rt hemi,10pcw,24_KI67.svs

All files were either imported as 2 (or 1) image - none with 3 images as before:

[wmoore@test120-omeroreadwrite ~]$ grep "1 file uploaded" /tmp/idr0070svs.err | wc
    400    5200   31598
[wmoore@test120-omeroreadwrite ~]$ grep "2 images imported" /tmp/idr0070svs.err | wc
    391    5083   30889
[wmoore@test120-omeroreadwrite ~]$ grep "1 image imported" /tmp/idr0070svs.err | wc
      2      26     156

will-moore · 2024-04-19T15:54:27Z

Re-annotate....

Remove label rows from csv on idr-testing...

[wmoore@test120-omeroreadwrite experimentA]$ sudo cat idr0070-experimentA-annotation.csv | grep -v label > /tmp/idr0070-experimentA-annotation2.csv
[wmoore@test120-omeroreadwrite experimentA]$ sudo rm idr0070-experimentA-annotation.csv
[wmoore@test120-omeroreadwrite experimentA]$ sudo mv /tmp/idr0070-experimentA-annotation2.csv idr0070-experimentA-annotation.csv

Reannotate as normal... check_annotations....

/opt/omero/server/venv3/bin/python /uod/idr/metadata/idr-utils/scripts/annotate/check_annotations.py Project:1104 idr0070-experimentA-annotation.csv --output /tmp/errors.csv


GAP43-10PCW,"HDBR_GAP43_IHC_10PCW/1634,6,Sl 2,10pcw,37_GAP43.svs",,,,,,,,,,,,,,,,,,,,Missing annotation
GAP43-12PCW,"HDBR_GAP43_IHC_12PCW/11761,12,Sl 3,12pcw,140_GAP43.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
GAP43-CS18,"HDBR_GAP43_IHC_CS18/1262,2,Embryo_Placenta,CS18,165_GAP43.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
MKI67-10PCW,"HDBR_MKI67_IHC/1634,6,Sl 2,10pcw,37_KI67.svs",,,,,,,,,,,,,,,,,,,,Missing annotation
NKX2-2-CS19,"HDBR_NKX2-2_IHC_CS19/352,1,Head,CS19,284_NKX2-2.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
PAX6-12PCW,"HDBR_PAX6_IHC_hires_12PCW/11610,9,brain,12pcw,114_PAX6.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
PAX6-CS19,"HDBR_PAX6_IHC_hires_CS19/352,1,Head,CS19,284_PAX6.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
WNT8B-12PCW,"HDBR_WNT8B_IHC_12PCW/11610,9,brain,12pcw,110_2014-05-20 14_35_30_wnt8b.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
WNT8B-12PCW,"HDBR_WNT8B_IHC_12PCW/11610,9,brain,12pcw,160_2014-05-20 14_17_11_wnt8b.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation

```

will-moore · 2024-04-22T10:44:06Z

Missing annotation errors from above are fixed in IDR/idr0070-kerwin-hdbr@4b08e7f as follows:

Checked all 7 of the [label image] images above and they are all working. So these are genuine label images and should be added back to the annotations.csv.
The other 2 Missing annotation errors are from 2 images that are now just a single image (no [. ] in the name) so these were re-named in the annotations.csv.

will-moore · 2024-04-22T11:54:49Z

Fixed - test:

(venv3) bash-4.2$ /opt/omero/server/venv3/bin/python /uod/idr/metadata/idr-utils/scripts/annotate/check_annotations.py Project:1104 idr0070-experimentA-annotation.csv --output /tmp/idr0070_errors.csv
All images are unique and have annotations.

Deleted previous OMERO.table:

(venv3) bash-4.2$ omero delete --report Annotation:25135328
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Annotation:25135328 ok
Steps: 6
Elapsed time: 0.248 secs.
Flags: []
Deleted objects
  FileAnnotation:25135328
  ProjectAnnotationLink:1667
  OriginalFile:29676048

Create OMERO.table

/opt/omero/server/OMERO.server/bin/omero metadata populate --report --batch 1000 --file idr0070-experimentA-annotation.csv Project:1104

Create Map annotations

/opt/omero/server/OMERO.server/bin/omero metadata populate --context bulkmap --batch 100 --cfg idr0070-experimentA-bulkmap-config.yml Project:1104

will-moore · 2024-04-24T13:07:40Z

The 3 Filesets in https://idr-testing.openmicroscopy.org/webclient/?show=dataset-10461 are still broken.
These weren't deleted & reimported for some reason...?

Fileset IDs are:
3551354, 3551355, 3551356

These are all found in idr0070_svs_filesetIds.csv above. So delete was attempted above.
Try again:

(venv3) bash-4.2$ omero delete Fileset:3551354 --report
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:3551354 failed: 'graph-fail'
failed: cannot read ome.model.fs.Fileset[3551354]
Steps: 6
Elapsed time: 0.032 secs.
Flags: [FAILURE, CANCELLED]

Also, none of these Filesets or Image IDs can be found in the DB via psql.
Using the Dataset ID, can see it contains different Images:

idr=> select * from DatasetImageLink where parent=10461;
   id    | permissions | version |  child   | creation_id | external_id | group_id | owner_id | update_id | parent 
---------+-------------+---------+----------+-------------+-------------+----------+----------+-----------+--------
 7310578 |         -56 |       0 | 15153328 |   389349671 |             |        3 |        2 | 389349671 |  10461
 7310579 |         -56 |       0 | 15153329 |   389349671 |             |        3 |        2 | 389349671 |  10461
 7310580 |         -56 |       0 | 15153330 |   389349700 |             |        3 |        2 | 389349700 |  10461
 7310581 |         -56 |       0 | 15153331 |   389349700 |             |        3 |        2 | 389349700 |  10461
 7310582 |         -56 |       0 | 15153332 |   389349731 |             |        3 |        2 | 389349731 |  10461
 7310583 |         -56 |       0 | 15153333 |   389349731 |             |        3 |        2 | 389349731 |  10461
(6 rows)

It turns out that the images, thumbnails, right panel etc showing in the webclient above were all cached in the browser or nginx and have all been deleted and replaced successfully!

will-moore · 2024-04-24T13:22:44Z

cc @francesw or @dominikl

I wonder if you could review the actions above that I have performed on idr-testing (starting at #691 (comment) above) and plan to apply to idr-next for the NGFF release?

Summary:

removed all annotations for idr0070
deleted and re-imported all svs images in idr0070 (400 in total)
updated annotations.csv accordingly (mostly just removing [macro] rows but some other tweaks too - see Label images removal idr0070-kerwin-hdbr#2
checked and re-apply annotations

You can see an example of the changes by looking at e.g.
https://idr.openmicroscopy.org/webclient/?show=dataset-10461
(and the equivalent Dataset on idr testing)

will-moore · 2024-04-25T09:19:05Z

Final test on pilot-idrngff...

$ cd /uod/idr/metadata/idr0070-kerwin-hdbr/
$  omero metadata populate --context deletemap --report --wait 300 --batch 100 --localcfg '{"ns":["openmicroscopy.org/mapr/organism", "openmicroscopy.org/mapr/antibody", "openmicroscopy.org/mapr/gene", "openmicroscopy.org/mapr/cell_line", "openmicroscopy.org/mapr/phenotype", "openmicroscopy.org/mapr/sirna", "openmicroscopy.org/mapr/compound", "openmicroscopy.org/mapr/protein"], "typesToIgnore":["Annotation"]}' --cfg experimentA/idr0070-experimentA-bulkmap-config.yml Project:1104

ConnectionTimout!
omero login -> internal server error

Restarted server and tried command above again... Success...
Then...

$ omero metadata populate --context deletemap --report --wait 300 --batch 100 --cfg experimentA/idr0070-experimentA-bulkmap-config.yml Project:1104

$ python /uod/idr/metadata/idr-utils/scripts/annotate/clean_orphaned_maps.py
...
INFO:root:Found 1 orphaned Organism maps
INFO:root:Deleting 1 maps
INFO:root:Found 74 orphaned Antibody maps
INFO:root:Deleting 74 maps
INFO:root:Found 2 orphaned Gene maps
INFO:root:Deleting 2 maps
INFO:root:Found 0 orphaned Cell Line maps
INFO:root:Found 0 orphaned Phenotype maps
INFO:root:Found 0 orphaned siRNA maps
INFO:root:Found 0 orphaned Compound maps
INFO:root:Found 0 orphaned Protein maps
INFO:root:Found 0 orphaned Notebook maps
INFO:root:Found 0 orphaned Study Info maps
INFO:root:Found 0 orphaned Study Components maps

psql -U omero -d idr -h $DBHOST -c "select fileset from image where format=360 and series=1 and (fileset < 3552156)" >> idr0070_svs_filesetIds.txt

$ for r in $(cat idr0070_svs_filesetIds.txt); do echo $r | tr -d '[:space:]' >> idr0070_svs_filesetIds.csv && echo "" >> idr0070_svs_filesetIds.csv; done

# remove first and last (non-ID) rows
vi idr0070_svs_filesetIds.csv

(venv3) bash-5.1$ wc !$
wc idr0070_svs_filesetIds.csv
 400  400 3200 idr0070_svs_filesetIds.csv

omero login

# start 11:09...
for r in $(cat idr0070_svs_filesetIds.csv); do echo $r && omero delete Fileset:$r; done

#... approx 20 mins

As wmoore, update filePaths.tsv and annotations.csv...

[wmoore@test120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr0070-kerwin-hdbr/experimentA/

sudo cat idr0070-experimentA-filePaths.tsv | grep svs >> /tmp/idr0070-experimentA-filePaths_svs.tsv
sudo mv /tmp/idr0070-experimentA-filePaths_svs.tsv ./
# update bulk.yml to point at the ...svs.tsv
sudo vi idr0070-experimentA-bulk.yml

wc idr0070-experimentA-filePaths_svs.tsv
  402  1400 95150 idr0070-experimentA-filePaths_svs.tsv


$ sudo -Es git remote add will https://github.com/will-moore/idr0070-kerwin-hdbr
$ sudo -Es git fetch will
$ sudo -Es git checkout will/label_images_removal

As omero-server, re-import...

11:32....

omero import --bulk experimentA/idr0070-experimentA-bulk.yml --file /tmp/idr0070svs.log  --errs /tmp/idr0070svs.err

dominikl · 2024-04-25T10:07:24Z

👍 Looks good to me! (Edit: on idr-testing)

will-moore · 2024-04-25T11:07:06Z

Import on pilot-idrngff above is failing with...

2024-04-25 10:33:34,114 4435       [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2024-04-25 10:33:34,288 4609       [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.0.0 revision: 3f8b3326cb578d59bd948fb84c838ff77e9f1b08 date: 1 August 2023
2024-04-25 10:33:34,693 5014       [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2024-04-25 10:33:34,694 5015       [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2024-04-25 10:33:34,694 5015       [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2024-04-25 10:33:34,697 5018       [      main] INFO   formats.importer.cli.CommandLineImporter - Adding exclusion: clientpath
2024-04-25 10:33:35,087 5408       [      main] INFO   formats.importer.cli.CommandLineImporter - Setting parallel upload: 8
2024-04-25 10:33:35,088 5409       [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2024-04-25 10:33:42,061 12382      [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2024-04-25 10:33:43,438 13759      [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /uod/idr/filesets/idr0070-kerwin-hdbr/20200414-Batch3-ftp/HDBR_SYP_IHC/11610,9,brain,12pcw,97_SYP.svs
java.lang.Exception: java.lang.NoSuchMethodError: 'java.lang.Object loci.formats.CoreMetadataList.remove(int, int)'
        at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:469)
        at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
        at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
        at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
        at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
Caused by: java.lang.NoSuchMethodError: 'java.lang.Object loci.formats.CoreMetadataList.remove(int, int)'
        at loci.formats.in.SVSReader.initStandardMetadata(SVSReader.java:652)
        at loci.formats.in.BaseTiffReader.initMetadata(BaseTiffReader.java:98)
        at loci.formats.in.BaseTiffReader.initFile(BaseTiffReader.java:610)
        at loci.formats.FormatReader.setId(FormatReader.java:1466)
        at loci.formats.ImageReader.setId(ImageReader.java:863)
        at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at loci.formats.Memoizer.setId(Memoizer.java:698)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
        ... 6 common frames omitted
2024-04-25 10:33:43,440 13761      [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 0 group(s) with 1 call(s) to setId in 1336ms. (1380ms total) [0 unknowns]
2024-04-25 10:33:47,596 17917      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2024-04-25 10:33:55,775 26096      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2024-04-25 10:34:00,118 30439      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2024-04-25 10:34:00,135 30456      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.9
2024-04-25 10:34:00,136 30457      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2024-04-25 10:34:00,136 30457      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.22
2024-04-25 10:34:00,136 30457      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2024-04-25 10:34:00,136 30457      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2024-04-25 10:34:00,136 30457      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 5.14.0-362.24.1.el9_3.0.1.x86_64
No imports due to errors!

will-moore · 2024-04-25T16:04:21Z

Try import again.. after updating jars

omero import --bulk experimentA/idr0070-experimentA-bulk.yml --file /tmp/idr0070svs_2.log  --errs /tmp/idr0070svs_2.err

Seems to be working!!

will-moore · 2024-05-08T16:16:36Z

Import complete...
Checkout the updated annotations.csv from IDR/idr0070-kerwin-hdbr#2

(base) [wmoore@pilot-idrngff-omeroreadwrite idr0070-kerwin-hdbr]$ sudo -Es git checkout will/label_images_removal
M	experimentA/idr0070-experimentA-bulk.yml
HEAD is now at 4b08e7f Fix annotations.csv for [label image] images

as omero-server

(venv3) bash-5.1$ /opt/omero/server/venv3/bin/python /uod/idr/metadata/idr-utils/scripts/annotate/check_annotations.py Project:1104 idr0070-experimentA-annotation.csv --output /tmp/idr0070_errors.csv
All images are unique and have annotations.


$ omero delete --report Annotation:25135328
Previous session expired for demo on localhost:4064
Server: [localhost:4064]
Username: [demo]
Password:
Created session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Annotation:25135328 ok
Steps: 6
Elapsed time: 0.492 secs.
Flags: []
Deleted objects
  FileAnnotation:25135328
  ProjectAnnotationLink:1667
  OriginalFile:29676048


/opt/omero/server/OMERO.server/bin/omero metadata populate --report --batch 1000 --file idr0070-experimentA-annotation.csv Project:1104

/opt/omero/server/OMERO.server/bin/omero metadata populate --context bulkmap --batch 100 --cfg idr0070-experimentA-bulkmap-config.yml Project:1104

will-moore mentioned this issue Mar 25, 2024

imgData JSON with no stateful services ome/omero-web#536

Closed

will-moore mentioned this issue Apr 22, 2024

Label images removal IDR/idr0070-kerwin-hdbr#2

Open

will-moore mentioned this issue Apr 25, 2024

idr-next steps for NGFF upgrade #675

Open

14 tasks

dominikl mentioned this issue Apr 25, 2024

Add stats for prod121 IDR/idr.openmicroscopy.org#195

Merged

will-moore mentioned this issue May 21, 2024

idr-testing May 2024 #696

Open

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

broken svs on Bio-Formats update #691

broken svs on Bio-Formats update #691

will-moore commented Mar 25, 2024 •

edited

Loading

will-moore commented Mar 25, 2024

will-moore commented Mar 27, 2024

will-moore commented Mar 27, 2024 •

edited

Loading

will-moore commented Mar 27, 2024

will-moore commented Mar 27, 2024 •

edited

Loading

will-moore commented Mar 27, 2024

will-moore commented Mar 27, 2024 •

edited

Loading

will-moore commented Apr 19, 2024

will-moore commented Apr 22, 2024

will-moore commented Apr 22, 2024

will-moore commented Apr 24, 2024

will-moore commented Apr 24, 2024

will-moore commented Apr 25, 2024 •

edited

Loading

dominikl commented Apr 25, 2024 •

edited

Loading

will-moore commented Apr 25, 2024

will-moore commented Apr 25, 2024

will-moore commented May 8, 2024

broken svs on Bio-Formats update #691

broken svs on Bio-Formats update #691

Comments

will-moore commented Mar 25, 2024 • edited Loading

will-moore commented Mar 25, 2024

will-moore commented Mar 27, 2024

will-moore commented Mar 27, 2024 • edited Loading

will-moore commented Mar 27, 2024

will-moore commented Mar 27, 2024 • edited Loading

will-moore commented Mar 27, 2024

will-moore commented Mar 27, 2024 • edited Loading

will-moore commented Apr 19, 2024

will-moore commented Apr 22, 2024

will-moore commented Apr 22, 2024

will-moore commented Apr 24, 2024

will-moore commented Apr 24, 2024

will-moore commented Apr 25, 2024 • edited Loading

dominikl commented Apr 25, 2024 • edited Loading

will-moore commented Apr 25, 2024

will-moore commented Apr 25, 2024

will-moore commented May 8, 2024

will-moore commented Mar 25, 2024 •

edited

Loading

will-moore commented Mar 27, 2024 •

edited

Loading

will-moore commented Mar 27, 2024 •

edited

Loading

will-moore commented Mar 27, 2024 •

edited

Loading

will-moore commented Apr 25, 2024 •

edited

Loading

dominikl commented Apr 25, 2024 •

edited

Loading