File Ripping Particulars #12
Comments
|
Hi! We were thinking the same thing. I've significantly rewrote a lot of the code. Here is the branch with all the updates. It now uses the option to keep the raw data around. The README has more of the details. Of course, I suggest trying it all out on a small, practice dataset to see how things work for you. As far as the software, we've included the necessary Prairie View code in this repo so you should not need to install the software. As proof, there is a Dockerfile that does run the ripping (via Linux+wine). However, it is very slow and I haven't figure out why yet. Likely we can build a Windows docker container for just the ripping (and use a Linux container for the rest of the pipeline), but I don't have much experience with that. |
|
Note that the version of software has to exactly match (ie 5.5 update 4) or Image ripping will choke (will raise error—not destructive from what I’ve seen). @chrisroat ripping is fast for Linux + wine when on a fast, local ext4 filesystem. I’ve had problems with software RAID0 (fails) or Lustre (sloooooow), however. |
|
The software is bundled with the repo, and we've only included 5.4 and 5.5
-- what we use in the lab. And yes, the python code raises an Exception
before trying anything, if your data does not match one of these. If you
find that you have a different version, we can either
- add in the newer version
- see if one of the current versions is compatible
@tyler Stephen Benster ***@***.***> - thanks for the info on
RAID0/Lustre. Was your work using the built Docker or Singularity
container, or your own setup? Have you tried on a Sherlock local scratch?
If scratch works well on Sherlock, we can update the containers to
(optionally) copy locally, rip, and copy output back.
C
…On Fri, Jun 11, 2021 at 12:50 PM Tyler Benster ***@***.***> wrote:
Note that the version of software has to exactly match (ie 5.5 update 4)
or Image ripping will choke (will raise error—not destructive from what
I’ve seen).
@chrisroat <https://github.com/chrisroat> ripping is fast for Linux +
wine when on a fast, local ext4 filesystem. I’ve had problems with software
RAID0 (fails) or Lustre (sloooooow), however.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#12 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAIBDYJY45MSGWUK5DABCMDTSGIXXANCNFSM46PUPELA>
.
|
|
Thanks for your responses! I'll be trying to get the container running for the lab today and test things out. Another thing that might be of interest for you both is that Michael Fox at Bruker told me that in early July there's going to be a new Prairie View release to 5.6! I'm not sure how you can prepare for that, but apparently it's coming soon. Edit:
Could you give a quick rundown of what you mean by a "local ext4 filesystem"? I'm not sure how our cluster/servers are set up yet. All I know is that we have 10gig lines connecting things which is promising. Edit 2: Edit 3: |
I would just benchmark it and if speed is ok wouldn’t worry :)
very common, you’ll have to build the singularity container. See https://github.com/deisseroth-lab/two-photon/blob/master/Singularity
I’ve seen the same error with prairie view 5.5 update 4. I believe this repo has 5.5 update 2 or 3. Unfortunately it seems we need to include the files for all minor versions |
Got it! Sounds good to me.
The IT group just got back to me and now I have Docker privileges! So I'll try to do that today.
Is this something I could help with somehow? If I was to copy the files you have in the Utilities from a computer with update 4 installed into a new folder would that be enough to get things working? I was also wondering: does the version the data is acquired with require the same version ripper? As in if I acquired data with 5.5 update 3 but have updated to 5.5 update 4, I now can't rip the files I made? |
|
You will have to ask Bruker about compatibility. In an ideal world, 5.5 updates 3 and 4 should be compatible. They may even guarantee the 5.6 ripper could support 5.5 data (i.e. newer software can still operate on older data) -- but I'm not sure. By the way, which dll file is missing? Can we just add it to the repo? |
|
Sorry for the delay in response Chris! I've been busy working on some other stuff. I apparently didn't write down the particular .dll file anywhere so I'll be trying to do this again this week and find out which one was missing... |
|
Hey everyone! In the next couple weeks I'll actually be collecting and processing some 2P data from our Bruker Scope and I've finally gotten around to installing your container onto our cluster! Unfortunately none of the Docker enabled machines have Wine installed on them, so I'm asking our IT group to install it on a machine for me. Hopefully they can do that soon. Thanks for your patience keeping the issue open.
Still need to look into this, once I have Wine installed on a machine I'll try installing the container and running it and see what happens. Edit: |
|
Hey Jeremy,
I do not think you need to have wine installed. The great thing about
containers is that they have all the necessary software installed. If you
are able to build the container, you should be able to run the container.
(And building could be done on your own computer, where you can install
docker yourself.)
HTH,
C
…On Sun, Sep 26, 2021 at 3:30 PM Jeremy Delahanty ***@***.***> wrote:
Hey everyone!
In the next couple weeks I'll actually be collecting and processing some
2P data from our Bruker Scope and I've finally gotten around to installing
your container onto our cluster! Unfortunately none of the Docker enabled
machines have Wine installed on them, so I'm asking our IT group to install
it on a machine for me. Hopefully they can do that soon. Thanks for your
patience keeping the issue open.
By the way, which dll file is missing? Can we just add it to the repo?
Still need to look into this, once I have Wine installed on a machine I'll
try installing the container and running it and see what happens.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#12 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAIBDYPEVNFAPSX4XNQ7YR3UD6NHHANCNFSM46PUPELA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
Edit 3: The container has been built and I think I'm super close!
Thanks again for your patience and help Chris! This is all really cool and I'm excited to learn. |
|
I'm happy to report some success! Thanks Tyler and Chris! I've gotten the ripper to at least run without any errors, but the problem is that I think I might be running into the issue where things are still pretty slow on Linux/Wine (if that's still an issue for you that is!). I've let it run for a bit now and so far I'm getting zero tiff files. Interestingly when I look at |
|
Yes, we've had similar reports. What kind of machine are you running on? When I set it up, I mostly worked from a small file that ripped pretty fast (just a few tiff files, I think). If you have a larger test file you can share, we can investigate further. |
|
Here's some specs since I'm not sure what would be most helpful: RAM Fileserver OS I do have a test set of 1k frames (I can collect a smaller one if that would be helpful), but I'm not sure how I'd share it! |
|
What OS are you running? Is it your 1k test set that does not rip?
You could try sharing the rawdata file for the 1k test set on Google Drive
or in a public cloud bucket.
C
…On Mon, Nov 1, 2021 at 8:57 PM Jeremy Delahanty ***@***.***> wrote:
Here's some specs since I'm not sure what would be most helpful:
CPU
***@***.***:/snlkt/data/bruker_pipeline$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
Stepping: 1
CPU MHz: 1200.500
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 5188.09
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 40960K
NUMA node0 CPU(s): 0-15,32-47
NUMA node1 CPU(s): 16-31,48-63
RAM
***@***.***:/snlkt/data/bruker_pipeline$ free -h
total used free shared buff/cache available
Mem: 251G 1.3G 119G 2.4G 130G 246G
Swap: 56G 80K 56G
Fileserver
As far as I'm aware, it's all connected with 10Gb lines to our file
directories.
I do have a test set of 1k frames (I can collect a smaller one if that
would be helpful), but I'm not sure how I'd share it!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#12 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAIBDYMZY55MB7AALY35QKLUJ5OS3ANCNFSM46PUPELA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
Updated the above comment for OS info! Here's a link to a Google Drive. Oh also, yes it is the 1k test set that doesn't rip. |
Turns out the answer to this is yes!
This did not solve things for the container, but I'm pretty sure it wasn't running because it didn't have the correct minor version of the ripper installed on it. I'll be trying to fix that next week... |
|
Thanks for the results on using a Windows VM. Just to confirm -- were you able to rip the 1k test set that was failing via the Docker setup? And how did you run the Windows VM (was it via Docker?)? Regarding the minor version, the repo has 5.4 and 5.5 of the ripper and your data dump seems to be done with 5.5. Since your Windows VM test worked, is it safe to assume the repo has all the necessary files? Can you elaborate on your statement about the minor version statement? [FYI, this isn't my top priority at the moment, so I won't be able to make much progress in the short term. Happy to keep commenting.] |
No problem! I had our IT Admin set one up on the cluster for me. I'm not experienced with setting them up so I had him help. As far as I know, the VM was not built with Docker. I think it was with Virtualbox. I can ask him and double check.
Yeah! It worked just like it does on the local microscope's machine with the GUI and everything.
Sure! I'm not familiar with how to properly talk about the different versions available in software, so I'll explain what I meant.
This is what I had meant, that it didn't have the correct "Update" Version available. I'm not sure which one is present in the repository specifically, but it looks like having a Here's the versions I have on our machine at Salk:
It appears that each one of these sub-updates has it's own version of a file called If you match the version of the So @tbenst was right! It looks like the repository needs to include each of these different versions at a more specific version. Would it be helpful to copy these versions in a pull request? I don't want to do that if it'll get in the way of anyone. The only thing is that we updated from 5.5 Update 5 to 5.6 Update 1. We don't have a copy of 5.5 Update 6 here. I'm not sure how we'd get that, maybe we can ask Bruker for it.
No problem! All of your help has been valuable and your guidance over comments is great. |
Were you using this repo? Or did you use a more recent ripper? If the latter, then we should not be surprised it worked. The test verified that a Windows VM is the same as a workstation -- which it should be. What I was hoping you found out was that this repo worked on a VM. But from the version mismatch errors you see below, I assume this is not true.
Very cool! I'm not sure why they break forward compatibility with an update release. It makes me sad, but perhaps they have documented this somewhere. If we are lucky, Bruker will respect backward compatibility: the latest ripper update for a minor version may be able to read data produced by older acquisition system updates for the same minor version. (i.e. 5.5 500 hopefully can read 5.5 400). If this is the case, we just have to keep the repo/image updated with the most recent update for a given minor version.
If you can test the most recent update for a minor version works, you can replace the 5.4 and 5.5 folders with the most recent update you have. You can include whatever latest update you have for 5.6 as well. If you put something on a branch and create a PR, it won't get in anybody's way. |
Unbelievably, this is not the case. Must be an exact version match. A real PITA |
|
In the meantime until I try out things in the container/give a more thorough set of answers to Chris, here's a message I received from Michael Fox from Bruker about this versioning requirement a couple weeks ago:
|
|
[edited when I realized the version is actually 4 pieces] Michael's statement is weaker than what we've actually found, so it might be worth confirming what we've found. He is saying the A.B release number is all that matters, but we are finding that A.B.C.D must match exactly (C always is 64?). Maybe it's safest just to check in every minor version we have and include the full version in the directory name (right now it is just X.Y). We also will need to update the regex to expand the version to include A.B.C.D: two-photon/two_photon/raw2tiff.py Lines 177 to 185 in 39afef0 |
|
Edit:
The reason for this was that I needed to use the I tried to initiate a PR that has some of those small changes but have no idea if I did things properly/helpfully! In the meantime, I'm running into something new which is encouraging.
|
|
The final 2 err statements are not normal. It shouldn't be trying to create a window. I see the pull request and will take a look later. |
|
Over the past couple days I've discovered a new problem with ripping in the container. Basically, Prairie View outputs The workaround I found was to use the Strangely, even if you include Another thing I have discovered is that, at least on our cluster system here at Salk, we see similar performance for the Windows VM I described before and the Docker container in terms of conversion speed for smaller datasets (less than 10k images). Since I'm still fighting the issue I described above with Docker I haven't had a chance to test a large (about 43k images) conversion there. On the VM, however, performance with large datasets keeps up with local performance for about 20k images and then starts slowing down to an eventual crawl. The VM has hardware that's superior to our local machine funny enough. I've been working with our IT admin to see if I can resolve this slowdown, but there's no obvious cause that we have found yet. I figured I'd share that info as well. I'm in contact with Michael at Bruker about this to see what he says, but won't hear back probably until after this weekend I'm imagining due to Thanksgiving. In the meantime, I'm going to try to run the ripper locally on the microscope's machine and then concatenate the OME.tiffs to H5 using the code in your repo and then copy stuff to our server with |
|
I've found Bruker output isn't always adhering to spec, which is too bad. It's also too bad they use v1.1 features, since (as you found out), most python parsers don't implement 1.1. One thing I noticed that runscript calls the With the Windows VM, my understanding is that you are using a Prairie View install and not the code in this repo? If so, that might be getting us closer to the root cause. Hopefully Bruker (or your IT team) can shed light on what the program is doing that won't work in a container. |
|
Hey Chris! I have some interesting updates as well as answers to your questions:
Definitely. It seems like they only output v1.1 some of the time. When we weren't using voltage outputs, which drive the stimulations we've implemented, we had no issues with the XML version.
I'm calling just rip.py in the script I've been testing which isn't part of the PR I started. Since I wasn't sure if it made sense to include that small of a thing in it I figured I'd leave it out.
We basically just copied the ripper utility to the VM, navigate into the Speaking of Wine environments, whenever I create a new Image, a temporary folder is generated for Wine wherever I try to run the actual ripping script. Any idea how to specify where these intermediate/temporary folders are stored when I create images? I talked with Michael over email today and he gave me some more information about what the ripper is really up to for a few things. He gave me the following information:
For the ripper to work, it must allocate 4 bytes per channel per pixel of memory to the process. If the ripper can't find enough contiguous memory when you call it, an error is raised that requires a restart of the computer. Memory is allocated just once per dataset, meaning once per Filelist.txt by a call to a convert function in the
The ripping utility is single core and essentially operates as a for loop iterating through the raw files on found in the Filelist.txt. Therefore, allocating additional cores to an individual instance of the ripper should not impact performance. As of 11/30/21, it has been found that the ripper does sometimes appear to use more than one core while operating, but it's possible that this is for the actual processing of reading and writing to/from disk. Michael says that " it’s basically just a giant (single threaded) loop reading/processing the raw data and writing out TIFF files."
It's possible that the software hooks used in Wine or in the virtualization process are causing some kind of fundamental issue with how Bruker's code actually executes. Our IT lead has mentioned that it's possible this is the reason things act strangely at some point. He also said that it is still strange that things work normally at first and then slow down, but virtualization can basically be weird is what he said. Today I tested a different Windows machine (I'm pretty sure it's not a VM...) that is connected to the local file system and has a RAID0 NVMe SSD available and it works just as well as locally. Lastly, I found out that many of the files currently on the repository for Bruker's Utilities are unnecessary for the ripper. All that is required is the |
|
Re the terrible ripping speeds on large files, I’m afraid I can report the
same behavior even on windows when ripping on NVME drives. It appears that
the ripper has quadratic performance with number of tiffs. I have reported
this issue multiple times to Bruker with no resolution. I would encourage
you to report the issue as well.
|
|
Dang, good to know about for sure that I'm not the only one seeing it... Thanks for the info Tyler! I mentioned it to Bruker and they basically said that the ripper shouldn't really experience a slow down unless there's some kind of file I/O issue or disk writing access issue. Interestingly when I ran the ripper on a recording of 53k images today on a machine in the cluster it didn't seem to suffer any slowdown. It also never peaked the processors that were available and didn't use much RAM at all. When I ran the same recording on the local machine that collected the images it doesn't look like it slows down either. There's definitely something strange going on sometimes and I'm wondering if it's that file I/O isn't consistent on networks even if they're sufficiently fast. I don't know enough about CS/hardware to say anything about that really. I'll keep you updated as I test things out tomorrow on our VM. |
|
Sounds like some good progress toward defining the problem. We have setups that work, and setups that don't. We should isolate what exactly the issue is, and try just changing one thing at a time in moving between "works" to "not works" (or vice versa). I don't know the exact setups, but I'll throw out some ideas you can try. I do think it's best to always be using a local filesystem. It's possible that the non-local filesystem is slow, and just has a local cache of about 20k images that fills up quickly. This would make it look like it writes the first 20k images fast, but it may not be. Can you take the setup that isn't working, and check if it's a local file system -- and if not, use the local filesystem? For this working setup:
can you try some experiments:
For the following:
|
|
Sorry for the delay in getting back to you! Been recovering from a COVID booster and busy imaging a bunch of mice over the the past couple days.
This set up was not connected to the local file system. Our IT is currently troubleshooting a problem with our "scratch space" used for testing out different things that I expect to be completed soon (hopefully today). Once that's ready I'll try it out again and report back.
Our IT admin needs to set up Docker on this computer for me to try this out. I asked him about it on a ticket already but he hasn't had a chance to look it over yet. I also don't think I was clear before, this machine doesn't have a full direct installation of Prairie View on it. All it has are these two files:
Will also see if this is possible pending IT.
This machine is the working setup I described above. I ran it using the
This one is not a VM.
Yes, NVME RAID0.
Yes, it's on the local file system. |
|
Hey all. This is my last week working in lab. I can still continue to comment here, but my help will be limited. My best thought given all this uncertainty is to add code that can optionally copy data to a local disk, and then copy the results from local disk back out to the main cluster file system. |
|
Good luck on your next adventure Chris! I'm still waiting for my IT manager to get me a VM on the local file system unfortunately so I can test it out and I found out today that the machine I was running the container on is NFS mounted to the directory I'm writing to. I think I'm at the mercy now of him giving me a writable drive for this purpose on a local filesystem... Edit: The compute I'm using for this case has a local file disk that I'm allowed to use and the container runs without slowing down! It looks like the issue truly is this NFS file mount situation! |
Hello @chrisroat!
I'm now in the process of automating our ETL pipeline for the Tye lab and will start using some of your code soon! The first thing I wanted to do is integrate Prairie View's image ripping utility correctly but I'm not sure how to change certain parameters of the program. I noticed in
rip.pythat there's a section written as follows:For now, I'm hoping to retain the Raw data until I'm comfortable that the pipeline works correctly. Does the
cmdlist here overwrite any defaults that the utility has? It seems that the tool by default has the"Delete Original Data after creating Images?"selected so I want to make sure that's not enabled.Any advice?
Edit: I was also wondering if it's necessary to have all the
Prairie View Utilitiesfiles installed on the machine as well of if having the Utility alone is sufficient. I'll be performing the ripping itself on a computing cluster and want to make sure I have everything the tool needs.Thanks again for responding to me over email! I really appreciate your help and the work you've made available here. It's really going to help our system get started.
The text was updated successfully, but these errors were encountered: