Dockerfile

[ashotmarg]

Hi @psathyrella, thanks for creating "partis".
I've started playing a bit with making a nextflow pipeline for partis. Among other things (ease of use and integration with other pipelines), this would easily solve the potential complications with docker images with batch systems.
I tried to make a docker image using the Dockerfile (running "docker build -t partis_docker ." inside the partis folder) and got an error.

The ides was to make sure this step works, so that I could gradually build/include the dependencies (e.g., R and R packages) for simulation, MDS and other analyses.
Any advice on this would be great, not sure if I missed something !
Thanks.

[psathyrella]

Great! Sounds like an awesome addition.

I just rebuilt with the same command as you on my machine and it worked fine, so I'm guessing it's something nextflow-specific? I'm not familiar with it at all, but your output looks a bit different so I'm imagining the env is also different. It looks like the build script isn't correctly cd'ing into the ham dir for you, so I'm guessing the script dir variable isn't getting set correctly? It looks like I changed it to this weird/complicated way pretty recently so it'd work in the linearham build, but in retrospect it's a bit silly to make it bash dependent. Maybe you can figure out what shell you're in there and suggest something that'll work in both?

edit: although, the build script does have #!/bin/bash at the top, so seems like it ought to reliably be running bash.

[ashotmarg]

Thanks for the prompt feedback! It's not Nextflow dependent, as Nextflow just calls the docker (or conda, singularity...) container and runs it in pipelines and/or submit jobs... Actually, I already tried partis with Nextflow, and it works nicely with your already build image (quay.io/matsengrp/partis), so that part is fine.
What I did there, was to just run docker build, and (re)create your image, in order to play around with adding R and packages. I tried this on macOS Ventura, will try it on Linux as well, see if that helps.
Cheers!

[psathyrella]

ah, ok, then sounds like a difference in the macos vs linux docker versions. If it's possible to suggest a fix that'd be great.

[KasapovicS]

When you do just the git clone on the repo the partis\packages\ham folder is empty.
This isnt the only one, there are many of them as this repository contains submodules, the entire repo should be ~550MB.
You should instead run

git clone --recursive https://github.com/psathyrella/partis.git

This populates all those folders and gets you the code for building the image.

[psathyrella]

That's a great point, running the wrong git clone command would cause this error. Do you know why there would be a difference in this (maybe in the COPY behavior) in different docker versions, though?

[KasapovicS]

That's a great point, running the wrong git clone command would cause this error. Do you know why there would be a difference in this (maybe in the COPY behavior) in different docker versions, though?

COPY behavior is the same once docker build starts, that's kinda the point of docker. There are some differences though in the build process on new Macs.

This major difference happens on newer MacOS, due to different processor instructions. On an Intel/AMD Machine (even an older Mac), docker pull will pull the linux AMD64 image with FROM instruction or docker pull. On a newer Mac using M1/2 processor, docker pull will the pull the linux/arm64/v8 image. This different image might miss some stuff with installations, most notorious one being missing of gcc or the distro not having all python wheels.

But here we quite clearly have a file missing. I might be a bit over confident, because I got the same error as he did - when I cloned it wrong :). Regular clone will give you the ham folder, but it will be empty. Since there was nothing in the ham folder, the Sconstruct file was not found.

[psathyrella]

Yes I think you're likely right that this is the issue. It seems absolutely wild to me that this kind of difference exists (like, isn't avoiding this the whole point of docker?). In any case, let me know if there's anything I can do to avoid it, otherwise I guess I can just put a note that it may not work on mac. I've tried setting submodule.recurse in the repo's .git/config, but i don't think there's a way for that to propagate.

[KasapovicS]

You could build an image yourself for each version you release and push it somewhere. Like dockerhub.

[psathyrella]

Does anyone know if this is still an issue with the new python 3 docker file?

[psathyrella]

Oh, and yeah, I used to manually build and push images to docker hub for the year or so between when dockerhub disabled auto builds and I switched to quay. If it would really solve a problem I could potentially do it occasionally, but I'm not sure I've understood how me building partis would get around the problems above? Maybe if I had access to a mac, but I don't -- perhaps a better fix would be adding documentation to help people build their own docker image on mac?