Trouble getting MCRIBS to run

[scott-huberty]

If I run a command like this:

docker run -e DOCKER_VERSION_8395080871=23.1.0 -it \
    -v /path/to/freesurfer_license/license.txt:/opt/freesurfer/license.txt:ro \
    -v /path/to/bids:/data:ro \
    -v /path/to/derivatives/Nibabies:/out \
    -v /path/to/derivatives/precomputed:/opt/derivatives/precomputed \
    -v /path/to/derivatives/work/nibabies_work:/scratch \
    -v /path/to/repos/nibabies/nibabies:/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies:ro \
    nipreps/nibabies:unstable /data /out participant \
    --age-months 1 \
    --participant-label 999 \
    --derivatives /opt/derivatives/precomputed \
    -w /scratch \
    --surface-recon-method mcribs \
    --cifti-output 91k \
    --verbose

My Nibabies run hangs during these workflows:

infant_single_anat_wf.mcribs_surface_recon_wf.aparc2nii

infant_single_anat_wf.mcribs_surface_recon_wf.brainmask2nii

And a bunch of crash logs are output to /path/to/derivatives/Nibabies/sub-999/ses-newborn/log/..., all of them pointing to various MCRIBS errors.

For example here is the first of the crash logs

Node: nibabies_24_1_wf.single_subject_1390_newborn_wf.infant_single_anat_wf.mcribs_sphere_reg_wf.get_spheres
Working directory: /scratch/nibabies_24_1_wf/single_subject_1390_newborn_wf/infant_single_anat_wf/mcribs_sphere_reg_wf/get_spheres

Node inputs:

function_str = def _get_dhcp_spheres(subject_id: str, subjects_dir: str) -> list:
    from pathlib import Path

    out = []
    for hemi in 'lr':
        sphere = Path(subjects_dir) / subject_id / 'surf' / f'{hemi}h.sphere.reg2'
        if not sphere.exists():
            raise OSError("MCRIBS spherical registration not found.")
        out.append(str(sphere))
    return out

subject_id = sub-1390
subjects_dir = /out/sourcedata/freesurfer

Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node get_spheres.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
	    out = function_handle(**args)
	  File "<string>", line 8, in _get_dhcp_spheres
	OSError: MCRIBS spherical registration not found.

I'm guessing that all the other crash logs are caused by the error in that log, which is:

OSError: MCRIBS spherical registration not found.

I can't tell if there is a certain file that MCRIBS is looking for (but can't find), that I am supposed to supply before calling Nibabies., or if this is a bug in Nibabies. I read the MCRIBS githbub README, and didn't find anything obvious.

FWIW, if I run the same exact command but specify --surface-recon-method infantfs, Nibabies completes without any error.

I'm still fairly new to MRI and the Ni[...preps, babel] ecosystems, so any help with debugging this would be appreciated!
If you want to punt this to the MCRIBS maintainers, I can open a thread there too.

[scott-huberty]

And FWIW... /path/to/derivatives/Nibabies/sourcedata/freesurfer/sub-999 is totally empty..

And

/path/to/derivatives/Nibabies/sourcedata/mcribs/sub-999/freesurfer/sub-999 Doesn't contain any .sphere.reg2 files, Which I think explains the error in the aforementioned log. I just haven't figure out yet why these files are missing.

[mgxd]

If those files are not found, MCRIBReconAll did not run successfully. I'm surprised an error wasn't thrown directly from the interface, as we've embedded a few checks to verify the results

nibabies/nibabies/interfaces/mcribs.py

Lines 232 to 284 in 7b1e91d

	@staticmethod
	def _verify_surfrecon_outputs(surfrecon_dir: Path, error: bool) -> bool:
	"""
	Sanity check to ensure the surface reconstruction was successful.
	MCRIBReconAll does not return a failing exit code if a step failed, which leads
	this interface to be marked as completed without error in such cases.
	"""
	# fmt:off
	surfrecon_files = {
	'meshes': (
	'pial-lh-reordered.vtp',
	'pial-rh-reordered.vtp',
	'white-rh.vtp',
	'white-lh.vtp',
	)
	}
	# fmt:on
	for d, fls in surfrecon_files.items():
	for fl in fls:
	if not (surfrecon_dir / d / fl).exists():
	if error:
	raise FileNotFoundError(f"SurfReconDeformable missing: {fl}")
	return False
	return True
	@staticmethod
	def _verify_autorecon_outputs(fs_dir: Path, error: bool) -> bool:
	"""
	Sanity check to ensure the necessary FreeSurfer files have been created.
	MCRIBReconAll does not return a failing exit code if a step failed, which leads
	this interface to be marked as completed without error in such cases.
	"""
	# fmt:off
	fs_files = {
	'mri': ('T2.mgz', 'aseg.presurf.mgz', 'ribbon.mgz', 'brain.mgz'),
	'label': ('lh.cortex.label', 'rh.cortex.label'),
	'stats': ('aseg.stats', 'brainvol.stats', 'lh.aparc.stats', 'rh.curv.stats'),
	'surf': (
	'lh.pial', 'rh.pial',
	'lh.white', 'rh.white',
	'lh.curv', 'rh.curv',
	'lh.thickness', 'rh.thickness'),
	}
	# fmt:on
	for d, fls in fs_files.items():
	for fl in fls:
	if not (fs_dir / d / fl).exists():
	if error:
	raise FileNotFoundError(f"FreeSurfer directory missing: {fl}")
	return False
	return True

If you check the working directory node, you may find more information on the specific crash.

FWIW, you will need to have a well defined precomputed segmentation to successfully run through MCRIBS, as it can be quite brittle with imperfect segs.

[scott-huberty]

Hey @mgxd sorry for the late reply. After a few attempts to simply clear my Nibabies cache and start the run from scratch, MCRIBS is now running smoothly, and I actually can't replicate the error I reported above. If I run into it again in the future, I'll inspect the code you just pointed to and see if anything can be improved regarding error throwing from the Nibabies end. I'll close this discussion as resolved.

But before I do, one more question: I'm now reliably hitting an error in one of the functional workflows.

FileNotFoundError: [Errno 2] No such file or directory: '/scratch/nibabies_24_1_wf/single_subject_1441_newborn_wf/func_preproc_ses_newborn_task_restPA_run_04_wf/bold_reg_wf/bbreg_wf/mri_coreg/registration.lta'

Basically, it looks like freesurfers mri_coreg is being called, pointing to a file that doesn't exist and wasn't previously generated. From a quick inspection of init_bold_reg_wf (which I think is the relevant code), the docstring states that this func computes the co-registration between the BOLD signal and a T1w space. My subject does not have a T1w space scan, only a T2w, which I thought would be okay after #305 .

So my thinking is that the lack of a T1w scan is the reason that this registration.lta is never generated, thus causing the error I'm seeing. Does this sound right to you?

Full crash log

Node: nibabies_24_1_wf.single_subject_1441_newborn_wf.func_preproc_ses_newborn_task_restPA_run_04_wf.bold_reg_wf.bbreg_wf.mri_coreg
Working directory: /scratch/nibabies_24_1_wf/single_subject_1441_newborn_wf/func_preproc_ses_newborn_task_restPA_run_04_wf/bold_reg_wf/bbreg_wf/mri_coreg

Node inputs:

args = <undefined>
brute_force_limit = <undefined>
brute_force_samples = <undefined>
compress_report = auto
conform_reference = <undefined>
dof = 6
environ = {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
ftol = 0.0001
initial_rotation = <undefined>
initial_scale = <undefined>
initial_shear = <undefined>
initial_translation = <undefined>
linmintol = 0.01
max_iters = <undefined>
no_brute_force = <undefined>
no_coord_dithering = <undefined>
no_cras0 = <undefined>
no_intensity_dithering = <undefined>
no_smooth = <undefined>
num_threads = 1
out_lta_file = True
out_params_file = <undefined>
out_reg_file = <undefined>
out_report = report.svg
ref_fwhm = <undefined>
reference_file = <undefined>
reference_mask = <undefined>
saturation_threshold = <undefined>
sep = [4]
source_file = <undefined>
source_mask = <undefined>
source_oob = <undefined>
subject_id = <undefined>
subjects_dir = /opt/freesurfer/subjects

Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node mri_coreg.

Cmdline:
	mri_coreg --s sub-1441 --dof 6 --ftol 1.000000e-04 --linmintol 1.000000e-02 --threads 1 --lta /scratch/nibabies_24_1_wf/single_subject_1441_newborn_wf/func_preproc_ses_newborn_task_restPA_run_04_wf/bold_reg_wf/bbreg_wf/mri_coreg/registration.lta --sep 4 --mov /scratch/nibabies_24_1_wf/single_subject_1441_newborn_wf/func_preproc_ses_newborn_task_restPA_run_04_wf/bold_reg_wf/bbreg_wf/mri_coreg/clipped.nii.gz --sd /out/sourcedata/freesurfer
Stdout:

Stderr:
	Segmentation fault
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._post_run_hook(runtime)
	  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/niworkflows/interfaces/freesurfer.py", line 248, in _post_run_hook
	    fix_lta_length(lta_file)
	  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/niworkflows/interfaces/freesurfer.py", line 399, in fix_lta_length
	    lines = Path(lta_file).read_text().splitlines(keepends=True)
	  File "/opt/conda/envs/nibabies/lib/python3.10/pathlib.py", line 1134, in read_text
	    with self.open(mode='r', encoding=encoding, errors=errors) as f:
	  File "/opt/conda/envs/nibabies/lib/python3.10/pathlib.py", line 1119, in open
	    return self._accessor.open(self, mode, buffering, encoding, errors,
	FileNotFoundError: [Errno 2] No such file or directory: '/scratch/nibabies_24_1_wf/single_subject_1441_newborn_wf/func_preproc_ses_newborn_task_restPA_run_04_wf/bold_reg_wf/bbreg_wf/mri_coreg/registration.lta'

[scott-huberty]

I actually think my issues have to do with running on an M1 Mac. On an Intel Mac I don't hit the same error. I'm closing this post. Thanks!