MRIQC: "Permission denied: '/home/mriqc/.cache'"

[jbwexler]

Getting this for all MRIQC runs for any dataset:

  File "/opt/conda/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/mriqc/.cache/datalad/sockets'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/mriqc/.cache/datalad'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 56, in build_workflow
    retval['workflow'] = init_mriqc_wf()
                         ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/workflows/core.py", line 48, in init_mriqc_wf
    workflow.add_nodes([fmri_qc_workflow()])
                        ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/workflows/functional/base.py", line 85, in fmri_qc_workflow
    _datalad_get(dataset)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 257, in _datalad_get
    return get(
           ^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/interface/base.py", line 773, in eval_func
    return return_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/interface/base.py", line 763, in return_func
    results = list(results)
              ^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad_next/patches/interface_utils.py", line 197, in
_execute_command_
    hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)
                                      ^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/distribution/dataset.py", line 331, in config
    repo = self.repo
           ^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/distribution/dataset.py", line 273, in repo
    self._repo = repo_from_path(self._path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/core/local/repo.py", line 61, in repo_from_path
    repo = cls(path, create=False, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/dataset/repo.py", line 163, in __call__
    instance = type.__call__(cls, *new_args, **new_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datalad/support/annexrepo.py", line 198, in __init__
    super(AnnexRepo, self).__init__(
  File "/opt/conda/lib/python3.11/site-packages/datalad/support/gitrepo.py", line 938, in __init__
    ssh_manager.ensure_initialized()
  File "/opt/conda/lib/python3.11/site-packages/datalad/support/sshconnector.py", line 738, in ensure_i
nitialized
    self._socket_dir.mkdir(exist_ok=True, parents=True)
  File "/opt/conda/lib/python3.11/pathlib.py", line 1120, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/opt/conda/lib/python3.11/pathlib.py", line 1120, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/opt/conda/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/home/mriqc/.cache'```

[yarikoptic]

could you remind how container executed in your case? if via our singularity_cmd then we might need to make sure that we have ~/.cache in our "fake HOME". but also we might need to fix datalad -- it should create all leading folders there and not assume ~/.cache to be present.

[jbwexler]

I'm using ReproMan to submit a slurm job on TACC's Frontera. I also add --bind /tmp:/node_tmp into the singularity command so that it will use the node's local temp rather than the scratch temp. I add that into the containers/.datalad/config file so it runs that way automatically when I submit via reproman. Here's an example of a command-array file:

code/containers/scripts/singularity_cmd run --bind /tmp:/node_tmp code/containers/images/bids/bids-mriqc--24.0.0.sing sourcedata/raw /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004488-mriqc participant --participant-label '01' -w '/node_tmp/work_dir/mriqc/ds004488_sub-01' -vv --nprocs 11 --ants-nthreads 8 --verbose-reports --dsname ds004488 --mem_gb 30

[jbwexler]

@yarikoptic Any ideas for a workaround?

[yarikoptic]

sorry, on this one I just filed

• Is it possible to not hardcode HOME in Dockerfile? nipreps/mriqc#1307
  and have not heard back yet. Workaround would be (didn't try) to just "manually" bind mount some tmp path into /home/mriqc/.cache for that call within .datalad/config I guess.

[jbwexler]

Thanks, I'll give that a try if it doesn't get resolved soon

[jbwexler]

I've finally gotten around to testing this - it worked! At least for v24.0.0 it did. With v24.0.1 I have a new permissions issue:

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/opt/conda/bin/python3.11'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/conda/lib/python3.11'
  sys._base_executable = '/opt/conda/bin/python3.11'
  sys.base_prefix = '/opt/conda'
  sys.base_exec_prefix = '/opt/conda'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/conda/bin/python3.11'
  sys.prefix = '/opt/conda'
  sys.exec_prefix = '/opt/conda'
  sys.path = [
    '/opt/conda/lib/python311.zip',
    '/opt/conda/lib/python3.11',
    '/opt/conda/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 936, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1073, in get_code
  File "<frozen importlib._bootstrap_external>", line 1130, in get_data
PermissionError: [Errno 13] Permission denied: '/opt/conda/lib/python3.11/encodings/__init__.py'

[yarikoptic]

I feel like I had fixed those permissions issues -- ReproNim/containers@78bfcd8 . apparently issue persists for new ones too, dang... I redid the same for newer version in bfe96681faad56b3c5a7e489b2aa653adf97d18b now and pushing . yet to figure out permanent solution

[jbwexler]

Thanks, I'll test it out once it's uploaded. Will that fix the "Permission denied: '/home/mriqc/.cache'" issue too or should I keep binding that?

[yarikoptic]

it finished uploading:

$> datalad push
copy(ok): images/bids/bids-mriqc--24.0.1.sing (file) [to datalad-public...]
publish(ok): . (dataset) [refs/heads/git-annex->datalad-public:refs/heads/git-annex 04f6e573..ed97304f]
publish(ok): . (dataset) [refs/heads/master->datalad-public:refs/heads/master d1b86f29..bfe96681]
publish(ok): . (dataset) [refs/heads/git-annex->origin:refs/heads/git-annex c897ad4a..ed97304f]
publish(ok): . (dataset) [refs/heads/master->origin:refs/heads/master d1b86f29..bfe96681]
                                                                                                                                                                                                                                                                  action summary:
  copy (ok: 1)
  publish (ok: 4)
datalad push  11.31s user 8.36s system 11% cpu 2:53.17 total
2 10548.....................................:Fri 23 Aug 2024 01:25:50 PM EDT:.

no -- likely it would not fix the .cache issue which is the still open

• Is it possible to not hardcode HOME in Dockerfile? nipreps/mriqc#1307
  so you would need to keep relying on the workaround.

[jbwexler]

Thanks, the new image worked perfectly.