Conventions for fs fetch-type plugins

[jennydaman]

Abstract

This document describes a convention for ChRIS fs-type plugins which can download files from network resources. Plugins conformant to this convention must have the name pull-{SCHEME} and must accept one required argument --uri which comes in the form {SCHEME}://{RESOURCE}. ChRIS clients such as the ChRIS_ui can benefit from this convention by being able to create feeds automatically given just a URI to some data.

Introduction

ChRIS fs-type plugins create ChRIS feeds by producing output data files to a given output directory. fs plugins are a means to import datasets into ChRIS.

Currently there exists some fs plugins, which have been described as "data pack plugins," which copy data from a hard-coded path inside their container image to the output directory. Some examples:

• https://github.com/FNNDSC/pl-mri10yr06mo01da_normal
• https://github.com/FNNDSC/pl-mri_sag_anon_192
• https://github.com/FNNDSC/pl-lungct

Distributing data inside of Docker containers has its advantages such as versioning and immutability, however it’s very inefficient. Moreover, it is repetitive to be pairing each of these datasets with their custom Python copy-files script.

A more efficient solution would be to have one fs plugin which is able to "pull" datasets from a public CDN, and make these datasets available from a public CDN.

Convention

Here, the "ChRIS URI 'Pull'" plugin convention is defined.

A plugin conformant to the "URI Pull" convention must:

• be a fs-type plugin
• have the plugin name pull-{SCHEME}
• have a required argument {"name": "uri", "long_flag": "--uri"}
• have the category "Pull"

The plugin should:

• expect that the value given for the argument uri is in the form {SCHEME}://{RESOURCE}
• retrieve data from the network resource indicated by {SCHEME}://{RESOURCE} and write it to the output directory

Where:

• SCHEME is a network protocol, e.g. http, https, git, ftp, ipfs, rsync

• {SCHEME}://{RESOURCE} is a URI

Integration

Plugins To Create

We plan to create the following ChRIS plugins which will conform to the "URI Pull" convention:

• pull-http and pull-https which does something like a wget --recursive

• pull-git for public git repositories, e.g. git://github.com/FNNDSC/SAG-anon-nii.git

• pull-ipfs: see https://ipfs.io/, https://en.wikipedia.org/wiki/InterPlanetary_File_System

• pull-datalad: see https://www.datalad.org/

Clients

ChRIS clients for end-users, such as the ChRIS_ui, can take advantage of the "URI Pull" convention and provide a feature where feeds can be created given a dataset URI.

Issues

What’s a better way to support clients which support multiple protocols? For instance, we would only want to create one pull-http which supports both http:// and https://.

One option is to have a single code base registered with two different plugin names. Another option is for clients to hard-code a mapping of schemes to plugin names.

A more difficult case would be for datalad and git repositories: a URI matching ^https://.+\.git$ could be any of: a HTTPS resource, a git repository, or a datalad repository.