Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container to container resolver XML file to YAML #868

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion group_vars/gxconfig.yml
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ base_app_main: &BASE_APP_MAIN
# If this is set to None, the default container resolvers loaded is
# determined by enable_mulled_containers. For available options see
# https://docs.galaxyproject.org/en/master/admin/container_resolvers.html
container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.xml"
container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"

# Rather than specifying a container_resolvers_config_file, the
# definition of the resolvers to enable can be embedded into Galaxy's
Expand Down
2 changes: 1 addition & 1 deletion group_vars/sn06.yml
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,7 @@ galaxy_config_templates:
dest: "{{ galaxy_server_dir }}/lib/galaxy/tools/toolbox/filters/global_host_filters.py"
- src: "{{ galaxy_config_template_src_dir }}/config/job_conf.yml.j2"
dest: "{{ galaxy_config_dir }}/job_conf.yml"
- src: "{{ galaxy_config_template_src_dir }}/config/container_resolvers_conf.xml.j2"
- src: "{{ galaxy_config_template_src_dir }}/config/container_resolvers_conf.yml.j2"
dest: "{{ galaxy_config_dir }}/container_resolvers_conf.xml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dest: "{{ galaxy_config_dir }}/container_resolvers_conf.xml"
dest: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"

- src: "{{ galaxy_config_template_src_dir }}/config/user_filters.py.j2"
dest: "{{ galaxy_server_dir }}/lib/galaxy/tools/toolbox/filters/user_filters.py"
Expand Down
2 changes: 1 addition & 1 deletion group_vars/sn07.yml
Original file line number Diff line number Diff line change
Expand Up @@ -451,7 +451,7 @@ galaxy_config_templates:
dest: "{{ galaxy_server_dir }}/lib/galaxy/tool_util/toolbox/filters/global_host_filters.py"
- src: "{{ galaxy_config_template_src_dir }}/config/job_conf.yml.j2"
dest: "{{ galaxy_config_dir }}/job_conf.yml"
- src: "{{ galaxy_config_template_src_dir }}/config/container_resolvers_conf.xml.j2"
- src: "{{ galaxy_config_template_src_dir }}/config/container_resolvers_conf.yml.j2"
dest: "{{ galaxy_config_dir }}/container_resolvers_conf.xml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dest: "{{ galaxy_config_dir }}/container_resolvers_conf.xml"
dest: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"

- src: "{{ galaxy_config_template_src_dir }}/config/user_filters.py.j2"
dest: "{{ galaxy_server_dir }}/lib/galaxy/tool_util/toolbox/filters/user_filters.py"
Expand Down
64 changes: 0 additions & 64 deletions templates/galaxy/config/container_resolvers_conf.xml.j2

This file was deleted.

128 changes: 128 additions & 0 deletions templates/galaxy/config/container_resolvers_conf.yml.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Sample file for container resolvers. Resolvers that are potentially used by default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Sample file for container resolvers. Resolvers that are potentially used by default
# Configuration file for container resolvers. Resolvers that are potentially used by default

Just because the file is used in production now.

# are uncommented (comments describe under which premises they are in the
# defaults).

# Note that commented yaml does not have a space after the #
# while additional explanations do.

# Explicit container resolvers
# ============================

# get a container description (URI) for an explicit docker container requirement
- type: explicit

# get a container description (URI) for an explicit singularity container requirement
- type: explicit_singularity

# get a cached container description (path) for singularity
# pulls the container into a cache directory if not yet there
#- type: cached_explicit_singularity
# set the cache directory for storing images
#cache_directory: database/container_cache/singularity/explicit

# Mulled container resolvers
# ==========================

# The following uncommented container resolvers are in the defaults
# if ``enable_mulled_containers`` is set in ``galaxy.yml`` (which is the default).

# get a container description to a cached mulled docker container
- type: cached_mulled
# This resolver type will check the image cache for `quay.io/NAMESPACE/containername`
# The image cache can be populated by `mulled_*` resolvers or
# with a manual `docker pull quay.io/NAMESPACE/containername`.
namespace: biocontainers
#- type: cached_mulled
# namespace: local
# # The local namespace refers to locally built images that are prefixed with quay.io/local/.
# #namespace: biocontainers
# # the version of the mulled hashing function (v2/v1)
# #hash_func: v2



# checks if the image file exists in `cache_directory`
- type: cached_mulled_singularity
#namespace: biocontainers
cache_directory: "/cvmfs/singularity.galaxyproject.org/all/"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cache_directory: "/cvmfs/singularity.galaxyproject.org/all/"
cache_directory: "{{ galaxy_singularity_images_cvmfs_path }}"

#
# the method for caching directory listings (not the method for image caching)
# can be uncached or dir_mtime (the latter only determines the directory listing
# if the modification time of the directory changed)
#cache_directory_cacher_type: uncached

#- type: cached_mulled_singularity
# namespace: local
# #cache_directory_cacher_type: uncached

# Resolves container images from quay.io/NAMESPACE/MULLED_HASH where the
# mulled hash describes which packages and versions should be in the container
#
# If the docker CLI is available the image will be pulled.
#
# These resolvers are generally listed after the cached_* resolvers, so that images
# are not pulled if they are already cached.
- type: mulled
namespace: biocontainers
#
# If true return quay.io/NAMESPACE/MULLED_HASH otherwise to the cached image
# which is also quay.io/NAMESPACE/MULLED_HASH in case of docker
#auto_install: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#auto_install: true
auto_install: true

We have auto install explicitly set to true in the XML version of the file, although judging by the description, I don't even think it makes a difference.


# Differences with the docker `mulled` resolver are
# - when pulling the image file will be stored in the configured cache dir
# - if auto_install is True the result will point to the cached image file
# and to quay.io/NAMESPACE/MULLED_HASH otherwise
- type: mulled_singularity
namespace: biocontainers
# In addition to the arguments of `mulled` there are cache_directory
# and cache_directory_cacher_type. See the description at `cached_explicit_singularity`
# and note the minor difference in the default for `cache_directory`
cache_directory: "/cvmfs/singularity.galaxyproject.org/all/"
#cache_directory_cacher_type: uncached

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
auto_install: true

Same story as before, we have true in the XML version.

# Building container resolvers
# ----------------------------
#
# The following uncommented container resolvers are included in the default
# if ``docker`` is available

#- type: build_mulled
# #namespace: local
# #hash_func: v2
# #auto_install: true
#- type: build_mulled_singularity
# #hash_func: v2
# #auto_install: true
# #cache_directory: database/container_cache/singularity/mulled
# #cache_directory_cacher_type: uncached


# Other explicit container resolvers
# ----------------------------------

#- type: fallback
#identifier: A_VALID_CONTAINER_IDENTIFIER

- type: fallback_singularity
identifier: "/data/0/singularity_base_images/centos:8.3.2011"
#- type: fallback_no_requirements
#identifier: A_VALID_CONTAINER_IDENTIFIER
- type: fallback_no_requirements_singularity
identifier: "/data/0/singularity_base_images/centos:8.3.2011"
#- type: requires_galaxy_environment
#identifier: A_VALID_CONTAINER_IDENTIFIER
#- type: requires_galaxy_environment_singularity
#identifier: A_VALID_CONTAINER_IDENTIFIER

# The mapping container resolver allows to specify a list of mappings from tools
# (tool_id) to containers (type and identifier).

#-type: mapping
#mappings:
#- container_type: docker
#tool_id: A_TOOL_ID
#identifier: A_VALID_CONTAINER_IDENTIFIER
#- container_type: singularity
#tool_id: A_TOOL_ID
#identifier: A_VALID_CONTAINER_IDENTIFIER# get a container description for a cached mulled singularity container