Merge pull request #336 from BU-ISCIII/develop

Develop merge for 2.2.0 release

.github/workflows/python_lint.yml

@@ -2,11 +2,10 @@ name: python_lint
 
 on:
   push:
-    paths:
-      - '**.py'
+    branches: "**"
   pull_request:
-    paths:
-      - '**.py'
+    types: [opened, reopened, synchronize, closed]
+    branches: "**"
 
 jobs:
   flake8_py3:
@@ -18,18 +17,43 @@ jobs:
           python-version: 3.9.x
           architecture: x64
       - name: Checkout PyTorch
-        uses: actions/checkout@master
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
       - name: Install flake8
         run: pip install flake8
+      - name: Check for Python file changes
+        id: file_check
+        uses: tj-actions/changed-files@v44
+        with:
+          sha: ${{ github.event.pull_request.head.sha }}
+          files: |
+             **.py
       - name: Run flake8
+        if: steps.file_check.outputs.any_changed == 'true'
         run: flake8 --ignore E501,W503,E203,W605
+      - name: No Python files changed
+        if: steps.file_check.outputs.any_changed != 'true'
+        run: echo "No Python files have been changed."
 
   black_lint:
     runs-on: ubuntu-latest
     steps:
         - name: Setup
           uses: actions/checkout@v2
+          with:
+            fetch-depth: 0
         - name: Install black in jupyter
           run: pip install black[jupyter]
+        - name: Check for Python file changes
+          id: file_check
+          uses: tj-actions/changed-files@v44
+          with:
+            sha: ${{ github.event.pull_request.head.sha }}
+            files: '**.py'
         - name: Check code lints with Black
+          if: steps.file_check.outputs.any_changed == 'true'
           uses: psf/black@stable
+        - name: No Python files changed
+          if: steps.file_check.outputs.any_changed != 'true'
+          run: echo "No Python files have been changed."

CHANGELOG.md

@@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [2.2.Xdev] - 2024-0X-XX : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.2.X
+## [2.X.Xdev] - 2024-0X-XX : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.X.X
 
 ### Credits
 
@@ -24,7 +24,7 @@ Code contributions to the new version:
 
 ### Requirements
 
-## [2.X.1hot] - 2024-0X-0X : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.X.1
+## [2.X.Xhot] - 2024-0X-0X : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.X.1
 
 ### Credits
 
@@ -44,6 +44,99 @@ Code contributions to the hotfix:
 
 ### Requirements
 
+## [2.2.0] - 2024-09-12 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.2.0
+
+### Credits
+
+Code contributions to the new version:
+- [Pablo Mata](https://github.com/Shettland)
+- [Jaime Ozáez](https://github.com/jaimeozaez)
+- [Sara Monzón](https://github.com/saramonzon)
+- [Sarai Varona](https://github.com/svarona)
+- [Daniel Valle](https://github.com/Daniel-VM)
+- [Víctor López](https://github.com/victor5lm)
+- [Juan Ledesma](https://github.com/juanledesma78)
+
+### Template fixes and updates
+
+- Updated documentation and results markdown for viralrecon, pikavirus and MAG [#247](https://github.com/BU-ISCIII/buisciii-tools/pull/247)
+- Added documentation and results markdown for RNAseq [#248](https://github.com/BU-ISCIII/buisciii-tools/pull/248)
+- Added documentation both output and results for plasmidID[#258](https://github.com/BU-ISCIII/buisciii-tools/pull/258)
+- Added markdown of assembly analysis procedure [#244](https://github.com/BU-ISCIII/buisciii-tools/pull/244)
+- Added output and results markdowns for ExomeEB, ExomeTrio and WGStrio [#249](https://github.com/BU-ISCIII/buisciii-tools/pull/249)
+- Added markdown of assembly results folder [#250](https://github.com/BU-ISCIII/buisciii-tools/pull/250)
+- Updated lablog results filenames where necessary (IRMA, seekndestroy, viralrecon and genomeev) [#253](https://github.com/BU-ISCIII/buisciii-tools/pull/253)
+- Added output and results markdowns for cgMLST/wgMLST [#255](https://github.com/BU-ISCIII/buisciii-tools/pull/255)
+- Added markdown for IRMA [#256](https://github.com/BU-ISCIII/buisciii-tools/pull/256)
+- Included RESULTS/lablog for exomeeb, exometrio and wgstrio templates and updated files to clean  [#260](https://github.com/BU-ISCIII/buisciii-tools/pull/260)
+- Changed scratch copy queue to middle_obx
+- Included missing folders in wgstrio template
+- Changed exomiser-html-description to png format and fixed location of irma-sarek markdowns [#261](https://github.com/BU-ISCIII/buisciii-tools/pull/261)
+- Updated configuration.json so that either idx or obx is used in case one of these queues is full [#263](https://github.com/BU-ISCIII/buisciii-tools/pull/263)
+- Updated lablog_viralrecon script for the automation of the setup of viralrecon services. [#264](https://github.com/BU-ISCIII/buisciii-tools/pull/264)
+- Included MULTIQC v.1.19 in viralrecon.config to fix error with string and numeric samples [#267](https://github.com/BU-ISCIII/buisciii-tools/pull/267)
+- Updated MTBSeq template to fit bacass pipeline. [#268](https://github.com/BU-ISCIII/buisciii-tools/pull/268)
+- IRMA template modified in order to avoid average overload.
+- Added "01" to results folder creation in assembly template.
+- Some prompt answers limited to 1 character in lablog_viralrecon.
+- Created lablog_mtbseq_results. [#270](https://github.com/BU-ISCIII/buisciii-tools/pull/270)
+- PR #271. Closes [#235](https://github.com/BU-ISCIII/buisciii-tools/issues/235), [#228](https://github.com/BU-ISCIII/buisciii-tools/issues/228) and [#196](https://github.com/BU-ISCIII/buisciii-tools/issues/196)
+- Included annotated tab description in exome-trios markdowns [#273](https://github.com/BU-ISCIII/buisciii-tools/pull/273)
+- Installed all necessary singularity images and modified all templates so that, instead of using conda environments or loaded modules, the corresponding singularity images are used [#272](https://github.com/BU-ISCIII/buisciii-tools/pull/272)
+- Updated sarek version in exomeeb, exometrio and wgstrio templates [#277](https://github.com/BU-ISCIII/buisciii-tools/pull/277)
+- Extension file of all_samples_virus_table_filtered (from csv to tsv) in lablog_viralrecon_results changed [#278](https://github.com/BU-ISCIII/buisciii-tools/pull/278)
+- Fixed singularity-images path when updating pangolin database in lablog_viralrecon. Added line break after prompted input. [#282](https://github.com/BU-ISCIII/buisciii-tools/pull/282)
+- Updated characterization and snippy templates to fit bacass pipeline. Corrected path in 05-iqtree in snippy template. [#283](https://github.com/BU-ISCIII/buisciii-tools/pull/283)
+- Included multiqc_report.html in RESULTS folder in every service, where necessary [#265] (https://github.com/BU-ISCIII/buisciii-tools/pull/265)
+- Added MAG tempalte and removed MAG from other templates [#288](https://github.com/BU-ISCIII/buisciii-tools/pull/288)
+- Added amrfinderplus to characterization template. [#289] (https://github.com/BU-ISCIII/buisciii-tools/pull/289)
+- Updated all files so that paths referring to /pipelines/ are updated according to the new structure [#287](https://github.com/BU-ISCIII/buisciii-tools/pull/287)
+- Updated assembly, ariba, snippy, amrfinderplus and iqtree templates, removed genomeev and mtbseq_assembly templates and updated services.json [#295](https://github.com/BU-ISCIII/buisciii-tools/pull/295)
+- Changed viralrecon's lablog so that references are available within refgenie [#296](https://github.com/BU-ISCIII/buisciii-tools/pull/296)
+- Updated services.json, mtbseq's lablog, viralrecon's lablog and assembly's config file [#299](https://github.com/BU-ISCIII/buisciii-tools/pull/299)
+- Added lablog to automate gene characterization with emmtyper, including unzipping assemblies. [#300](https://github.com/BU-ISCIII/buisciii-tools/pull/300)
+- Fixed 99-stats (MAG) template. [#301](https://github.com/BU-ISCIII/buisciii-tools/pull/301)
+- Created a python script to process IRMA's results and create a standard vcf file against reference. [#304](https://github.com/BU-ISCIII/buisciii-tools/pull/304)
+- Fixed IRMA's lablog so that the sequences of the samples are not displayed several times neither in the .txt files of each influenza type nor in all_samples_completo.txt [#305](https://github.com/BU-ISCIII/buisciii-tools/pull/305)
+- Modified bioinfo_doc.py so that new lines in the delivery message are applied in the email [#307](https://github.com/BU-ISCIII/buisciii-tools/pull/307)
+- Added several improvements in lablog_viralrecon (created log files, modified check_references function behaviour, enabled config files regeneration) [#306](https://github.com/BU-ISCIII/buisciii-tools/pull/306)
+- Fixed bug when lablog_viralrecon tries to download references that don't belong to any family. [#310](https://github.com/BU-ISCIII/buisciii-tools/pull/310)
+- Added mvmoneo to SFTP users. [#317](https://github.com/BU-ISCIII/buisciii-tools/pull/317)
+- Added scripts for time series RNAseq and updated differential expression code for differentially expressed transcripts [#316](https://github.com/BU-ISCIII/buisciii-tools/pull/316).
+- Added bbaladron to SFTP users [#316](https://github.com/BU-ISCIII/buisciii-tools/pull/316).
+- Added new template for comprehensive taxonomy profiling using the nf-core/taxprofiler pipeline [#320](https://github.com/BU-ISCIII/buisciii-tools/pull/320).
+- Added full execution support for the MAG template [#321](https://github.com/BU-ISCIII/buisciii-tools/pull/321).
+- Added labels to services.json and updated bioinfo_doc.py and jinja_template_delivery.j2 so that software versions data is displayed in the delivery pdf [#330](https://github.com/BU-ISCIII/buisciii-tools/pull/330).
+- Updated several templates (singularity images, outdated paths, improvements, etc) [#331](https://github.com/BU-ISCIII/buisciii-tools/pull/331)
+- Added permissions fixing after running scratch_copy, as well as a new fix-permissions module in the tools [#332](https://github.com/BU-ISCIII/buisciii-tools/pull/332).
+- Updated MAG lablogs and utils.py [#334](https://github.com/BU-ISCIII/buisciii-tools/pull/334).
+- Updated some files (setup.py, __main__.py, README, etc) for the 2.2.0 release [#335](https://github.com/BU-ISCIII/buisciii-tools/pull/335).
+
+### Modules
+
+#### Added enhancements
+
+- PR [#274](https://github.com/BU-ISCIII/buisciii-tools/pull/274): added `--dev` option, configuration dev and test folder structure.
+- PR [#276](https://github.com/BU-ISCIII/buisciii-tools/pull/276): wkhtmlpdf does not need absolute path to executable. Added better error handling when executable does not exists.
+- PR [#288](https://github.com/BU-ISCIII/buisciii-tools/pull/288) Allowed to handle more than one service at a time, related to issue [#217](https://github.com/BU-ISCIII/buisciii-tools/issues/217)
+
+#### Fixes
+
+- Fixed archive module. Updated correct header for scout tsv [#258](https://github.com/BU-ISCIII/buisciii-tools/pull/258).
+- Fixed clean module. Corrected purge_files function. Renaming stage moved from clean to rename_nocopy option. Updated services.json file with correct paths for some services. [#280](https://github.com/BU-ISCIII/buisciii-tools/pull/280)
+- Fixed autoclean-sftp function. [#281](https://github.com/BU-ISCIII/buisciii-tools/pull/281)
+- Fixed bioinfo_doc.py. Modified it so that this module creates a .pdf file including new-line characters, without merging lines into one single line [#259](https://github.com/BU-ISCIII/buisciii-tools/pull/259).
+- PR [#288](https://github.com/BU-ISCIII/buisciii-tools/pull/288) Fixed updating service's state to in_progress multiple times, related with issue [#285](https://github.com/BU-ISCIII/buisciii-tools/issues/285)
+- Review and update of services.json for files and folders cleaning [#318](https://github.com/BU-ISCIII/buisciii-tools/pull/318).
+
+#### Changed
+
+- Forcing python lint to success if no .py files are in PR [#279](https://github.com/BU-ISCIII/buisciii-tools/pull/279)
+
+#### Removed
+
+### Requirements
+
 ## [2.1.0] - 2024-04-19 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.1.0
 
 ### Credits

README.md

@@ -7,9 +7,8 @@ BU-ISCIII provides a serie or services in its portfolio for supporting bioinform
 
 - [buisciii-tools](#buisciii-tools)
   - [Installation](#installation)
-    - [Bioconda](#bioconda)
-    - [Pip](#pip)
-    - [Development version](#development-version)
+    - [Micromamba and pip](#micromamba-and-pip)
+    - [Dev version](#dev-version)
   - [Usage](#usage)
     - [Command-line](#command-line)
       - [list](#list)
@@ -22,34 +21,47 @@ BU-ISCIII provides a serie or services in its portfolio for supporting bioinform
       - [bioinfo\_doc](#bioinfo_doc)
       - [archive](#archive)
       - [autoclean\_sftp](#autoclean_sftp)
+      - [fix-permissions](#fix-permissions)
   - [Acknowledgements](#acknowledgements)
 
 ## Installation
 
-### Bioconda
+### Micromamba and pip
 
 ```bash
-conda create -n buisciii-tools pip
-conda activate 
-conda env update --file environment.yml
+micromamba create -n buisciii -f environment.yml
+micromamba activate buisciii
+pip install --force-reinstall --upgrade git+https://github.com/bu-isciii/buisciii-tools.git@main
 ```
 
-### Pip
+or
 
 ```bash
+git checkout main
+conda create -n buisciii -f environment.yml
 conda activate 
 pip install .
 ```
 
-### Development version
+### Dev version
 
 If you want to install the latest code in the repository:
 
 ```bash
-conda create -n buisciii_dev pip
+micromamba create -n buisciii_dev -f environment.yml
+micromamba activate buisciii_dev
 pip install --force-reinstall --upgrade git+https://github.com/bu-isciii/buisciii-tools.git@develop
 ```
 
+or locally:
+
+```bash
+git checkout develop
+micromamba create -n buisciii_dev -f environment.yml
+micromamba activate buisciii_dev
+pip install .
+```
+
 ## Usage
 
 ### Command-line
@@ -72,7 +84,7 @@ Options:
   -u, --api_user TEXT        User for the API logging
   -p, --api_password TEXT    Password for the API logging
   -c, --cred_file TEXT       Config file with API logging credentials
-  --help                     Show this message and exit.
+  --help                     Show this message and exit
 
 Commands:
   list         List available bu-isciii services.
@@ -83,6 +95,8 @@ Commands:
   finish       Service cleaning, remove big files, rename folders before...
   bioinfo-doc  Create the folder documentation structure in bioinfo_doc...
   archive      Archive services or retrieve services from archive
+  autoclean-sftp   Clean old sftp services
+  fix-permissions  Fix permissions
 ```
 
 #### list
@@ -137,9 +151,10 @@ Output:
 │                        │ control, host removal and exploratory     │                                            │
 │                        │ analysis of samples.                      │                                            │
 │ ariba_characterization │                                           │                                            │
-│                mag_met │ Bioinformatics best-practise analysis     │ https://github.com/nf-core/mag             │
-│                        │ pipeline for assembly, binning and        │                                            │
-│                        │ annotation of metagenomes.                │                                            │
+│                mag_met │ 1- Bioinformatics best-practise analysis  │ https://github.com/nf-core/mag or          │
+│                        │ for taxonomic classification and          │ https://github.com/nf-core/taxprofiler     │
+│                        │ profiling; 2- Bioinformatics best-practise│                                            │
+│                        │ analysis pipeline for assembly, binning   │                                            │
 └────────────────────────┴───────────────────────────────────────────┴────────────────────────────────────────────┘
 ```
 
@@ -377,6 +392,26 @@ Options:
   --help                  Show this message and exit.
 ```
 
+#### fix-permissions
+
+Example of usage:
+
+```bash
+bu-isciii fix-permissions -d /data/bi
+```
+
+Help:
+
+```bash
+Usage: bu-isciii fix-permissions [OPTIONS]
+
+  Fix permissions
+
+Options:
+  -d, --input_directory PATH  Input directory to fix permissions (absolute path) [required]
+  --help                  Show this message and exit.
+```
+
 ## Acknowledgements
 
 Python package idea and design is really inspired in [nf-core/tools](https://github.com/nf-core/tools).