Test data from S3 Access Denied

[AlexWeinreb]

Description of the bug

When running the test pipeline, after removing the modules section of test.config, I get an error:

Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5Y09AVCCQ6XN047R; S3 Extended Request ID: KpBg8rVOT5mMawlT7bdlqpEoPVdGzCbvvKnBnZzPZr9AxmOaTlqDM6R3YTrwZ6vIPW8peeMbd+Q=; Proxy: null)

I tried creating an AWS account and adding the IDs to ~/.aws but this does not seem to solve the issue (although, as I'm not familiar with AWS, I might have failed to configure everything properly).

When trying to access directly the files linked in test.config, I also get an access denied:
salzman-lab/fastqs-for-testing/MACA_24m_M_BLADDER_58_S5_L001_R1_001.subsample-10-percent.fastq.gz

<Error>
  <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
    <RequestId>VV31AWYW5PX90QGX</RequestId>
    <HostId>
      0S+VOJqxAISBQR5QYYIA/jBy7Yna7SBfaFmArUg5NcNnsi+j8GbWTsdDq7sJVMPnZdeMtb5/iFM=
    </HostId>
</Error>

I suspect this can be solved by setting permissions on the S3 side? Alternatively, it should be possible to download the files without using S3 and change test.config to use local files? I could not find the files in here nor that repo, which mentions them but only provides reprocessed files.

Steps to reproduce

nextflow clone salzmanlab/nf-sicilian nf-sicilian
cd nf-sicilian
sed -i.bak -e '52,57d' conf/test.config
nextflow run main.nf -profile test

Note: similar result using nextflow run main.nf -profile test,singularity

Log files

executor >  slurm (3)
[c7/4fd446] process > PREPARE_GENOME:GUNZIP_GTF (GCF_000001635.26_GRCm38.p6_genomic.chr19.gtf.gz)            [100%] 1 of 1 ✔
[65/7fc943] process > PREPARE_GENOME:STAR_GENOMEGENERATE (mm10_chrom19.fa)                                   [  0%] 0 of 1
[23/377588] process > PREPARE_GENOME:SICILIAN_CREATEANNOTATOR (GCF_000001635.26_GRCm38.p6_genomic.chr19.gtf) [  0%] 0 of 1
executor >  slurm (3)
[c7/4fd446] process > PREPARE_GENOME:GUNZIP_GTF (GCF_000001635.26_GRCm38.p6_genomic.chr19.gtf.gz)            [100%] 1 of 1 ✔
[65/7fc943] process > PREPARE_GENOME:STAR_GENOMEGENERATE (mm10_chrom19.fa)                                   [  0%] 0 of 1
[23/377588] process > PREPARE_GENOME:SICILIAN_CREATEANNOTATOR (GCF_000001635.26_GRCm38.p6_genomic.chr19.gtf) [  0%] 0 of 1
[-        ] process > UMITOOLS_WHITELIST                                                                     -
[-        ] process > UMITOOLS_EXTRACT                                                                       -
[-        ] process > STAR_ALIGN                                                                             -
[-        ] process > SICILIAN:SICILIAN_CLASSINPUT                                                           -
[-        ] process > SICILIAN:SICILIAN_GLM                                                                  -
[-        ] process > SICILIAN:SICILIAN_ANNSPLICES                                                           -
[-        ] process > SICILIAN:SICILIAN_CONSOLIDATE                                                          -
[-        ] process > SICILIAN:SICILIAN_PROCESS_CI_10X                                                       -
[-        ] process > SICILIAN:SICILIAN_POSTPROCESS                                                          -
[-        ] process > GET_SOFTWARE_VERSIONS                                                                  -
[-        ] process > MULTIQC                                                                                -
[]
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 0Z7KE8KCEGH25XBX; S3 Extended Request ID: +UFUBHwrBgm3/RT/1cSHToixuB9Di8G90i4wV+0PYHJDw0SgFaXZ+UkHm2vmcUi6rCESQpahnpiNaUTMtuJ3NQ==; Proxy: null)

 -- Check script './subworkflows/local/input_check.nf' at line: 49 or see '.nextflow.log' file for more details
-[nf-core/sicilian] Pipeline completed with errors-
WARN: Killing running tasks (2)
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.

System

• Hardware: HPC
• Executor: SLURM

Nextflow Installation

• Version: 22.04.5.5708

Container engine

Same error with or without Singularity.

Additional context

I tried creating a file /.aws/credentials with:

AWS_ACCESS_KEY_ID='AAAAAAAAAA'
AWS_SECRET_ACCESS_KEY='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
AWS_DEFAULT_REGION='us-east-1'

but that didn't seem to help.

[fomightez]

To extend the note about, "I could not find the files in here nor that repo, which mentions them but only provides reprocessed files":
It isn't just the data that is/was in the S3 bucket that is a problem due to the czbiohub/test-datasets repo not being accessible. There's several lines getting additional resources from that repo there in aws_maca_chr19.conf