Bug: Archive in InitialWorkDirRequirement error with cwltool

[aleidel]

There is a bug in for fixed inputs when dealing with archived data (e.g. RData files). Currently s4n tool create -i preprocessed.RData -i metadata.RData -o reshaped.RData Rscript workflows/reshape_data/reshape_data.R would create the following cwl file:

#!/usr/bin/env cwl-runner

cwlVersion: v1.2
class: CommandLineTool

requirements:
- class: InitialWorkDirRequirement
  listing:
  - entryname: preprocessed.RData
    entry:
      $include: ../../preprocessed.RData
  - entryname: metadata.RData
    entry:
      $include: ../../metadata.RData
  - entryname: reshape_data.R
    entry:
      $include: reshape_data.R

inputs: []
outputs:
- id: reshaped
  type: File
  outputBinding:
    glob: reshaped.RData

baseCommand:
- Rscript
- reshape_data.R

s4n execution is successful but cwltool execution is not working
cwltool workflows/reshape_data/reshape_data.cwl due to RData file.
error:

ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
The error was: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte`

s4n execute local workflows/reshape_data/reshape_data.cwl does work.

It would be possible to have a cwl file and a yaml file (or call with arguments) that looks like this:

#!/usr/bin/env cwl-runner

cwlVersion: v1.2
class: CommandLineTool

requirements:
- class: InitialWorkDirRequirement
  listing:
  - entryname: preprocessed.RData
    entry: $(inputs.preprocessed_RData)
  - entryname: metadata.RData
    entry: $(inputs.metadata_RData)
  - entryname: reshape_data.R
    entry:
      $include: reshape_data.R

inputs:
- id: preprocessed_RData
  type: File
- id: metadata_RData
  type: File

outputs:
- id: reshaped
  type: File
  outputBinding:
    glob: reshaped.RData

baseCommand:
- Rscript
- reshape_data.R

This would allow execution with cwltool or with s4n execute local --runner cwltool workflows/reshape_data.cwl --preprocessed_RData preprocessed.RData --metadata_RData metadata.RData or s4n execute local --runner cwltool workflows/reshape_data/reshape_data.cwl workflows/reshape_data/reshape_data_inputs.yml but not without --runner cwltool (or maybe I did something wrong)
s4n execute local workflows/reshape_data/reshape_data.cwl --preprocessed_RData preprocessed.RData --metadata_RData metadata.RData

📁 Created staging directory: "/tmp/.tmpXz3xVr"
thread 'main' panicked at src/io.rs:162:62:
Could not read file ./preprocessed.RData
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[aleidel]

Could change cwl file creation to 2. version (above) if one of the inputs is an archive and create a yml file with the parameters. Then user would need to have cwltool installed to run it???

[JensKrumsieck]

$include appends the text to the cwl file and does not work with binaries. So V2 would be more correct.
s4n execute local is something we have control of, so it could be fixed at a later point

[aleidel]

Okay, I will change it

[JensKrumsieck]

What is the reason for the execute_cwl_local-fn? And why does it say CWLTool is needed? We can not use cwltool in s4n tool create!! Furthermore everythin in tool create happens in the users local folder.

m4.4_sciwin_client/src/commands/tool.rs

Lines 88 to 130 in c94c8f5

	fn execute_cwl_local(
	mut cwl: CommandLineTool,
	args: &CreateToolArgs,
	repo: &Repository,
	inputs: &[String],
	) -> Result<CommandLineTool, Box<dyn std::error::Error>> {
	if args.is_raw {
	warn!("Temporary files are created for local execution");
	}
	let path = get_qualified_filename(&cwl.base_command, args.name.clone());
	let mut yaml_cwl = cwl.save(&path);
	yaml_cwl = format_cwl(&yaml_cwl)?;
	std::fs::write(&path, yaml_cwl.as_bytes())?;
	warn!("Execution requires CWLTool");
	let input_refs: Vec<&str> = inputs.iter().map(|s| s.as_str()).collect();
	let yaml = create_input_yml(&input_refs, &path)?;
	let yaml_str = path.replace(".cwl", "_inputs.yml");
	if !args.no_commit && !args.is_raw {
	stage_file(repo, path.as_str())?;
	stage_file(repo, &yaml_str)?;
	commit(repo, &format!("Execution of `{}`", args.command.join(" ")))?;
	}
	// Execute locally
	let args_exec = LocalExecuteArgs {
	runner: Runner::CWLTool,
	out_dir: None,
	is_quiet: false,
	file: PathBuf::from(&path),
	args: vec![yaml.clone()],
	};
	execute_local(&args_exec)?;
	if args.is_raw {
	fs::remove_file(yaml)?;
	fs::remove_file(&path)?;
	}
	Ok(cwl)
	}