CWL v1.0 improvements over sbg:draft-2

Overview

The following command line tool changes are implemented for CWL v1.0 in comparison to sbg:draft-2.

No more $job for BiX

$job does not exist in JavaScript expressions. Available contexts are $inputs and $runtime.

Entering a JS expression

Expressions are denoted by the syntax $(...) or ${...}. A code fragment wrapped in the $(...) syntax is used as a one-line expression. A code fragment wrapped in ${...} behaves like expressions in sbg:draft-2.

Commands relating to file paths (basename, dirname, nameroot, nameext)

Experience improved commands relating to file paths. For instance, use .nameroot to get the input basename. Learn more from CWL's documentation.

One-line expressions

If the input is a BAM and the output is a VCF, you can define the output name as $(inputs.input_bam.nameroot).vcf.

No expressions in the base command

All the expressions must be inserted via arguments.

This means that if you want to insert pre-commands (such as un-TAR reference files), you need to define the whole command through arguments and leave the base command empty.

Expressions in secondary files

Secondary files can now be defined with an expression.

For example, if the input is VCF or VCF.GZ, the secondary file is either .idx or .tbi.: $(self.nameext == 'gz' ? "tbi" : "idx").

StepInputExpressionRequirement

Workflow connection type incompatibility can now be resolved by applying JS expressions on step inputs. This simple example converts an incoming file into a list with one file, because the tool expects a list of files. Any custom value manipulation can be performed, such as converting single-item list to a single element, converting a single element into a list, list flattening, removing nulls from a list and much more. Whatever you need to make the connected tools compatible without changing the tools.
The value of self is the incoming value. in case of multiple connections to one input port, linkMerge is applied first. In case of scatter, scatter is applied first and valueFrom is performed separately for each element in the scattered list.

in:
- id: array_of_files
  source: single_file
  valueFrom: "$(self ? [self] : self)"

InitialWorkDirRequirement

You can create a file in workdir on runtime or make a file available in the workdir on runtime using InitialWorkDirRequirement.

Create literal content file example

requirements:
 
  - class: InitialWorkDirRequirement
    listing:
    - entry: $(JSON.stringify(inputs))
      entryname: cwl.inputs.json

Create expression content file example:

requirements:
 
  - class: InitialWorkDirRequirement
    listing:
    - entry: |-
 
         Some contents
 
      entryname: dynamic_input.txt

Stage an input example:

requirements:
 
 - class: InitialWorkDirRequirement
   listing:
   - "$(inputs.bam)"
   - "$(inputs.fastq_list)"

Example with staging inputs and creating files

requirements:
 
 - class: InitialWorkDirRequirement
   listing:
   - entry: $(JSON.stringify(inputs))
     entryname: cwl.inputs.json
   - entry: |-
     ${
        return JSON.stringify(inputs)
     }
     entryname: '${return "_1_cwl.inputs.json"}'
   - "$(inputs.bam)"
   - "$(inputs.fastq_list)"

ExpressionLibRequirement

Use ExpressionLibRequirement to write a JS function in one place and use it in multiple places.

Define a function as:

requirements:
 - class: InlineJavascriptRequirement
 
   expressionLib:
     - var ext = function(){
       var x = inputs.vcf.nameext == 'gz' ? "tbi" : "idx";
       return x
     };

And call it in a different place:

secondaryFiles:
 ${
 
    return ext()
 
  }

SchemaDefRequirement

Custom input and output structures can be defined in SchemaDefRequirement.

class: SchemaDefRequirement
types:
- name: FileRecord
  type: record
  fields:
    - name: file
      type: File
    - name: metadata
      type: map   
      values: string

shellQuote

Keep this as false.

EnvVarRequirement

Instead of starting a command with export MY_CUSTOM_VARIABLE=DzoniJovanovic, use EnvVarRequirement.

requirements:
 EnvVarRequirement:
   envDef:
     FILIP: $(inputs.last_name)

ExpressionTool

Similar to command-line tool except it has no command line and does not start a Docker container. The only purpose of this is to reshape stuff.

Example:

• https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/decider_bwa_expression.cwl
• https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/sort_scatter_expression.cwl

Input/Output Type Directory

The input can be type directory.

Input/Output Union Type

An input or output can be defined to be one or more possible types.

Example: Intervals input string or BED file

Input/Output Format Ontology

inputs:
 aligned_sequences:
   type: File
   label: Aligned sequences in BAM format
   format: edam:format_2572
 
 
 
$namespaces:
 edam: http://edamontology.org/
$schemas:
 - http://edamontology.org/EDAM_1.18.owl

SoftwareRequirement

hints:
 SoftwareRequirement:
   packages:
     interproscan:
       specs: [ "https://identifiers.org/rrid/RRID:SCR_005829" ]
       version: [ "5.21-60" ]