{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","settings":"","results":{"codes":[]},"params":[]},"next":{"description":"","pages":[]},"title":"CWL v1.0 improvements over sbg:draft-2","type":"basic","slug":"cwl-v10-improvements-over-sbgdraft-2","excerpt":"","body":"##Overview\n\nThe following command line tool changes are implemented for CWL v1.0 in comparison to sbg:draft-2.\n\n## No more $job for BiX\n`$job` does not exist in JavaScript expressions. Available contexts are `$inputs` and `$runtime`.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"sbg:draft-2\",\n    \"h-1\": \"CWL v1.0\",\n    \"0-0\": \"`$job.inputs.input_bam.path`\",\n    \"0-1\": \"`$(inputs.input_bam.path)`\",\n    \"1-0\": \"`$job.allocatedResources.cpu`\",\n    \"1-1\": \"`$(runtime.cores)`\"\n  },\n  \"cols\": 2,\n  \"rows\": 2\n}\n[/block]\n##Entering a JS expression\n\nExpressions are denoted by the syntax `$(...)` or `${...}`. A code fragment wrapped in the `$(...)` syntax is used as a one-line expression. A code fragment wrapped in `${...}` behaves like expressions in sbg:draft-2.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"sbg:draft-2\",\n    \"h-1\": \"CWL v1.0\",\n    \"0-0\": \"`$job.inputs.input_bam.path + ‘.vcf’`\",\n    \"0-1\": \"`$(inputs.input_bam.path).vcf`\",\n    \"1-0\": \"`{ return $job.inputs.input_bam.path + ‘.vcf’ }`\",\n    \"1-1\": \"`${ return inputs.input_bam.path + ‘.vcf’}`\"\n  },\n  \"cols\": 2,\n  \"rows\": 2\n}\n[/block]\n##Commands relating to file paths (basename, dirname, nameroot, nameext)\n\nExperience improved commands relating to file paths. For instance, use `.nameroot` to get the input basename. Learn more from CWL's [documentation](http://www.commonwl.org/v1.0/CommandLineTool.html#File).\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Expression\",\n    \"h-1\": \"Return\",\n    \"0-0\": \"`$(inputs.input_bam.path)`\",\n    \"0-1\": \"`/path/to/file.sorted.bam`\",\n    \"1-0\": \"`$(inputs.input_bam.basename)`\",\n    \"1-1\": \"`file.sorted.bam`\",\n    \"2-0\": \"`$(inputs.input_bam.dirname)`\",\n    \"2-1\": \"`/path/to`\",\n    \"3-0\": \"`$(inputs.input_bam.nameroot)`\",\n    \"3-1\": \"`file.sorted`\",\n    \"4-0\": \"`$(inputs.input_bam.nameext)`\",\n    \"4-1\": \"`bam`\"\n  },\n  \"cols\": 2,\n  \"rows\": 5\n}\n[/block]\n##One-line expressions\n\nIf the input is a BAM and the output is a VCF, you can define the output name as `$(inputs.input_bam.nameroot).vcf`.\n\n##No expressions in the base command\n\nAll the expressions must be inserted via arguments.\n\nThis means that if you want to insert pre-commands (such as un-TAR reference files), you need to define the whole command through arguments and leave the base command empty.\n\n##Expressions in secondary files\n\nSecondary files can now be defined with an expression.\n\nFor example, if the input is VCF or VCF.GZ, the secondary file is either .idx or .tbi.: `$(self.nameext == 'gz' ? \"tbi\" : \"idx\")`.\n\n##StepInputExpressionRequirement\n\nWorkflow connection type incompatibility can now be resolved by applying JS expressions on step inputs. This simple example converts an incoming file into a list with one file, because the tool expects a list of files. Any custom value manipulation can be performed, such as converting single-item list to a single element, converting a single element into a list, list flattening, removing nulls from a list and much more. Whatever you need to make the connected tools compatible without changing the tools.\nThe value of `self` is the incoming value. in case of multiple connections to one input port, linkMerge is applied first. In case of scatter, scatter is applied first and `valueFrom` is performed separately for each element in the scattered list. \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"in:\\n- id: array_of_files\\n  source: single_file\\n  valueFrom: \\\"$(self ? [self] : self)\\\"\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n\n##InitialWorkDirRequirement\n\nYou can create a file in `workdir` on runtime or make a file available in the `workdir` on runtime using `InitialWorkDirRequirement`.\n\n###Create literal content file example\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n \\n  - class: InitialWorkDirRequirement\\n    listing:\\n    - entry: $(JSON.stringify(inputs))\\n      entryname: cwl.inputs.json\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n###Create expression content file example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n \\n  - class: InitialWorkDirRequirement\\n    listing:\\n    - entry: |-\\n \\n         Some contents\\n \\n      entryname: dynamic_input.txt\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n###Stage an input example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n \\n - class: InitialWorkDirRequirement\\n   listing:\\n   - \\\"$(inputs.bam)\\\"\\n   - \\\"$(inputs.fastq_list)\\\"\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n###Example with staging inputs and creating files\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n \\n - class: InitialWorkDirRequirement\\n   listing:\\n   - entry: $(JSON.stringify(inputs))\\n     entryname: cwl.inputs.json\\n   - entry: |-\\n     ${\\n        return JSON.stringify(inputs)\\n     }\\n     entryname: '${return \\\"_1_cwl.inputs.json\\\"}'\\n   - \\\"$(inputs.bam)\\\"\\n   - \\\"$(inputs.fastq_list)\\\"\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n##ExpressionLibRequirement\n\nUse `ExpressionLibRequirement` to write a JS function in one place and use it in multiple places.\n\nDefine a function as:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n - class: InlineJavascriptRequirement\\n \\n   expressionLib:\\n     - var ext = function(){\\n       var x = inputs.vcf.nameext == 'gz' ? \\\"tbi\\\" : \\\"idx\\\";\\n       return x\\n     };\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nAnd call it in a different place:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"secondaryFiles:\\n ${\\n \\n    return ext()\\n \\n  }\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n##SchemaDefRequirement\n\nCustom input and output structures can be defined in `SchemaDefRequirement`.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"class: SchemaDefRequirement\\ntypes:\\n- name: FileRecord\\n  type: record\\n  fields:\\n    - name: file\\n      type: File\\n    - name: metadata\\n      type: map   \\n      values: string\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n##shellQuote\n\nKeep this as false.\n\n##EnvVarRequirement\n\nInstead of starting a command with `export MY_CUSTOM_VARIABLE=DzoniJovanovic`, use `EnvVarRequirement`.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n EnvVarRequirement:\\n   envDef:\\n     FILIP: $(inputs.last_name)\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n##ExpressionTool\n\nSimilar to command-line tool except it has no command line and does not start a Docker container. The only purpose of this is to reshape stuff.\n\nExample:\n* https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/decider_bwa_expression.cwl\n* https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/sort_scatter_expression.cwl\n\n##Input/Output Type Directory\n\nThe input can be type directory.\n\n##Input/Output Union Type\n\nAn input or output can be defined to be one or more possible types.\n\nExample: Intervals input string or BED file\n\n##Input/Output Format Ontology\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"inputs:\\n aligned_sequences:\\n   type: File\\n   label: Aligned sequences in BAM format\\n   format: edam:format_2572\\n \\n \\n \\n$namespaces:\\n edam: http://edamontology.org/\\n$schemas:\\n - http://edamontology.org/EDAM_1.18.owl\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\n##SoftwareRequirement\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"hints:\\n SoftwareRequirement:\\n   packages:\\n     interproscan:\\n       specs: [ \\\"https://identifiers.org/rrid/RRID:SCR_005829\\\" ]\\n       version: [ \\\"5.21-60\\\" ]\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]","updates":[],"order":2,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"5eb01045d4a054003bee6eb7","createdAt":"2020-05-04T12:53:25.281Z","user":"5767bc73bb15f40e00a28777","category":{"sync":{"isSync":false,"url":""},"pages":[],"title":"BRING YOUR TOOLS","slug":"sdk-overview","order":7,"from_sync":false,"reference":false,"_id":"5eb00899b36ba5002d35b0c1","createdAt":"2020-05-04T12:20:41.310Z","version":"5773dcfc255e820e00e1cd50","project":"5773dcfc255e820e00e1cd4d","__v":0},"version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["5773dcfc255e820e00e1cd51","5773df36904b0c0e00ef05ff","577baf92451b1e0e006075ac","577bb183b7ee4a0e007c4e8d","577ce77a1cf3cb0e0048e5ea","577d11865fd4de0e00cc3dab","578e62792c3c790e00937597","578f4fd98335ca0e006d5c84","578f5e5c3d04570e00976ebb","57bc35f7531e000e0075d118","57f801b3760f3a1700219ebb","5804d55d1642890f00803623","581c8d55c0dc651900aa9350","589dcf8ba8c63b3b00c3704f","594cebadd8a2f7001b0b53b2","59a562f46a5d8c00238e309a","5a2aa096e25025003c582b58","5a2e79566c771d003ca0acd4","5a3a5166142db90026f24007","5a3a52b5bcc254001c4bf152","5a3a574a2be213002675c6d2","5a3a66bb2be213002675cb73","5a3a6e4854faf60030b63159","5c8a68278e883901341de571","5cb9971e57bf020024523c7b","5cbf1683e2a36d01d5012ecd","5dc15666a4f788004c5fd7d7","5eaff69e844d67003642a020","5eb00899b36ba5002d35b0c1","5eb0172be179b70073dc936e","5eb01b42b36ba5002d35ebba","5eb01f202654a20136813093","5eb918ef149186021c9a76c8","5f0839d3f4b24e005ebbbc29","5f893e508c9862002d0614a9","6024033e2b2f6f004dfe994c"],"_id":"5773dcfc255e820e00e1cd50","__v":36,"createdAt":"2016-06-29T14:36:44.812Z","releaseDate":"2016-06-29T14:36:44.812Z","project":"5773dcfc255e820e00e1cd4d"},"project":"5773dcfc255e820e00e1cd4d","__v":0,"parentDoc":null}

Get started

Cavatica Projects

Find apps

Bring data to Cavatica

Set metadata associated with a file

Collaborate securely

BRING YOUR TOOLS

Docker

Developer tools

EDIT AN APP

Tool wrapping tips and tricks

Describe your tool (Legacy)

Browse public datasets

Run an analysis

Task Execution

Review a task

Kids First studies

Cavatica Datasets

Cavatica API

Connect cloud storage

SBFS (Beta)

BULK API ACTIONS

ASYNC BULK ACTIONS

DRS API

WES API

CAVATICA DATASETS API (Advanced Access)

Data Cruncher

SBFS (Beta)

SBFS Command reference

CWL v1.0 improvements over sbg:draft-2

##Overview The following command line tool changes are implemented for CWL v1.0 in comparison to sbg:draft-2. ## No more $job for BiX `$job` does not exist in JavaScript expressions. Available contexts are `$inputs` and `$runtime`. [block:parameters] { "data": { "h-0": "sbg:draft-2", "h-1": "CWL v1.0", "0-0": "`$job.inputs.input_bam.path`", "0-1": "`$(inputs.input_bam.path)`", "1-0": "`$job.allocatedResources.cpu`", "1-1": "`$(runtime.cores)`" }, "cols": 2, "rows": 2 } [/block] ##Entering a JS expression Expressions are denoted by the syntax `$(...)` or `${...}`. A code fragment wrapped in the `$(...)` syntax is used as a one-line expression. A code fragment wrapped in `${...}` behaves like expressions in sbg:draft-2. [block:parameters] { "data": { "h-0": "sbg:draft-2", "h-1": "CWL v1.0", "0-0": "`$job.inputs.input_bam.path + ‘.vcf’`", "0-1": "`$(inputs.input_bam.path).vcf`", "1-0": "`{ return $job.inputs.input_bam.path + ‘.vcf’ }`", "1-1": "`${ return inputs.input_bam.path + ‘.vcf’}`" }, "cols": 2, "rows": 2 } [/block] ##Commands relating to file paths (basename, dirname, nameroot, nameext) Experience improved commands relating to file paths. For instance, use `.nameroot` to get the input basename. Learn more from CWL's [documentation](http://www.commonwl.org/v1.0/CommandLineTool.html#File). [block:parameters] { "data": { "h-0": "Expression", "h-1": "Return", "0-0": "`$(inputs.input_bam.path)`", "0-1": "`/path/to/file.sorted.bam`", "1-0": "`$(inputs.input_bam.basename)`", "1-1": "`file.sorted.bam`", "2-0": "`$(inputs.input_bam.dirname)`", "2-1": "`/path/to`", "3-0": "`$(inputs.input_bam.nameroot)`", "3-1": "`file.sorted`", "4-0": "`$(inputs.input_bam.nameext)`", "4-1": "`bam`" }, "cols": 2, "rows": 5 } [/block] ##One-line expressions If the input is a BAM and the output is a VCF, you can define the output name as `$(inputs.input_bam.nameroot).vcf`. ##No expressions in the base command All the expressions must be inserted via arguments. This means that if you want to insert pre-commands (such as un-TAR reference files), you need to define the whole command through arguments and leave the base command empty. ##Expressions in secondary files Secondary files can now be defined with an expression. For example, if the input is VCF or VCF.GZ, the secondary file is either .idx or .tbi.: `$(self.nameext == 'gz' ? "tbi" : "idx")`. ##StepInputExpressionRequirement Workflow connection type incompatibility can now be resolved by applying JS expressions on step inputs. This simple example converts an incoming file into a list with one file, because the tool expects a list of files. Any custom value manipulation can be performed, such as converting single-item list to a single element, converting a single element into a list, list flattening, removing nulls from a list and much more. Whatever you need to make the connected tools compatible without changing the tools. The value of `self` is the incoming value. in case of multiple connections to one input port, linkMerge is applied first. In case of scatter, scatter is applied first and `valueFrom` is performed separately for each element in the scattered list. [block:code] { "codes": [ { "code": "in:\n- id: array_of_files\n source: single_file\n valueFrom: \"$(self ? [self] : self)\"", "language": "yaml" } ] } [/block] ##InitialWorkDirRequirement You can create a file in `workdir` on runtime or make a file available in the `workdir` on runtime using `InitialWorkDirRequirement`. ###Create literal content file example [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - entry: $(JSON.stringify(inputs))\n entryname: cwl.inputs.json", "language": "yaml" } ] } [/block] ###Create expression content file example: [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - entry: |-\n \n Some contents\n \n entryname: dynamic_input.txt", "language": "yaml" } ] } [/block] ###Stage an input example: [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - \"$(inputs.bam)\"\n - \"$(inputs.fastq_list)\"", "language": "yaml" } ] } [/block] ###Example with staging inputs and creating files [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - entry: $(JSON.stringify(inputs))\n entryname: cwl.inputs.json\n - entry: |-\n ${\n return JSON.stringify(inputs)\n }\n entryname: '${return \"_1_cwl.inputs.json\"}'\n - \"$(inputs.bam)\"\n - \"$(inputs.fastq_list)\"", "language": "yaml" } ] } [/block] ##ExpressionLibRequirement Use `ExpressionLibRequirement` to write a JS function in one place and use it in multiple places. Define a function as: [block:code] { "codes": [ { "code": "requirements:\n - class: InlineJavascriptRequirement\n \n expressionLib:\n - var ext = function(){\n var x = inputs.vcf.nameext == 'gz' ? \"tbi\" : \"idx\";\n return x\n };", "language": "yaml" } ] } [/block] And call it in a different place: [block:code] { "codes": [ { "code": "secondaryFiles:\n ${\n \n return ext()\n \n }", "language": "yaml" } ] } [/block] ##SchemaDefRequirement Custom input and output structures can be defined in `SchemaDefRequirement`. [block:code] { "codes": [ { "code": "class: SchemaDefRequirement\ntypes:\n- name: FileRecord\n type: record\n fields:\n - name: file\n type: File\n - name: metadata\n type: map \n values: string", "language": "yaml" } ] } [/block] ##shellQuote Keep this as false. ##EnvVarRequirement Instead of starting a command with `export MY_CUSTOM_VARIABLE=DzoniJovanovic`, use `EnvVarRequirement`. [block:code] { "codes": [ { "code": "requirements:\n EnvVarRequirement:\n envDef:\n FILIP: $(inputs.last_name)", "language": "yaml" } ] } [/block] ##ExpressionTool Similar to command-line tool except it has no command line and does not start a Docker container. The only purpose of this is to reshape stuff. Example: * https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/decider_bwa_expression.cwl * https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/sort_scatter_expression.cwl ##Input/Output Type Directory The input can be type directory. ##Input/Output Union Type An input or output can be defined to be one or more possible types. Example: Intervals input string or BED file ##Input/Output Format Ontology [block:code] { "codes": [ { "code": "inputs:\n aligned_sequences:\n type: File\n label: Aligned sequences in BAM format\n format: edam:format_2572\n \n \n \n$namespaces:\n edam: http://edamontology.org/\n$schemas:\n - http://edamontology.org/EDAM_1.18.owl", "language": "yaml" } ] } [/block] ##SoftwareRequirement [block:code] { "codes": [ { "code": "hints:\n SoftwareRequirement:\n packages:\n interproscan:\n specs: [ \"https://identifiers.org/rrid/RRID:SCR_005829\" ]\n version: [ \"5.21-60\" ]", "language": "yaml" } ] } [/block]