CWL v1.0 improvements over sbg:draft-2

{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","settings":"","results":{"codes":[]},"params":[]},"next":{"description":"","pages":[]},"title":"CWL v1.0 improvements over sbg:draft-2","type":"basic","slug":"cwl-v10-improvements-over-sbgdraft-2","excerpt":"","body":"##Overview\n\nThe following command line tool changes are implemented for CWL v1.0 in comparison to sbg:draft-2.\n\n## No more $job for BiX8\n`$job` does not exist in JavaScript expressions. Available contexts are `$inputs` and `$runtime`.\n[block:parameters]\n{\n \"data\": {\n \"h-0\": \"sbg:draft-2\",\n \"h-1\": \"CWL v1.0\",\n \"0-0\": \"`$job.inputs.input_bam.path`\",\n \"0-1\": \"`$(inputs.input_bam.path)`\",\n \"1-0\": \"`$job.allocatedResources.cpu`\",\n \"1-1\": \"`$(runtime.cores)`\"\n },\n \"cols\": 2,\n \"rows\": 2\n}\n[/block]\n##Entering a JS expression\n\nExpressions are denoted by the syntax `$(...)` or `${...}`. A code fragment wrapped in the `$(...)` syntax is used as a one-line expression. A code fragment wrapped in `${...}` behaves like expressions in sbg:draft-2.\n[block:parameters]\n{\n \"data\": {\n \"h-0\": \"sbg:draft-2\",\n \"h-1\": \"CWL v1.0\",\n \"0-0\": \"`$job.inputs.input_bam.path + ‘.vcf’`\",\n \"0-1\": \"`$(inputs.input_bam.path).vcf`\",\n \"1-0\": \"`{ return $job.inputs.input_bam.path + ‘.vcf’ }`\",\n \"1-1\": \"`${ return inputs.input_bam.path + ‘.vcf’}`\"\n },\n \"cols\": 2,\n \"rows\": 2\n}\n[/block]\n##Commands relating to file paths (basename, dirname, nameroot, nameext)\n\nExperience improved commands relating to file paths. For instance, use `.nameroot` to get the input basename. Learn more from CWL's [documentation](http://www.commonwl.org/v1.0/CommandLineTool.html#File).\n[block:parameters]\n{\n \"data\": {\n \"h-0\": \"Expression\",\n \"h-1\": \"Return\",\n \"0-0\": \"`$(inputs.input_bam.path)`\",\n \"0-1\": \"`/path/to/file.sorted.bam`\",\n \"1-0\": \"`$(inputs.input_bam.basename)`\",\n \"1-1\": \"`file.sorted.bam`\",\n \"2-0\": \"`$(inputs.input_bam.dirname)`\",\n \"2-1\": \"`/path/to`\",\n \"3-0\": \"`$(inputs.input_bam.nameroot)`\",\n \"3-1\": \"`file.sorted`\",\n \"4-0\": \"`$(inputs.input_bam.nameext)`\",\n \"4-1\": \"`bam`\"\n },\n \"cols\": 2,\n \"rows\": 5\n}\n[/block]\n##One-line expressions\n\nIf the input is a BAM and the output is a VCF, you can define the output name as `$(inputs.input_bam.nameroot).vcf`.\n\n##No expressions in the base command\n\nAll the expressions must be inserted via arguments.\n\nThis means that if you want to insert pre-commands (such as un-TAR reference files), you need to define the whole command through arguments and leave the base command empty.\n\n##Expressions in secondary files\n\nSecondary files can now be defined with an expression.\n\nFor example, if the input is VCF or VCF.GZ, the secondary file is either .idx or .tbi.: `$(self.nameext == 'gz' ? \"tbi\" : \"idx\")`.\n\n##InitialWorkDirRequirement\n\nYou can create a file in `workdir` on runtime or make a file available in the `workdir` on runtime using `InitialWorkDirRequirement`.\n\n###Create literal content file example\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"requirements:\\n \\n - class: InitialWorkDirRequirement\\n listing:\\n - entry: $(JSON.stringify(inputs))\\n entryname: cwl.inputs.json\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n###Create expression content file example:\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"requirements:\\n \\n - class: InitialWorkDirRequirement\\n listing:\\n - entry: |-\\n \\n Some contents\\n \\n entryname: dynamic_input.txt\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n###Stage an input example:\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"requirements:\\n \\n - class: InitialWorkDirRequirement\\n listing:\\n - \\\"$(inputs.bam)\\\"\\n - \\\"$(inputs.fastq_list)\\\"\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n###Example with staging inputs and creating files\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"requirements:\\n \\n - class: InitialWorkDirRequirement\\n listing:\\n - entry: $(JSON.stringify(inputs))\\n entryname: cwl.inputs.json\\n - entry: |-\\n ${\\n return JSON.stringify(inputs)\\n }\\n entryname: '${return \\\"_1_cwl.inputs.json\\\"}'\\n - \\\"$(inputs.bam)\\\"\\n - \\\"$(inputs.fastq_list)\\\"\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n##ExpressionLibRequirement\n\nUse `ExpressionLibRequirement` to write a JS function in one place and use it in multiple places.\n\nDefine a function as:\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"requirements:\\n - class: InlineJavascriptRequirement\\n \\n expressionLib:\\n - var ext = function(){\\n var x = inputs.vcf.nameext == 'gz' ? \\\"tbi\\\" : \\\"idx\\\";\\n return x\\n };\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\nAnd call it in a different place:\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"secondaryFiles:\\n ${\\n \\n return ext()\\n \\n }\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n##SchemaDefRequirement\n\nCustom input and output structures can be defined in `SchemaDefRequirement`.\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"class: SchemaDefRequirement\\ntypes:\\n- name: FileRecord\\n type: record\\n fields:\\n - name: file\\n type: File\\n - name: metadata\\n type: map \\n values: string\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n##shellQuote\n\nKeep this as false.\n\n##EnvVarRequirement\n\nInstead of starting a command with `export MY_CUSTOM_VARIABLE=DzoniJovanovic`, use `EnvVarRequirement`.\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"requirements:\\n EnvVarRequirement:\\n envDef:\\n FILIP: $(inputs.last_name)\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n##ExpressionTool\n\nSimilar to command-line tool except it has no command line and does not start a Docker container. The only purpose of this is to reshape stuff.\n\nExample:\n* https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/decider_bwa_expression.cwl\n* https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/sort_scatter_expression.cwl\n\n##Input/Output Type Directory\n\nThe input can be type directory.\n\n##Input/Output Union Type\n\nAn input or output can be defined to be one or more possible types.\n\nExample: Intervals input string or BED file\n\n##Input/Output Format Ontology\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"inputs:\\n aligned_sequences:\\n type: File\\n label: Aligned sequences in BAM format\\n format: edam:format_2572\\n \\n \\n \\n$namespaces:\\n edam: http://edamontology.org/\\n$schemas:\\n - http://edamontology.org/EDAM_1.18.owl\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]\n##SoftwareRequirement\n[block:code]\n{\n \"codes\": [\n {\n \"code\": \"hints:\\n SoftwareRequirement:\\n packages:\\n interproscan:\\n specs: [ \\\"https://identifiers.org/rrid/RRID:SCR_005829\\\" ]\\n version: [ \\\"5.21-60\\\" ]\",\n \"language\": \"yaml\"\n }\n ]\n}\n[/block]","updates":[],"order":2,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"5eb01045d4a054003bee6eb7","createdAt":"2020-05-04T12:53:25.281Z","user":"5767bc73bb15f40e00a28777","category":{"sync":{"isSync":false,"url":""},"pages":[],"title":"BRING YOUR TOOLS","slug":"sdk-overview","order":6,"from_sync":false,"reference":false,"_id":"5eb00899b36ba5002d35b0c1","createdAt":"2020-05-04T12:20:41.310Z","version":"5773dcfc255e820e00e1cd50","project":"5773dcfc255e820e00e1cd4d","__v":0},"version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["5773dcfc255e820e00e1cd51","5773df36904b0c0e00ef05ff","577baf92451b1e0e006075ac","577bb183b7ee4a0e007c4e8d","577ce77a1cf3cb0e0048e5ea","577d11865fd4de0e00cc3dab","578e62792c3c790e00937597","578f4fd98335ca0e006d5c84","578f5e5c3d04570e00976ebb","57bc35f7531e000e0075d118","57f801b3760f3a1700219ebb","5804d55d1642890f00803623","581c8d55c0dc651900aa9350","589dcf8ba8c63b3b00c3704f","594cebadd8a2f7001b0b53b2","59a562f46a5d8c00238e309a","5a2aa096e25025003c582b58","5a2e79566c771d003ca0acd4","5a3a5166142db90026f24007","5a3a52b5bcc254001c4bf152","5a3a574a2be213002675c6d2","5a3a66bb2be213002675cb73","5a3a6e4854faf60030b63159","5c8a68278e883901341de571","5cb9971e57bf020024523c7b","5cbf1683e2a36d01d5012ecd","5dc15666a4f788004c5fd7d7","5eaff69e844d67003642a020","5eb00899b36ba5002d35b0c1","5eb0172be179b70073dc936e","5eb01b42b36ba5002d35ebba","5eb01f202654a20136813093","5eb918ef149186021c9a76c8","5f0839d3f4b24e005ebbbc29","5f893e508c9862002d0614a9"],"_id":"5773dcfc255e820e00e1cd50","__v":35,"createdAt":"2016-06-29T14:36:44.812Z","releaseDate":"2016-06-29T14:36:44.812Z","project":"5773dcfc255e820e00e1cd4d"},"project":"5773dcfc255e820e00e1cd4d","__v":0,"parentDoc":null}

##Overview The following command line tool changes are implemented for CWL v1.0 in comparison to sbg:draft-2. ## No more $job for BiX8 `$job` does not exist in JavaScript expressions. Available contexts are `$inputs` and `$runtime`. [block:parameters] { "data": { "h-0": "sbg:draft-2", "h-1": "CWL v1.0", "0-0": "`$job.inputs.input_bam.path`", "0-1": "`$(inputs.input_bam.path)`", "1-0": "`$job.allocatedResources.cpu`", "1-1": "`$(runtime.cores)`" }, "cols": 2, "rows": 2 } [/block] ##Entering a JS expression Expressions are denoted by the syntax `$(...)` or `${...}`. A code fragment wrapped in the `$(...)` syntax is used as a one-line expression. A code fragment wrapped in `${...}` behaves like expressions in sbg:draft-2. [block:parameters] { "data": { "h-0": "sbg:draft-2", "h-1": "CWL v1.0", "0-0": "`$job.inputs.input_bam.path + ‘.vcf’`", "0-1": "`$(inputs.input_bam.path).vcf`", "1-0": "`{ return $job.inputs.input_bam.path + ‘.vcf’ }`", "1-1": "`${ return inputs.input_bam.path + ‘.vcf’}`" }, "cols": 2, "rows": 2 } [/block] ##Commands relating to file paths (basename, dirname, nameroot, nameext) Experience improved commands relating to file paths. For instance, use `.nameroot` to get the input basename. Learn more from CWL's [documentation](http://www.commonwl.org/v1.0/CommandLineTool.html#File). [block:parameters] { "data": { "h-0": "Expression", "h-1": "Return", "0-0": "`$(inputs.input_bam.path)`", "0-1": "`/path/to/file.sorted.bam`", "1-0": "`$(inputs.input_bam.basename)`", "1-1": "`file.sorted.bam`", "2-0": "`$(inputs.input_bam.dirname)`", "2-1": "`/path/to`", "3-0": "`$(inputs.input_bam.nameroot)`", "3-1": "`file.sorted`", "4-0": "`$(inputs.input_bam.nameext)`", "4-1": "`bam`" }, "cols": 2, "rows": 5 } [/block] ##One-line expressions If the input is a BAM and the output is a VCF, you can define the output name as `$(inputs.input_bam.nameroot).vcf`. ##No expressions in the base command All the expressions must be inserted via arguments. This means that if you want to insert pre-commands (such as un-TAR reference files), you need to define the whole command through arguments and leave the base command empty. ##Expressions in secondary files Secondary files can now be defined with an expression. For example, if the input is VCF or VCF.GZ, the secondary file is either .idx or .tbi.: `$(self.nameext == 'gz' ? "tbi" : "idx")`. ##InitialWorkDirRequirement You can create a file in `workdir` on runtime or make a file available in the `workdir` on runtime using `InitialWorkDirRequirement`. ###Create literal content file example [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - entry: $(JSON.stringify(inputs))\n entryname: cwl.inputs.json", "language": "yaml" } ] } [/block] ###Create expression content file example: [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - entry: |-\n \n Some contents\n \n entryname: dynamic_input.txt", "language": "yaml" } ] } [/block] ###Stage an input example: [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - \"$(inputs.bam)\"\n - \"$(inputs.fastq_list)\"", "language": "yaml" } ] } [/block] ###Example with staging inputs and creating files [block:code] { "codes": [ { "code": "requirements:\n \n - class: InitialWorkDirRequirement\n listing:\n - entry: $(JSON.stringify(inputs))\n entryname: cwl.inputs.json\n - entry: |-\n ${\n return JSON.stringify(inputs)\n }\n entryname: '${return \"_1_cwl.inputs.json\"}'\n - \"$(inputs.bam)\"\n - \"$(inputs.fastq_list)\"", "language": "yaml" } ] } [/block] ##ExpressionLibRequirement Use `ExpressionLibRequirement` to write a JS function in one place and use it in multiple places. Define a function as: [block:code] { "codes": [ { "code": "requirements:\n - class: InlineJavascriptRequirement\n \n expressionLib:\n - var ext = function(){\n var x = inputs.vcf.nameext == 'gz' ? \"tbi\" : \"idx\";\n return x\n };", "language": "yaml" } ] } [/block] And call it in a different place: [block:code] { "codes": [ { "code": "secondaryFiles:\n ${\n \n return ext()\n \n }", "language": "yaml" } ] } [/block] ##SchemaDefRequirement Custom input and output structures can be defined in `SchemaDefRequirement`. [block:code] { "codes": [ { "code": "class: SchemaDefRequirement\ntypes:\n- name: FileRecord\n type: record\n fields:\n - name: file\n type: File\n - name: metadata\n type: map \n values: string", "language": "yaml" } ] } [/block] ##shellQuote Keep this as false. ##EnvVarRequirement Instead of starting a command with `export MY_CUSTOM_VARIABLE=DzoniJovanovic`, use `EnvVarRequirement`. [block:code] { "codes": [ { "code": "requirements:\n EnvVarRequirement:\n envDef:\n FILIP: $(inputs.last_name)", "language": "yaml" } ] } [/block] ##ExpressionTool Similar to command-line tool except it has no command line and does not start a Docker container. The only purpose of this is to reshape stuff. Example: * https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/decider_bwa_expression.cwl * https://github.com/NCI-GDC/gdc-dnaseq-cwl/blob/master/tools/sort_scatter_expression.cwl ##Input/Output Type Directory The input can be type directory. ##Input/Output Union Type An input or output can be defined to be one or more possible types. Example: Intervals input string or BED file ##Input/Output Format Ontology [block:code] { "codes": [ { "code": "inputs:\n aligned_sequences:\n type: File\n label: Aligned sequences in BAM format\n format: edam:format_2572\n \n \n \n$namespaces:\n edam: http://edamontology.org/\n$schemas:\n - http://edamontology.org/EDAM_1.18.owl", "language": "yaml" } ] } [/block] ##SoftwareRequirement [block:code] { "codes": [ { "code": "hints:\n SoftwareRequirement:\n packages:\n interproscan:\n specs: [ \"https://identifiers.org/rrid/RRID:SCR_005829\" ]\n version: [ \"5.21-60\" ]", "language": "yaml" } ] } [/block]

Cavatica

Get started

Cavatica Projects

Find apps

Bring your private data

Set metadata associated with a file

Collaborate securely

BRING YOUR TOOLS

Docker

Developer tools

EDIT AN APP

Tool wrapping tips and tricks

Describe your tool (Legacy)

Browse public datasets

Run an analysis

Task Execution

Review a task

Kids First studies

Cavatica Datasets

Cavatica API

Connect cloud storage

SBFS (Beta)

BULK API ACTIONS

ASYNC BULK ACTIONS

DRS API

WES API

CAVATICA DATASETS API (Advanced Access)

Data Cruncher

SBFS (Beta)

SBFS Command reference

CWL v1.0 improvements over sbg:draft-2