{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","settings":"","results":{"codes":[]},"params":[]},"next":{"description":"","pages":[]},"title":"Maintaining and versioning CWL on external tool repositories","type":"basic","slug":"maintaining-and-versioning-cwl-on-external-tool-repositories","excerpt":"","body":"##Summary\n\nCAVATICA enables users to develop their own Common Workflow Language (CWL) tools and workflows directly on the Platform using the Seven Bridges Software Development Kit (SDK). The SDK consists of the Tool Editor, Workflow Editor, and other helpful utilities. Both the Tool Editor and Workflow Editor have a visual editor and code editor component.\n\nThis tutorial presents best practices for writing and maintaining CWL tools/workflows in an external tool repository, such as github, so that users can better manage versions of their tools. Users should follow these best practices if they would like to publish and share their CWL tools and workflows in the [Dockstore repository](https://dockstore.org/) since Dockstore has the ability to automatically pull changes from github. \n\nThese best practices will ensure that the CWL is fully portable and can run successfully not only on Seven Bridges Platforms, but also on other CWL executors such as cwltool and Toil.\n\nThis tutorial will guide you through using two open source tools for working with CWL, and you will need to be comfortable with or willing to learn how to use command line tools.\n\nThe tutorial provides guidance both for developers and researchers writing CWL in a local Integrated Development Environment (IDE) or code editor, but also for developers using the graphical “low code” Seven Bridges Tool Editor and Workflow Editor.\n\n###Overview of flow\n\nThe overall flow can be summarized in the diagram below. Developing a CWL workflow that runs portably follows the same general practices as any other good software development though there are some CWL-specific aspects:\n\n1. Use small tests to check each part of the workflow\n2. Test each tool separately before testing the parent workflow\n3. Test and verify execution on more than one environment to ensure there are no environment-specific bugs\n4. Keep the amount of Javascript to a minimum and ensure it is strict mode ECMAscript 5\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/5d3c2ca-BDC_Dockstore_CWL_tutorial.png\",\n        \"BDC_Dockstore CWL tutorial.png\",\n        800,\n        600,\n        \"#c4c8cb\"\n      ]\n    }\n  ]\n}\n[/block]\nAn example of a set of workflows developed according to these principles are the Seven Bridges CWL wrappers for the association testing tools, GENESIS, available on [github](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl) and [Dockstore](https://dockstore.org/organizations/SevenBridges/collections/genesispipelines).\n\n##Tools used in this guide\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Tool\",\n    \"h-1\": \"Note\",\n    \"h-2\": \"Installation\",\n    \"0-0\": \"[cwltool](https://github.com/common-workflow-language/cwltool)\",\n    \"0-1\": \"CWL reference runner. Will be used for checking CWL for correctness and for running tests.\",\n    \"0-2\": \"`pipx install cwltool`\",\n    \"1-0\": \"[sbpack](https://github.com/rabix/sbpack)\",\n    \"1-1\": \"Tool to upload and download CWL code from Seven Bridges Platforms.\",\n    \"1-2\": \"`pipx install sbpack`\",\n    \"2-0\": \"[git](https://git-scm.com/)\",\n    \"2-1\": \"Popular versioning system we will use, Github is built around git.\",\n    \"2-2\": \"`via website`\",\n    \"3-0\": \"[benten](https://github.com/rabix/benten) (optional)\",\n    \"3-1\": \"CWL language server, offers code intelligence for a variety of code editors.\",\n    \"3-2\": \"`pipx install benten` OR let VS Code extension install it for you.\",\n    \"4-0\": \"[VS Code](https://code.visualstudio.com/) (optional)\",\n    \"4-1\": \"A popular and powerful open source code editor with a CWL editing plugin\",\n    \"4-2\": \"`via website`\"\n  },\n  \"cols\": 3,\n  \"rows\": 5\n}\n[/block]\nYou may choose to install the tools any way you choose. The Seven Bridges team recommends using pipx because it is straightforward; works on Linux, macOS and Windows; and installs each tool in an isolated virtual environment to avoid dependencies.\n\n##Developing locally vs. developing on CAVATICA\n\nYou can follow the practices outlined here by either developing CWL locally using a code editor or by developing CWL on the cloud using the Tool Editor and Workflow Editor feature of CAVATICA. Use the sbpack tool to upload/download the CWL from/to the Platform.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/8572442-develop-locally-vs-on-cavatica.png\",\n        \"develop-locally-vs-on-cavatica.png\",\n        641,\n        397,\n        \"#333\"\n      ]\n    }\n  ]\n}\n[/block]\nIt is currently difficult to use the two development models simultaneously. If you attempt to edit \nand develop CWL locally -AND- edit on CAVATICA at the same time, you will encounter issues with the formatting and layout of the CWL code due to how the Platform packs the code into one file. Therefore, it is best to use one development model.\n\n##Collect small test data sets and test cases\n\nWhen you are ready to wrap a tool or workflow in CWL please collect small data sets that will allow testing of each CWL tool all the way up to the final workflows. This greatly facilitates debugging. \n\n  * **On Seven Bridges with the Tool Editor and Workflow Editor**: Add these test files to your development project and run tasks periodically as you develop the tool/workflow.\n  * **Local development**: For each CWL, create a test job file using the test data as inputs. This is used to periodically run the tool and workflow to ensure the CWL being developed is correct. An example of such a test job file can be found [here](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/master/nullmodel/job-null-model.yml).\n  * **Continuous integration**: Use continuous integration to ensure the tests are being run automatically. github makes this easy using [actions](https://docs.github.com/en/actions/learn-github-actions/introduction-to-github-actions). As an example, look at this [github action](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/master/.github/workflows/nullmodel.yml) that invokes cwltool using a job file and test inputs.\n\n##Develop, test, fix, repeat\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/5504e91-develop-on-cavatica.png\",\n        \"develop-on-cavatica.png\",\n        550,\n        555,\n        \"#333\"\n      ]\n    }\n  ]\n}\n[/block]\nOnce the test sets are up and continuous integration has been put in place, you enter a standard cycle of development, testing, debugging and fixing of the CWL. \n\nDeveloping CWL in this way should ensure that the CWL executes flawlessly on multiple platforms. However, please note that there may be particular aspects of CWL where different executors may interpret the specification differently, have different bugs, or have different affordances.\n\nFor this reason, we recommend using a common software engineering practice and test the CWL on multiple platforms. In general, it is sufficient to test on two different platforms to ensure that we are not touching on aspects of CWL that may be buggy or inconsistent in one or more executors.\n\nIn addition to Seven Bridges, we recommend using the executor “cwltool” which is the community developed CWL reference runner.\n\n###Test tools separately\n\nWe will follow an iterative process while developing the workflow. First, we will wrap the individual tools. Next, we will test them with cwltool. We will then push the CWL to Seven Bridges and test with the Seven Bridges executor.\n\nOnce we are satisfied that a particular iteration of the code works, we will commit that code to our repository. Finally, once we have tested all the individual tools, we will wrap the workflow with the component tools and repeat the process.\n\n##Brief guidelines to improve portability of CWL\n\n###Minimize use of Javascript\n\n####Use bash scripts and parameter references\n\n  * The bash script runs in the container, guaranteeing reproducible and stable execution\n  * Use parameter references or short JS expressions\n  * Use `set -x` at the top of the bash script to see the execution steps in the stdout log. Excellent for debugging\n\nConsider this [embedded script](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/5f17ca875ec5b0e324fa899ed0e3175ef9ddf9d0/single-variant-association/assoc_single_r.cwl#L37):\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"- entryname: script.sh\\n      entry: |\\n        set -x\\n        # This is a bit of cleverness we have to do to extract the chromosome\\n        # number from the segments file and pass it to the R script\\n        CHROM=\\\"$(\\\"$\\\")(awk 'NR==${return parseInt(inputs.segment) + 1} {print $1}' $(inputs.segment_file.path))\\\"\\n        Rscript /usr/local/analysis_pipeline/R/assoc_single.R assoc_single.config --chromosome $CHROM --segment $(inputs.segment)\",\n      \"language\": \"shell\",\n      \"name\": \"embedded script\"\n    }\n  ]\n}\n[/block]\nIn order to make the underlying tool work, some cleverness has to be done: we have to parse an input file and read one of its columns.\n\nThis cleverness is done in bash and ensures that it runs in the container and runs identically everywhere. Note the use of `set -x` which prints all commands and what is actually executed out to stdout. \n\nThis helps with recordkeeping and debugging and is easier to read than complicated, long command lines. Please note the use of the `$(“$”)` trick to make the embedded script work on both Seven Bridges executor and cwltool.\n\n####Prefer parameter references and short expressions\n\nWhen creating files (e.g. config files or manifests) prefer parameter references or short expressions that embed input variables into the raw text. This is easier to debug, to read and results in less problems overall. For example instead of\n\n\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"      - class InitialWorkDirRequirement\\n        listing:\\n        - entryname: null_model.config\\n          writable: false\\n          entry: |-\\n            ${   \\n              \\n                 var arguments = {};\\n                 if(inputs.output_prefix){\\n                     var filename = input.output_prefix + \\\"_null_model\\\";\\n                     arguments.push('out_prefix \\\\\\\"' + filename '\\\\\\\"');\\n                     var phenotype_filename = inputs.output_prefix + \\\"_phenotypes.RData\\\";\\n                     arguments.push('out_phenotype_file \\\\\\\"' + phenotype_filename + '\\\\\\\"\\\\);\\n                     \\n                 }\\n                 else{\\n                     arguments.push{'out_prefix \\\"null_model:\\\"'};\\n                     arguments.piush('out_phenotype_file \\\"phenotypes.RData\\\"');\\n                 }\\n                 arguments.push('outcome ' + inputs.outcome);\\n                 arguments.push('phenotype_file \\\"' + inputs.phenotype_file.path _ '\\\"');\\n                 if(inputs.gds_files){\\n                     arguments.push('gds_file \\\"' + inputs.gds_files[0].path.split('chr')[0] + 'chr .gds\\\"')\\n                 }\\n                 if(inputs.pca_file){\\n                     arguments.push('pca_file \\\"' + inputs.pca_file.path + '\\\"')\\n                 }\\n                 if(inputs.binary){\\n                     arguments.push('binary ' + inputs.binary)\\n                 }\\n                 if(inputs.conditional_variant_file){\\n                     arguments.push('conditional_variant_file \\\"' + inputs.conditional_variant_file.path + '\\\"'\\n                 }\\n                 if(inputs.covars){\\n                     temp = []\\n                     for(var i=0; i<inputs.covars.length; i++){\\n                         temp.push(inputs.covars[i])\\n                     }\\n                     arguments,push('covars \\\"' + temp.join(' ') + '\\\"'\\n                 }\\n                 if(inputs.group_var){\\n                     arguments.push('group_var \\\"' + inputs.group_var _ '\\\"')\\n                 }\\n                 if(inputs,inverse_normal){\\n                     arguments.push('inverse_normal ' + inputs.inverse_normal)\\n                 }\\n                 if(inputs.n_pcs){\\n                     if(inputs.n_pcs > 0)\\n                         arguments.push('n_pcs ' + inputs.n_pcs)\\n                 }\\n                 if(inputs.rescale_variance){\\n                     arguments.push('rescale_variance \\\"' + inputs.rescale_variance + '\\\"')\\n                 }\\n                 if(inputs.resid_covars){\\n                     arguments.push('resid_covars ' + inputs.resid_covars)\\n                 }\\n                 if(inputs.sample_include_file){\\n                     arguments.push('sample_include_file \\\"' + inputs.sample_include_file.path + '\\\"')\\n                 }\\n                 if(inputs.norm_bygroup){\\n                     arguments.push(norm_bygroup ' + inputs.norm_bygroup)\\n                 }\\n                 rturn arguments.join('\\\\n')\\n              }      \\n\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": []\n    }\n  ]\n}\n[/block]\n[Prefer](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/5f17ca875ec5b0e324fa899ed0e3175ef9ddf9d0/nullmodel/null-model.cwl#L71)\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"  baseCommand:  []\\n  arguments:\\n  - prefix: ''\\n    position: 1\\n    valueFrom: |-\\n      ${\\n          if(inputs.null_model_files == null)\\n          {\\n            \\n          }\\n          else\\n          }\\n            command = \\\"cp\\\" + inputs.null_model_file.path + \\\" \\\" + inputs.phenotype_file.path + \\\" . && \\\"\\n            return command + \\\"echo 'Passing null_model.R.'\\\"\\n          }\\n       }\\n    shellQuote: false    \",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nThis makes the generated file more readable and allows individual expressions to be debugged more easily.\n\n####Avoid complicated, dynamic command lines\n\n  * If a complicated command line with multiple commands is needed, use bash scripts instead\n  * Avoid doing conditionals embedded in arguments: these are hard to read. Use bash scripts instead\n  * Avoid complicated dynamic globs\n\nTherefore avoid constructs like this:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"  baseCommand: []\\n  arguments\\n  - prefix: ''\\n    position: 1\\n    valueFrom: |-\\n      ${\\n          if(inputs.null_model_files == null)\\n          {\\n            return Rscript /usr/local/analysis_pipeline/R/null_model.R null_model.config\\\"\\n            \\n          }\\n          else\\n          }\\n          {\\n            command = \\\"cp\\\" + inputs.null_model_file.path + \\\"\\\" + inputs.phenotype_file.path + \\\" . &&\\\"\\n            return command + \\\"echo 'Passing null_model.R.'\\\"\\n          }\\n      }\\n    shellQuote: false \",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nThis is intended as a conditional, but it is hard to read.\n\nNote: there is a bug in here. The input variable name is actually null_model_file . The Seven Bridges executor resolves this to undefined and the test always returns false. This is an uncaught bug. Code review or more test cases might have caught this.\n\nAlso, the complexity here results in us needing more complicated constructs in the output binding.\n\n###Appendix\n\n####Suggested CWL git project structure\n\nBy pushing CWL to github your software project now has a high level of version control and collaboration opportunities. As a community we should work towards a standardization of CWL project structure. Seven Bridges suggests a very simple folder structure.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/dfb2301-folder-structure.png\",\n        \"folder-structure.png\",\n        633,\n        251,\n        \"#f7f8fa\"\n      ]\n    }\n  ]\n}\n[/block]\nResearch projects and software toolkits often contain more than one workflow. In this case, make a directory for each workflow and have a directory for small test data. \n\nThe “.github/workflows” is used for continuous integration tests only. Finally add a `README.md` with notes about how the workflows are intended to be used (see below for test badges). Most of the README notes can be copied directly from your CWL descriptions. \n\nPopulate each workflow directory with the individual tool steps. This allows for better organization and for other researchers/developers to incorporate your steps in their workflows. This folder structure can be created manually if following the local development style or by using the SDK utility sbpack to represent the workflow as one CWL file per step. \n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/7a75b1f-languages.png\",\n        \"languages.png\",\n        442,\n        156,\n        \"#f0eaec\"\n      ]\n    }\n  ]\n}\n[/block]\nGithub can identify Common Workflow Language and will tag your project as containing it. By keeping the project dedicated to CWL this will help other users easily find your workflow and tools wrappers.\n\n####Run CWL validation using a git pre-commit hook\n\nWe can use this feature to run a quick check each time we go to commit code. A git pre-commit hook is a script that is run before code is committed. We can use this feature to run a quick check each time we go to commit code.  We will create a bash script which is simply a set of calls to `cwltool --validate` to check each CWL file that is being committed.\n\nThe only clever thing we will do is use a git command to retrieve the files we are about to commit and only validate those.\n\nWe will create a file:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"for f in `git diff --staged --name-only`\\ndo\\n cwltool --validate \\\"$f\\\"\\ndone\\n\",\n      \"language\": \"text\",\n      \"name\": \"check-changed.sh\"\n    }\n  ]\n}\n[/block]\nWe will then invoke this script from another script called `pre-commit`. The `pre-commit` file will be put under the `.git/hooks` directory and will be automatically invoked during a commit action. The script `.git/hooks/pre-commit` is invoked from the root of the repository:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"check-changed.sh\",\n      \"language\": \"text\",\n      \"name\": \".git/hooks/pre-commit\"\n    }\n  ]\n}\n[/block]\nYou can read up a bit more about git hooks [here](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks).\n\n####Add test badges to your repository\n\nSince you have done all the hard work of setting up tests and continuous integration why not add some test badges to show this off in your repository on github?\n\nThis is how the Readme looks from the [Seven Bridges GENESIS CWL repository](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl) linked at the start.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/a4e6b94-cwl-wappers.png\",\n        \"cwl-wappers.png\",\n        634,\n        241,\n        \"#f2f4f4\"\n      ]\n    }\n  ]\n}\n[/block]\nThe badges are available from the actions and are included in the Readme [here](https://raw.githubusercontent.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/master/README.md).","updates":[],"order":2,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"60e41ad9358510000f5ecebc","createdAt":"2021-07-06T08:56:57.192Z","user":"5dbff97bc91e080029778657","category":{"sync":{"isSync":false,"url":""},"pages":[],"title":"Tutorials","slug":"tutorials","order":1,"from_sync":false,"reference":false,"_id":"6024033e2b2f6f004dfe994c","createdAt":"2021-02-10T16:01:02.224Z","version":"5773dcfc255e820e00e1cd50","project":"5773dcfc255e820e00e1cd4d","__v":0},"version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["5773dcfc255e820e00e1cd51","5773df36904b0c0e00ef05ff","577baf92451b1e0e006075ac","577bb183b7ee4a0e007c4e8d","577ce77a1cf3cb0e0048e5ea","577d11865fd4de0e00cc3dab","578e62792c3c790e00937597","578f4fd98335ca0e006d5c84","578f5e5c3d04570e00976ebb","57bc35f7531e000e0075d118","57f801b3760f3a1700219ebb","5804d55d1642890f00803623","581c8d55c0dc651900aa9350","589dcf8ba8c63b3b00c3704f","594cebadd8a2f7001b0b53b2","59a562f46a5d8c00238e309a","5a2aa096e25025003c582b58","5a2e79566c771d003ca0acd4","5a3a5166142db90026f24007","5a3a52b5bcc254001c4bf152","5a3a574a2be213002675c6d2","5a3a66bb2be213002675cb73","5a3a6e4854faf60030b63159","5c8a68278e883901341de571","5cb9971e57bf020024523c7b","5cbf1683e2a36d01d5012ecd","5dc15666a4f788004c5fd7d7","5eaff69e844d67003642a020","5eb00899b36ba5002d35b0c1","5eb0172be179b70073dc936e","5eb01b42b36ba5002d35ebba","5eb01f202654a20136813093","5eb918ef149186021c9a76c8","5f0839d3f4b24e005ebbbc29","5f893e508c9862002d0614a9","6024033e2b2f6f004dfe994c","60a7a12f9a06c70052b7c4db","60a7ab97266a4700161507c4","60b0c84babba720010a8b0b5"],"_id":"5773dcfc255e820e00e1cd50","__v":39,"createdAt":"2016-06-29T14:36:44.812Z","releaseDate":"2016-06-29T14:36:44.812Z","project":"5773dcfc255e820e00e1cd4d"},"project":"5773dcfc255e820e00e1cd4d","__v":0,"parentDoc":null}

Maintaining and versioning CWL on external tool repositories


##Summary CAVATICA enables users to develop their own Common Workflow Language (CWL) tools and workflows directly on the Platform using the Seven Bridges Software Development Kit (SDK). The SDK consists of the Tool Editor, Workflow Editor, and other helpful utilities. Both the Tool Editor and Workflow Editor have a visual editor and code editor component. This tutorial presents best practices for writing and maintaining CWL tools/workflows in an external tool repository, such as github, so that users can better manage versions of their tools. Users should follow these best practices if they would like to publish and share their CWL tools and workflows in the [Dockstore repository](https://dockstore.org/) since Dockstore has the ability to automatically pull changes from github. These best practices will ensure that the CWL is fully portable and can run successfully not only on Seven Bridges Platforms, but also on other CWL executors such as cwltool and Toil. This tutorial will guide you through using two open source tools for working with CWL, and you will need to be comfortable with or willing to learn how to use command line tools. The tutorial provides guidance both for developers and researchers writing CWL in a local Integrated Development Environment (IDE) or code editor, but also for developers using the graphical “low code” Seven Bridges Tool Editor and Workflow Editor. ###Overview of flow The overall flow can be summarized in the diagram below. Developing a CWL workflow that runs portably follows the same general practices as any other good software development though there are some CWL-specific aspects: 1. Use small tests to check each part of the workflow 2. Test each tool separately before testing the parent workflow 3. Test and verify execution on more than one environment to ensure there are no environment-specific bugs 4. Keep the amount of Javascript to a minimum and ensure it is strict mode ECMAscript 5 [block:image] { "images": [ { "image": [ "https://files.readme.io/5d3c2ca-BDC_Dockstore_CWL_tutorial.png", "BDC_Dockstore CWL tutorial.png", 800, 600, "#c4c8cb" ] } ] } [/block] An example of a set of workflows developed according to these principles are the Seven Bridges CWL wrappers for the association testing tools, GENESIS, available on [github](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl) and [Dockstore](https://dockstore.org/organizations/SevenBridges/collections/genesispipelines). ##Tools used in this guide [block:parameters] { "data": { "h-0": "Tool", "h-1": "Note", "h-2": "Installation", "0-0": "[cwltool](https://github.com/common-workflow-language/cwltool)", "0-1": "CWL reference runner. Will be used for checking CWL for correctness and for running tests.", "0-2": "`pipx install cwltool`", "1-0": "[sbpack](https://github.com/rabix/sbpack)", "1-1": "Tool to upload and download CWL code from Seven Bridges Platforms.", "1-2": "`pipx install sbpack`", "2-0": "[git](https://git-scm.com/)", "2-1": "Popular versioning system we will use, Github is built around git.", "2-2": "`via website`", "3-0": "[benten](https://github.com/rabix/benten) (optional)", "3-1": "CWL language server, offers code intelligence for a variety of code editors.", "3-2": "`pipx install benten` OR let VS Code extension install it for you.", "4-0": "[VS Code](https://code.visualstudio.com/) (optional)", "4-1": "A popular and powerful open source code editor with a CWL editing plugin", "4-2": "`via website`" }, "cols": 3, "rows": 5 } [/block] You may choose to install the tools any way you choose. The Seven Bridges team recommends using pipx because it is straightforward; works on Linux, macOS and Windows; and installs each tool in an isolated virtual environment to avoid dependencies. ##Developing locally vs. developing on CAVATICA You can follow the practices outlined here by either developing CWL locally using a code editor or by developing CWL on the cloud using the Tool Editor and Workflow Editor feature of CAVATICA. Use the sbpack tool to upload/download the CWL from/to the Platform. [block:image] { "images": [ { "image": [ "https://files.readme.io/8572442-develop-locally-vs-on-cavatica.png", "develop-locally-vs-on-cavatica.png", 641, 397, "#333" ] } ] } [/block] It is currently difficult to use the two development models simultaneously. If you attempt to edit and develop CWL locally -AND- edit on CAVATICA at the same time, you will encounter issues with the formatting and layout of the CWL code due to how the Platform packs the code into one file. Therefore, it is best to use one development model. ##Collect small test data sets and test cases When you are ready to wrap a tool or workflow in CWL please collect small data sets that will allow testing of each CWL tool all the way up to the final workflows. This greatly facilitates debugging. * **On Seven Bridges with the Tool Editor and Workflow Editor**: Add these test files to your development project and run tasks periodically as you develop the tool/workflow. * **Local development**: For each CWL, create a test job file using the test data as inputs. This is used to periodically run the tool and workflow to ensure the CWL being developed is correct. An example of such a test job file can be found [here](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/master/nullmodel/job-null-model.yml). * **Continuous integration**: Use continuous integration to ensure the tests are being run automatically. github makes this easy using [actions](https://docs.github.com/en/actions/learn-github-actions/introduction-to-github-actions). As an example, look at this [github action](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/master/.github/workflows/nullmodel.yml) that invokes cwltool using a job file and test inputs. ##Develop, test, fix, repeat [block:image] { "images": [ { "image": [ "https://files.readme.io/5504e91-develop-on-cavatica.png", "develop-on-cavatica.png", 550, 555, "#333" ] } ] } [/block] Once the test sets are up and continuous integration has been put in place, you enter a standard cycle of development, testing, debugging and fixing of the CWL. Developing CWL in this way should ensure that the CWL executes flawlessly on multiple platforms. However, please note that there may be particular aspects of CWL where different executors may interpret the specification differently, have different bugs, or have different affordances. For this reason, we recommend using a common software engineering practice and test the CWL on multiple platforms. In general, it is sufficient to test on two different platforms to ensure that we are not touching on aspects of CWL that may be buggy or inconsistent in one or more executors. In addition to Seven Bridges, we recommend using the executor “cwltool” which is the community developed CWL reference runner. ###Test tools separately We will follow an iterative process while developing the workflow. First, we will wrap the individual tools. Next, we will test them with cwltool. We will then push the CWL to Seven Bridges and test with the Seven Bridges executor. Once we are satisfied that a particular iteration of the code works, we will commit that code to our repository. Finally, once we have tested all the individual tools, we will wrap the workflow with the component tools and repeat the process. ##Brief guidelines to improve portability of CWL ###Minimize use of Javascript ####Use bash scripts and parameter references * The bash script runs in the container, guaranteeing reproducible and stable execution * Use parameter references or short JS expressions * Use `set -x` at the top of the bash script to see the execution steps in the stdout log. Excellent for debugging Consider this [embedded script](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/5f17ca875ec5b0e324fa899ed0e3175ef9ddf9d0/single-variant-association/assoc_single_r.cwl#L37): [block:code] { "codes": [ { "code": "- entryname: script.sh\n entry: |\n set -x\n # This is a bit of cleverness we have to do to extract the chromosome\n # number from the segments file and pass it to the R script\n CHROM=\"$(\"$\")(awk 'NR==${return parseInt(inputs.segment) + 1} {print $1}' $(inputs.segment_file.path))\"\n Rscript /usr/local/analysis_pipeline/R/assoc_single.R assoc_single.config --chromosome $CHROM --segment $(inputs.segment)", "language": "shell", "name": "embedded script" } ] } [/block] In order to make the underlying tool work, some cleverness has to be done: we have to parse an input file and read one of its columns. This cleverness is done in bash and ensures that it runs in the container and runs identically everywhere. Note the use of `set -x` which prints all commands and what is actually executed out to stdout. This helps with recordkeeping and debugging and is easier to read than complicated, long command lines. Please note the use of the `$(“$”)` trick to make the embedded script work on both Seven Bridges executor and cwltool. ####Prefer parameter references and short expressions When creating files (e.g. config files or manifests) prefer parameter references or short expressions that embed input variables into the raw text. This is easier to debug, to read and results in less problems overall. For example instead of [block:code] { "codes": [ { "code": " - class InitialWorkDirRequirement\n listing:\n - entryname: null_model.config\n writable: false\n entry: |-\n ${ \n \n var arguments = {};\n if(inputs.output_prefix){\n var filename = input.output_prefix + \"_null_model\";\n arguments.push('out_prefix \\\"' + filename '\\\"');\n var phenotype_filename = inputs.output_prefix + \"_phenotypes.RData\";\n arguments.push('out_phenotype_file \\\"' + phenotype_filename + '\\\"\\);\n \n }\n else{\n arguments.push{'out_prefix \"null_model:\"'};\n arguments.piush('out_phenotype_file \"phenotypes.RData\"');\n }\n arguments.push('outcome ' + inputs.outcome);\n arguments.push('phenotype_file \"' + inputs.phenotype_file.path _ '\"');\n if(inputs.gds_files){\n arguments.push('gds_file \"' + inputs.gds_files[0].path.split('chr')[0] + 'chr .gds\"')\n }\n if(inputs.pca_file){\n arguments.push('pca_file \"' + inputs.pca_file.path + '\"')\n }\n if(inputs.binary){\n arguments.push('binary ' + inputs.binary)\n }\n if(inputs.conditional_variant_file){\n arguments.push('conditional_variant_file \"' + inputs.conditional_variant_file.path + '\"'\n }\n if(inputs.covars){\n temp = []\n for(var i=0; i<inputs.covars.length; i++){\n temp.push(inputs.covars[i])\n }\n arguments,push('covars \"' + temp.join(' ') + '\"'\n }\n if(inputs.group_var){\n arguments.push('group_var \"' + inputs.group_var _ '\"')\n }\n if(inputs,inverse_normal){\n arguments.push('inverse_normal ' + inputs.inverse_normal)\n }\n if(inputs.n_pcs){\n if(inputs.n_pcs > 0)\n arguments.push('n_pcs ' + inputs.n_pcs)\n }\n if(inputs.rescale_variance){\n arguments.push('rescale_variance \"' + inputs.rescale_variance + '\"')\n }\n if(inputs.resid_covars){\n arguments.push('resid_covars ' + inputs.resid_covars)\n }\n if(inputs.sample_include_file){\n arguments.push('sample_include_file \"' + inputs.sample_include_file.path + '\"')\n }\n if(inputs.norm_bygroup){\n arguments.push(norm_bygroup ' + inputs.norm_bygroup)\n }\n rturn arguments.join('\\n')\n } \n", "language": "json" } ] } [/block] [block:image] { "images": [ { "image": [] } ] } [/block] [Prefer](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/blob/5f17ca875ec5b0e324fa899ed0e3175ef9ddf9d0/nullmodel/null-model.cwl#L71) [block:code] { "codes": [ { "code": " baseCommand: []\n arguments:\n - prefix: ''\n position: 1\n valueFrom: |-\n ${\n if(inputs.null_model_files == null)\n {\n \n }\n else\n }\n command = \"cp\" + inputs.null_model_file.path + \" \" + inputs.phenotype_file.path + \" . && \"\n return command + \"echo 'Passing null_model.R.'\"\n }\n }\n shellQuote: false ", "language": "json" } ] } [/block] This makes the generated file more readable and allows individual expressions to be debugged more easily. ####Avoid complicated, dynamic command lines * If a complicated command line with multiple commands is needed, use bash scripts instead * Avoid doing conditionals embedded in arguments: these are hard to read. Use bash scripts instead * Avoid complicated dynamic globs Therefore avoid constructs like this: [block:code] { "codes": [ { "code": " baseCommand: []\n arguments\n - prefix: ''\n position: 1\n valueFrom: |-\n ${\n if(inputs.null_model_files == null)\n {\n return Rscript /usr/local/analysis_pipeline/R/null_model.R null_model.config\"\n \n }\n else\n }\n {\n command = \"cp\" + inputs.null_model_file.path + \"\" + inputs.phenotype_file.path + \" . &&\"\n return command + \"echo 'Passing null_model.R.'\"\n }\n }\n shellQuote: false ", "language": "json" } ] } [/block] This is intended as a conditional, but it is hard to read. Note: there is a bug in here. The input variable name is actually null_model_file . The Seven Bridges executor resolves this to undefined and the test always returns false. This is an uncaught bug. Code review or more test cases might have caught this. Also, the complexity here results in us needing more complicated constructs in the output binding. ###Appendix ####Suggested CWL git project structure By pushing CWL to github your software project now has a high level of version control and collaboration opportunities. As a community we should work towards a standardization of CWL project structure. Seven Bridges suggests a very simple folder structure. [block:image] { "images": [ { "image": [ "https://files.readme.io/dfb2301-folder-structure.png", "folder-structure.png", 633, 251, "#f7f8fa" ] } ] } [/block] Research projects and software toolkits often contain more than one workflow. In this case, make a directory for each workflow and have a directory for small test data. The “.github/workflows” is used for continuous integration tests only. Finally add a `README.md` with notes about how the workflows are intended to be used (see below for test badges). Most of the README notes can be copied directly from your CWL descriptions. Populate each workflow directory with the individual tool steps. This allows for better organization and for other researchers/developers to incorporate your steps in their workflows. This folder structure can be created manually if following the local development style or by using the SDK utility sbpack to represent the workflow as one CWL file per step. [block:image] { "images": [ { "image": [ "https://files.readme.io/7a75b1f-languages.png", "languages.png", 442, 156, "#f0eaec" ] } ] } [/block] Github can identify Common Workflow Language and will tag your project as containing it. By keeping the project dedicated to CWL this will help other users easily find your workflow and tools wrappers. ####Run CWL validation using a git pre-commit hook We can use this feature to run a quick check each time we go to commit code. A git pre-commit hook is a script that is run before code is committed. We can use this feature to run a quick check each time we go to commit code. We will create a bash script which is simply a set of calls to `cwltool --validate` to check each CWL file that is being committed. The only clever thing we will do is use a git command to retrieve the files we are about to commit and only validate those. We will create a file: [block:code] { "codes": [ { "code": "for f in `git diff --staged --name-only`\ndo\n cwltool --validate \"$f\"\ndone\n", "language": "text", "name": "check-changed.sh" } ] } [/block] We will then invoke this script from another script called `pre-commit`. The `pre-commit` file will be put under the `.git/hooks` directory and will be automatically invoked during a commit action. The script `.git/hooks/pre-commit` is invoked from the root of the repository: [block:code] { "codes": [ { "code": "check-changed.sh", "language": "text", "name": ".git/hooks/pre-commit" } ] } [/block] You can read up a bit more about git hooks [here](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks). ####Add test badges to your repository Since you have done all the hard work of setting up tests and continuous integration why not add some test badges to show this off in your repository on github? This is how the Readme looks from the [Seven Bridges GENESIS CWL repository](https://github.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl) linked at the start. [block:image] { "images": [ { "image": [ "https://files.readme.io/a4e6b94-cwl-wappers.png", "cwl-wappers.png", 634, 241, "#f2f4f4" ] } ] } [/block] The badges are available from the actions and are included in the Readme [here](https://raw.githubusercontent.com/sevenbridges-openworkflows/uw-genesis-topmed-cwl/master/README.md).