{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","settings":"","results":{"codes":[]},"params":[]},"next":{"description":"","pages":[]},"title":"Bring WDL apps to CAVATICA","type":"basic","slug":"bring-wdl-apps-to-cavatica","excerpt":"","body":"[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"To get access to this feature, please contact our [Support Team](mailto:support:::at:::sevenbridges.com).\"\n}\n[/block]\n## Overview\n\n[Workflow Description Language (WDL)](https://github.com/openwdl/wdl) is an open, standardized, _human readable and writable_ language for expressing tasks and workflows. WDL is designed to be a general-purpose workflow language, but it is most widely used in the field of bioinformatics. If you already have tools or execution pipelines that are described in WDL, this page will provide details on how you can push them to CAVATICA and run your analysis at scale, using the full potential and benefits of the Seven Bridges execution environment.\n\n## Prerequisites\n\n* An account on CAVATICA.\n* Installed `sbpack`. This is a command-line tool you'll use to adapt and push WDL apps to CAVATICA. For more details on what `sbpack` can do, how to install it and its main use cases, see [About sbpack](#section-about-sbpack) below.\n* _(Optional)_ Downloaded [WOMtool](https://cromwell.readthedocs.io/en/develop/WOMtool). The WOMtool `jar` executable is used to generate the JSON input file that specifies all of the WDL app's inputs.\n\n### About sbpack\n\nThe primary use of `sbpack` is to provide an easy way to upload (`sbpack`) and download (`sbpull`) apps to/from any Seven Bridges powered platform. Since it is a command-line tool, it can be particularly useful as a part of continuous development and integration pipelines for bioinformatics apps, as it allows seamless automated deployment of new app versions to CAVATICA. It works with apps described using the following workflow description standards:\n\n* **Common Workflow Language (CWL)**. Apart from enabling the standard app pull and push flows, also provides advanced functionalities such as resolution of linked processes, schemadefs and $includes and $imports.\n* **Nextflow**. Adapts, prepares and pushes Nextflow apps for execution in Seven Bridges environments using a special `sbpack_nf` command.\n* **Workflow Description Language (WDL)**. Uses the `sbpack_wdl` command to convert and push WDL apps for execution in Seven Bridges environments.\n\nTo install `sbpack`, use the standard install method through `pip`:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"pip install sbpack\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n## Procedure\n\nThe procedure of publishing WDL apps for use on CAVATICA is a process that consists of the following two stages:\n\n* **Initial app conversion.** In this step, your WDL app will be converted to a format that is executable on CAVATICA. However, to optimize your app for execution on CAVATICA and make the most out of CAVATICA's execution environment it is _strongly recommended_ to go through the next step.\n* **Optimizing the converted app for execution in Seven Bridges environments.** The app that has been initially converted now contains an additional configuration file that you will use to define CAVATICA-specific options and fully optimize it for use in the Seven Bridges execution environment. Once the optimized configuration is prepared, the app configuration is pushed to CAVATICA again.\n\n### Initial app conversion\n\nThis step adapts the WDL app for execution on CAVATICA. It is performed by executing the `sbpack_wdl` command in the following format:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --womtool-input WOMTOOL_INPUT\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n\nIn the command above, replace the placeholders as follows:\n\n* `PROFILE_NAME` is the CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries).\n* `APPID` specifies the identifier of the app on CAVATICA, in the `{user}/{project}/{app_name}` format. The `{user}` part is your CAVATICA username. The `{project}` part is the project to which you want to push the app and `{app_name}` is the name you want to assign to the app. For example the full app ID can be `rfranklin/my-new-project/my-wdl-app`. If the specified app ID does not exist, it will be created. If it exists, a new [revision (version)](doc:app-versions) of the app will be created.\n* `WORKFLOW_PATH` needs to be replaced with the path where the WDL app files are located on your local machine. The `WORKFLOW_PATH` folder should include all the dependencies that the workflow needs in order to run.\n* `ENTRYPOINT` should be replaced with the path to the actual .wdl file, relative to the root of `WORKFLOW_PATH`.\n* `WOMTOOL_INPUT` should be replaced with the path to the JSON inputs file containing your app's inputs schema generated using [WOMtool](https://cromwell.readthedocs.io/en/develop/WOMtool/). If you don't have the inputs file, replace `--womtool-input` with the `--womtool-path` argument and provide the path to the [WOMtool jar executable](https://github.com/broadinstitute/cromwell/releases) on your local machine, which will generate the inputs and pass them to `sbpack_wdl` automatically.\n\nHere is a sample of the command:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbpack_wdl --profile cavatica --appid rfranklin/wdl-project/test-app --workflow-path /Users/rfranklin/apps/wdl/demo --entrypoint cellranger_workflow.wdl --womtool-input /Users/rfranklin/apps/wdl/demo/inputs.json\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nOnce executed successfully, this command will convert the WDL app for use on CAVATICA and push it to the CAVATICA project specified as the value of the `--appid` argument. The local directory specified as the value of `--workflow-path` will now contain an additional `sb_wdl_schema.yaml` file. The file contains configuration parameters that can be adjusted and optimized for execution on CAVATICA.\n\nOptionally, to avoid pushing the app to CAVATICA at this stage and perform [optimizations for the Seven Bridges execution environment](#section-optimizing-the-converted-app-for-execution-in-seven-bridges-environments) beforehand, use the `--dump-sb-app` flag at the end of the command. For a full list of available arguments to the `sbpack_wdl` command, see the [sbpack_wdl command reference](#section--sbpack_wdl-command-reference).\n\n### Optimizing the converted app for execution in Seven Bridges environments\n\nWhen you have performed the initial conversion step, the generated `sb_wdl_schema.yaml` file is important as it contains confguration parameters that will help you optimize the app for execution on CAVATICA. The file consists of the following major sections:\n\n* The initial section that includes general app information and the documentation content describing the app (if any):\n    * `app_content`: Contains details about app's package and WDL file:  \n    * `code_package`: CAVATICA ID of the file that contains the WDL code. This is replaced by the `git_pull` key if the `--no-package` option was used to set a git repository as the source of the app's code. See the [sbpack_wdl command reference](#section--sbpack_wdl-command-reference) for details.\n    * `entrypoint`: Relative path to the file containing the WDL code, relative to the root directory of the ZIP file defined in `code_package`.\n    * `class`: Defines the type of workflow description language used for the app. The value will always be `wdl` for WDL apps.\n    * `cwlVersion`: Defines the version of CWL used to describe the app. The value will always be `None` for WDL apps.\n    * `doc` _(Optional)_: The Markdown-formatted text describing the app. \n    * The `inputs` section that defines details of the app inputs.\n    * The `outputs` section that defines details of app outputs.\n    * The `requirements` section that defines app execution requirements such as initial working directory.\n\n#### Configuring inputs\n\nEach of the app inputs that is present in the `inputs` section contains the following basic details:\n\n* `id`: Unique identifier of the input.\n* `inputBinding`: Defines the mapping of the input to the command line of the app that is being executed. If `inputBinding` is omitted, input is made available in the execution environment, but is not passed on to the WDL executor.\n    * `prefix`: The ID of the WDL workflow input, in the form that would be added to the WDL inputs JSON file. For example, if you provide a value for an input that is labelled as `workflow.input_file` in the WDL inputs JSON file, the prefix would be defined as `workflow.input_file.`\n* `default`: The default value for the input. If the input value is not set on task execution, this default value is taken and passed on to the executor as defined with `inputBinding.prefix`.\n* `label`: Text description of the input. \n* `sbg:toolDefaultValue`: Default value of the input in the WDL workflow. Value provided here is not used in execution and is descriptive (for information purpose) only.\n* `sbg:fileTypes`: Comma separated (with spaces) value of file extensions that are used in the file picker when setting up tasks. For example: sbg:fileTypes: `“FASTQ, FASTQ.GZ”`.\n* `type`: The type of value expected on the input. If there is a question mark at the end, for example `int?`, the input is optional. Otherwise, the input is mandatory. Learn more about [available types of inputs](page:about-app-input-and-output-types).\n\nTo accommodate for the transition between WDL and the Seven Bridges execution environment, the `sb_wdl_schema.yaml` file will always contain an additional input whose ID is `auxiliary_files`, which contains the list of files not added as explicit workflow inputs but are required for workflow execution. To enable proper execution on CAVATICA, please do not remove this input from `sb_wdl_schema.yaml`.\n\n**Note: File and Directory inputs**\n\nIn some WDL versions, `Directory` inputs are marked as `File`. The type of such inputs will consequently be `File` on CAVATICA. If the input should actually be a directory, simply change its `type` property from `File` to `Directory` in `sb_wdl_schema.yaml`.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"\",\n  \"body\": \"There may be cases where input types are not properly set by `sbpack_wdl` during initial conversion, especially for complex input types. These cases may need to be handled manually based on each use case.\"\n}\n[/block]\n#### Configuring outputs\n\nIn addition to executing WDL apps on CAVATICA, you also need to optimize app outputs to produce and save only files that match the defined criteria, To achieve this and be able to further configure your app outputs, see the details about configuration parameters contained in the `outputs` section of the `sb_wdl_schema.yaml` file:\n\n* `id`: Unique identifier of the output. You can change this value to provide a more adequate and descriptive one if necessary.\n* `outputBinding`: Defines the `glob` expression or pattern that will be used to select the output file.\n    * `glob`: The [glob](page:glob) expression that defines the items to keep as outputs on the output port.\n* `type`: The type of output value.\n\n**Example: Configuring an output**\n\nThe `sb_wdl_schema.yaml` file always contains one automatically generated app output:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"outputs:\\n  - id: wdl_workdir\\n    outputBinding:\\n        glob: '*'\\n    type: File[]\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nTo configure your outputs, create an entry for each of the output ports you want to have, following the pattern below:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"outputs:\\n  - id: csv_file\\n    outputBinding:\\n        glob: '*.csv'\\n    type: File\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nIn the example above, replace `csv_file` with an ID that describes your output and replace `*.csv` with the glob that matches the type and naming of the data you want to produce on the output. Learn more about available [types of outputs](page:about-app-input-and-output-types).\n\n**Example: Configuring a dynamic output directory name**\n\nYou can also use the `sb_wdl_schema.yaml` file to set the name of the output directory by defining it in an app input, provided that the tool itself supports the option of defining the output directory name using the corresponding input argument and its value. The first step is to define the input that takes the output directory name (in the `sb_wdl_schema.yaml inputs` section):\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"inputs:\\n  - id: my_workflow_outdir_name\\n    inputBinding:\\n        prefix: my_workflow.outdir_name\\n    type: string?\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nOnce you have defined the input, define an output, where the output directory `glob` will be a variable that gets the value defined in the input above.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"outputs:\\n  - id: output_directory\\n    outputBinding:\\n        glob: '$(inputs.outdir_name)'\\n    type: Directory\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nThe `$(inputs.outdir)` value is a variable that will be replaced with the actual value entered in the `outdir` input when the app is executed.\n\n#### Configuring requirements\n\nThe `requirements` sections is primarily used for the following execution-related parameters:\n\n* Setting input staging (making input files available in the app's working directory).\n* Setting computation instances that are used for app executions on CAVATICA.\n\n**Setting input staging**\n\nFiles that are named as inputs to a tool are not, by default, in the tool's working directory. In most apps this access is sufficient, since most tools only need to read their input files, process the data contained in them, and write new output files on the basis of this data to their working directory. However, in some cases an app might require input files to be placed directly in its working directory. If this is the case with your app, modify the `requirements` section in the `sb_wdl_schema.yaml` file as follows:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n  - class: InitialWorkDirRequirement\\n    listing:\\n      - '$(inputs.tsv_files)'\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nEntries under `listing` define files and directories that will be made available in the app’s working directory before the command is executed. The files and directories are usually defined as variables named after their respective input IDs.\n\nAnother useful option is creation of a file directly in the working directory. This is done by defining `entryname` and `entry` keys in the `InitialWorkDirRequirement` class, as follows:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"requirements:\\n- class: InitialWorkDirRequirement\\n  listing:\\n  - entryname: input_nanoseq.csv   \\n    entry: |\\n    ${\\n    if (inputs.auxiliary_files && !inputs.in_csv_file){\\n        var content = 'group,barcode';\\n        for (var i = 0; i  < inputs.auxiliary_files.length; i++){\\n            if (inputs.auxiliary_files[i].metadata['barcode']){\\n                var barcode = inputs.auxiliary_files[i].metadata['barcode'];\\n            }\\n            else {\\n                var barcode = '';\\n            }\\n            if (inputs.auxiliary_files[i].metadata['group']){\\n                var group = inputs.auxiliary_files[i].metadata['group'];\\n            }\\n            else {\\n                var group = '';\\n            }\\n            content = content.concat(group,',',barcode,'\\\\\\\\n');\\n        }\\n        return content\\n    }\\n    else {\\n        return ''\\n    }\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nIn the example code above, `entryname `defines the name of the file generated in the working directory, which is `input_nanoseq.csv`, while `entry` contains a Javascript expression that populates the generated file by getting `barcode` and `group` metadata values from input files and concatenating them in a single CSV file. As we're using a Javascript expression, it is recommended to use the default option of generating a YAML schema when using the `sbpack_wdl` command, as YAML is more convenient and reduces the possibility of making errors compared to using Javascript expressions in a JSON file. The expression can be defined to match your needs and intended use. Read more about [dynamic expressions in tool descriptions](doc:dynamic-expressions-in-tool-descriptions-1) or see some of the most common expression examples in our [Javascript Cookbook](doc:javascript-cookbook-1).\n\n**Setting execution instances**\n\nAnother useful option that is available for configuration in the `hints` section is the definition of the computation instance used for app execution on CAVATICA. This is also done by defining key-value pairs as follows:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"hints:\\n  - class: sbg:AWSInstanceType\\n    value: c4.8xlarge;ebs-gp2;2000\",\n      \"language\": \"yaml\"\n    }\n  ]\n}\n[/block]\nIn this case, the workflow uses a `c4.8xlarge` instance with 2000 GB of attached EBS storage. The value consists of the following three parts (separated by `;`):\n* Instance type, e.g. `c4.8xlarge`.\n* Attached disk type: always `ebs-gp2` for all instances with EBS storage.\n* Disk size in GB.\n\nSee the list of [AWS](doc:list-of-available-amazon-web-services-instances) instances that are available for task execution on CAVATICA. \n\n### Pushing the optimized app configuration to CAVATICA\n\nWhen you are done with changes to the `sb_wdl_schema.yaml` file, push the optimized app configuration to CAVATICA. As we just making configuration changes to an app that has already been pushed to CAVATICA, this can be done using the regular `sbpack` command in the following format:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbpack PROFILE_NAME APPID CONFIG_FILE\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nIn the command above, `PROFILE_NAME` refers to the CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries). The  `APPID` parameter specifies the identifier of the app on CAVATICA. _Use the same  `APPID` you used in the initial conversion step_. Finally, `CONFIG_FILE` is the `sb_wdl_schema.yaml` in which you made app execution optimizations. The final command should be, for example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbpack cavatica-profile rfranklin/wdl-project/test-app sb_wdl_schema.yaml\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThis pushes the modified app configuration to CAVATICA and creates a new [revision (version)](doc:app-versions) of the app. Once this is done, you are ready to [run a task](doc:run-a-task-1) using the app.\n\n### Updating already converted and optimized apps\nIf you have already converted your app, made optimizations in the `sb_wdl_schema.yaml` file, and pushed the app to CAVATICA, all subsequent updates to the app's WDL code and the process of propagating the update to CAVATICA are quite straightforward. If the updates you made do not require changes to manually configured parameters in the `sb_wdl_schema.yaml` file (such as inputs, outputs, requirements, etc.), create a new code package by running a command in the following format:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --sb-schema SB_SCHEMA\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThis command is almost the same as the initial app conversion step, but differs in the additional `--sb-schema` argument. This argument allows you to provide and reuse an existing `sb_wdl_schema.yaml` configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on CAVATICA. The command will generate a new code package based on your updated WDL code provided through `--workflow-path` and `--entrypoint` and the YAML or JSON configuration file provided through `--sb-schema`, and push the updated app to CAVATICA creating a new [revision (version)](doc:app-versions).\n\n### Copying WDL apps between projects on CAVATICA\n\nWhen an app is on CAVATICA, you can copy it and use it on other CAVATICA projects. To copy WDL apps between projects, use the `sbcopy` command that is a part of the `sbpack` utility:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbcopy [--profile PROFILE] --appid APPID --projectid PROJECTID\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThe command takes the following arguments:\n\n* `PROFILE`: refers to the CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries).\n* `APPID`: specifies the identifier of the app on CAVATICA. Takes the form `{user}/{project}/{app_id}`. The `{user}` part is your CAVATICA username. The `{project}` part is the source project where the app is located and `{app_id}` is the ID of the app you want to copy; for example `rfranklin/my-new-project/my-wdl-app`.\n* `PROJECTID`: is the identifier of the destination project where the app will be copied. Takes the form `{user}/{project}`.\n\nThe final command should be, for example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sbcopy cavatica-profile rfranklin/wdl-project/test-app jsmith/my-wdl-project\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"Note that WDL app copies made through standard [visual interface](https://docs.sevenbridges.com/docs/copy-an-app-from-a-project) or [API](doc:copy-an-app) methods instead of using `sbcopy` will still point to the originally pushed code package and the original project where it is located. This might cause failures due to lack of permissions, if users who need to run the copied instances of the app aren't [added](doc:add-a-collaborator-to-a-project) to the project where the original code package is located. To avoid this, please use `sbcopy` to copy WDL apps between projects on CAVATICA, as described above.\"\n}\n[/block]\n## `sbpack_wdl` command reference\n\nHere is a list describing all available arguments od the `sbpack_wdl` command that is used to convert and push WDL apps for execution on CAVATICA.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Argument\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`-h`, `--help`\",\n    \"0-1\": \"\",\n    \"0-2\": \"Shows the list of all arguments and their corresponding explanations.\",\n    \"1-0\": \"`--profile PROFILE`\",\n    \"1-1\": \"\",\n    \"1-2\": \"CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries). If you are using the default profile, this parameter can be omitted.\",\n    \"2-0\": \"`--appid APPID`\",\n    \"2-1\": \"Required\",\n    \"2-2\": \"The ID of the WDL app once it is pushed to CAVATICA. Takes the form of `{user}/{project}/{app_id}`. The `{user}` part is your CAVATICA username. The `{project}` part is the project to which you want to push the app and `{app_id}` is the ID you want to assign to the app, for example my-`rfranklin/my-new-project/my-wdl-app`.\",\n    \"3-0\": \"`--workflow-path WORKFLOW_PATH`\",\n    \"3-1\": \"Required\",\n    \"3-2\": \"Path to the main workflow directory (the local directory where the app's files are located).\",\n    \"4-0\": \"`--entrypoint ENTRYPOINT`\",\n    \"4-1\": \"Required\",\n    \"4-2\": \"Relative path to the the file that contains the app's WDL code from the main workflow directory defined in `--workflow-path`.\",\n    \"5-0\": \"`--sb-package-id SB_PACKAGE_ID`\",\n    \"5-1\": \"\",\n    \"5-2\": \"ID of an already uploaded package. If you have already converted and pushed the app to CAVATICA, it has its own code package ID, as shown in the `code_package` key in the `sb_wdl_schema.yaml` file. When the package ID is provided, the conversion script will skip the upload step and thus take less time to execute.\",\n    \"6-0\": \"`--sb-doc SB_DOC`\",\n    \"6-1\": \"\",\n    \"6-2\": \"Path to the app description document written in Markdown. The document is meant to provide additional details about the app and will be shown when viewing app details on CAVATICA. If not provided, `README.md` will be used if available in the same directory where `entrypoint` file is located.\",\n    \"7-0\": \"`--womtool-input WOMTOOL_INPUT`\",\n    \"7-1\": \"Required if `--womtool-path` isn't used\",\n    \"7-2\": \"Path to JSON inputs file containing your app's inputs schema generated using [WOMtool](https://cromwell.readthedocs.io/en/develop/WOMtool/).\",\n    \"8-0\": \"`--womtool-path WOMTOOL_PATH`\",\n    \"8-1\": \"Required if `--womtool-input` isn't used\",\n    \"8-2\": \"Path to the [WOMtool](https://cromwell.readthedocs.io/en/stable/WOMtool/) jar file on your local machine. If you don't already have a JSON inputs file that you can provide via `--womtool-input`, this command will use WOMtool to generate the file on the fly and provide it to `sbpack_wdl` automatically.\",\n    \"10-0\": \"`--dump-sb-app`\",\n    \"10-1\": \"\",\n    \"10-2\": \"Dumps the converted app to a local file without pushing it to CAVATICA. Using this option will enable you to convert the app and generate the `sb_wdl_schema.yaml`  file to make configuration optimizations prior to pushing the app to CAVATICA.\",\n    \"11-0\": \"`--no-package`\",\n    \"11-1\": \"\",\n    \"11-2\": \"References a git repository containing the app's code instead of pushing the app's code to CAVATICA. Git repository address is specified as the `git_pull` key in the `sb_wdl_schema.yaml` file, instead of `code_package`. For example: `git_pull: https://git.domain.com/repository`. The value for the `git_pull` key is the URL you would normally use to clone the repository to your local environment.\",\n    \"12-0\": \"`--json`\",\n    \"12-2\": \"Creates the `sb_wdl_schema` file in JSON format instead of the default YAML.\",\n    \"9-0\": \"`--sb-schema SB_SCHEMA`\",\n    \"9-2\": \"Path to an existing `sb_wdl_schema` file in JSON or YAML format. This allows you to use an existing configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on CAVATICA.\"\n  },\n  \"cols\": 3,\n  \"rows\": 13\n}\n[/block]\n## Important notes for executing WDL apps on CAVATICA\n\n* Workflows are executed in Local mode. Make sure your workflow can run in Local mode before porting it to CAVATICA.\n* Use of Docker is required. See how to [create and upload a Docker image](doc:upload-your-docker-image) containing your app and make sure to edit the WDL code to [use the newly created image](https://cromwell.readthedocs.io/en/stable/tutorials/Containers#specifying-containers-in-your-workflow). If the Docker image is not specified for a process, a default alpine image will be used.\n* Execution is done in a dedicated working directory and all the WDL work is done in the `wdl_workdir` directory inside the working directory. Avoid using and relying on hard-coded paths in workflows.\n\n## Differences between running CWL and WDL apps on CAVATICA\n\nExecutions on CAVATICA normally result in separate [jobs](doc:about-task-execution#section-jobs) (steps) being created for each tool in the workflow. When a WDL pipeline is executed on CAVATICA, a single execution job will be created, regardless of the number of tools within the pipeline. This \"one app, one job\" approach results in the following differences compared to CWL app executions:\n\n* [Memoization](doc:about-memoization) is not available for WDL apps. As memoization relies on using previous job outputs to skip identical jobs in new executions, it is not useful in situations when there is only one job in an app execution.\n* Handling of [spot instance](doc:about-spot-instances) interruption can't be used with WDL apps. If a spot instance gets terminated when a WDL app is running, the app would have to be rerun on an on-demand instance from the beginning, which would result in increased costs instead of savings. Therefore, it is recommended to disable spot instances for WDL app executions.\n* [Task stats](doc:view-task-stats) and [logs](doc:view-task-logs) are organized differently. As task statistics and logs are usually shown and organized per job, WDL apps will have cumulative stats for the single job in the execution, while logs will be saved in a single folder.","updates":[],"order":6,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"622a0afeb4fde20071b7ad54","createdAt":"2022-03-10T14:28:14.998Z","user":"5767bc73bb15f40e00a28777","category":{"sync":{"isSync":false,"url":""},"pages":[],"title":"BRING YOUR TOOLS","slug":"sdk-overview","order":9,"from_sync":false,"reference":false,"_id":"5eb00899b36ba5002d35b0c1","createdAt":"2020-05-04T12:20:41.310Z","version":"5773dcfc255e820e00e1cd50","project":"5773dcfc255e820e00e1cd4d","__v":0},"version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["5773dcfc255e820e00e1cd51","5773df36904b0c0e00ef05ff","577baf92451b1e0e006075ac","577bb183b7ee4a0e007c4e8d","577ce77a1cf3cb0e0048e5ea","577d11865fd4de0e00cc3dab","578e62792c3c790e00937597","578f4fd98335ca0e006d5c84","578f5e5c3d04570e00976ebb","57bc35f7531e000e0075d118","57f801b3760f3a1700219ebb","5804d55d1642890f00803623","581c8d55c0dc651900aa9350","589dcf8ba8c63b3b00c3704f","594cebadd8a2f7001b0b53b2","59a562f46a5d8c00238e309a","5a2aa096e25025003c582b58","5a2e79566c771d003ca0acd4","5a3a5166142db90026f24007","5a3a52b5bcc254001c4bf152","5a3a574a2be213002675c6d2","5a3a66bb2be213002675cb73","5a3a6e4854faf60030b63159","5c8a68278e883901341de571","5cb9971e57bf020024523c7b","5cbf1683e2a36d01d5012ecd","5dc15666a4f788004c5fd7d7","5eaff69e844d67003642a020","5eb00899b36ba5002d35b0c1","5eb0172be179b70073dc936e","5eb01b42b36ba5002d35ebba","5eb01f202654a20136813093","5eb918ef149186021c9a76c8","5f0839d3f4b24e005ebbbc29","5f893e508c9862002d0614a9","6024033e2b2f6f004dfe994c","60a7a12f9a06c70052b7c4db","60a7ab97266a4700161507c4","60b0c84babba720010a8b0b5"],"_id":"5773dcfc255e820e00e1cd50","__v":39,"createdAt":"2016-06-29T14:36:44.812Z","releaseDate":"2016-06-29T14:36:44.812Z","project":"5773dcfc255e820e00e1cd4d"},"project":"5773dcfc255e820e00e1cd4d","__v":0,"parentDoc":null}

Bring WDL apps to CAVATICA


[block:callout] { "type": "info", "body": "To get access to this feature, please contact our [Support Team](mailto:[email protected])." } [/block] ## Overview [Workflow Description Language (WDL)](https://github.com/openwdl/wdl) is an open, standardized, _human readable and writable_ language for expressing tasks and workflows. WDL is designed to be a general-purpose workflow language, but it is most widely used in the field of bioinformatics. If you already have tools or execution pipelines that are described in WDL, this page will provide details on how you can push them to CAVATICA and run your analysis at scale, using the full potential and benefits of the Seven Bridges execution environment. ## Prerequisites * An account on CAVATICA. * Installed `sbpack`. This is a command-line tool you'll use to adapt and push WDL apps to CAVATICA. For more details on what `sbpack` can do, how to install it and its main use cases, see [About sbpack](#section-about-sbpack) below. * _(Optional)_ Downloaded [WOMtool](https://cromwell.readthedocs.io/en/develop/WOMtool). The WOMtool `jar` executable is used to generate the JSON input file that specifies all of the WDL app's inputs. ### About sbpack The primary use of `sbpack` is to provide an easy way to upload (`sbpack`) and download (`sbpull`) apps to/from any Seven Bridges powered platform. Since it is a command-line tool, it can be particularly useful as a part of continuous development and integration pipelines for bioinformatics apps, as it allows seamless automated deployment of new app versions to CAVATICA. It works with apps described using the following workflow description standards: * **Common Workflow Language (CWL)**. Apart from enabling the standard app pull and push flows, also provides advanced functionalities such as resolution of linked processes, schemadefs and $includes and $imports. * **Nextflow**. Adapts, prepares and pushes Nextflow apps for execution in Seven Bridges environments using a special `sbpack_nf` command. * **Workflow Description Language (WDL)**. Uses the `sbpack_wdl` command to convert and push WDL apps for execution in Seven Bridges environments. To install `sbpack`, use the standard install method through `pip`: [block:code] { "codes": [ { "code": "pip install sbpack", "language": "shell" } ] } [/block] ## Procedure The procedure of publishing WDL apps for use on CAVATICA is a process that consists of the following two stages: * **Initial app conversion.** In this step, your WDL app will be converted to a format that is executable on CAVATICA. However, to optimize your app for execution on CAVATICA and make the most out of CAVATICA's execution environment it is _strongly recommended_ to go through the next step. * **Optimizing the converted app for execution in Seven Bridges environments.** The app that has been initially converted now contains an additional configuration file that you will use to define CAVATICA-specific options and fully optimize it for use in the Seven Bridges execution environment. Once the optimized configuration is prepared, the app configuration is pushed to CAVATICA again. ### Initial app conversion This step adapts the WDL app for execution on CAVATICA. It is performed by executing the `sbpack_wdl` command in the following format: [block:code] { "codes": [ { "code": "sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --womtool-input WOMTOOL_INPUT", "language": "shell" } ] } [/block] In the command above, replace the placeholders as follows: * `PROFILE_NAME` is the CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries). * `APPID` specifies the identifier of the app on CAVATICA, in the `{user}/{project}/{app_name}` format. The `{user}` part is your CAVATICA username. The `{project}` part is the project to which you want to push the app and `{app_name}` is the name you want to assign to the app. For example the full app ID can be `rfranklin/my-new-project/my-wdl-app`. If the specified app ID does not exist, it will be created. If it exists, a new [revision (version)](doc:app-versions) of the app will be created. * `WORKFLOW_PATH` needs to be replaced with the path where the WDL app files are located on your local machine. The `WORKFLOW_PATH` folder should include all the dependencies that the workflow needs in order to run. * `ENTRYPOINT` should be replaced with the path to the actual .wdl file, relative to the root of `WORKFLOW_PATH`. * `WOMTOOL_INPUT` should be replaced with the path to the JSON inputs file containing your app's inputs schema generated using [WOMtool](https://cromwell.readthedocs.io/en/develop/WOMtool/). If you don't have the inputs file, replace `--womtool-input` with the `--womtool-path` argument and provide the path to the [WOMtool jar executable](https://github.com/broadinstitute/cromwell/releases) on your local machine, which will generate the inputs and pass them to `sbpack_wdl` automatically. Here is a sample of the command: [block:code] { "codes": [ { "code": "sbpack_wdl --profile cavatica --appid rfranklin/wdl-project/test-app --workflow-path /Users/rfranklin/apps/wdl/demo --entrypoint cellranger_workflow.wdl --womtool-input /Users/rfranklin/apps/wdl/demo/inputs.json", "language": "shell" } ] } [/block] Once executed successfully, this command will convert the WDL app for use on CAVATICA and push it to the CAVATICA project specified as the value of the `--appid` argument. The local directory specified as the value of `--workflow-path` will now contain an additional `sb_wdl_schema.yaml` file. The file contains configuration parameters that can be adjusted and optimized for execution on CAVATICA. Optionally, to avoid pushing the app to CAVATICA at this stage and perform [optimizations for the Seven Bridges execution environment](#section-optimizing-the-converted-app-for-execution-in-seven-bridges-environments) beforehand, use the `--dump-sb-app` flag at the end of the command. For a full list of available arguments to the `sbpack_wdl` command, see the [sbpack_wdl command reference](#section--sbpack_wdl-command-reference). ### Optimizing the converted app for execution in Seven Bridges environments When you have performed the initial conversion step, the generated `sb_wdl_schema.yaml` file is important as it contains confguration parameters that will help you optimize the app for execution on CAVATICA. The file consists of the following major sections: * The initial section that includes general app information and the documentation content describing the app (if any): * `app_content`: Contains details about app's package and WDL file:   * `code_package`: CAVATICA ID of the file that contains the WDL code. This is replaced by the `git_pull` key if the `--no-package` option was used to set a git repository as the source of the app's code. See the [sbpack_wdl command reference](#section--sbpack_wdl-command-reference) for details. * `entrypoint`: Relative path to the file containing the WDL code, relative to the root directory of the ZIP file defined in `code_package`. * `class`: Defines the type of workflow description language used for the app. The value will always be `wdl` for WDL apps. * `cwlVersion`: Defines the version of CWL used to describe the app. The value will always be `None` for WDL apps. * `doc` _(Optional)_: The Markdown-formatted text describing the app.  * The `inputs` section that defines details of the app inputs. * The `outputs` section that defines details of app outputs. * The `requirements` section that defines app execution requirements such as initial working directory. #### Configuring inputs Each of the app inputs that is present in the `inputs` section contains the following basic details: * `id`: Unique identifier of the input. * `inputBinding`: Defines the mapping of the input to the command line of the app that is being executed. If `inputBinding` is omitted, input is made available in the execution environment, but is not passed on to the WDL executor. * `prefix`: The ID of the WDL workflow input, in the form that would be added to the WDL inputs JSON file. For example, if you provide a value for an input that is labelled as `workflow.input_file` in the WDL inputs JSON file, the prefix would be defined as `workflow.input_file.` * `default`: The default value for the input. If the input value is not set on task execution, this default value is taken and passed on to the executor as defined with `inputBinding.prefix`. * `label`: Text description of the input.  * `sbg:toolDefaultValue`: Default value of the input in the WDL workflow. Value provided here is not used in execution and is descriptive (for information purpose) only. * `sbg:fileTypes`: Comma separated (with spaces) value of file extensions that are used in the file picker when setting up tasks. For example: sbg:fileTypes: `“FASTQ, FASTQ.GZ”`. * `type`: The type of value expected on the input. If there is a question mark at the end, for example `int?`, the input is optional. Otherwise, the input is mandatory. Learn more about [available types of inputs](page:about-app-input-and-output-types). To accommodate for the transition between WDL and the Seven Bridges execution environment, the `sb_wdl_schema.yaml` file will always contain an additional input whose ID is `auxiliary_files`, which contains the list of files not added as explicit workflow inputs but are required for workflow execution. To enable proper execution on CAVATICA, please do not remove this input from `sb_wdl_schema.yaml`. **Note: File and Directory inputs** In some WDL versions, `Directory` inputs are marked as `File`. The type of such inputs will consequently be `File` on CAVATICA. If the input should actually be a directory, simply change its `type` property from `File` to `Directory` in `sb_wdl_schema.yaml`. [block:callout] { "type": "info", "title": "", "body": "There may be cases where input types are not properly set by `sbpack_wdl` during initial conversion, especially for complex input types. These cases may need to be handled manually based on each use case." } [/block] #### Configuring outputs In addition to executing WDL apps on CAVATICA, you also need to optimize app outputs to produce and save only files that match the defined criteria, To achieve this and be able to further configure your app outputs, see the details about configuration parameters contained in the `outputs` section of the `sb_wdl_schema.yaml` file: * `id`: Unique identifier of the output. You can change this value to provide a more adequate and descriptive one if necessary. * `outputBinding`: Defines the `glob` expression or pattern that will be used to select the output file. * `glob`: The [glob](page:glob) expression that defines the items to keep as outputs on the output port. * `type`: The type of output value. **Example: Configuring an output** The `sb_wdl_schema.yaml` file always contains one automatically generated app output: [block:code] { "codes": [ { "code": "outputs:\n - id: wdl_workdir\n outputBinding:\n glob: '*'\n type: File[]", "language": "yaml" } ] } [/block] To configure your outputs, create an entry for each of the output ports you want to have, following the pattern below: [block:code] { "codes": [ { "code": "outputs:\n - id: csv_file\n outputBinding:\n glob: '*.csv'\n type: File", "language": "yaml" } ] } [/block] In the example above, replace `csv_file` with an ID that describes your output and replace `*.csv` with the glob that matches the type and naming of the data you want to produce on the output. Learn more about available [types of outputs](page:about-app-input-and-output-types). **Example: Configuring a dynamic output directory name** You can also use the `sb_wdl_schema.yaml` file to set the name of the output directory by defining it in an app input, provided that the tool itself supports the option of defining the output directory name using the corresponding input argument and its value. The first step is to define the input that takes the output directory name (in the `sb_wdl_schema.yaml inputs` section): [block:code] { "codes": [ { "code": "inputs:\n - id: my_workflow_outdir_name\n inputBinding:\n prefix: my_workflow.outdir_name\n type: string?", "language": "yaml" } ] } [/block] Once you have defined the input, define an output, where the output directory `glob` will be a variable that gets the value defined in the input above. [block:code] { "codes": [ { "code": "outputs:\n - id: output_directory\n outputBinding:\n glob: '$(inputs.outdir_name)'\n type: Directory", "language": "yaml" } ] } [/block] The `$(inputs.outdir)` value is a variable that will be replaced with the actual value entered in the `outdir` input when the app is executed. #### Configuring requirements The `requirements` sections is primarily used for the following execution-related parameters: * Setting input staging (making input files available in the app's working directory). * Setting computation instances that are used for app executions on CAVATICA. **Setting input staging** Files that are named as inputs to a tool are not, by default, in the tool's working directory. In most apps this access is sufficient, since most tools only need to read their input files, process the data contained in them, and write new output files on the basis of this data to their working directory. However, in some cases an app might require input files to be placed directly in its working directory. If this is the case with your app, modify the `requirements` section in the `sb_wdl_schema.yaml` file as follows: [block:code] { "codes": [ { "code": "requirements:\n - class: InitialWorkDirRequirement\n listing:\n - '$(inputs.tsv_files)'", "language": "yaml" } ] } [/block] Entries under `listing` define files and directories that will be made available in the app’s working directory before the command is executed. The files and directories are usually defined as variables named after their respective input IDs. Another useful option is creation of a file directly in the working directory. This is done by defining `entryname` and `entry` keys in the `InitialWorkDirRequirement` class, as follows: [block:code] { "codes": [ { "code": "requirements:\n- class: InitialWorkDirRequirement\n listing:\n - entryname: input_nanoseq.csv \n entry: |\n ${\n if (inputs.auxiliary_files && !inputs.in_csv_file){\n var content = 'group,barcode';\n for (var i = 0; i < inputs.auxiliary_files.length; i++){\n if (inputs.auxiliary_files[i].metadata['barcode']){\n var barcode = inputs.auxiliary_files[i].metadata['barcode'];\n }\n else {\n var barcode = '';\n }\n if (inputs.auxiliary_files[i].metadata['group']){\n var group = inputs.auxiliary_files[i].metadata['group'];\n }\n else {\n var group = '';\n }\n content = content.concat(group,',',barcode,'\\\\n');\n }\n return content\n }\n else {\n return ''\n }", "language": "yaml" } ] } [/block] In the example code above, `entryname `defines the name of the file generated in the working directory, which is `input_nanoseq.csv`, while `entry` contains a Javascript expression that populates the generated file by getting `barcode` and `group` metadata values from input files and concatenating them in a single CSV file. As we're using a Javascript expression, it is recommended to use the default option of generating a YAML schema when using the `sbpack_wdl` command, as YAML is more convenient and reduces the possibility of making errors compared to using Javascript expressions in a JSON file. The expression can be defined to match your needs and intended use. Read more about [dynamic expressions in tool descriptions](doc:dynamic-expressions-in-tool-descriptions-1) or see some of the most common expression examples in our [Javascript Cookbook](doc:javascript-cookbook-1). **Setting execution instances** Another useful option that is available for configuration in the `hints` section is the definition of the computation instance used for app execution on CAVATICA. This is also done by defining key-value pairs as follows: [block:code] { "codes": [ { "code": "hints:\n - class: sbg:AWSInstanceType\n value: c4.8xlarge;ebs-gp2;2000", "language": "yaml" } ] } [/block] In this case, the workflow uses a `c4.8xlarge` instance with 2000 GB of attached EBS storage. The value consists of the following three parts (separated by `;`): * Instance type, e.g. `c4.8xlarge`. * Attached disk type: always `ebs-gp2` for all instances with EBS storage. * Disk size in GB. See the list of [AWS](doc:list-of-available-amazon-web-services-instances) instances that are available for task execution on CAVATICA.  ### Pushing the optimized app configuration to CAVATICA When you are done with changes to the `sb_wdl_schema.yaml` file, push the optimized app configuration to CAVATICA. As we just making configuration changes to an app that has already been pushed to CAVATICA, this can be done using the regular `sbpack` command in the following format: [block:code] { "codes": [ { "code": "sbpack PROFILE_NAME APPID CONFIG_FILE", "language": "shell" } ] } [/block] In the command above, `PROFILE_NAME` refers to the CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries). The  `APPID` parameter specifies the identifier of the app on CAVATICA. _Use the same  `APPID` you used in the initial conversion step_. Finally, `CONFIG_FILE` is the `sb_wdl_schema.yaml` in which you made app execution optimizations. The final command should be, for example: [block:code] { "codes": [ { "code": "sbpack cavatica-profile rfranklin/wdl-project/test-app sb_wdl_schema.yaml", "language": "shell" } ] } [/block] This pushes the modified app configuration to CAVATICA and creates a new [revision (version)](doc:app-versions) of the app. Once this is done, you are ready to [run a task](doc:run-a-task-1) using the app. ### Updating already converted and optimized apps If you have already converted your app, made optimizations in the `sb_wdl_schema.yaml` file, and pushed the app to CAVATICA, all subsequent updates to the app's WDL code and the process of propagating the update to CAVATICA are quite straightforward. If the updates you made do not require changes to manually configured parameters in the `sb_wdl_schema.yaml` file (such as inputs, outputs, requirements, etc.), create a new code package by running a command in the following format: [block:code] { "codes": [ { "code": "sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --sb-schema SB_SCHEMA", "language": "shell" } ] } [/block] This command is almost the same as the initial app conversion step, but differs in the additional `--sb-schema` argument. This argument allows you to provide and reuse an existing `sb_wdl_schema.yaml` configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on CAVATICA. The command will generate a new code package based on your updated WDL code provided through `--workflow-path` and `--entrypoint` and the YAML or JSON configuration file provided through `--sb-schema`, and push the updated app to CAVATICA creating a new [revision (version)](doc:app-versions). ### Copying WDL apps between projects on CAVATICA When an app is on CAVATICA, you can copy it and use it on other CAVATICA projects. To copy WDL apps between projects, use the `sbcopy` command that is a part of the `sbpack` utility: [block:code] { "codes": [ { "code": "sbcopy [--profile PROFILE] --appid APPID --projectid PROJECTID", "language": "shell" } ] } [/block] The command takes the following arguments: * `PROFILE`: refers to the CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries). * `APPID`: specifies the identifier of the app on CAVATICA. Takes the form `{user}/{project}/{app_id}`. The `{user}` part is your CAVATICA username. The `{project}` part is the source project where the app is located and `{app_id}` is the ID of the app you want to copy; for example `rfranklin/my-new-project/my-wdl-app`. * `PROJECTID`: is the identifier of the destination project where the app will be copied. Takes the form `{user}/{project}`. The final command should be, for example: [block:code] { "codes": [ { "code": "sbcopy cavatica-profile rfranklin/wdl-project/test-app jsmith/my-wdl-project", "language": "shell" } ] } [/block] [block:callout] { "type": "info", "body": "Note that WDL app copies made through standard [visual interface](https://docs.sevenbridges.com/docs/copy-an-app-from-a-project) or [API](doc:copy-an-app) methods instead of using `sbcopy` will still point to the originally pushed code package and the original project where it is located. This might cause failures due to lack of permissions, if users who need to run the copied instances of the app aren't [added](doc:add-a-collaborator-to-a-project) to the project where the original code package is located. To avoid this, please use `sbcopy` to copy WDL apps between projects on CAVATICA, as described above." } [/block] ## `sbpack_wdl` command reference Here is a list describing all available arguments od the `sbpack_wdl` command that is used to convert and push WDL apps for execution on CAVATICA. [block:parameters] { "data": { "h-0": "Argument", "h-1": "Required", "h-2": "Description", "0-0": "`-h`, `--help`", "0-1": "", "0-2": "Shows the list of all arguments and their corresponding explanations.", "1-0": "`--profile PROFILE`", "1-1": "", "1-2": "CAVATICA profile containing the CAVATICA API endpoint and authentication token, as set in the [Seven Bridges credentials file](doc:store-credentials-to-access-seven-bridges-client-applications-and-libraries). If you are using the default profile, this parameter can be omitted.", "2-0": "`--appid APPID`", "2-1": "Required", "2-2": "The ID of the WDL app once it is pushed to CAVATICA. Takes the form of `{user}/{project}/{app_id}`. The `{user}` part is your CAVATICA username. The `{project}` part is the project to which you want to push the app and `{app_id}` is the ID you want to assign to the app, for example my-`rfranklin/my-new-project/my-wdl-app`.", "3-0": "`--workflow-path WORKFLOW_PATH`", "3-1": "Required", "3-2": "Path to the main workflow directory (the local directory where the app's files are located).", "4-0": "`--entrypoint ENTRYPOINT`", "4-1": "Required", "4-2": "Relative path to the the file that contains the app's WDL code from the main workflow directory defined in `--workflow-path`.", "5-0": "`--sb-package-id SB_PACKAGE_ID`", "5-1": "", "5-2": "ID of an already uploaded package. If you have already converted and pushed the app to CAVATICA, it has its own code package ID, as shown in the `code_package` key in the `sb_wdl_schema.yaml` file. When the package ID is provided, the conversion script will skip the upload step and thus take less time to execute.", "6-0": "`--sb-doc SB_DOC`", "6-1": "", "6-2": "Path to the app description document written in Markdown. The document is meant to provide additional details about the app and will be shown when viewing app details on CAVATICA. If not provided, `README.md` will be used if available in the same directory where `entrypoint` file is located.", "7-0": "`--womtool-input WOMTOOL_INPUT`", "7-1": "Required if `--womtool-path` isn't used", "7-2": "Path to JSON inputs file containing your app's inputs schema generated using [WOMtool](https://cromwell.readthedocs.io/en/develop/WOMtool/).", "8-0": "`--womtool-path WOMTOOL_PATH`", "8-1": "Required if `--womtool-input` isn't used", "8-2": "Path to the [WOMtool](https://cromwell.readthedocs.io/en/stable/WOMtool/) jar file on your local machine. If you don't already have a JSON inputs file that you can provide via `--womtool-input`, this command will use WOMtool to generate the file on the fly and provide it to `sbpack_wdl` automatically.", "10-0": "`--dump-sb-app`", "10-1": "", "10-2": "Dumps the converted app to a local file without pushing it to CAVATICA. Using this option will enable you to convert the app and generate the `sb_wdl_schema.yaml`  file to make configuration optimizations prior to pushing the app to CAVATICA.", "11-0": "`--no-package`", "11-1": "", "11-2": "References a git repository containing the app's code instead of pushing the app's code to CAVATICA. Git repository address is specified as the `git_pull` key in the `sb_wdl_schema.yaml` file, instead of `code_package`. For example: `git_pull: https://git.domain.com/repository`. The value for the `git_pull` key is the URL you would normally use to clone the repository to your local environment.", "12-0": "`--json`", "12-2": "Creates the `sb_wdl_schema` file in JSON format instead of the default YAML.", "9-0": "`--sb-schema SB_SCHEMA`", "9-2": "Path to an existing `sb_wdl_schema` file in JSON or YAML format. This allows you to use an existing configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on CAVATICA." }, "cols": 3, "rows": 13 } [/block] ## Important notes for executing WDL apps on CAVATICA * Workflows are executed in Local mode. Make sure your workflow can run in Local mode before porting it to CAVATICA. * Use of Docker is required. See how to [create and upload a Docker image](doc:upload-your-docker-image) containing your app and make sure to edit the WDL code to [use the newly created image](https://cromwell.readthedocs.io/en/stable/tutorials/Containers#specifying-containers-in-your-workflow). If the Docker image is not specified for a process, a default alpine image will be used. * Execution is done in a dedicated working directory and all the WDL work is done in the `wdl_workdir` directory inside the working directory. Avoid using and relying on hard-coded paths in workflows. ## Differences between running CWL and WDL apps on CAVATICA Executions on CAVATICA normally result in separate [jobs](doc:about-task-execution#section-jobs) (steps) being created for each tool in the workflow. When a WDL pipeline is executed on CAVATICA, a single execution job will be created, regardless of the number of tools within the pipeline. This "one app, one job" approach results in the following differences compared to CWL app executions: * [Memoization](doc:about-memoization) is not available for WDL apps. As memoization relies on using previous job outputs to skip identical jobs in new executions, it is not useful in situations when there is only one job in an app execution. * Handling of [spot instance](doc:about-spot-instances) interruption can't be used with WDL apps. If a spot instance gets terminated when a WDL app is running, the app would have to be rerun on an on-demand instance from the beginning, which would result in increased costs instead of savings. Therefore, it is recommended to disable spot instances for WDL app executions. * [Task stats](doc:view-task-stats) and [logs](doc:view-task-logs) are organized differently. As task statistics and logs are usually shown and organized per job, WDL apps will have cumulative stats for the single job in the execution, while logs will be saved in a single folder.