Set task output destinations

📘
Navigation

Set output destinations via the visual interface

Use output destinations in API calls:

Create a new draft task

Get details of a task

Modify a task

Run a task

Rerun a task

Overview

In order to improve the way your data is organized, when running a task on CAVATICA you have the option of defining where the output files will be placed once the task has been completed. This output location can either be within the folder structure of the project where the task is being executed, but can also be on a volume that is attached to CAVATICA.

Set the output destination

The exact output destination is configured for each task individually, in the Outputs section of the draft task page. Other actions on the draft task page, such as configuring task inputs and app settings, are described in the instructions on how to run a task.

To set the output destinations when running a task, follow the steps below:

Navigate to a project.
Open the Apps tab.
Click Run next to the app you want to run. The draft task page opens.
In the Outputs section define the location of your outputs. There are two main settings:

Main destination path - Select or enter the root path where task outputs will be saved.
Choose subfolder for individual outputs - When enabled, allows you to choose a destination subfolder under the main destination path for each individual output. This enables you to create a more detailed and granular subfolder structure for the outputs.

To understand the relation between the main destination path and subfolders for individual outputs, take a look at the following diagram:

📘
This structure shown in the diagram above can be overridden for individual task outputs, by using paths that start with /. This allows you to set paths that are relative to the root of the output location instead of the main destination path.

For each of the outputs, you can define the destination path by setting existing folders as destinations, creating new folders, and using tokens to create and name folders dynamically. To learn how you can set the destination for your task and task outputs, read the paragraphs below.

By clicking next to the main destination path or an output, you get the Task outputs destination dialog. There are two tabs in the dialog:

Current project - allows you to select a location from the project you are currently working in.
Volumes - allows you to select a location on a volume that is attached to CAVATICA. To be able to use the volume as an output destination, it has to be attached in RW (Read and Write) mode with the Direct Export option enabled, and you need to have permissions to access the volume and write data to it.
Both of the selections listed above also allow you to create a new folder and select it as the output destination.

Apart from selecting existing folders or creating new ones, you can also use tokens to create and name folders dynamically during task runtime. Tokens are placeholders that will be replaced with their corresponding values during task execution. For example, a path that is defined as ./processed/<username> will be resolved as ./processed/rfranklin for a task that is run by a user whose username is rfranklin. As you can see, tokens can be combined with named folders to create paths.

Available tokens

Tokens are used to dynamically define parts of output destination paths and they are replaced with appropriate values during task execution. For example, if you want your task outputs to be stored in a directory named after your username, you will use the <username> token in the output path. If your username is rfranklin, the token will be replaced with this value and the outputs will be placed in a directory named rfranklin.

For a list of available tokens, their descriptions, and samples of their rendered values, see the table below:

Token	Description	Note
`<task_id>`	ID of the task you are executing, for example `ea078c48-f73b-12c3-a43b-4a1ab89e7f15`.
`<task_name>`	Slugified name of the task you are executing, for example `bwa-mem-bundle-run-03-16-2021`.
`<app_slug>`	The slug of the app you are executing, without the `<user>`, `<project>` and `<rev>` parts, for example `whole-exome-sequencing-bwa-gatk-4-0`.
`<app_name>`	Slugified name of the app you are executing, for example `gatk-baserecalibrator`.
`<app_rev>`	App revision number, for example `1`.
`<project_slug>`	Slug of the project in which the task is being executed, for example `my-project`.
`<project_name>`	Slugified name of the project in which the task is being executed, for example `dna2protein-update`.
`<username>`	Your username, for example `rfranklin`.
`<parent_task_id>`	ID of the parent task, for example `ea078c48-12c3-5460-a43b-4a1ab89e7f15`.	Available only for batch tasks.
`<parent_task_name>`	Slugified name of the parent task, for example `samtools-mpileup-parallel-run-03-16-2021`.	Available only for batch tasks.
<metadata:field_name> • `<metadata:sample_id>` • `<metadata:case_id>` • all other metadata fields that are available on CAVATICA • custom metadata fields	Sample ID, Case ID or any of the available metadata fields, including custom fields. Metadata tokens are available only in batch tasks. If used in tasks where batching is disabled, a warning is displayed. Tokens based on metadata fields are always defined in the `<metadata:field_name>` format, for example `<metadata:case_id>`. When rendered, field name is replaced with the corresponding value, for example `<metadata:sample_id>` can be `HCC1143BL`.	Available only for batch tasks.
`<output_id>`	ID of the output for which you are defining the destination, for example `aligned_reads`.
`<output_name>`	Slugified name of the output for which you are defining the destination, for example `summary_metrics`.

Troubleshooting

If a task output shows an error that prevents the task from being executed, here are some guidelines that can help you understand and fix these errors:

The output path contains invalid segments.

This error is usually related to wrong syntax of a token used in the output path. Make sure the tokens you specified in the path are spelled correctly and meet the following requirements:

Belong to the list of valid tokens.
Are available for the type of task you are trying to execute. For example, metadata tokens are available for batch tasks only.

Aliases must be project paths. Volume paths can't be used as aliases.

Aliases are objects within a project that are used as pointers to files that are located on a volume. Think of an alias as a symbolic link on UNIX-like operating systems. Aliases are exclusively project locations, so paths that begin with volume:// can't be used as aliases.

Output folder name contains invalid characters.

This error is displayed only if the invalid characters could not be automatically replaced to create a valid path. To fix the error, make sure that the path contains only valid characters for the output destination you are using:

CAVATICA folder: 0-9, a-Z, A-Z, hyphen (-), underscore (_), period (.). The name must be up to 255 characters long.
Amazon Web Services (AWS) volume folder: 0-9, a-z, A-Z, hyphen (-), underscore (_), period (.), tilde (~), equals (=), colon (:). The name must be at least one character long. Empty folder names are not allowed.
Google Cloud Storage (GCS) volume folder: 0-9, a-z, A-Z, hyphen (-), underscore (_), space ( ). The folder's display name must start and end with a letter or digit. The name must be between 3 and 30 characters. You can nest folders up to 10 (ten) levels deep.

Illegal characters found in the output path, will be automatically replaced.

This is a warning indicating that there are invalid characters in the specified path, but the path will be automatically fixed by replacing invalid characters with valid ones.

Too many nested folders (Google Cloud Storage only).

This indicates that you have exceeded the maximum allowed folder nesting level, which is 10 for a Google Cloud Storage (GCS) location. The limitation is defined by GCS as the cloud storage provider. To fix this error, please make sure you define an output destination that is nested up to 10 levels deep.