{"_id":"5a3a57db9fb104001ecd58fe","project":"5773dcfc255e820e00e1cd4d","version":{"_id":"5773dcfc255e820e00e1cd50","__v":27,"project":"5773dcfc255e820e00e1cd4d","createdAt":"2016-06-29T14:36:44.812Z","releaseDate":"2016-06-29T14:36:44.812Z","categories":["5773dcfc255e820e00e1cd51","5773df36904b0c0e00ef05ff","577baf92451b1e0e006075ac","577bb183b7ee4a0e007c4e8d","577ce77a1cf3cb0e0048e5ea","577d11865fd4de0e00cc3dab","578e62792c3c790e00937597","578f4fd98335ca0e006d5c84","578f5e5c3d04570e00976ebb","57bc35f7531e000e0075d118","57f801b3760f3a1700219ebb","5804d55d1642890f00803623","581c8d55c0dc651900aa9350","589dcf8ba8c63b3b00c3704f","594cebadd8a2f7001b0b53b2","59a562f46a5d8c00238e309a","5a2aa096e25025003c582b58","5a2e79566c771d003ca0acd4","5a3a5166142db90026f24007","5a3a52b5bcc254001c4bf152","5a3a574a2be213002675c6d2","5a3a66bb2be213002675cb73","5a3a6e4854faf60030b63159","5c8a68278e883901341de571","5cb9971e57bf020024523c7b","5cbf1683e2a36d01d5012ecd","5dc15666a4f788004c5fd7d7"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"category":{"_id":"5a3a574a2be213002675c6d2","project":"5773dcfc255e820e00e1cd4d","version":"5773dcfc255e820e00e1cd50","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2017-12-20T12:27:54.317Z","from_sync":false,"order":3,"slug":"bring-your-data","title":"Bring your private data"},"user":"566590c83889610d0008a253","githubsync":"","__v":0,"parentDoc":null,"metadata":{"title":"","description":"","image":[]},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2017-12-20T12:30:19.002Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":2,"body":"##Overview\n\nLarge datasets hosted on corporate or academic clusters and workstations can be uploaded to Cavatica using the Command Line Uploader. This is a fast and secure upload client that has been optimized to efficiently upload files to Cavatica, taking advantage of parallelization where possible.\n\n##Prerequisites\n\nThe command line uploader requires Java 1.8 or newer. Issue the following command in a terminal window.\n\n`$ java -version`\n\nand look for the version number in the first line of the output. It should look something like this:\n\n`java version \"1.8.0_20\"`\n`Java(TM) SE Runtime Environment (build 1.8.0_20-b26)`\n`Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)`\n\n##Install the command line uploader\n\n1. Click **Data** in the top navigation bar and choose **Data Tools**. \n2. Click **Download** in the \"Command Line Uploader\" section. \n3. Unpack the uploader to a directory of your choice. Your home directory is a good default location. To do this, enter: \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"$ tar zxvf ~/Downloads/cavatica-uploader.tgz -C ~\",\n      \"language\": \"text\",\n      \"name\": \"Unpack: \"\n    }\n  ]\n}\n[/block]\n3. Run the uploader with the -h switch to list all the available command line options (listed below) and their usage: \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"$ ~/sbg-uploader/bin/cavatica-uploader.sh -h\",\n      \"language\": \"text\",\n      \"name\": \"Run:\"\n    }\n  ]\n}\n[/block]\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n##Command line options\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"cavatica-uploader.sh [-h] [-l] -p id [-t token] [-u username] [-x url] [-f project_folder] [-pf] file/folder [- -tag \\\"enter tag here\\\"]\",\n      \"language\": \"text\",\n      \"name\": \"Syntax\"\n    }\n  ]\n}\n[/block]\n\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Option\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"-h\\n- -help &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\",\n    \"0-1\": \"This option prints a short usage summary and will cause the uploader to ignore any other options. It will exit with a status of 0 (see Exit Statuses below).\",\n    \"1-0\": \"-l\\n- -list-projects\",\n    \"1-1\": \"This option gives you a list of projects available as upload targets with the following two columns:\\n\\n* **The project identifier** which is used for specifying the target project (e.g. `rfranklin/samtools` where \\\"rfranklin\\\" is the project owner and \\\"samtools\\\" is the name of the project).\\n* **The project name** (e.g. “my new project”).\",\n    \"2-0\": \"-p <id>\\n- -project <id>\",\n    \"2-1\": \"This option is used for choosing the target project by specifying its unique identifier, e.g. `username/projectname`.\\n\\nUse the `- -list-projects` option to find the project identifier for your project.\\n\\nTo upload files to a project, you must be a member of that project and must have the write permission granted by the project administrator. \\n\\nThis option is mandatory.\",\n    \"3-0\": \"-t <token>\\n- -token <token>\",\n    \"3-1\": \"This option is used to specify an authorization token. This option overrides the credentials read from the configuration file (see below).\",\n    \"4-0\": \"-u <username>\\n- -user <username>\",\n    \"4-1\": \"This option is used to specify a username for Cavatica. If omitted and not using the - -token option, you will be prompted for a username.\\n*This option can be used only by users who haven't connected their Cavatica account with an eRA account.*\",\n    \"5-0\": \"-x <url>\\n- -proxy <url>\",\n    \"5-1\": \"This option specifies a proxy server through which the uploader should connect. The proxy parameter should be of the form proto://[username:password:::at:::]host[:port]. proto can be 'http’ or 'socks’. HTTP proxies must allow the CONNECT command to port 443. SOCKS proxies can be both SOCKS4 and SOCKS5.\\n  * username and password are optional and will be used if the proxy requires authentication.\\n  * host is required.\\n  * port is optional. If omitted, the uploader will use 8080 for HTTP and 1080 for SOCKS proxies.\",\n    \"8-0\": \"- -tag\",\n    \"8-1\": \"Use this option to enter tags composed of strings for your files. Format your tags as such: `- -tag \\\"first tag here\\\" - -tag \\\"second tag here\\\" - -tag \\\"third tag here\\\"`\",\n    \"9-0\": \"- -list-tags\",\n    \"9-1\": \"Use this option to list all the tags in your destination project.\",\n    \"10-0\": \"-mm\\n- -manifest-metadata\",\n    \"10-1\": \"Use this option to upload multiple files and set their metadata using the [manifest file](doc:set-metadata-using-the-command-line-uploader#section-set-metadata-for-multiple-files-using-a-manifest-file).\\n\\nTo only apply individual metadata fields from the manifest, list them after the `- -manifest-metadata` option, e.g. `- -manifest-metadata sample paired_end`.\\n\\nThis option should be used in combination with the `- -manifest-file` option.\",\n    \"11-0\": \"-mf\\n- -manifest-file\",\n    \"11-1\": \"Use this option to specify the name of the manifest file. \\n\\nThis option should be used in combination with `- -manifest-metadata` , e.g. `- -manifest-metadata - -manifest-file filename.csv`.\\n \\nTo only upload files while omitting the metadata use `- -manifest-file filename.csv`.\",\n    \"12-0\": \"- -dry-run\",\n    \"12-1\": \"Use this option to only output data in the terminal and check the settings without uploading anything.\\n\\nTo output information about specific metadata fields, list them after the `- -dry-run` option, e.g. `- -dry-run sample library`.\",\n    \"6-0\": \"-f <project_folder>\\n- -folder <project_folder>\",\n    \"6-1\": \"Use this option to specify the destination folder in the target project on Cavatica, into which the items will be uploaded. If the specified folder does not exist in the destination project on Cavatica, it is created. If it does exist, items are uploaded into the specified folder.\",\n    \"7-0\": \"-pf\\n- -preserve-folder\",\n    \"7-1\": \"Specify one or more local folders to upload to Cavatica, while replicating the folder structure in the destination project based on the structure on your local machine. This option can be combined with `-f` - you can specify the destination folder into which to upload the local folder structure. If `-pf` is not used, the folder structure will be flattened when uploaded to the destination project on Cavatica.\"\n  },\n  \"cols\": 2,\n  \"rows\": 13\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"* Allowed characters in **file names** are all alphanumerics and special characters except slash (`/`). File names can’t be longer than 255 characters.\\n* Allowed characters in **folder names** are `a-z`, `A-Z`, `0-9`, and special characters (`_`), (`-`), and (`.`). Folder names can’t be longer than 255 characters.\"\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"success\",\n  \"title\": \"Tagging files\",\n  \"body\": \"You can also tag files as you upload them [via Desktop Uploader](upload-via-desktop-uploader) or [from an FTP or HTTP(S) server](upload-from-an-ftp-server).\\n\\n Learn more about [tagging your files](tag-your-files) in general on Cavatica and how tags are beneficial to organizing your data.\"\n}\n[/block]\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n### Upload command examples\n\nLet's assume we want to upload three files into a folder named `samples` that does not currently exist within our project on Cavatica:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples sample1.fastq  sample2.fastq  sample3.fastq\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThe folder is automatically created and the three .fastq files are uploaded into the folder. The resulting folder structure is as follows:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/012c2b6-sbpla-upload-via-the-command-line-1.jpg\",\n        \"sbpla-upload-via-the-command-line-1.jpg\",\n        206,\n        257,\n        \"#dde7ee\"\n      ]\n    }\n  ]\n}\n[/block]\nNow we are uploading additional two files into the same samples directory:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples sample1.fastq sample4.fastq\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nAs `sample1.fastq` has the same name and size as the file that is already present at the destination, it is skipped. The other file, `sample4.fastq` is uploaded to the folder within the project.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/93a6f97-sbpla-upload-via-the-command-line-2.jpg\",\n        \"sbpla-upload-via-the-command-line-2.jpg\",\n        207,\n        327,\n        \"#dee8ee\"\n      ],\n      \"border\": false\n    }\n  ]\n}\n[/block]\n If the size of `sample1.fastq` was different from the size of the file already present on Cavatica, the file would be uploaded and automatically prefixed with a number and underscore, e.g. `1_sample1.fastq`.\n\nLet's say we now want to add a file to a non-existing subfolder on Cavatica:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples/march/week4 sample5.fastq\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\nThis command will result in creation of the `march/week4` subfolder structure within the target folder, while the specified file will be uploaded into the subfolder named `week4`.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/1c1fa78-sbpla-upload-via-the-command-line-3.jpg\",\n        \"sbpla-upload-via-the-command-line-3.jpg\",\n        324,\n        529,\n        \"#eaf0f5\"\n      ],\n      \"border\": false\n    }\n  ]\n}\n[/block]\nFinally, let's assume that we want to upload another folder to the samples folder on Cavatica and preserve its folder structure at the same time:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples -pf april\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nAssuming that the structure of the `april` folder is:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/e93559a-sbpla-upload-via-the-command-line-4.jpg\",\n        \"sbpla-upload-via-the-command-line-4.jpg\",\n        272,\n        188,\n        \"#e6eef4\"\n      ]\n    }\n  ]\n}\n[/block]\nThe final folder structure on Cavatica is:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/415b573-sbpla-upload-via-the-command-line-5.jpg\",\n        \"sbpla-upload-via-the-command-line-5.jpg\",\n        326,\n        737,\n        \"#eaf0f5\"\n      ]\n    }\n  ]\n}\n[/block]\n\n###Upload multiple files\n\n To upload larger volumes of data, use the command line uploader with a [manifest file](doc:set-metadata-using-the-command-line-uploader#section-upload-files-without-setting-metadata).\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"cavatica-uploader.sh -u username - -manifest-file filename.csv\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\n##Authentication\n[block:callout]\n{\n  \"type\": \"success\",\n  \"body\": \"You can obtain [an authentication token ](doc:get-your-authentication-token) for your Cavatica account from the Developer Dashboard.\",\n  \"title\": \"Getting the authentication token\"\n}\n[/block]\nThe Command Line Uploader looks for credentials in the following locations in order:\nIf the -u username option is given, the uploader prompts for and reads the password from standard input.\n\n1. If the -t token option is given, the uploader uses your authentication token.\n2. The uploader looks for an authentication token in the configuration file (see below).\n3. The uploader looks for the username and password in the configuration file (see below).\n4. The uploader prompts for and reads the username and password from standard input.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n##Configuration files\nTo avoid providing your credentials each time you use the uploader, you can store them in a configuration file. This file is called .sbgrc and resides in your home directory. This location varies across operating systems, but would typically be:\n\n/home/$USER/.sbgrc on UNIX;\n/Users/$USER/.sbgrc on OS X;\nC:\\Documents and Settings\\%USERNAME% on Windows XP, 2000 and 2003; and\nC:\\Users\\%USERNAME% on Windows Vista, 7, 8 and 10.\n\nThe .sbgrc configuration file should contain key-value pairs of the following form:\n`username = johndoe`\n`password = supersecret123`\n`auth-token = ec43d6dce3c54193ac18e3855f734ccf`\n\n\nYou can specify the username and password, or the authentication token, or both. If both are given, then the authentication token will take precedence. The uploader will use these values only if no other authentication options are provided on the command line.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"Please keep in mind that `.sbgrc` configuration file may only contain a single set of credentials. If multiple \\\"username\\\", \\\"password\\\" or \\\"token\\\" lines are encountered, the uploader will disregard all values but the last.\"\n}\n[/block]\n##Metadata\nFiles on Cavatica are accompanied by metadata describing, amongst other things, their file type, origin, sample ID, and information about the sequencing technology used to create it. This metadata is often required by tools and workflows, and must be set before a file becomes fully usable.\n\nClick [here](doc:set-metadata-using-the-command-line-uploader) to learn how to set metadata using the command line uploader. You can also [manually set metadata later](doc:set-metadata-using-the-visual-interface).\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n##Exit statuses\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Code\",\n    \"h-1\": \"Status\",\n    \"0-0\": \"0\",\n    \"1-0\": \"100\",\n    \"2-0\": \"101\",\n    \"3-0\": \"102\",\n    \"4-0\": \"103\",\n    \"5-0\": \"104\",\n    \"6-0\": \"200\",\n    \"0-1\": \"Normal termination. The upload has either finished successfully, or usage information was written to standard output.\",\n    \"1-1\": \"The upload has failed in the pre-processing phase or the uploader was unable to initialize it properly.\",\n    \"2-1\": \"Input arguments were not properly set.\",\n    \"3-1\": \"Mandatory options were not set.\",\n    \"4-1\": \"Authentication error; invalid user credentials were used.\",\n    \"5-1\": \"Bad metadata file.\",\n    \"6-1\": \"Abnormal termination; an unknown error caused the upload to fail.\"\n  },\n  \"cols\": 2,\n  \"rows\": 7\n}\n[/block]\nFor example, if you want to use the Command Line Uploader to upload the FASTQ file sample1.fastq, with the associated metadata file, sample1.fastq.meta, to a project whose ID is 1234. Then, you should enter\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"cavatica-uploader$ bin/cavatica-uploader -t $AUTH_TOKEN -p 1234 sample1.fastq - -tag \\\"fastq\\\" - -tag \\\"sample 1\\\"\",\n      \"language\": \"text\",\n      \"name\": \"Example:\"\n    }\n  ]\n}\n[/block]\nand replace `$AUTH_TOKEN` with your own authentication token.\n[block:callout]\n{\n  \"type\": \"success\",\n  \"body\": \"As shown in the example above, don't forget to change directory to the one containing the sbg-uploader, and to prefix the executable name with `bin/`.\"\n}\n[/block]","excerpt":"","slug":"upload-via-the-command-line","type":"basic","title":"Upload via the command line"}

Upload via the command line


##Overview Large datasets hosted on corporate or academic clusters and workstations can be uploaded to Cavatica using the Command Line Uploader. This is a fast and secure upload client that has been optimized to efficiently upload files to Cavatica, taking advantage of parallelization where possible. ##Prerequisites The command line uploader requires Java 1.8 or newer. Issue the following command in a terminal window. `$ java -version` and look for the version number in the first line of the output. It should look something like this: `java version "1.8.0_20"` `Java(TM) SE Runtime Environment (build 1.8.0_20-b26)` `Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)` ##Install the command line uploader 1. Click **Data** in the top navigation bar and choose **Data Tools**. 2. Click **Download** in the "Command Line Uploader" section. 3. Unpack the uploader to a directory of your choice. Your home directory is a good default location. To do this, enter: [block:code] { "codes": [ { "code": "$ tar zxvf ~/Downloads/cavatica-uploader.tgz -C ~", "language": "text", "name": "Unpack: " } ] } [/block] 3. Run the uploader with the -h switch to list all the available command line options (listed below) and their usage: [block:code] { "codes": [ { "code": "$ ~/sbg-uploader/bin/cavatica-uploader.sh -h", "language": "text", "name": "Run:" } ] } [/block] <div align="right"><a href="#top">top</a></div> ##Command line options [block:code] { "codes": [ { "code": "cavatica-uploader.sh [-h] [-l] -p id [-t token] [-u username] [-x url] [-f project_folder] [-pf] file/folder [- -tag \"enter tag here\"]", "language": "text", "name": "Syntax" } ] } [/block] [block:parameters] { "data": { "h-0": "Option", "h-1": "Description", "0-0": "-h\n- -help &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;", "0-1": "This option prints a short usage summary and will cause the uploader to ignore any other options. It will exit with a status of 0 (see Exit Statuses below).", "1-0": "-l\n- -list-projects", "1-1": "This option gives you a list of projects available as upload targets with the following two columns:\n\n* **The project identifier** which is used for specifying the target project (e.g. `rfranklin/samtools` where \"rfranklin\" is the project owner and \"samtools\" is the name of the project).\n* **The project name** (e.g. “my new project”).", "2-0": "-p <id>\n- -project <id>", "2-1": "This option is used for choosing the target project by specifying its unique identifier, e.g. `username/projectname`.\n\nUse the `- -list-projects` option to find the project identifier for your project.\n\nTo upload files to a project, you must be a member of that project and must have the write permission granted by the project administrator. \n\nThis option is mandatory.", "3-0": "-t <token>\n- -token <token>", "3-1": "This option is used to specify an authorization token. This option overrides the credentials read from the configuration file (see below).", "4-0": "-u <username>\n- -user <username>", "4-1": "This option is used to specify a username for Cavatica. If omitted and not using the - -token option, you will be prompted for a username.\n*This option can be used only by users who haven't connected their Cavatica account with an eRA account.*", "5-0": "-x <url>\n- -proxy <url>", "5-1": "This option specifies a proxy server through which the uploader should connect. The proxy parameter should be of the form proto://[username:password@]host[:port]. proto can be 'http’ or 'socks’. HTTP proxies must allow the CONNECT command to port 443. SOCKS proxies can be both SOCKS4 and SOCKS5.\n * username and password are optional and will be used if the proxy requires authentication.\n * host is required.\n * port is optional. If omitted, the uploader will use 8080 for HTTP and 1080 for SOCKS proxies.", "8-0": "- -tag", "8-1": "Use this option to enter tags composed of strings for your files. Format your tags as such: `- -tag \"first tag here\" - -tag \"second tag here\" - -tag \"third tag here\"`", "9-0": "- -list-tags", "9-1": "Use this option to list all the tags in your destination project.", "10-0": "-mm\n- -manifest-metadata", "10-1": "Use this option to upload multiple files and set their metadata using the [manifest file](doc:set-metadata-using-the-command-line-uploader#section-set-metadata-for-multiple-files-using-a-manifest-file).\n\nTo only apply individual metadata fields from the manifest, list them after the `- -manifest-metadata` option, e.g. `- -manifest-metadata sample paired_end`.\n\nThis option should be used in combination with the `- -manifest-file` option.", "11-0": "-mf\n- -manifest-file", "11-1": "Use this option to specify the name of the manifest file. \n\nThis option should be used in combination with `- -manifest-metadata` , e.g. `- -manifest-metadata - -manifest-file filename.csv`.\n \nTo only upload files while omitting the metadata use `- -manifest-file filename.csv`.", "12-0": "- -dry-run", "12-1": "Use this option to only output data in the terminal and check the settings without uploading anything.\n\nTo output information about specific metadata fields, list them after the `- -dry-run` option, e.g. `- -dry-run sample library`.", "6-0": "-f <project_folder>\n- -folder <project_folder>", "6-1": "Use this option to specify the destination folder in the target project on Cavatica, into which the items will be uploaded. If the specified folder does not exist in the destination project on Cavatica, it is created. If it does exist, items are uploaded into the specified folder.", "7-0": "-pf\n- -preserve-folder", "7-1": "Specify one or more local folders to upload to Cavatica, while replicating the folder structure in the destination project based on the structure on your local machine. This option can be combined with `-f` - you can specify the destination folder into which to upload the local folder structure. If `-pf` is not used, the folder structure will be flattened when uploaded to the destination project on Cavatica." }, "cols": 2, "rows": 13 } [/block] [block:callout] { "type": "info", "body": "* Allowed characters in **file names** are all alphanumerics and special characters except slash (`/`). File names can’t be longer than 255 characters.\n* Allowed characters in **folder names** are `a-z`, `A-Z`, `0-9`, and special characters (`_`), (`-`), and (`.`). Folder names can’t be longer than 255 characters." } [/block] [block:callout] { "type": "success", "title": "Tagging files", "body": "You can also tag files as you upload them [via Desktop Uploader](upload-via-desktop-uploader) or [from an FTP or HTTP(S) server](upload-from-an-ftp-server).\n\n Learn more about [tagging your files](tag-your-files) in general on Cavatica and how tags are beneficial to organizing your data." } [/block] <div align="right"><a href="#top">top</a></div> ### Upload command examples Let's assume we want to upload three files into a folder named `samples` that does not currently exist within our project on Cavatica: [block:code] { "codes": [ { "code": "./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples sample1.fastq sample2.fastq sample3.fastq", "language": "shell" } ] } [/block] The folder is automatically created and the three .fastq files are uploaded into the folder. The resulting folder structure is as follows: [block:image] { "images": [ { "image": [ "https://files.readme.io/012c2b6-sbpla-upload-via-the-command-line-1.jpg", "sbpla-upload-via-the-command-line-1.jpg", 206, 257, "#dde7ee" ] } ] } [/block] Now we are uploading additional two files into the same samples directory: [block:code] { "codes": [ { "code": "./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples sample1.fastq sample4.fastq", "language": "shell" } ] } [/block] As `sample1.fastq` has the same name and size as the file that is already present at the destination, it is skipped. The other file, `sample4.fastq` is uploaded to the folder within the project. [block:image] { "images": [ { "image": [ "https://files.readme.io/93a6f97-sbpla-upload-via-the-command-line-2.jpg", "sbpla-upload-via-the-command-line-2.jpg", 207, 327, "#dee8ee" ], "border": false } ] } [/block] If the size of `sample1.fastq` was different from the size of the file already present on Cavatica, the file would be uploaded and automatically prefixed with a number and underscore, e.g. `1_sample1.fastq`. Let's say we now want to add a file to a non-existing subfolder on Cavatica: [block:code] { "codes": [ { "code": "./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples/march/week4 sample5.fastq", "language": "text" } ] } [/block] This command will result in creation of the `march/week4` subfolder structure within the target folder, while the specified file will be uploaded into the subfolder named `week4`. [block:image] { "images": [ { "image": [ "https://files.readme.io/1c1fa78-sbpla-upload-via-the-command-line-3.jpg", "sbpla-upload-via-the-command-line-3.jpg", 324, 529, "#eaf0f5" ], "border": false } ] } [/block] Finally, let's assume that we want to upload another folder to the samples folder on Cavatica and preserve its folder structure at the same time: [block:code] { "codes": [ { "code": "./cavatica-uploader.sh -t token1 -p usr1/prj1 -f samples -pf april", "language": "shell" } ] } [/block] Assuming that the structure of the `april` folder is: [block:image] { "images": [ { "image": [ "https://files.readme.io/e93559a-sbpla-upload-via-the-command-line-4.jpg", "sbpla-upload-via-the-command-line-4.jpg", 272, 188, "#e6eef4" ] } ] } [/block] The final folder structure on Cavatica is: [block:image] { "images": [ { "image": [ "https://files.readme.io/415b573-sbpla-upload-via-the-command-line-5.jpg", "sbpla-upload-via-the-command-line-5.jpg", 326, 737, "#eaf0f5" ] } ] } [/block] ###Upload multiple files To upload larger volumes of data, use the command line uploader with a [manifest file](doc:set-metadata-using-the-command-line-uploader#section-upload-files-without-setting-metadata). [block:code] { "codes": [ { "code": "cavatica-uploader.sh -u username - -manifest-file filename.csv", "language": "text" } ] } [/block] ##Authentication [block:callout] { "type": "success", "body": "You can obtain [an authentication token ](doc:get-your-authentication-token) for your Cavatica account from the Developer Dashboard.", "title": "Getting the authentication token" } [/block] The Command Line Uploader looks for credentials in the following locations in order: If the -u username option is given, the uploader prompts for and reads the password from standard input. 1. If the -t token option is given, the uploader uses your authentication token. 2. The uploader looks for an authentication token in the configuration file (see below). 3. The uploader looks for the username and password in the configuration file (see below). 4. The uploader prompts for and reads the username and password from standard input. <div align="right"><a href="#top">top</a></div> ##Configuration files To avoid providing your credentials each time you use the uploader, you can store them in a configuration file. This file is called .sbgrc and resides in your home directory. This location varies across operating systems, but would typically be: /home/$USER/.sbgrc on UNIX; /Users/$USER/.sbgrc on OS X; C:\Documents and Settings\%USERNAME% on Windows XP, 2000 and 2003; and C:\Users\%USERNAME% on Windows Vista, 7, 8 and 10. The .sbgrc configuration file should contain key-value pairs of the following form: `username = johndoe` `password = supersecret123` `auth-token = ec43d6dce3c54193ac18e3855f734ccf` You can specify the username and password, or the authentication token, or both. If both are given, then the authentication token will take precedence. The uploader will use these values only if no other authentication options are provided on the command line. <div align="right"><a href="#top">top</a></div> [block:callout] { "type": "info", "body": "Please keep in mind that `.sbgrc` configuration file may only contain a single set of credentials. If multiple \"username\", \"password\" or \"token\" lines are encountered, the uploader will disregard all values but the last." } [/block] ##Metadata Files on Cavatica are accompanied by metadata describing, amongst other things, their file type, origin, sample ID, and information about the sequencing technology used to create it. This metadata is often required by tools and workflows, and must be set before a file becomes fully usable. Click [here](doc:set-metadata-using-the-command-line-uploader) to learn how to set metadata using the command line uploader. You can also [manually set metadata later](doc:set-metadata-using-the-visual-interface). <div align="right"><a href="#top">top</a></div> ##Exit statuses [block:parameters] { "data": { "h-0": "Code", "h-1": "Status", "0-0": "0", "1-0": "100", "2-0": "101", "3-0": "102", "4-0": "103", "5-0": "104", "6-0": "200", "0-1": "Normal termination. The upload has either finished successfully, or usage information was written to standard output.", "1-1": "The upload has failed in the pre-processing phase or the uploader was unable to initialize it properly.", "2-1": "Input arguments were not properly set.", "3-1": "Mandatory options were not set.", "4-1": "Authentication error; invalid user credentials were used.", "5-1": "Bad metadata file.", "6-1": "Abnormal termination; an unknown error caused the upload to fail." }, "cols": 2, "rows": 7 } [/block] For example, if you want to use the Command Line Uploader to upload the FASTQ file sample1.fastq, with the associated metadata file, sample1.fastq.meta, to a project whose ID is 1234. Then, you should enter [block:code] { "codes": [ { "code": "cavatica-uploader$ bin/cavatica-uploader -t $AUTH_TOKEN -p 1234 sample1.fastq - -tag \"fastq\" - -tag \"sample 1\"", "language": "text", "name": "Example:" } ] } [/block] and replace `$AUTH_TOKEN` with your own authentication token. [block:callout] { "type": "success", "body": "As shown in the example above, don't forget to change directory to the one containing the sbg-uploader, and to prefix the executable name with `bin/`." } [/block]