{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","results":{"codes":[]},"settings":"","params":[]},"next":{"description":"","pages":[]},"title":"Upload via the command line","type":"basic","slug":"upload-via-the-command-line","excerpt":"","body":"##Overview\n\nThe Seven Bridges CLI Uploader is the recommended tool for uploading data to the Seven Bridges Platform. It is integrated into the [Seven Bridges CLI](doc:command-line-interface).\n\nThe Command Line Uploader is recommended for large scale uploads. If you need to upload smaller scale uploads instead, we recommend using the [Web uploader](doc:upload-from-your-computer).\n\nThe maximum number of files you can submit for upload is 250,000.  \n\n## COMMAND LINE OPTIONS\n\n**Syntax:**\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb [global-parameters] upload <upload-subcommand> [command-parameters]\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThe supported global parameters are:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Option\",\n    \"h-1\": \"Short parameter\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`--config <string>`\",\n    \"0-1\": \"\",\n    \"0-2\": \"Configuration file to use instead of the default one.\",\n    \"1-0\": \"`--help`\",\n    \"1-1\": \"-h\",\n    \"1-2\": \"Display help.\",\n    \"2-0\": \"`--profile <string>`\",\n    \"2-1\": \"\",\n    \"2-2\": \"Configuration profile to use from credentials file. Please note that this parameter is applicable only to \\\"start\\\" subcommand, \\\"resume\\\" and \\\"delete\\\" subcommands will use the same profile as the one used for starting the upload job, while for \\\"status\\\" subcommand this parameter is not applicable. (default \\\"default\\\")\",\n    \"3-0\": \"`--debug`\",\n    \"3-1\": \"\",\n    \"3-2\": \"Run the command with debug information in the output.\"\n  },\n  \"cols\": 3,\n  \"rows\": 4\n}\n[/block]\n## Subcommands\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Subcommand\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`start`\",\n    \"0-1\": \"Start the upload job\",\n    \"1-0\": \"`status`\",\n    \"1-1\": \"Check the upload status\",\n    \"2-0\": \"`resume`\",\n    \"2-1\": \"Resume the upload job\",\n    \"3-0\": \"`delete`\",\n    \"3-1\": \"Delete an upload job. Please note that only jobs that are paused can be deleted. To pause an upload job, use CTRL+C.\"\n  },\n  \"cols\": 2,\n  \"rows\": 4\n}\n[/block]\nRead below for detailed information and instructions for all of the commands.\n\n## Start the upload\n\nThis command initializes a new upload job. Each job is identified by its unique name which can be assigned manually (using the corresponding command parameter) or automatically. Job name can be used to track the upload status and resume its execution (in case it is paused).\n\nThe upload is performed in the foreground, in the current active command shell session. The command shell session needs to remain open during the upload command execution, otherwise the upload will be paused and can be resumed. During the upload providing any other input will not be possible.\n\nYou can perform multiple uploads (i.e. start multiple upload jobs) on a single machine. The following are the statuses for an upload job:\n\n* **Initializing -**preparing the list of items (files and folders) which will be uploaded\n* **Running**- the items are being uploaded.\n* **Paused** - the upload job can be interrupted by terminating active command shell or issuing the CTRL+C command \n* **Completed**- this status denotes that all items are processed and the upload job is done; it doesn't necessarily mean all items have been uploaded successfully. The log file contains more detailed information about the upload job, including failed items. The [status command](#section-check-the-upload-status) provides an overview of the upload job, including the number of uploaded, failed, skipped, and remaining items.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb [global-parameters] upload start [<input> | --manifest-file <manifest-file>] --destination <path> [command-parameters]\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n### Required parameters\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`--destination <string>`\",\n    \"0-1\": \"Upload destination, which can be project root or a specific folder inside a Platform project. Destination can be specified either by name or ID (e.g. \\\"sbguser/sbgproject\\\" or \\\"sbguser/sbgproject/directory1\\\" or \\\"5dc01c9ae4b01e2d090a700a\\\").\"\n  },\n  \"cols\": 2,\n  \"rows\": 1\n}\n[/block]\n### Optional parameters\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`<input>`\",\n    \"0-1\": \"The Item (file or folder) or list of items to be uploaded. By default, the source folder structure is preserved on the Platform. The `<input>` and `--manifest-file` parameters are mutually exclusive, and one of them must be included within the `start` command.\",\n    \"1-0\": \"`--manifest-file <manifest-file>`\",\n    \"1-1\": \"The manifest file which contains the list of items that will be uploaded as well as the accompanying metadata (only applies to files). By default, the source folder structure is preserved on the Platform after the upload. The `<input>` and `--manifest-file` parameters are mutually exclusive, and one of them must be included within the `start` command. Learn about the [manifest file format](doc:format-of-a-manifest-file).\",\n    \"2-0\": \"`--name <string>`\",\n    \"2-1\": \"The upload job name which must be unique. The allowed characters are A-Za-z0-9_-. The maximum number of characters is 20. If omitted, the name will be automatically assigned using the following format \\\"DAY_DDMMMYYYY_HHMMSS\\\", for example Thu_27Feb2020_231455).\",\n    \"3-0\": \"`--autorename`\",\n    \"3-1\": \"If a file with the same name already exists at the upload destination, an underscore followed by a serial number will be added as a prefix. In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The `--autorename` and `--overwrite` parameters are mutually exclusive, if omitted SKIP will be used as default method.\",\n    \"4-0\": \"`--overwrite`\",\n    \"4-1\": \"If a file with the same name already exists at the upload destination, it will be overwritten by the file that is uploaded.  In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The `--autorename` and `--overwrite` parameters are mutually exclusive, if omitted SKIP will be used as default method.\",\n    \"5-0\": \"`--tag <string>`\",\n    \"5-1\": \"Tag which will be set for each file once its upload is complete. Please note that tagging is not applicable to folders on the Seven Bridges Platform. It is possible to set maximum 32 tags, with each tag being maximum 64 characters long.\",\n    \"6-0\": \"`--chunk-size <int>`\",\n    \"6-1\": \"Preferred size of the upload part in bytes. If omitted, the default value of 64MiB will be used.\\n Unstable network connection: use for example --chunk-size 8000000 (note that min value is 5243000)
.\\n\\nIf you have an unstable network connection, we recommend using '--chunk-size 8000000'.\",\n    \"8-0\": \"`--speed-limit <int>`\",\n    \"8-1\": \"Maximum allowed network bandwidth for the process that executes this upload job. Should be specified in kbps (kilobits per second). If omitted, the maximum possible bandwidth will be used.\",\n    \"7-0\": \"`--parallel`\",\n    \"7-1\": \"Maximum number of parallel file uploads. The allowed range for this value is 1 to 8.  The default value is 8.
\\n\\nIf you have a low upload speed or low (filesystem) read speed (eg. magnetic discs) or if you want to limit system resources used by the uploader, we recommend using `--parallel 4` or `--parallel 2`.\"\n  },\n  \"cols\": 2,\n  \"rows\": 9\n}\n[/block]\nThe following table illustrates the entire naming conflict resolution mechanism:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/c5d4805-f378e95-conflict-resolution-matrix.png\",\n        \"f378e95-conflict-resolution-matrix.png\",\n        471,\n        296,\n        \"#eaebeb\"\n      ]\n    }\n  ]\n}\n[/block]\n## Check the upload status\n\nUse the `status` command to check the upload status. You can check the status for all upload jobs by omitting the `name` parameter.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb [global-parameters] upload status [<name>]\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThis will return the following information about each of the upload jobs:\n\n* Upload job name\n* Status\n* Processed (percent completed, bytes uploaded / total bytes)\n* Average upload speed (total bytes uploaded / total time spent in \"Running\" status)\n* ETA (Estimated Time of Arrival - expected remaining time for job completion)\nTo check a specific upload job, you should specify the `name` of the upload job. This will return more detailed information about the upload job:\n\n* Upload job name\n* Status\n* Log file path\n* Time submitted\n* Upload command (note: if wildcard is used in upload command, expanded command will be displayed)\n* Total number of submitted files\n* Number of uploaded files (successfully uploaded to the Platform)\n* Number of skipped files\n* Number of failed files\n* Number of remaining files (files in \"Queued\" and \"Uploading\" state)\n* Processed (percent completed, bytes uploaded / total bytes)\n* Average upload speed - total bytes uploaded / total time spent in \"Running\" status\n* ETA (Estimated Time of Arrival - expected remaining time for job completion)\nThe information about completed jobs will be kept in job history for at least 1 month after it has been completed or paused, unless it's been deleted (using the `sb upload delete` option, see below). After this time, the completed and paused jobs will be removed from the list of upload jobs.\n\n### Optional parameters\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name \",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`<name>`\",\n    \"0-1\": \"The name of the upload job. Use this parameter to obtain more detailed information about a specified job.\"\n  },\n  \"cols\": 2,\n  \"rows\": 1\n}\n[/block]\n## Resume an upload job\n\nUse this command to resume a previously paused upload job. The upload will resume its execution from where it had been paused.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb [global-parameters] upload resume [<name>]\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n### Optional parameters\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`<name>`\",\n    \"0-1\": \"Specify the name of the paused job you wish to resume. This parameter is required unless there's only one paused job.\"\n  },\n  \"cols\": 2,\n  \"rows\": 1\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\n## Delete an upload job\n\nUse this command to delete a specified upload job or all upload jobs. Only jobs that have been completed or paused can be deleted. To pause the job, use CTRL+C.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb [global-parameters] upload delete [<name> | --all ]\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nPlease keep in mind that an upload job is kept in the job history for (at least) 1 month after it has been completed or paused and is automatically deleted from the list after that time.\n\n### Optional parameters\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`<name>`\",\n    \"0-1\": \"Upload job name, required if the -all parameter is not used. The `<name>` and `-all` parameters are mutually exclusive. One or the other is required.\",\n    \"1-0\": \"`-all`\",\n    \"1-1\": \"Deletes all upload jobs which are COMPLETED or PAUSED. The `<name>` and `-all` parameters are mutually exclusive. One or the other is required.\"\n  },\n  \"cols\": 2,\n  \"rows\": 2\n}\n[/block]\n##Examples\n\nThis section will list several examples on how to upload files. The destination project that is used in all of the examples is:  `rfanklin/my-project`.\n\nPlease note that tags will not be shown in the examples.\n\n###Initial state\n\nThe following snippet shows the example of a directory tree on a local computer (from the local path where sb commands are executed):\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"├── dir1\\n├── dir2\\n│   ├── dir2-1\\n│   │   └── file4.bam\\n│   └── dir2-2\\n├── file1.bam\\n├── file2.bam\\n└── file3.bam\\n\",\n      \"language\": \"text\",\n      \"name\": \"Example of the initial state\"\n    }\n  ]\n}\n[/block]\n \n###  Example 1\n\n Upload a folder and 2 files into the project root and tag the uploaded files.\n\n**Command** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload start dir1/ file1.bam file2.bam --destination rfranklin/my-project --tag upload1\",\n      \"language\": \"c\",\n      \"name\": \"Command\"\n    }\n  ]\n}\n[/block]\n**Terminal output** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"upload job name: Thu_08Apr2021_182030\\nCOMPLETE\\nSuccessfully uploaded 2 of 2 files\\n\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]\n** Result on the Platform**\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"├── dir1\\n├── file1.bam\\n└── file2.bam\\n\",\n      \"language\": \"c\",\n      \"name\": \"Result\"\n    }\n  ]\n}\n[/block]\n### Example 2\n\nUpload a folder (with underlying folder structure and one file included) and 3 files into the existing folder within the project.\n \n * assigning name to the upload job (for easier status tracking)\n  * tagging uploaded files\n  * and specifying a custom setting for the number of parallel file uploads (covering the \n low upload speed case).\n\n\n*Command*\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2\",\n      \"language\": \"c\",\n      \"name\": \"Command\"\n    }\n  ]\n}\n[/block]\n**Terminal output** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"upload job name: upload2\\nCOMPLETE\\nSuccessfully uploaded 4 of 4 files\\n\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]\n**Result on the Platform**\n\nThe following will be the result on the Platform, assuming that the Example 1 (see above) is already executed and that the result remains intact:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"├── dir1\\n│   ├── dir2\\n│   │   ├── dir2-1\\n│   │   │   └── file4.bam\\n│   │   └── dir2-2\\n│   ├── file1.bam\\n│   ├── file2.bam\\n│   └── file3.bam\\n├── file1.bam\\n└── file2.bam\\n\",\n      \"language\": \"c\",\n      \"name\": \"Result\"\n    }\n  ]\n}\n[/block]\n### Example 3\n\nUpload 2 folders (with underlying folder structure and one file included) and 1 file into the project root, while:\n\n  * choosing auto-rename as a method for resolving name conflict\n  * tagging uploaded files\n  * and specifying a custom setting for the chunk size (covering the unstable network connection case).\n\n**Command**\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload start dir1/ dir2/ file1.bam --destination rfranklin/my-project --autorename --tag upload3 --chunk-size 8000000\",\n      \"language\": \"c\",\n      \"name\": \"Command\"\n    }\n  ]\n}\n[/block]\n**Terminal output**\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"upload job name: Thu_08Apr2021_182550\\nCOMPLETE\\nSuccessfully uploaded 2 of 2 files\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]\n**Result on the Platform**\n\nThe following will be the result on the Platform, assuming that the two previous examples are already executed:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"├── _1_file1.bam\\n├── dir1\\n│   ├── dir2\\n│   │   ├── dir2-1\\n│   │   │   └── file4.bam\\n│   │   └── dir2-2\\n│   ├── file1.bam\\n│   ├── file2.bam\\n│   └── file3.bam\\n├── dir2\\n│   ├── dir2-1\\n│   │   └── file4.bam\\n│   └── dir2-2\\n├── file1.bam\\n└── file2.bam\\n\",\n      \"language\": \"c\",\n      \"name\": \"Result\"\n    }\n  ]\n}\n[/block]\n###Example 4\n\nCheck the status for all upload jobs.\n\n**Command** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload status\",\n      \"language\": \"c\",\n      \"name\": \"Command\"\n    }\n  ]\n}\n[/block]\n**Terminal output** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"Upload job name       Status                 Processed  Average speed  Estimated time  \\n                                                                                       \\nThu_08Apr2021_182030  COMPLETED    100% (85.00/85.00B)      85.00 Bps  N/A  \\nupload2               COMPLETED  100% (114.00/114.00B)      38.00 Bps  N/A  \\nThu_08Apr2021_182550  COMPLETED    100% (68.00/68.00B)      68.00 Bps  N/A  \\n\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]\n###Example 5\n\nGet the detailed status for an upload job.\n\n**Command** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload status upload2\",\n      \"language\": \"c\",\n      \"name\": \"Command\"\n    }\n  ]\n}\n[/block]\n**Terminal output** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"Upload job name:       upload2  \\nStatus:                COMPLETED  \\nLog file path:         /home/nikola/.sevenbridges/sb/logs/sb.log  \\nTime submitted:        08.04.2021 18:23  \\nCommand:               sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2  \\nTotal files:           4  \\nTotal size:            114.00 B  \\n# uploaded:            4  \\n# skipped:             0  \\n# failed:              0  \\n# remaining:           0  \\n% processed:           100%  \\nAverage upload speed:  38.00 Bps  \\nETA                    N/A  \\n\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]\n###Example 6\n\nDelete single upload job and check the status.\n\n**Commands**\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload delete upload2\\nsb upload status\",\n      \"language\": \"c\",\n      \"name\": \"Commands\"\n    }\n  ]\n}\n[/block]\n**Terminal output**\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"Upload job name       Status               Processed  Average speed  Estimated time  \\n                                                                                     \\nThu_08Apr2021_182030  COMPLETED  100% (85.00/85.00B)      85.00 Bps  N/A  \\nThu_08Apr2021_182550  COMPLETED  100% (68.00/68.00B)      68.00 Bps  N/A  \\n\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]\n###Example 7\n\nDelete all upload jobs and check the status.\n\n**Commands** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"sb upload delete --all\\nsb upload status\",\n      \"language\": \"c\",\n      \"name\": \"Commands\"\n    }\n  ]\n}\n[/block]\n**Terminal output** \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"Upload job name       Status               Processed  Average speed  Estimated time\",\n      \"language\": \"c\",\n      \"name\": \"Terminal output\"\n    }\n  ]\n}\n[/block]","updates":[],"order":3,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"5a3a57db9fb104001ecd58fe","project":"5773dcfc255e820e00e1cd4d","version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["5773dcfc255e820e00e1cd51","5773df36904b0c0e00ef05ff","577baf92451b1e0e006075ac","577bb183b7ee4a0e007c4e8d","577ce77a1cf3cb0e0048e5ea","577d11865fd4de0e00cc3dab","578e62792c3c790e00937597","578f4fd98335ca0e006d5c84","578f5e5c3d04570e00976ebb","57bc35f7531e000e0075d118","57f801b3760f3a1700219ebb","5804d55d1642890f00803623","581c8d55c0dc651900aa9350","589dcf8ba8c63b3b00c3704f","594cebadd8a2f7001b0b53b2","59a562f46a5d8c00238e309a","5a2aa096e25025003c582b58","5a2e79566c771d003ca0acd4","5a3a5166142db90026f24007","5a3a52b5bcc254001c4bf152","5a3a574a2be213002675c6d2","5a3a66bb2be213002675cb73","5a3a6e4854faf60030b63159","5c8a68278e883901341de571","5cb9971e57bf020024523c7b","5cbf1683e2a36d01d5012ecd","5dc15666a4f788004c5fd7d7","5eaff69e844d67003642a020","5eb00899b36ba5002d35b0c1","5eb0172be179b70073dc936e","5eb01b42b36ba5002d35ebba","5eb01f202654a20136813093","5eb918ef149186021c9a76c8","5f0839d3f4b24e005ebbbc29","5f893e508c9862002d0614a9","6024033e2b2f6f004dfe994c","60a7a12f9a06c70052b7c4db","60a7ab97266a4700161507c4","60b0c84babba720010a8b0b5"],"_id":"5773dcfc255e820e00e1cd50","__v":39,"createdAt":"2016-06-29T14:36:44.812Z","releaseDate":"2016-06-29T14:36:44.812Z","project":"5773dcfc255e820e00e1cd4d"},"category":{"sync":{"isSync":false,"url":""},"pages":[],"title":"Bring data to CAVATICA","slug":"bring-your-data","order":6,"from_sync":false,"reference":false,"_id":"5a3a574a2be213002675c6d2","project":"5773dcfc255e820e00e1cd4d","version":"5773dcfc255e820e00e1cd50","createdAt":"2017-12-20T12:27:54.317Z","__v":1},"user":"566590c83889610d0008a253","createdAt":"2017-12-20T12:30:19.002Z","githubsync":"","__v":0,"parentDoc":null}

Upload via the command line


##Overview The Seven Bridges CLI Uploader is the recommended tool for uploading data to the Seven Bridges Platform. It is integrated into the [Seven Bridges CLI](doc:command-line-interface). The Command Line Uploader is recommended for large scale uploads. If you need to upload smaller scale uploads instead, we recommend using the [Web uploader](doc:upload-from-your-computer). The maximum number of files you can submit for upload is 250,000. ## COMMAND LINE OPTIONS **Syntax:** [block:code] { "codes": [ { "code": "sb [global-parameters] upload <upload-subcommand> [command-parameters]", "language": "shell" } ] } [/block] The supported global parameters are: [block:parameters] { "data": { "h-0": "Option", "h-1": "Short parameter", "h-2": "Description", "0-0": "`--config <string>`", "0-1": "", "0-2": "Configuration file to use instead of the default one.", "1-0": "`--help`", "1-1": "-h", "1-2": "Display help.", "2-0": "`--profile <string>`", "2-1": "", "2-2": "Configuration profile to use from credentials file. Please note that this parameter is applicable only to \"start\" subcommand, \"resume\" and \"delete\" subcommands will use the same profile as the one used for starting the upload job, while for \"status\" subcommand this parameter is not applicable. (default \"default\")", "3-0": "`--debug`", "3-1": "", "3-2": "Run the command with debug information in the output." }, "cols": 3, "rows": 4 } [/block] ## Subcommands [block:parameters] { "data": { "h-0": "Subcommand", "h-1": "Description", "0-0": "`start`", "0-1": "Start the upload job", "1-0": "`status`", "1-1": "Check the upload status", "2-0": "`resume`", "2-1": "Resume the upload job", "3-0": "`delete`", "3-1": "Delete an upload job. Please note that only jobs that are paused can be deleted. To pause an upload job, use CTRL+C." }, "cols": 2, "rows": 4 } [/block] Read below for detailed information and instructions for all of the commands. ## Start the upload This command initializes a new upload job. Each job is identified by its unique name which can be assigned manually (using the corresponding command parameter) or automatically. Job name can be used to track the upload status and resume its execution (in case it is paused). The upload is performed in the foreground, in the current active command shell session. The command shell session needs to remain open during the upload command execution, otherwise the upload will be paused and can be resumed. During the upload providing any other input will not be possible. You can perform multiple uploads (i.e. start multiple upload jobs) on a single machine. The following are the statuses for an upload job: * **Initializing -**preparing the list of items (files and folders) which will be uploaded * **Running**- the items are being uploaded. * **Paused** - the upload job can be interrupted by terminating active command shell or issuing the CTRL+C command  * **Completed**- this status denotes that all items are processed and the upload job is done; it doesn't necessarily mean all items have been uploaded successfully. The log file contains more detailed information about the upload job, including failed items. The [status command](#section-check-the-upload-status) provides an overview of the upload job, including the number of uploaded, failed, skipped, and remaining items. [block:code] { "codes": [ { "code": "sb [global-parameters] upload start [<input> | --manifest-file <manifest-file>] --destination <path> [command-parameters]", "language": "shell" } ] } [/block] ### Required parameters [block:parameters] { "data": { "h-0": "Name", "h-1": "Description", "0-0": "`--destination <string>`", "0-1": "Upload destination, which can be project root or a specific folder inside a Platform project. Destination can be specified either by name or ID (e.g. \"sbguser/sbgproject\" or \"sbguser/sbgproject/directory1\" or \"5dc01c9ae4b01e2d090a700a\")." }, "cols": 2, "rows": 1 } [/block] ### Optional parameters [block:parameters] { "data": { "h-0": "Name", "h-1": "Description", "0-0": "`<input>`", "0-1": "The Item (file or folder) or list of items to be uploaded. By default, the source folder structure is preserved on the Platform. The `<input>` and `--manifest-file` parameters are mutually exclusive, and one of them must be included within the `start` command.", "1-0": "`--manifest-file <manifest-file>`", "1-1": "The manifest file which contains the list of items that will be uploaded as well as the accompanying metadata (only applies to files). By default, the source folder structure is preserved on the Platform after the upload. The `<input>` and `--manifest-file` parameters are mutually exclusive, and one of them must be included within the `start` command. Learn about the [manifest file format](doc:format-of-a-manifest-file).", "2-0": "`--name <string>`", "2-1": "The upload job name which must be unique. The allowed characters are A-Za-z0-9_-. The maximum number of characters is 20. If omitted, the name will be automatically assigned using the following format \"DAY_DDMMMYYYY_HHMMSS\", for example Thu_27Feb2020_231455).", "3-0": "`--autorename`", "3-1": "If a file with the same name already exists at the upload destination, an underscore followed by a serial number will be added as a prefix. In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The `--autorename` and `--overwrite` parameters are mutually exclusive, if omitted SKIP will be used as default method.", "4-0": "`--overwrite`", "4-1": "If a file with the same name already exists at the upload destination, it will be overwritten by the file that is uploaded.  In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The `--autorename` and `--overwrite` parameters are mutually exclusive, if omitted SKIP will be used as default method.", "5-0": "`--tag <string>`", "5-1": "Tag which will be set for each file once its upload is complete. Please note that tagging is not applicable to folders on the Seven Bridges Platform. It is possible to set maximum 32 tags, with each tag being maximum 64 characters long.", "6-0": "`--chunk-size <int>`", "6-1": "Preferred size of the upload part in bytes. If omitted, the default value of 64MiB will be used.\n Unstable network connection: use for example --chunk-size 8000000 (note that min value is 5243000)
.\n\nIf you have an unstable network connection, we recommend using '--chunk-size 8000000'.", "8-0": "`--speed-limit <int>`", "8-1": "Maximum allowed network bandwidth for the process that executes this upload job. Should be specified in kbps (kilobits per second). If omitted, the maximum possible bandwidth will be used.", "7-0": "`--parallel`", "7-1": "Maximum number of parallel file uploads. The allowed range for this value is 1 to 8. The default value is 8.
\n\nIf you have a low upload speed or low (filesystem) read speed (eg. magnetic discs) or if you want to limit system resources used by the uploader, we recommend using `--parallel 4` or `--parallel 2`." }, "cols": 2, "rows": 9 } [/block] The following table illustrates the entire naming conflict resolution mechanism: [block:image] { "images": [ { "image": [ "https://files.readme.io/c5d4805-f378e95-conflict-resolution-matrix.png", "f378e95-conflict-resolution-matrix.png", 471, 296, "#eaebeb" ] } ] } [/block] ## Check the upload status Use the `status` command to check the upload status. You can check the status for all upload jobs by omitting the `name` parameter. [block:code] { "codes": [ { "code": "sb [global-parameters] upload status [<name>]", "language": "shell" } ] } [/block] This will return the following information about each of the upload jobs: * Upload job name * Status * Processed (percent completed, bytes uploaded / total bytes) * Average upload speed (total bytes uploaded / total time spent in "Running" status) * ETA (Estimated Time of Arrival - expected remaining time for job completion) To check a specific upload job, you should specify the `name` of the upload job. This will return more detailed information about the upload job: * Upload job name * Status * Log file path * Time submitted * Upload command (note: if wildcard is used in upload command, expanded command will be displayed) * Total number of submitted files * Number of uploaded files (successfully uploaded to the Platform) * Number of skipped files * Number of failed files * Number of remaining files (files in "Queued" and "Uploading" state) * Processed (percent completed, bytes uploaded / total bytes) * Average upload speed - total bytes uploaded / total time spent in "Running" status * ETA (Estimated Time of Arrival - expected remaining time for job completion) The information about completed jobs will be kept in job history for at least 1 month after it has been completed or paused, unless it's been deleted (using the `sb upload delete` option, see below). After this time, the completed and paused jobs will be removed from the list of upload jobs. ### Optional parameters [block:parameters] { "data": { "h-0": "Name ", "h-1": "Description", "0-0": "`<name>`", "0-1": "The name of the upload job. Use this parameter to obtain more detailed information about a specified job." }, "cols": 2, "rows": 1 } [/block] ## Resume an upload job Use this command to resume a previously paused upload job. The upload will resume its execution from where it had been paused. [block:code] { "codes": [ { "code": "sb [global-parameters] upload resume [<name>]", "language": "shell" } ] } [/block] ### Optional parameters [block:parameters] { "data": { "h-0": "Name", "h-1": "Description", "0-0": "`<name>`", "0-1": "Specify the name of the paused job you wish to resume. This parameter is required unless there's only one paused job." }, "cols": 2, "rows": 1 } [/block] [block:code] { "codes": [ { "code": "", "language": "text" } ] } [/block] ## Delete an upload job Use this command to delete a specified upload job or all upload jobs. Only jobs that have been completed or paused can be deleted. To pause the job, use CTRL+C. [block:code] { "codes": [ { "code": "sb [global-parameters] upload delete [<name> | --all ]", "language": "shell" } ] } [/block] Please keep in mind that an upload job is kept in the job history for (at least) 1 month after it has been completed or paused and is automatically deleted from the list after that time. ### Optional parameters [block:parameters] { "data": { "h-0": "Name", "h-1": "Description", "0-0": "`<name>`", "0-1": "Upload job name, required if the -all parameter is not used. The `<name>` and `-all` parameters are mutually exclusive. One or the other is required.", "1-0": "`-all`", "1-1": "Deletes all upload jobs which are COMPLETED or PAUSED. The `<name>` and `-all` parameters are mutually exclusive. One or the other is required." }, "cols": 2, "rows": 2 } [/block] ##Examples This section will list several examples on how to upload files. The destination project that is used in all of the examples is: `rfanklin/my-project`. Please note that tags will not be shown in the examples. ###Initial state The following snippet shows the example of a directory tree on a local computer (from the local path where sb commands are executed): [block:code] { "codes": [ { "code": "├── dir1\n├── dir2\n│ ├── dir2-1\n│ │ └── file4.bam\n│ └── dir2-2\n├── file1.bam\n├── file2.bam\n└── file3.bam\n", "language": "text", "name": "Example of the initial state" } ] } [/block] ### Example 1 Upload a folder and 2 files into the project root and tag the uploaded files. **Command** [block:code] { "codes": [ { "code": "sb upload start dir1/ file1.bam file2.bam --destination rfranklin/my-project --tag upload1", "language": "c", "name": "Command" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "upload job name: Thu_08Apr2021_182030\nCOMPLETE\nSuccessfully uploaded 2 of 2 files\n", "language": "c", "name": "Terminal output" } ] } [/block] ** Result on the Platform** [block:code] { "codes": [ { "code": "├── dir1\n├── file1.bam\n└── file2.bam\n", "language": "c", "name": "Result" } ] } [/block] ### Example 2 Upload a folder (with underlying folder structure and one file included) and 3 files into the existing folder within the project. * assigning name to the upload job (for easier status tracking) * tagging uploaded files * and specifying a custom setting for the number of parallel file uploads (covering the low upload speed case). *Command* [block:code] { "codes": [ { "code": "sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2", "language": "c", "name": "Command" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "upload job name: upload2\nCOMPLETE\nSuccessfully uploaded 4 of 4 files\n", "language": "c", "name": "Terminal output" } ] } [/block] **Result on the Platform** The following will be the result on the Platform, assuming that the Example 1 (see above) is already executed and that the result remains intact: [block:code] { "codes": [ { "code": "├── dir1\n│ ├── dir2\n│ │ ├── dir2-1\n│ │ │ └── file4.bam\n│ │ └── dir2-2\n│ ├── file1.bam\n│ ├── file2.bam\n│ └── file3.bam\n├── file1.bam\n└── file2.bam\n", "language": "c", "name": "Result" } ] } [/block] ### Example 3 Upload 2 folders (with underlying folder structure and one file included) and 1 file into the project root, while: * choosing auto-rename as a method for resolving name conflict * tagging uploaded files * and specifying a custom setting for the chunk size (covering the unstable network connection case). **Command** [block:code] { "codes": [ { "code": "sb upload start dir1/ dir2/ file1.bam --destination rfranklin/my-project --autorename --tag upload3 --chunk-size 8000000", "language": "c", "name": "Command" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "upload job name: Thu_08Apr2021_182550\nCOMPLETE\nSuccessfully uploaded 2 of 2 files", "language": "c", "name": "Terminal output" } ] } [/block] **Result on the Platform** The following will be the result on the Platform, assuming that the two previous examples are already executed: [block:code] { "codes": [ { "code": "├── _1_file1.bam\n├── dir1\n│ ├── dir2\n│ │ ├── dir2-1\n│ │ │ └── file4.bam\n│ │ └── dir2-2\n│ ├── file1.bam\n│ ├── file2.bam\n│ └── file3.bam\n├── dir2\n│ ├── dir2-1\n│ │ └── file4.bam\n│ └── dir2-2\n├── file1.bam\n└── file2.bam\n", "language": "c", "name": "Result" } ] } [/block] ###Example 4 Check the status for all upload jobs. **Command** [block:code] { "codes": [ { "code": "sb upload status", "language": "c", "name": "Command" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "Upload job name Status Processed Average speed Estimated time \n \nThu_08Apr2021_182030 COMPLETED 100% (85.00/85.00B) 85.00 Bps N/A \nupload2 COMPLETED 100% (114.00/114.00B) 38.00 Bps N/A \nThu_08Apr2021_182550 COMPLETED 100% (68.00/68.00B) 68.00 Bps N/A \n", "language": "c", "name": "Terminal output" } ] } [/block] ###Example 5 Get the detailed status for an upload job. **Command** [block:code] { "codes": [ { "code": "sb upload status upload2", "language": "c", "name": "Command" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "Upload job name: upload2 \nStatus: COMPLETED \nLog file path: /home/nikola/.sevenbridges/sb/logs/sb.log \nTime submitted: 08.04.2021 18:23 \nCommand: sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2 \nTotal files: 4 \nTotal size: 114.00 B \n# uploaded: 4 \n# skipped: 0 \n# failed: 0 \n# remaining: 0 \n% processed: 100% \nAverage upload speed: 38.00 Bps \nETA N/A \n", "language": "c", "name": "Terminal output" } ] } [/block] ###Example 6 Delete single upload job and check the status. **Commands** [block:code] { "codes": [ { "code": "sb upload delete upload2\nsb upload status", "language": "c", "name": "Commands" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "Upload job name Status Processed Average speed Estimated time \n \nThu_08Apr2021_182030 COMPLETED 100% (85.00/85.00B) 85.00 Bps N/A \nThu_08Apr2021_182550 COMPLETED 100% (68.00/68.00B) 68.00 Bps N/A \n", "language": "c", "name": "Terminal output" } ] } [/block] ###Example 7 Delete all upload jobs and check the status. **Commands** [block:code] { "codes": [ { "code": "sb upload delete --all\nsb upload status", "language": "c", "name": "Commands" } ] } [/block] **Terminal output** [block:code] { "codes": [ { "code": "Upload job name Status Processed Average speed Estimated time", "language": "c", "name": "Terminal output" } ] } [/block]