{"id":1228,"date":"2018-07-18T16:49:29","date_gmt":"2018-07-18T21:49:29","guid":{"rendered":"http:\/\/blog.ls-al.com\/?p=1228"},"modified":"2018-07-18T16:49:29","modified_gmt":"2018-07-18T21:49:29","slug":"tar-to-object-storage-using-rclone","status":"publish","type":"post","link":"https:\/\/blog.ls-al.com\/tar-to-object-storage-using-rclone\/","title":{"rendered":"Tar to Object Storage Using rclone"},"content":{"rendered":"<p>Sometimes using curl and uploading\/downloading with an object storage back end will work just fine but in this case I was looking to tar straight into object storage.  One option is using rclone with the rcat command.  Some example below.<\/p>\n<p>This test was done using Oracle Cloud Infrastructure Object Storage with an Amazon S3 Compatibility API Key.  This test consists of:<br \/>\n- 2 196 914 files<br \/>\n- size using df -h 122G<br \/>\n- local tar\/gzip file for comparison 52G<br \/>\n- correct rclone.conf setup for the API Key and OCI policies if required for this user<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# rclone ls s3_servers_ashburn:SERVERS\r\n 10738097 oci01-20180717_\/etc.tgz\r\n  2132252 oci01-20180718_\/home\/opc.tgz\r\n   286946 oci01-20180717_\/home\/opc\/terraform.tgz\r\n\r\n# time tar zcpf - \/opt\/app2\/12.2 | rclone rcat s3_servers_ashburn:SERVERS\/oci01-20180718_\/opt\/app2\/12.2.tgz\r\ntar: Removing leading `\/' from member names\r\nreal\t149m48.812s\r\nuser\t78m13.544s\r\nsys\t11m42.817s\r\n\r\n# rclone ls s3_servers_ashburn:SERVERS\r\n 10738097 oci01-20180717_\/etc.tgz\r\n  2132252 oci01-20180718_\/home\/opc.tgz\r\n40476682243 oci01-20180718_\/opt\/app2\/12.2.tgz\r\n   286946 ocil01-20180717_\/home\/opc\/terraform.tgz\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes using curl and uploading\/downloading with an object storage back end will work just fine but in this case I<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[96],"tags":[],"class_list":["post-1228","post","type-post","status-publish","format-standard","hentry","category-rclone"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1228","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1228"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1228\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1228"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1228"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1228"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}