Tar to Object Storage Using rclone

July 18, 2018 admin

Sometimes using curl and uploading/downloading with an object storage back end will work just fine but in this case I was looking to tar straight into object storage. One option is using rclone with the rcat command. Some example below.

This test was done using Oracle Cloud Infrastructure Object Storage with an Amazon S3 Compatibility API Key. This test consists of:
- 2 196 914 files
- size using df -h 122G
- local tar/gzip file for comparison 52G
- correct rclone.conf setup for the API Key and OCI policies if required for this user

# rclone ls s3_servers_ashburn:SERVERS
 10738097 oci01-20180717_/etc.tgz
  2132252 oci01-20180718_/home/opc.tgz
   286946 oci01-20180717_/home/opc/terraform.tgz

# time tar zcpf - /opt/app2/12.2 | rclone rcat s3_servers_ashburn:SERVERS/oci01-20180718_/opt/app2/12.2.tgz
tar: Removing leading `/' from member names
real	149m48.812s
user	78m13.544s
sys	11m42.817s

# rclone ls s3_servers_ashburn:SERVERS
 10738097 oci01-20180717_/etc.tgz
  2132252 oci01-20180718_/home/opc.tgz
40476682243 oci01-20180718_/opt/app2/12.2.tgz
   286946 ocil01-20180717_/home/opc/terraform.tgz