Sometimes using curl and uploading/downloading with an object storage back end will work just fine but in this case I was looking to tar straight into object storage. One option is using rclone with the rcat command. Some example below.
This test was done using Oracle Cloud Infrastructure Object Storage with an Amazon S3 Compatibility API Key. This test consists of:
- 2 196 914 files
- size using df -h 122G
- local tar/gzip file for comparison 52G
- correct rclone.conf setup for the API Key and OCI policies if required for this user
# rclone ls s3_servers_ashburn:SERVERS 10738097 oci01-20180717_/etc.tgz 2132252 oci01-20180718_/home/opc.tgz 286946 oci01-20180717_/home/opc/terraform.tgz # time tar zcpf - /opt/app2/12.2 | rclone rcat s3_servers_ashburn:SERVERS/oci01-20180718_/opt/app2/12.2.tgz tar: Removing leading `/' from member names real 149m48.812s user 78m13.544s sys 11m42.817s # rclone ls s3_servers_ashburn:SERVERS 10738097 oci01-20180717_/etc.tgz 2132252 oci01-20180718_/home/opc.tgz 40476682243 oci01-20180718_/opt/app2/12.2.tgz 286946 ocil01-20180717_/home/opc/terraform.tgz