Riaan's SysAdmin Blog

My tips, howtos, gotchas, snippets and stuff. Use at your own risk!


Using tar and AWS S3

Example of tar straight to object storage and untar back.

$ tar -cP /ARCHIVE/temp/ | gzip | aws s3 cp - s3://sites2-ziparchives.ls-al.com/temp.tgz

$ aws s3 ls s3://sites2-ziparchives.ls-al.com | grep temp.tgz
2020-05-07 15:40:28    7344192 temp.tgz

$ aws s3 cp s3://sites2-ziparchives.ls-al.com/temp.tgz - | tar zxvp
tar: Removing leading `/' from member names

$ ls ARCHIVE/temp/
'March 30-April 3 Kinder Lesson Plans.pdf'   RCAT

Individual Amazon S3 objects can range in size from 1 byte to 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

When using "aws s3 cp" command you need to specify the --expected-size flag.



Bio Info for Riaan