Riaan's SysAdmin Blog

My tips, howtos, gotchas, snippets and stuff. Use at your own risk!

rclone

Object Storage Listing with Rclone and jq

Some examples of using rclone and jq to see object listing in a bucket. These examples was using Oracle (OCI) Object Storage but since this is rclone it should not matter what the target is.

Rclone retrieving JSON object listing of a bucket:

$ rclone lsjson -R s3_ashburn:APPS 
[
{"Path":"config","Name":"config","Size":155,"MimeType":"application/octet-stream","ModTime":"2018-11-02T18:01:31.028653533Z","IsDir":false},
{"Path":"data","Name":"data","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533157179Z","IsDir":true},
{"Path":"index","Name":"index","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533226556Z","IsDir":true},
{"Path":"keys","Name":"keys","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533246534Z","IsDir":true},
{"Path":"snapshots","Name":"snapshots","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533266804Z","IsDir":true},
{"Path":"index/6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Name":"6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Size":37828
350,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:05Z","IsDir":false},
{"Path":"index/b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Name":"b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Size":36726
493,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:02Z","IsDir":false},

Use jq select to grab older than certain dates:

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-12-01")|.Name'
ffea09b644533ddcde68a93095bc512646fd0ac0557d39e6e06e004bf73b6bed
ffef7980ade85ea2d9b436c40df46384bbbe8e7e6e71219aff0757ad90f1652f
fff3f56e384ab055c3aa4b6e2dd527c368bf2280863d357e577402460fe9d41a

Use jq csv filter and specific fields

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | [.Name,.Size] | @csv'

Use jq select for older than certain date, specific fields and csv

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-11-01") | [.Name,.Size,.ModTime] | @csv'

Rclone size

$ rclone size s3_ashburn:APPS --json 
{"count":8088,"bytes":38670955795}

Rclone size and jq csv filter

$ rclone size s3_ashburn:APPS --json | jq -r '[.count,.bytes] | @csv'
8088,38670955795

admin

Bio Info for Riaan