Object Storage Listing with Rclone and jq

January 4, 2019 admin

Some examples of using rclone and jq to see object listing in a bucket. These examples was using Oracle (OCI) Object Storage but since this is rclone it should not matter what the target is.

Rclone retrieving JSON object listing of a bucket:

$ rclone lsjson -R s3_ashburn:APPS 
[
{"Path":"config","Name":"config","Size":155,"MimeType":"application/octet-stream","ModTime":"2018-11-02T18:01:31.028653533Z","IsDir":false},
{"Path":"data","Name":"data","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533157179Z","IsDir":true},
{"Path":"index","Name":"index","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533226556Z","IsDir":true},
{"Path":"keys","Name":"keys","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533246534Z","IsDir":true},
{"Path":"snapshots","Name":"snapshots","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533266804Z","IsDir":true},
{"Path":"index/6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Name":"6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Size":37828
350,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:05Z","IsDir":false},
{"Path":"index/b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Name":"b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Size":36726
493,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:02Z","IsDir":false},

Use jq select to grab older than certain dates:

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-12-01")|.Name'
ffea09b644533ddcde68a93095bc512646fd0ac0557d39e6e06e004bf73b6bed
ffef7980ade85ea2d9b436c40df46384bbbe8e7e6e71219aff0757ad90f1652f
fff3f56e384ab055c3aa4b6e2dd527c368bf2280863d357e577402460fe9d41a

Use jq csv filter and specific fields

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | [.Name,.Size] | @csv'

Use jq select for older than certain date, specific fields and csv

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-11-01") | [.Name,.Size,.ModTime] | @csv'

Rclone size

$ rclone size s3_ashburn:APPS --json 
{"count":8088,"bytes":38670955795}

Rclone size and jq csv filter

$ rclone size s3_ashburn:APPS --json | jq -r '[.count,.bytes] | @csv'
8088,38670955795