{"id":1282,"date":"2019-01-04T10:59:31","date_gmt":"2019-01-04T16:59:31","guid":{"rendered":"http:\/\/blog.ls-al.com\/?p=1282"},"modified":"2019-01-04T11:03:26","modified_gmt":"2019-01-04T17:03:26","slug":"object-storage-listing-with-rclone-and-jq","status":"publish","type":"post","link":"https:\/\/blog.ls-al.com\/object-storage-listing-with-rclone-and-jq\/","title":{"rendered":"Object Storage Listing with Rclone and jq"},"content":{"rendered":"\n<p>Some examples of using rclone and jq to see object listing in a bucket. These examples was using Oracle (OCI) Object Storage but since this is rclone it should not matter what the target is.<\/p>\n\n\n\n<p><strong>Rclone retrieving JSON object listing of a bucket:<\/strong><br><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ rclone lsjson -R s3_ashburn:APPS \n&#x5B;\n{\"Path\":\"config\",\"Name\":\"config\",\"Size\":155,\"MimeType\":\"application\/octet-stream\",\"ModTime\":\"2018-11-02T18:01:31.028653533Z\",\"IsDir\":false},\n{\"Path\":\"data\",\"Name\":\"data\",\"Size\":0,\"MimeType\":\"inode\/directory\",\"ModTime\":\"2019-01-04T15:31:54.533157179Z\",\"IsDir\":true},\n{\"Path\":\"index\",\"Name\":\"index\",\"Size\":0,\"MimeType\":\"inode\/directory\",\"ModTime\":\"2019-01-04T15:31:54.533226556Z\",\"IsDir\":true},\n{\"Path\":\"keys\",\"Name\":\"keys\",\"Size\":0,\"MimeType\":\"inode\/directory\",\"ModTime\":\"2019-01-04T15:31:54.533246534Z\",\"IsDir\":true},\n{\"Path\":\"snapshots\",\"Name\":\"snapshots\",\"Size\":0,\"MimeType\":\"inode\/directory\",\"ModTime\":\"2019-01-04T15:31:54.533266804Z\",\"IsDir\":true},\n{\"Path\":\"index\/6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533\",\"Name\":\"6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533\",\"Size\":37828\n350,\"MimeType\":\"application\/octet-stream\",\"ModTime\":\"2019-01-03T21:27:05Z\",\"IsDir\":false},\n{\"Path\":\"index\/b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae\",\"Name\":\"b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae\",\"Size\":36726\n493,\"MimeType\":\"application\/octet-stream\",\"ModTime\":\"2019-01-03T21:27:02Z\",\"IsDir\":false},\n<\/pre><\/div>\n\n\n<p><strong>Use jq select to grab older than certain dates:<\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ rclone lsjson -R s3_ashburn:APPS | jq -r &#039;.&#x5B;] | select (.&quot;ModTime&quot; &lt; &quot;2018-12-01&quot;)|.Name&#039;\nffea09b644533ddcde68a93095bc512646fd0ac0557d39e6e06e004bf73b6bed\nffef7980ade85ea2d9b436c40df46384bbbe8e7e6e71219aff0757ad90f1652f\nfff3f56e384ab055c3aa4b6e2dd527c368bf2280863d357e577402460fe9d41a\n<\/pre><\/div>\n\n\n<p><strong>Use jq csv filter and specific fields<\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ rclone lsjson -R s3_ashburn:APPS | jq -r '.&#x5B;] | &#x5B;.Name,.Size] | @csv'\n<\/pre><\/div>\n\n\n<p><strong>Use jq select for older than certain date,&nbsp;specific&nbsp;fields&nbsp;and&nbsp;csv<\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ rclone lsjson -R s3_ashburn:APPS | jq -r &#039;.&#x5B;] | select (.&quot;ModTime&quot; &lt; &quot;2018-11-01&quot;) | &#x5B;.Name,.Size,.ModTime] | @csv&#039;\n<\/pre><\/div>\n\n\n<p><strong>Rclone size<\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ rclone size s3_ashburn:APPS --json \n{\"count\":8088,\"bytes\":38670955795}\n<\/pre><\/div>\n\n\n<p><strong>Rclone size and jq csv filter<\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ rclone size s3_ashburn:APPS --json | jq -r '&#x5B;.count,.bytes] | @csv'\n8088,38670955795\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"<p>Some examples of using rclone and jq to see object listing in a bucket. These examples was using Oracle (OCI)<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[96],"tags":[],"class_list":["post-1282","post","type-post","status-publish","format-standard","hentry","category-rclone"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1282","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1282"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1282\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1282"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1282"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1282"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}