{"id":1240,"date":"2018-08-07T10:26:26","date_gmt":"2018-08-07T15:26:26","guid":{"rendered":"http:\/\/blog.ls-al.com\/?p=1240"},"modified":"2018-08-07T11:58:36","modified_gmt":"2018-08-07T16:58:36","slug":"object-storage-with-duplicity-and-rclone","status":"publish","type":"post","link":"https:\/\/blog.ls-al.com\/object-storage-with-duplicity-and-rclone\/","title":{"rendered":"Object Storage with Duplicity and Rclone"},"content":{"rendered":"<p>At this point I prefer using restic for my object storage backup needs but since I did a POC for duplicity and specifically using rclone with duplicity I am writing down my notes. A good description of duplicity and restic here:<\/p>\n<blockquote class=\"wp-embedded-content\" data-secret=\"ox8k1IkbZO\"><p><a href=\"https:\/\/www.backblaze.com\/blog\/backing-linux-backblaze-b2-duplicity-restic\/\">Backing Up Linux to Backblaze B2 with Duplicity and Restic<\/a><\/p><\/blockquote>\n<p><iframe loading=\"lazy\" class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;Backing Up Linux to Backblaze B2 with Duplicity and Restic&#8221; &#8212; Backblaze Blog | Cloud Storage &amp; Cloud Backup\" src=\"https:\/\/www.backblaze.com\/blog\/backing-linux-backblaze-b2-duplicity-restic\/embed\/#?secret=ox8k1IkbZO\" data-secret=\"ox8k1IkbZO\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe><br \/>\nWe\u2019re highlighting Duplicity and Restic because they exemplify two different philosophical approaches to data backup: \u201cOld School\u201d (Duplicity) vs \u201cNew School\u201d (Restic).<\/p>\n<p>Since I am doing my tests with Oracle Cloud Infrastructure (OCI) Object Storage and so far it's Amazon S3 Compatibility Interface does not work out of the box with most tools except with rclone, I am using rclone as a backend. With restic using rclone as a back-end worked pretty smooth but duplicity does not have good rclone support so I used a python back-end written by Francesco Magno and hosted here: https:\/\/github.com\/GilGalaad\/duplicity-rclone\/blob\/master\/README.md<\/p>\n<p>I had a couple issues with getting duplicity to work with this back-end so I will show how to get around it.<\/p>\n<p>First:<br \/>\n1. Make sure rclone is working with your rclone config and can at least \"ls\" your bucket.<br \/>\n2. Setup a gpg key.<br \/>\n3. Copy rclonebackend.py to duplicity backends folder. In my case \/usr\/lib64\/python2.7\/site-packages\/duplicity\/backends<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# PASSPHRASE=&quot;mypassphrase&quot; duplicity --encrypt-key 094CA414 \/tmp rclone:\/\/mycompany-POC-phoenix:dr01-duplicity\r\nInvalidBackendURL: Syntax error (port) in: rclone:\/\/mycompany-POC-phoenix:dr01-duplicity AFalse BNone Cmycompany-POC-phoenix:dr01-duplicity\r\n<\/pre>\n<p>## Hack backends.py<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# diff \/usr\/lib64\/python2.7\/site-packages\/duplicity\/backend.py \/tmp\/backend.py \r\n303c303\r\n&lt; if not (self.scheme in &#x5B;'rsync'] and re.search('::&#x5B;^:]*$', self.url_string) or (self.scheme in &#x5B;'rclone']) ): --- &gt;             if not (self.scheme in &#x5B;'rsync'] and re.search('::&#x5B;^:]*$', self.url_string)):\r\n<\/pre>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# PASSPHRASE=&quot;mypassphrase&quot; duplicity --encrypt-key 094CA414 \/tmp rclone:\/\/mycompany-POC-phoenix:dr01-duplicity\r\nLocal and Remote metadata are synchronized, no sync needed.\r\nLast full backup date: none\r\nNo signatures found, switching to full backup.\r\n--------------&#x5B; Backup Statistics ]--------------\r\nStartTime 1533652997.49 (Tue Aug  7 14:43:17 2018)\r\nEndTime 1533653022.35 (Tue Aug  7 14:43:42 2018)\r\nElapsedTime 24.86 (24.86 seconds)\r\nSourceFiles 50\r\nSourceFileSize 293736179 (280 MB)\r\nNewFiles 50\r\nNewFileSize 136467418 (130 MB)\r\nDeletedFiles 0\r\nChangedFiles 0\r\nChangedFileSize 0 (0 bytes)\r\nChangedDeltaSize 0 (0 bytes)\r\nDeltaEntries 50\r\nRawDeltaSize 293723433 (280 MB)\r\nTotalDestinationSizeChange 279406571 (266 MB)\r\nErrors 0\r\n-------------------------------------------------\r\n\r\n# rclone ls mycompany-POC-phoenix:dr01-duplicity\r\n  1773668 duplicity-full-signatures.20180807T144317Z.sigtar.gpg\r\n      485 duplicity-full.20180807T144317Z.manifest.gpg\r\n209763240 duplicity-full.20180807T144317Z.vol1.difftar.gpg\r\n 69643331 duplicity-full.20180807T144317Z.vol2.difftar.gpg\r\n\r\n# PASSPHRASE=&quot;mypassphrase&quot; duplicity --encrypt-key 094CA414 collection-status rclone:\/\/mycompany-POC-phoenix:dr01-duplicity\r\nLast full backup date: Tue Aug  7 14:43:17 2018\r\nCollection Status\r\n-----------------\r\nConnecting with backend: BackendWrapper\r\nArchive dir: \/root\/.cache\/duplicity\/df529824ba5d10f9e31329e440c5efa6\r\n\r\nFound 0 secondary backup chains.\r\n\r\nFound primary backup chain with matching signature chain:\r\n-------------------------\r\nChain start time: Tue Aug  7 14:43:17 2018\r\nChain end time: Tue Aug  7 14:50:12 2018\r\nNumber of contained backup sets: 2\r\nTotal number of contained volumes: 3\r\n Type of backup set:                            Time:      Num volumes:\r\n                Full         Tue Aug  7 14:43:17 2018                 2\r\n         Incremental         Tue Aug  7 14:50:12 2018                 1\r\n-------------------------\r\nNo orphaned or incomplete backup sets found.\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>At this point I prefer using restic for my object storage backup needs but since I did a POC for<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[100,87,96],"tags":[],"class_list":["post-1240","post","type-post","status-publish","format-standard","hentry","category-dupliciti","category-oci","category-rclone"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1240","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1240"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1240\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}