Since btrfs has send and receive capabilities I took a look at it. The title is replication but if you are interested in enterprise level sophisticated storage level replication for disaster recover or better yet mature data set cloning for non production instances you will need to look further. For example the Oracle ZFS appliance has a mature replication engine built on send and receive but handles all the replication magic for you. I am not aware of commercial solutions built on btrfs that has the mature functionality the ZFS appliance can offer yet. Note we are not just talking replication but also snapshot cloning, sharing and protection of snapshots on the target end. So for now here is what I have tested for pure btrfs send and receive.<\/p>\n
Some details on machine 1:<\/p>\n
\r\nroot@u1604b1-m1:~# more \/etc\/issue\r\nUbuntu Xenial Xerus (development branch) \\n \\l\r\n\r\nroot@u1604b1-m1:~# df -h\r\nFilesystem Size Used Avail Use% Mounted on\r\n[..]\r\n\/dev\/sda1 7.5G 3.9G 3.1G 56% \/\r\n\/dev\/sda1 7.5G 3.9G 3.1G 56% \/home\r\n[..]\r\n\r\nroot@u1604b1-m1:~# mount\r\n[..]\r\n\/dev\/sda1 on \/ type btrfs (rw,relatime,space_cache,subvolid=257,subvol=\/@)\r\n\/dev\/sda1 on \/home type btrfs (rw,relatime,space_cache,subvolid=258,subvol=\/@home)\r\n[..]\r\n\r\nroot@u1604b1-m1:~# btrfs --version\r\nbtrfs-progs v4.4\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume list \/\r\nID 257 gen 47 top level 5 path @\r\nID 258 gen 47 top level 5 path @home\r\n<\/pre>\nTest ssh to machine 2:<\/p>\n
\r\nroot@u1604b1-m1:~# ssh root@192.168.2.29 uptime\r\nroot@192.168.2.29's password: \r\n 10:33:23 up 5 min, 1 user, load average: 0.22, 0.37, 0.19\r\n<\/pre>\nMachine 2 subvolumes before we receive:<\/p>\n
\r\nroot@u1604b1-m2:~# btrfs subvolume list \/\r\nID 257 gen 40 top level 5 path @\r\nID 258 gen 40 top level 5 path @home\r\n<\/pre>\nCreate a subvolume, add a file and take a snapshot:<\/p>\n
\r\nroot@u1604b1-m1:~# btrfs subvolume create \/tank1\r\nCreate subvolume '\/\/tank1'\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume list \/\r\nID 257 gen 53 top level 5 path @\r\nID 258 gen 50 top level 5 path @home\r\nID 264 gen 53 top level 257 path tank1\r\n\r\nroot@u1604b1-m1:~# ls \/tank1\r\n\r\nroot@u1604b1-m1:~# touch \/tank1\/rr_test1\r\nroot@u1604b1-m1:~# ls -l \/tank1\/\r\ntotal 0\r\n-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume snapshot \/tank1 \/tank1_snapshot\r\nCreate a snapshot of '\/tank1' in '\/\/tank1_snapshot'\r\n\r\nroot@u1604b1-m1:~# ls -l \/tank1_snapshot\/\r\ntotal 0\r\n-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume list \/\r\nID 257 gen 63 top level 5 path @\r\nID 258 gen 58 top level 5 path @home\r\nID 264 gen 59 top level 257 path tank1\r\nID 265 gen 59 top level 257 path tank1_snapshot\r\n<\/pre>\nDelete a snapshot:<\/p>\n
\r\nroot@u1604b1-m1:~# btrfs subvolume delete \/tank1_snapshot\r\nDelete subvolume (no-commit): '\/\/tank1_snapshot'\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume list \/\r\nID 257 gen 64 top level 5 path @\r\nID 258 gen 58 top level 5 path @home\r\nID 264 gen 59 top level 257 path tank1\r\n<\/pre>\nTake a read-only snapshot and send to machine 2:<\/p>\n
\r\nroot@u1604b1-m1:~# btrfs subvolume snapshot -r \/tank1 \/tank1_snapshot\r\n\r\nCreate a readonly snapshot of '\/tank1' in '\/\/tank1_snapshot'\r\n\r\nroot@u1604b1-m1:~# btrfs send \/tank1_snapshot | ssh root@192.168.2.29 "btrfs receive \/" \r\nAt subvol \/tank1_snapshot\r\nroot@192.168.2.29's password: \r\nAt subvol tank1_snapshot\r\n\r\nMachine 2 after receiving snapshot1:\r\n\r\n[bash]\r\nroot@u1604b1-m2:~# btrfs subvolume list \/\r\nID 257 gen 61 top level 5 path @\r\nID 258 gen 60 top level 5 path @home\r\nID 264 gen 62 top level 257 path tank1_snapshot\r\n\r\nroot@u1604b1-m2:~# ls -l \/tank1_snapshot\/\r\ntotal 0\r\n-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1\r\n<\/pre>\nCreate one more file:<\/p>\n
\r\nroot@u1604b1-m1:~# touch \/tank1\/rr_test2\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume snapshot -r \/tank1 \/tank1_snapshot2\r\nCreate a readonly snapshot of '\/tank1' in '\/\/tank1_snapshot2'\r\n\r\nroot@u1604b1-m1:~# btrfs send \/tank1_snapshot2 | ssh root@192.168.2.29 "btrfs receive \/" \r\nAt subvol \/tank1_snapshot2\r\nroot@192.168.2.29's password: \r\nAt subvol tank1_snapshot2\r\n<\/pre>\nMachine 2 after receiving snapshot 2:<\/p>\n
\r\nroot@u1604b1-m2:~# btrfs subvolume list \/\r\nID 257 gen 65 top level 5 path @\r\nID 258 gen 60 top level 5 path @home\r\nID 264 gen 62 top level 257 path tank1_snapshot\r\nID 265 gen 66 top level 257 path tank1_snapshot2\r\n\r\nroot@u1604b1-m2:~# ls -l \/tank1_snapshot2\/\r\ntotal 0\r\n-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1\r\n-rw-r--r-- 1 root root 0 Mar 10 10:53 rr_test2\r\n<\/pre>\nIncremental send(adding -v for now to see more detail):<\/p>\n
\r\nroot@u1604b1-m1:~# btrfs subvolume snapshot -r \/tank1 \/tank1_snapshot3\r\nCreate a readonly snapshot of '\/tank1' in '\/\/tank1_snapshot3'\r\n\r\nroot@u1604b1-m1:~# btrfs send -vp \/tank1_snapshot2 \/tank1_snapshot3 | ssh root@192.168.2.29 "btrfs receive \/" \r\nAt subvol \/tank1_snapshot3\r\nBTRFS_IOC_SEND returned 0\r\njoining genl thread\r\nroot@192.168.2.29's password: \r\nAt snapshot tank1_snapshot3\r\n<\/pre>\nUsing larger files to see effect of incremental better:<\/p>\n
\r\nroot@u1604b1-m1:~# cp \/media\/sf_E_DRIVE\/ISO\/ubuntu-gnome-15.10-desktop-amd64.iso \/tank1\/\r\nroot@u1604b1-m1:~# du -sh \/tank1\r\n1.1G\t\/tank1\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume snapshot -r \/tank1 \/tank1_snapshot6\r\nCreate a readonly snapshot of '\/tank1' in '\/\/tank1_snapshot6'\r\n\r\nroot@u1604b1-m1:~# time btrfs send -vp \/tank1_snapshot5 \/tank1_snapshot6 | ssh root@192.168.2.29 "btrfs receive -v \/" \r\nAt subvol \/tank1_snapshot6\r\nroot@192.168.2.29's password: \r\nreceiving snapshot tank1_snapshot6 uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, ctransid=272 parent_uuid=ec3f1fb5-9bed-3e4c-9c5b-a6c586b10531, parent_ctransid=201\r\nBTRFS_IOC_SEND returned 0\r\njoining genl thread\r\nBTRFS_IOC_SET_RECEIVED_SUBVOL uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, stransid=272\r\nAt snapshot tank1_snapshot6\r\n\r\nreal\t1m10.578s\r\nuser\t0m0.696s\r\nsys\t0m16.064s\r\n<\/pre>\nMachine 2 after snapshot6:<\/p>\n
\r\ntotal 1.1G\r\n-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1\r\n-rw-r--r-- 1 root root 22 Mar 10 12:19 rr_test2\r\n-rwxr-x--- 1 root root 1.1G Mar 10 13:04 ubuntu-gnome-15.10-desktop-amd64.iso\r\n<\/pre>\n\r\nroot@u1604b1-m1:~# cp \/media\/sf_E_DRIVE\/ISO\/ubuntu-gnome-15.10-desktop-i386.iso \/tank1\/\r\n\r\nroot@u1604b1-m1:~# btrfs subvolume snapshot -r \/tank1 \/tank1_snapshot7\r\nCreate a readonly snapshot of '\/tank1' in '\/\/tank1_snapshot7'\r\n\r\nroot@u1604b1-m1:~# time btrfs send -vp \/tank1_snapshot6 \/tank1_snapshot7 | ssh root@192.168.2.29 "btrfs receive -v \/" \r\nAt subvol \/tank1_snapshot7\r\nroot@192.168.2.29's password: \r\nreceiving snapshot tank1_snapshot7 uuid=5c255311-0f60-4149-91f7-99d9d5acf64c, ctransid=276 parent_uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, parent_ctransid=272\r\n\r\nBTRFS_IOC_SEND returned 0\r\njoining genl thread\r\nBTRFS_IOC_SET_RECEIVED_SUBVOL uuid=5c255311-0f60-4149-91f7-99d9d5acf64c, stransid=276\r\nAt snapshot tank1_snapshot7\r\n\r\nreal\t1m17.393s\r\nuser\t0m0.640s\r\nsys\t0m16.716s\r\n\r\nMachine 2 after snapshot7:\r\n\r\n[bash]\r\nroot@u1604b1-m2:~# ls -lh \/tank1_snapshot7\r\ntotal 2.0G\r\n-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1\r\n-rw-r--r-- 1 root root 22 Mar 10 12:19 rr_test2\r\n-rwxr-x--- 1 root root 1.1G Mar 10 13:04 ubuntu-gnome-15.10-desktop-amd64.iso\r\n-rwxr-x--- 1 root root 1.1G Mar 10 13:07 ubuntu-gnome-15.10-desktop-i386.iso\r\n<\/pre>\nExperiment: On the target I sent a snapshot to a new btrfs subvolume. So in effect become independent. This does not really help us with cloning since with large datasets it takes too long as well as duplicate the space which nullifies why we like COW.<\/p>\n
\r\nroot@u1604b1-m2:~# btrfs subvolume create \/tank1_clone\r\nCreate subvolume '\/\/tank1_clone'\r\n\r\nroot@u1604b1-m2:~# btrfs send \/tank1_snapshot3 | btrfs receive \/tank1_clone\r\nAt subvol \/tank1_snapshot3\r\nAt subvol tank1_snapshot3\r\n<\/pre>\nThis was just my initial look see on what btrfs is capable of and how similar it is to ZFS and ZFS appliance functionality.<\/p>\n
So far at least it seems promising that send and receive is being addressed in btrfs but I don't think you can easily roll your own solution for A) replication and B) writable snapshots(clones) with btrfs yet. There will be too much work around building the replication discipline and framework. <\/p>\n
A few links that I came across that are useful while looking at btrfs and the topics of replication and database cloning.<\/p>\n
1. http:\/\/rockstor.com\/blog\/snapshots\/data-replication-with-rockstor\/
\n2. http:\/\/blog.contractoracle.com\/2013\/02\/oracle-database-on-btrfs-reduce-costs.html
\n3. http:\/\/www.cybertec.at\/2015\/01\/forking-databases-the-art-of-copying-without-copying\/
\n4. http:\/\/blog.contractoracle.com\/2013\/02\/oracle-database-on-btrfs-reduce-costs.html
\n5. https:\/\/bdrouvot.wordpress.com\/2014\/04\/25\/reduce-resource-consumption-and-clone-in-seconds-your-oracle-virtual-environment-on-your-laptop-using-linux-containers-and-btrfs\/
\n6. https:\/\/docs.opensvc.com\/storage.btrfs.html
\n7. https:\/\/ilmarkerm.eu\/blog\/2014\/08\/cloning-pluggable-database-with-custom-snapshot\/
\n8. http:\/\/blog.ronnyegner-consulting.de\/2010\/02\/17\/creating-database-clones-with-zfs-really-fast\/
\n9. http:\/\/www.seedsofgenius.net\/solaris\/zfs-vs-btrfs-a-reference<\/p>\n","protected":false},"excerpt":{"rendered":"Since btrfs has send and receive capabilities I took a look at it. The title is replication but if you<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[62,30],"tags":[],"class_list":["post-935","post","type-post","status-publish","format-standard","hentry","category-btrfs","category-zfs"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/935","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=935"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/935\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=935"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=935"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=935"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}