Riaan's SysAdmin Blog

My tips, howtos, gotchas, snippets and stuff. Use at your own risk!

BtrfsZFS

Btrfs Replication

admin

Since btrfs has send and receive capabilities I took a look at it. The title is replication but if you are interested in enterprise level sophisticated storage level replication for disaster recover or better yet mature data set cloning for non production instances you will need to look further. For example the Oracle ZFS appliance has a mature replication engine built on send and receive but handles all the replication magic for you. I am not aware of commercial solutions built on btrfs that has the mature functionality the ZFS appliance can offer yet. Note we are not just talking replication but also snapshot cloning, sharing and protection of snapshots on the target end. So for now here is what I have tested for pure btrfs send and receive.

Some details on machine 1:

root@u1604b1-m1:~# more /etc/issue
Ubuntu Xenial Xerus (development branch) \n \l

root@u1604b1-m1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
[..]
/dev/sda1       7.5G  3.9G  3.1G  56% /
/dev/sda1       7.5G  3.9G  3.1G  56% /home
[..]

root@u1604b1-m1:~# mount
[..]
/dev/sda1 on / type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/@)
/dev/sda1 on /home type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@home)
[..]

root@u1604b1-m1:~# btrfs --version
btrfs-progs v4.4

root@u1604b1-m1:~# btrfs subvolume list /
ID 257 gen 47 top level 5 path @
ID 258 gen 47 top level 5 path @home

Test ssh to machine 2:

root@u1604b1-m1:~# ssh root@192.168.2.29 uptime
root@192.168.2.29's password: 
 10:33:23 up 5 min,  1 user,  load average: 0.22, 0.37, 0.19

Machine 2 subvolumes before we receive:

root@u1604b1-m2:~# btrfs subvolume list /
ID 257 gen 40 top level 5 path @
ID 258 gen 40 top level 5 path @home

Create a subvolume, add a file and take a snapshot:

root@u1604b1-m1:~# btrfs subvolume create /tank1
Create subvolume '//tank1'

root@u1604b1-m1:~# btrfs subvolume list /
ID 257 gen 53 top level 5 path @
ID 258 gen 50 top level 5 path @home
ID 264 gen 53 top level 257 path tank1

root@u1604b1-m1:~# ls /tank1

root@u1604b1-m1:~# touch /tank1/rr_test1
root@u1604b1-m1:~# ls -l /tank1/
total 0
-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1

root@u1604b1-m1:~# btrfs subvolume snapshot /tank1 /tank1_snapshot
Create a snapshot of '/tank1' in '//tank1_snapshot'

root@u1604b1-m1:~# ls -l /tank1_snapshot/
total 0
-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1

root@u1604b1-m1:~# btrfs subvolume list /
ID 257 gen 63 top level 5 path @
ID 258 gen 58 top level 5 path @home
ID 264 gen 59 top level 257 path tank1
ID 265 gen 59 top level 257 path tank1_snapshot

Delete a snapshot:

root@u1604b1-m1:~# btrfs subvolume delete /tank1_snapshot
Delete subvolume (no-commit): '//tank1_snapshot'

root@u1604b1-m1:~# btrfs subvolume list /
ID 257 gen 64 top level 5 path @
ID 258 gen 58 top level 5 path @home
ID 264 gen 59 top level 257 path tank1

Take a read-only snapshot and send to machine 2:

root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot

Create a readonly snapshot of '/tank1' in '//tank1_snapshot'

root@u1604b1-m1:~# btrfs send /tank1_snapshot | ssh root@192.168.2.29 "btrfs receive /" 
At subvol /tank1_snapshot
root@192.168.2.29's password: 
At subvol tank1_snapshot

Machine 2 after receiving snapshot1:

[bash]
root@u1604b1-m2:~# btrfs subvolume list /
ID 257 gen 61 top level 5 path @
ID 258 gen 60 top level 5 path @home
ID 264 gen 62 top level 257 path tank1_snapshot

root@u1604b1-m2:~# ls -l /tank1_snapshot/
total 0
-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1

Create one more file:

root@u1604b1-m1:~# touch /tank1/rr_test2

root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot2
Create a readonly snapshot of '/tank1' in '//tank1_snapshot2'

root@u1604b1-m1:~# btrfs send /tank1_snapshot2 | ssh root@192.168.2.29 "btrfs receive /" 
At subvol /tank1_snapshot2
root@192.168.2.29's password: 
At subvol tank1_snapshot2

Machine 2 after receiving snapshot 2:

root@u1604b1-m2:~# btrfs subvolume list /
ID 257 gen 65 top level 5 path @
ID 258 gen 60 top level 5 path @home
ID 264 gen 62 top level 257 path tank1_snapshot
ID 265 gen 66 top level 257 path tank1_snapshot2

root@u1604b1-m2:~# ls -l /tank1_snapshot2/
total 0
-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1
-rw-r--r-- 1 root root 0 Mar 10 10:53 rr_test2

Incremental send(adding -v for now to see more detail):

root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot3
Create a readonly snapshot of '/tank1' in '//tank1_snapshot3'

root@u1604b1-m1:~# btrfs send -vp /tank1_snapshot2 /tank1_snapshot3 | ssh root@192.168.2.29 "btrfs receive /" 
At subvol /tank1_snapshot3
BTRFS_IOC_SEND returned 0
joining genl thread
root@192.168.2.29's password: 
At snapshot tank1_snapshot3

Using larger files to see effect of incremental better:

root@u1604b1-m1:~# cp /media/sf_E_DRIVE/ISO/ubuntu-gnome-15.10-desktop-amd64.iso /tank1/
root@u1604b1-m1:~# du -sh /tank1
1.1G	/tank1

root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot6
Create a readonly snapshot of '/tank1' in '//tank1_snapshot6'

root@u1604b1-m1:~# time btrfs send -vp /tank1_snapshot5 /tank1_snapshot6 | ssh root@192.168.2.29 "btrfs receive -v /"  
At subvol /tank1_snapshot6
root@192.168.2.29's password: 
receiving snapshot tank1_snapshot6 uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, ctransid=272 parent_uuid=ec3f1fb5-9bed-3e4c-9c5b-a6c586b10531, parent_ctransid=201
BTRFS_IOC_SEND returned 0
joining genl thread
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, stransid=272
At snapshot tank1_snapshot6

real	1m10.578s
user	0m0.696s
sys	0m16.064s

Machine 2 after snapshot6:

total 1.1G
-rw-r--r-- 1 root root    0 Mar 10 10:38 rr_test1
-rw-r--r-- 1 root root   22 Mar 10 12:19 rr_test2
-rwxr-x--- 1 root root 1.1G Mar 10 13:04 ubuntu-gnome-15.10-desktop-amd64.iso

root@u1604b1-m1:~# cp /media/sf_E_DRIVE/ISO/ubuntu-gnome-15.10-desktop-i386.iso /tank1/

root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot7
Create a readonly snapshot of '/tank1' in '//tank1_snapshot7'

root@u1604b1-m1:~# time btrfs send -vp /tank1_snapshot6 /tank1_snapshot7 | ssh root@192.168.2.29 "btrfs receive -v /" 
At subvol /tank1_snapshot7
root@192.168.2.29's password: 
receiving snapshot tank1_snapshot7 uuid=5c255311-0f60-4149-91f7-99d9d5acf64c, ctransid=276 parent_uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, parent_ctransid=272

BTRFS_IOC_SEND returned 0
joining genl thread
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=5c255311-0f60-4149-91f7-99d9d5acf64c, stransid=276
At snapshot tank1_snapshot7

real	1m17.393s
user	0m0.640s
sys	0m16.716s

Machine 2 after snapshot7:

[bash]
root@u1604b1-m2:~# ls -lh /tank1_snapshot7
total 2.0G
-rw-r--r-- 1 root root    0 Mar 10 10:38 rr_test1
-rw-r--r-- 1 root root   22 Mar 10 12:19 rr_test2
-rwxr-x--- 1 root root 1.1G Mar 10 13:04 ubuntu-gnome-15.10-desktop-amd64.iso
-rwxr-x--- 1 root root 1.1G Mar 10 13:07 ubuntu-gnome-15.10-desktop-i386.iso

Experiment: On the target I sent a snapshot to a new btrfs subvolume. So in effect become independent. This does not really help us with cloning since with large datasets it takes too long as well as duplicate the space which nullifies why we like COW.

root@u1604b1-m2:~# btrfs subvolume create /tank1_clone
Create subvolume '//tank1_clone'

root@u1604b1-m2:~# btrfs send /tank1_snapshot3 | btrfs receive  /tank1_clone
At subvol /tank1_snapshot3
At subvol tank1_snapshot3

This was just my initial look see on what btrfs is capable of and how similar it is to ZFS and ZFS appliance functionality.

So far at least it seems promising that send and receive is being addressed in btrfs but I don't think you can easily roll your own solution for A) replication and B) writable snapshots(clones) with btrfs yet. There will be too much work around building the replication discipline and framework.

A few links that I came across that are useful while looking at btrfs and the topics of replication and database cloning.

1. http://rockstor.com/blog/snapshots/data-replication-with-rockstor/
2. http://blog.contractoracle.com/2013/02/oracle-database-on-btrfs-reduce-costs.html
3. http://www.cybertec.at/2015/01/forking-databases-the-art-of-copying-without-copying/
4. http://blog.contractoracle.com/2013/02/oracle-database-on-btrfs-reduce-costs.html
5. https://bdrouvot.wordpress.com/2014/04/25/reduce-resource-consumption-and-clone-in-seconds-your-oracle-virtual-environment-on-your-laptop-using-linux-containers-and-btrfs/
6. https://docs.opensvc.com/storage.btrfs.html
7. https://ilmarkerm.eu/blog/2014/08/cloning-pluggable-database-with-custom-snapshot/
8. http://blog.ronnyegner-consulting.de/2010/02/17/creating-database-clones-with-zfs-really-fast/
9. http://www.seedsofgenius.net/solaris/zfs-vs-btrfs-a-reference

admin

Bio Info for Riaan