Recording my test with zfsbackup-go. While I am playing around with backup\/DR\/object storage I also compared the concept here with a previous test around restic\/rclone\/object storage. <\/p>\n\n\n\n
In general ZFS snapshot and replication should work much better with file systems containing huge numbers of files. Most solutions struggle with millions of files and rsync on file level and restic\/rclone on object storage level. Walking the tree is just never efficient. So this test works well but has not been scaled yet. I plan to work on that as well as seeing how well the bucket can be synced to different regions.<\/p>\n\n\n\n
Minio server <\/strong><\/p>\n\n\n\n Tip: minio server has a nice browser interface<\/p>\n\n\n server 1:<\/strong><\/p>\n\n\n\n This server simulate our \"prod\" server. We create an initial data set in \/DATA on our server, take snapshot and backup to object storage.<\/p>\n\n\n server 2:<\/strong><\/p>\n\n\n\n This server is a possible DR or new server but the idea is somewhere else preferably another cloud region or data center.<\/p>\n\n\n That concludes one test. In theory that is a cold DR situation where you have nothing really ready until you need it. So think build a server and recover \/DATA from zfs backup in object storage. So initial restore will be very long depending on your size.<\/p>\n\n\n\n Read on if you are thinking you want to go more towards pilot light or warm DR we can run incremental backups, then on the target server keep receiving snapshots periodically into our target ZFS file system DATA. You may observe why not just do real ZFS send\/receive an no object storage in between. There is no good answer except there are many ways you could solve DR and this is one of them. In this case I could argue object storage is cheap and has some very good redundancy\/availability features. And your replication between regions may be using a back haul very fast\/cheap channel where your VPN or fastconnect WAN between regions may be slow and\/or expensive.<\/p>\n\n\n\n You could also be thinking something between cold and warm DR is where you want to be and therefore only apply the full DATA receive when you are ready. That could mean a lot of snapshots likely to apply afterwards. Or maybe not I have not checked on that aspect of a recovery process.<\/p>\n\n\n\n Regardless I like the idea of leveraging zfs with object storage so you may not have a use for this but I definitely will.<\/p>\n\n\n\n Incremental snapshots:<\/strong><\/p>\n\n\n\n server 1:<\/strong><\/p>\n\n\n\n Add more data to source, snapshot and backup to object storage.<\/p>\n\n\n server 2:<\/p>\n\n\n LINK: https:\/\/github.com\/someone1\/zfsbackup-go<\/p>\n","protected":false},"excerpt":{"rendered":" Recording my test with zfsbackup-go. While I am playing around with backup\/DR\/object storage I also compared the concept here with<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1391","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1391","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1391"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1391\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n#\u00a0docker run -p 9000:9000 --name minio1 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" -e "MINIO_SECRET_KEY=wJalrXUtnFEMI\/K7MDENG\/bPxRfiCYEXAMPLEKEY" -v \/DATA\/minio-repos\/:\/minio-repos minio\/minio server \/minio-repos\n\n You are running an older version of MinIO released 1 week ago \n Update: docker pull minio\/minio:RELEASE.2019-07-17T22-54-12Z \n\n\nEndpoint: http:\/\/172.17.0.2:9000 http:\/\/127.0.0.1:9000\n\nBrowser Access:\n http:\/\/172.17.0.2:9000 http:\/\/127.0.0.1:9000\n\nObject API (Amazon S3 compatible):\n Go: https:\/\/docs.min.io\/docs\/golang-client-quickstart-guide\n Java: https:\/\/docs.min.io\/docs\/java-client-quickstart-guide\n Python: https:\/\/docs.min.io\/docs\/python-client-quickstart-guide\n JavaScript: https:\/\/docs.min.io\/docs\/javascript-client-quickstart-guide\n .NET: https:\/\/docs.min.io\/docs\/dotnet-client-quickstart-guide\n<\/pre><\/div>\n\n\n
\n# rsync -a \/media\/sf_DATA\/MyWorkDocs \/DATA\/\n\n# du -sh \/DATA\/MyWorkDocs\/\n1.5G\t\/DATA\/MyWorkDocs\/\n\n#\u00a0export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n#\u00a0export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI\/K7MDENG\/bPxRfiCYEXAMPLEKEY\n#\u00a0export AWS_S3_CUSTOM_ENDPOINT=http:\/\/192.168.1.112:9000\n# export AWS_REGION=us-east-1\n\n# zfs snapshot DATA@20190721-0752\n\n# \/usr\/local\/bin\/zfsbackup-go send --full DATA s3:\/\/zfs-poc\n2019\/07\/21 07:53:12 Ignoring user provided number of cores (2) and using the number of detected cores (1).\nDone.\n\tTotal ZFS Stream Bytes: 1514016976 (1.4 GiB)\n\tTotal Bytes Written: 1176757570 (1.1 GiB)\n\tElapsed Time: 1m17.522630438s\n\tTotal Files Uploaded: 7\n\n#\u00a0\/usr\/local\/bin\/zfsbackup-go list s3:\/\/zfs-poc\n2019\/07\/21 07:56:57 Ignoring user provided number of cores (2) and using the number of detected cores (1).\nFound 1 backup sets:\n\nVolume: DATA\n\tSnapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)\n\tReplication: false\n\tArchives: 6 - 1176757570 bytes (1.1 GiB)\n\tVolume Size (Raw): 1514016976 bytes (1.4 GiB)\n\tUploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)\n\n\nThere are 4 manifests found locally that are not on the target destination.\n<\/pre><\/div>\n\n\n
\n# export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n#\u00a0export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI\/K7MDENG\/bPxRfiCYEXAMPLEKEY\n#\u00a0export AWS_S3_CUSTOM_ENDPOINT=http:\/\/192.168.1.112:9000\n#\u00a0export AWS_REGION=us-east-1\n#\u00a0\/usr\/local\/bin\/zfsbackup-go list s3:\/\/zfs-poc\n2019\/07\/21 07:59:16 Ignoring user provided number of cores (2) and using the number of detected cores (1).\nFound 1 backup sets:\n\nVolume: DATA\n\tSnapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)\n\tReplication: false\n\tArchives: 6 - 1176757570 bytes (1.1 GiB)\n\tVolume Size (Raw): 1514016976 bytes (1.4 GiB)\n\tUploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)\n\n#\u00a0zfs list\nNAME USED AVAIL REFER MOUNTPOINT\nDATA 2.70M 96.4G 26K \/DATA\n#\u00a0zfs list -t snapshot\nno datasets available\n# ls \/DATA\/\n\n** using -F. This is a COLD DR style test with no existing infrastructure\/ZFS sets on target systems\n# \/usr\/local\/bin\/zfsbackup-go receive --auto DATA s3:\/\/zfs-poc DATA -F\n2019\/07\/21 08:05:28 Ignoring user provided number of cores (2) and using the number of detected cores (1).\n2019\/07\/21 08:06:42 Done. Elapsed Time: 1m13.968871681s\n2019\/07\/21 08:06:42 Done.\n#\u00a0ls \/DATA\/\nMyWorkDocs\n#\u00a0du -sh \/DATA\/MyWorkDocs\/\n1.5G\t\/DATA\/MyWorkDocs\/\n#\u00a0zfs list\nNAME USED AVAIL REFER MOUNTPOINT\nDATA 1.41G 95.0G 1.40G \/DATA\n#\u00a0zfs list -t snapshot\nNAME USED AVAIL REFER MOUNTPOINT\nDATA@20190721-0752 247K - 1.40G -\n<\/pre><\/div>\n\n\n
\n# rsync -a \/media\/sf_DATA\/MySrc \/DATA\/\n#\u00a0du -sh \/DATA\/MySrc\/\n1.1M\t\/DATA\/MySrc\/\n\n#\u00a0zfs snapshot DATA@20190721-0809\n#\u00a0zfs list -t snapshot\nNAME USED AVAIL REFER MOUNTPOINT\nDATA@20190721-0752 31K - 1.40G -\nDATA@20190721-0809 0B - 1.41G -\n\n# \/usr\/local\/bin\/zfsbackup-go send --increment DATA s3:\/\/zfs-poc\n2019\/07\/21 08:10:49 Ignoring user provided number of cores (2) and using the number of detected cores (1).\nDone.\n\tTotal ZFS Stream Bytes: 1202792 (1.1 MiB)\n\tTotal Bytes Written: 254909 (249 KiB)\n\tElapsed Time: 228.123591ms\n\tTotal Files Uploaded: 2\n\n# \/usr\/local\/bin\/zfsbackup-go list s3:\/\/zfs-poc\n2019\/07\/21 08:11:17 Ignoring user provided number of cores (2) and using the number of detected cores (1).\nFound 2 backup sets:\n\nVolume: DATA\n\tSnapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)\n\tReplication: false\n\tArchives: 6 - 1176757570 bytes (1.1 GiB)\n\tVolume Size (Raw): 1514016976 bytes (1.4 GiB)\n\tUploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)\n\n\nVolume: DATA\n\tSnapshot: 20190721-0809 (2019-07-21 08:09:47 -0500 CDT)\n\tIncremental From Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)\n\tIntermediary: false\n\tReplication: false\n\tArchives: 1 - 254909 bytes (249 KiB)\n\tVolume Size (Raw): 1202792 bytes (1.1 MiB)\n\tUploaded: 2019-07-21 08:10:49.3280703 -0500 CDT (took 214.139056ms)\n\nThere are 4 manifests found locally that are not on the target destination.\n<\/pre><\/div>\n\n\n
\n# \/usr\/local\/bin\/zfsbackup-go list s3:\/\/zfs-poc\n2019\/07\/21 08:11:44 Ignoring user provided number of cores (2) and using the number of detected cores (1).\nFound 2 backup sets:\n\nVolume: DATA\n\tSnapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)\n\tReplication: false\n\tArchives: 6 - 1176757570 bytes (1.1 GiB)\n\tVolume Size (Raw): 1514016976 bytes (1.4 GiB)\n\tUploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)\n\n\nVolume: DATA\n\tSnapshot: 20190721-0809 (2019-07-21 08:09:47 -0500 CDT)\n\tIncremental From Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)\n\tIntermediary: false\n\tReplication: false\n\tArchives: 1 - 254909 bytes (249 KiB)\n\tVolume Size (Raw): 1202792 bytes (1.1 MiB)\n\tUploaded: 2019-07-21 08:10:49.3280703 -0500 CDT (took 214.139056ms)\n\n** not sure why I need to force (-F) maybe because data set is mounted? message like this:\n** cannot receive incremental stream: destination DATA has been modified since most recent snapshot\n*** 2019\/07\/21 08:12:25 Error while trying to read from volume DATA|20190721-0752|to|20190721-0809.zstream.gz.vol1 - io: read\/write on closed pipe\n\n# \/usr\/local\/bin\/zfsbackup-go receive --auto DATA s3:\/\/zfs-poc DATA -F\n2019\/07\/21 08:12:53 Ignoring user provided number of cores (2) and using the number of detected cores (1).\n2019\/07\/21 08:12:54 Done. Elapsed Time: 379.712693ms\n2019\/07\/21 08:12:54 Done.\n\n#\u00a0ls \/DATA\/\nMySrc MyWorkDocs\n#\u00a0du -sh \/DATA\/MySrc\/\n1.1M\t\/DATA\/MySrc\/\n#\u00a0zfs list -t snapshot\nNAME USED AVAIL REFER MOUNTPOINT\nDATA@20190721-0752 30K - 1.40G -\nDATA@20190721-0809 34K - 1.41G -\n\n<\/pre><\/div>\n\n\n