zfsbackup-go test with minio server
Recording my test with zfsbackup-go. While I am playing around with backup/DR/object storage I also compared the concept here with a previous test around restic/rclone/object storage.
In general ZFS snapshot and replication should work much better with file systems containing huge numbers of files. Most solutions struggle with millions of files and rsync on file level and restic/rclone on object storage level. Walking the tree is just never efficient. So this test works well but has not been scaled yet. I plan to work on that as well as seeing how well the bucket can be synced to different regions.
Minio server
Tip: minio server has a nice browser interface
# docker run -p 9000:9000 --name minio1 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" -v /DATA/minio-repos/:/minio-repos minio/minio server /minio-repos
You are running an older version of MinIO released 1 week ago
Update: docker pull minio/minio:RELEASE.2019-07-17T22-54-12Z
Endpoint: http://172.17.0.2:9000 http://127.0.0.1:9000
Browser Access:
http://172.17.0.2:9000 http://127.0.0.1:9000
Object API (Amazon S3 compatible):
Go: https://docs.min.io/docs/golang-client-quickstart-guide
Java: https://docs.min.io/docs/java-client-quickstart-guide
Python: https://docs.min.io/docs/python-client-quickstart-guide
JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
.NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
server 1:
This server simulate our "prod" server. We create an initial data set in /DATA on our server, take snapshot and backup to object storage.
# rsync -a /media/sf_DATA/MyWorkDocs /DATA/
# du -sh /DATA/MyWorkDocs/
1.5G /DATA/MyWorkDocs/
# export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# export AWS_S3_CUSTOM_ENDPOINT=http://192.168.1.112:9000
# export AWS_REGION=us-east-1
# zfs snapshot DATA@20190721-0752
# /usr/local/bin/zfsbackup-go send --full DATA s3://zfs-poc
2019/07/21 07:53:12 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Done.
Total ZFS Stream Bytes: 1514016976 (1.4 GiB)
Total Bytes Written: 1176757570 (1.1 GiB)
Elapsed Time: 1m17.522630438s
Total Files Uploaded: 7
# /usr/local/bin/zfsbackup-go list s3://zfs-poc
2019/07/21 07:56:57 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Found 1 backup sets:
Volume: DATA
Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
Replication: false
Archives: 6 - 1176757570 bytes (1.1 GiB)
Volume Size (Raw): 1514016976 bytes (1.4 GiB)
Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)
There are 4 manifests found locally that are not on the target destination.
server 2:
This server is a possible DR or new server but the idea is somewhere else preferably another cloud region or data center.
# export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# export AWS_S3_CUSTOM_ENDPOINT=http://192.168.1.112:9000
# export AWS_REGION=us-east-1
# /usr/local/bin/zfsbackup-go list s3://zfs-poc
2019/07/21 07:59:16 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Found 1 backup sets:
Volume: DATA
Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
Replication: false
Archives: 6 - 1176757570 bytes (1.1 GiB)
Volume Size (Raw): 1514016976 bytes (1.4 GiB)
Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
DATA 2.70M 96.4G 26K /DATA
# zfs list -t snapshot
no datasets available
# ls /DATA/
** using -F. This is a COLD DR style test with no existing infrastructure/ZFS sets on target systems
# /usr/local/bin/zfsbackup-go receive --auto DATA s3://zfs-poc DATA -F
2019/07/21 08:05:28 Ignoring user provided number of cores (2) and using the number of detected cores (1).
2019/07/21 08:06:42 Done. Elapsed Time: 1m13.968871681s
2019/07/21 08:06:42 Done.
# ls /DATA/
MyWorkDocs
# du -sh /DATA/MyWorkDocs/
1.5G /DATA/MyWorkDocs/
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
DATA 1.41G 95.0G 1.40G /DATA
# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
DATA@20190721-0752 247K - 1.40G -
That concludes one test. In theory that is a cold DR situation where you have nothing really ready until you need it. So think build a server and recover /DATA from zfs backup in object storage. So initial restore will be very long depending on your size.
Read on if you are thinking you want to go more towards pilot light or warm DR we can run incremental backups, then on the target server keep receiving snapshots periodically into our target ZFS file system DATA. You may observe why not just do real ZFS send/receive an no object storage in between. There is no good answer except there are many ways you could solve DR and this is one of them. In this case I could argue object storage is cheap and has some very good redundancy/availability features. And your replication between regions may be using a back haul very fast/cheap channel where your VPN or fastconnect WAN between regions may be slow and/or expensive.
You could also be thinking something between cold and warm DR is where you want to be and therefore only apply the full DATA receive when you are ready. That could mean a lot of snapshots likely to apply afterwards. Or maybe not I have not checked on that aspect of a recovery process.
Regardless I like the idea of leveraging zfs with object storage so you may not have a use for this but I definitely will.
Incremental snapshots:
server 1:
Add more data to source, snapshot and backup to object storage.
# rsync -a /media/sf_DATA/MySrc /DATA/
# du -sh /DATA/MySrc/
1.1M /DATA/MySrc/
# zfs snapshot DATA@20190721-0809
# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
DATA@20190721-0752 31K - 1.40G -
DATA@20190721-0809 0B - 1.41G -
# /usr/local/bin/zfsbackup-go send --increment DATA s3://zfs-poc
2019/07/21 08:10:49 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Done.
Total ZFS Stream Bytes: 1202792 (1.1 MiB)
Total Bytes Written: 254909 (249 KiB)
Elapsed Time: 228.123591ms
Total Files Uploaded: 2
# /usr/local/bin/zfsbackup-go list s3://zfs-poc
2019/07/21 08:11:17 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Found 2 backup sets:
Volume: DATA
Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
Replication: false
Archives: 6 - 1176757570 bytes (1.1 GiB)
Volume Size (Raw): 1514016976 bytes (1.4 GiB)
Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)
Volume: DATA
Snapshot: 20190721-0809 (2019-07-21 08:09:47 -0500 CDT)
Incremental From Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
Intermediary: false
Replication: false
Archives: 1 - 254909 bytes (249 KiB)
Volume Size (Raw): 1202792 bytes (1.1 MiB)
Uploaded: 2019-07-21 08:10:49.3280703 -0500 CDT (took 214.139056ms)
There are 4 manifests found locally that are not on the target destination.
server 2:
# /usr/local/bin/zfsbackup-go list s3://zfs-poc
2019/07/21 08:11:44 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Found 2 backup sets:
Volume: DATA
Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
Replication: false
Archives: 6 - 1176757570 bytes (1.1 GiB)
Volume Size (Raw): 1514016976 bytes (1.4 GiB)
Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)
Volume: DATA
Snapshot: 20190721-0809 (2019-07-21 08:09:47 -0500 CDT)
Incremental From Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
Intermediary: false
Replication: false
Archives: 1 - 254909 bytes (249 KiB)
Volume Size (Raw): 1202792 bytes (1.1 MiB)
Uploaded: 2019-07-21 08:10:49.3280703 -0500 CDT (took 214.139056ms)
** not sure why I need to force (-F) maybe because data set is mounted? message like this:
** cannot receive incremental stream: destination DATA has been modified since most recent snapshot
*** 2019/07/21 08:12:25 Error while trying to read from volume DATA|20190721-0752|to|20190721-0809.zstream.gz.vol1 - io: read/write on closed pipe
# /usr/local/bin/zfsbackup-go receive --auto DATA s3://zfs-poc DATA -F
2019/07/21 08:12:53 Ignoring user provided number of cores (2) and using the number of detected cores (1).
2019/07/21 08:12:54 Done. Elapsed Time: 379.712693ms
2019/07/21 08:12:54 Done.
# ls /DATA/
MySrc MyWorkDocs
# du -sh /DATA/MySrc/
1.1M /DATA/MySrc/
# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
DATA@20190721-0752 30K - 1.40G -
DATA@20190721-0809 34K - 1.41G -
LINK: https://github.com/someone1/zfsbackup-go
