Replication from unencrypted to encrypted set

My experiment to:

• create an Ubuntu 20.04 (not final release as of Feb 20) server in Virtualbox
• setup server with a ZFS root disk
• enable serial console
• Virtualbox export to OCI

As you probably know newer desktop versions of Ubuntu will offer ZFS for root volume during installation. I am not sure if that is true in Ubuntu 20.04 server installs and when I looked at the 19.10 ZFS installation I did not necessarily want to use the ZFS layout they did. My experiment is my custom case and also tested on Ubuntu 16.04 and 18.04.

Note this is an experiment and ZFS layout, boot partition type, LUKS, EFI, multiple boot disks(mirrored) and netplan are all debatable configurations. Mine may not be ideal but it works for my use case.

Also my goal here was to export a bootable/usable OCI (Oracle Cloud Infrastructure) compute instance.

Start by booting a recent desktop live CD. Since I am testing 20.04 (focal) I used that. In the live cd environment open a terminal, sudo and apt install ssh. Start the ssh service and set ubuntu user password.

$ ssh ubuntu@192.168.1.142
$ sudo -i

##
apt-add-repository universe
apt update
apt install --yes debootstrap gdisk zfs-initramfs

## find correct device name for below
DISK=/dev/disk/by-id/ata-VBOX_HARDDISK_VB26c080f2-2bd16227
USER=ubuntu
HOST=server
POOL=ubuntu

##
sgdisk --zap-all $DISK
sgdisk --zap-all $DISK
sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK
sgdisk     -n2:1M:+512M   -t2:EF00 $DISK
sgdisk     -n3:0:+1G      -t3:BF01 $DISK
sgdisk     -n4:0:0        -t4:BF01 $DISK
sgdisk --print $DISK

##
zpool create -o ashift=12 -d \
    -o feature@async_destroy=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@extensible_dataset=enabled \
    -o feature@filesystem_limits=enabled \
    -o feature@hole_birth=enabled \
    -o feature@large_blocks=enabled \
    -o feature@lz4_compress=enabled \
    -o feature@spacemap_histogram=enabled \
    -o feature@userobj_accounting=enabled \
    -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \
    -O normalization=formD -O relatime=on -O xattr=sa \
    -O mountpoint=/ -R /mnt bpool ${DISK}-part3

zpool create -o ashift=12 \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
    -O mountpoint=/ -R /mnt rpool ${DISK}-part4

zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=off -o mountpoint=none bpool/BOOT

zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu
zfs mount rpool/ROOT/ubuntu

zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/ubuntu
zfs mount bpool/BOOT/ubuntu

## Note: I skipped creating datasets for home, root, var/lib/ /var/log etc etc

##
debootstrap focal /mnt
zfs set devices=off rpool

## 
cat > /mnt/etc/netplan/01-netcfg.yaml<< EOF
network:
  version: 2
  ethernets:
    enp0s3:
      dhcp4: true
EOF

##
cat > /mnt/etc/apt/sources.list<< EOF
deb http://archive.ubuntu.com/ubuntu focal main universe
EOF

##
mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login

##
locale-gen --purge en_US.UTF-8
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
echo US/Central > /etc/timezone    
dpkg-reconfigure -f noninteractive tzdata

##
passwd

##
apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs
apt install --yes grub-pc
grub-probe /boot

##update-initramfs -u -k all  <- this does not work. try below 
KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

# edit /etc/default/grub
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=5
GRUB_CMDLINE_LINUX_DEFAULT=
GRUB_CMDLINE_LINUX=root=ZFS=rpool/ROOT/ubuntu console=tty1 console=ttyS0,115200
GRUB_TERMINAL=serial console
GRUB_SERIAL_COMMAND=serial --unit=0 --speed=115200

##
update-grub
grub-install $DISK

##
cat > /etc/systemd/system/zfs-import-bpool.service<< EOF
[Unit]
  DefaultDependencies=no
  Before=zfs-import-scan.service
  Before=zfs-import-cache.service

[Service]
  Type=oneshot
  RemainAfterExit=yes
  ExecStart=/sbin/zpool import -N -o cachefile=none bpool

[Install]
  WantedBy=zfs-import.target
EOF

systemctl enable zfs-import-bpool.service

##
zfs set mountpoint=legacy bpool/BOOT/ubuntu
echo bpool/BOOT/ubuntu /boot zfs \
    nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab
zfs snapshot bpool/BOOT/ubuntu@install
zfs snapshot rpool/ROOT/ubuntu@install

##
systemctl enable serial-getty@ttyS0
apt install ssh
systemctl enable ssh

##
exit
##
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

** reboot

** detach live cd

NOTE: grub had a prompt on reboot but no options. try below

KERNEL=ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'
update-initramfs -u -k $KERNEL
update-grub

REF: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS

Most of you will know that Ubuntu 16.04 will have ZFS merged into the kernel. Despite licensing arguments I see this as a positive move. I recently tested btrfs replication (http://blog.ls-al.com/btrfs-replication/) but being a long time Solaris admin and understanding how easy ZFS makes things I welcome this development. Here is a quick test of ZFS replication between two Ubuntu 16.04 hosts.

Install zfs utils on both hosts.

# apt-get install zfsutils-linux

Quick and dirty create zpools using an image just for the test.

root@u1604b1-m1:~# dd if=/dev/zero of=/tank1.img bs=1G count=1 &> /dev/null
root@u1604b1-m1:~# zpool create tank1 /tank1.img 
root@u1604b1-m1:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank1  1008M    50K  1008M         -     0%     0%  1.00x  ONLINE  -

root@u1604b1-m2:~# dd if=/dev/zero of=/tank1.img bs=1G count=1 &> /dev/null
root@u1604b1-m2:~# zpool create tank1 /tank1.img
root@u1604b1-m2:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank1  1008M    64K  1008M         -     0%     0%  1.00x  ONLINE  -
root@u1604b1-m2:~# zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
tank1    55K   976M    19K  /tank1

Copy a file into the source file system.

root@u1604b1-m1:~# cp /media/sf_E_DRIVE/W.pdf /tank1/
root@u1604b1-m1:~# ls -lh /tank1
total 12M
-rwxr-x--- 1 root root 12M Apr 20 19:22 W.pdf

Take a snapshot.

root@u1604b1-m1:~# zfs snapshot tank1@snapshot1
root@u1604b1-m1:~# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank1@snapshot1      0      -  11.2M  -

Verify empty target

root@u1604b1-m2:~# zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
tank1    55K   976M    19K  /tank1

root@u1604b1-m2:~# zfs list -t snapshot
no datasets available

Send initial

root@u1604b1-m1:~# zfs send tank1@snapshot1 | ssh root@192.168.2.29 zfs recv tank1
root@192.168.2.29's password: 
cannot receive new filesystem stream: destination 'tank1' exists
must specify -F to overwrite it
warning: cannot send 'tank1@snapshot1': Broken pipe

root@u1604b1-m1:~# zfs send tank1@snapshot1 | ssh root@192.168.2.29 zfs recv -F tank1
root@192.168.2.29's password: 

Check target.

root@u1604b1-m2:~# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank1@snapshot1      0      -  11.2M  -
root@u1604b1-m2:~# ls -lh /tank1
total 12M
-rwxr-x--- 1 root root 12M Apr 20 19:22 W.pdf

Lets populate one more file and take a new snapshot.

root@u1604b1-m1:~# cp /media/sf_E_DRIVE/S.pdf /tank1
root@u1604b1-m1:~# zfs snapshot tank1@snapshot2

Incremental send

root@u1604b1-m1:~# zfs send -i tank1@snapshot1 tank1@snapshot2 | ssh root@192.168.2.29 zfs recv tank1
root@192.168.2.29's password: 

Check target

root@u1604b1-m2:~# ls -lh /tank1
total 12M
-rwxr-x--- 1 root root 375K Apr 20 19:27 S.pdf
-rwxr-x--- 1 root root  12M Apr 20 19:22 W.pdf

root@u1604b1-m2:~# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank1@snapshot1     9K      -  11.2M  -
tank1@snapshot2      0      -  11.5M  -

Start Update 03.18.15:
This is untested but I suspect if you are upgrading the kernel to 3.19.0 and you have issues you may need to change to the daily Vivid ppa. In my initial post I used stable and Utopic since Vivid was very new.
End Update 03.18.15:

This is what I did to make an Ubuntu 15.04 virtualbox guest (works for Ubuntu 14.10 also) boot with the ZFS file system.

Previous articles:

Ubuntu On a ZFS Root File System for Ubuntu 14.04

Booting Ubuntu on a ZFS Root File System

SYSTEM REQUIREMENTS
64-bit Ubuntu Live CD. (Not the alternate or 32-bit installer)
AMD64 or EM64T compatible computer. (ie: x86-64)
15GB disk, 2GB memory minimum, Virtualbox
Create new VM. I use bridged networking in case I want to use ssh during the setup.
Start the Ubuntu LiveCD and open a terminal at the desktop. I used the 15.04 64-bit alpha CD.
Control-F1 to first text terminal.

1. Setup repo.

$ sudo -i
# /etc/init.d/lightdm stop
# apt-add-repository --yes ppa:zfs-native/stable

** Change /etc/apt/sources.list.d/... down to trusty. Or utopic possibly. I did not test.

2. Install zfs.

# apt-get update
# apt-get install debootstrap ubuntu-zfs
# dmesg | grep ZFS:
[ 3900.114234] ZFS: Loaded module v0.6.3-4~trusty, ZFS pool version 5000, ZFS filesystem version 5

** takes a long time to compile initial module for 3.16.0-28-generic

3. Install ssh.

Using a ssh terminal makes it easier to copy and paste for both command execution and documentation. However with this bare bones environment at this point openssh might not install very clean. I played with it a little to get at least sshd to run.

# apt-get install ssh
# /etc/init.d/ssh start
# /usr/sbin/sshd

** check with ps if ssh process is running
** edit sshd_config and allow root login
** set root passwd

4. Setup disk partitions.

# fdisk -l
Disk /dev/loop0: 1 GiB, 1103351808 bytes, 2154984 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc2c7def9

Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 411647 409600 200M be Solaris boot
/dev/sda2 411648 31457279 31045632 14.8G bf Solaris

5. Format partitions.

# mke2fs -m 0 -L /boot/grub -j /dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part1
# zpool create -o ashift=9 rpool /dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part2

6. ZFS Setup and Mountpoints.

# zpool create -o ashift=9 rpool /dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part2
# zfs create rpool/ROOT
# zfs create rpool/ROOT/ubuntu-1
# zfs umount -a
# zfs set mountpoint=/ rpool/ROOT/ubuntu-1
# zpool set bootfs=rpool/ROOT/ubuntu-1 rpool
# zpool export rpool
# zpool import -d /dev/disk/by-id -R /mnt rpool
# mkdir -p /mnt/boot/grub
# mount /dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part1 /mnt/boot/grub

7. Install Ubuntu 15.04 on /mnt

# debootstrap vivid /mnt
I: Retrieving Release
...
I: Base system installed successfully.

# cp /etc/hostname /mnt/etc/
# cp /etc/hosts /mnt/etc/
# vi /mnt/etc/fstab
# cat /mnt/etc/fstab
/dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part1 /boot/grub auto defaults 0 1

# cat /mnt/etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

8. Setup chroot to update and install ubuntu-minimal

# mount --bind /dev /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys /mnt/sys
# chroot /mnt /bin/bash --login
# locale-gen en_US.UTF-8

# apt-get update
# apt-get install ubuntu-minimal software-properties-common

9. Setup ZOL repo

# apt-add-repository --yes ppa:zfs-native/stable

** leave grub repo off for now.

# cat /etc/apt/sources.list.d/zfs-native-ubuntu-stable-vivid.list
deb http://ppa.launchpad.net/zfs-native/stable/ubuntu trusty main
# deb-src http://ppa.launchpad.net/zfs-native/stable/ubuntu vivid main
# apt-get update
# apt-get install --no-install-recommends linux-image-generic linux-headers-generic

# apt-get install ubuntu-zfs

** skipped grub stuff for this pass

# apt-get install zfs-initramfs
# apt-get dist-upgrade

10. Make sure root has access

# passwd root

11. Test grub

# grub-probe /
bash: grub-probe: command not found

12. Use older patched grub from ZOL project

# apt-add-repository --yes ppa:zfs-native/grub
gpg: keyring `/tmp/tmp5urr4u7g/secring.gpg' created
gpg: keyring `/tmp/tmp5urr4u7g/pubring.gpg' created
gpg: requesting key F6B0FC61 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmp5urr4u7g/trustdb.gpg: trustdb created
gpg: key F6B0FC61: public key "Launchpad PPA for Native ZFS for Linux" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK

# cat /etc/apt/sources.list.d/zfs-native-ubuntu-grub-vivid.list
deb http://ppa.launchpad.net/zfs-native/grub/ubuntu raring main

# apt-get install grub2-common grub-pc
Installation finished. No error reported.
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part2'.

** As you can see grub has issues with dev path.

# ln -s /dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part2 /dev/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52-part2
# apt-get install grub2-common grub-pc

# grub-probe /
zfs
# ls /boot/grub/i386-pc/zfs*
/boot/grub/i386-pc/zfscrypt.mod /boot/grub/i386-pc/zfsinfo.mod /boot/grub/i386-pc/zfs.mod

** Note at the end I show a udev rule that can help work around this path issue.

# update-initramfs -c -k all

# grep "boot=zfs" /boot/grub/grub.cfg
linux /ROOT/ubuntu-1@/boot/vmlinuz-3.16.0-28-generic root=ZFS=rpool/ROOT/ubuntu-1 ro boot=zfs quiet splash $vt_handoff

# grep "boot=zfs" /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash boot=zfs"

# update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-3.16.0-28-generic
Found initrd image: /boot/initrd.img-3.16.0-28-generic
done

# grub-install $(readlink -f /dev/disk/by-id/ata-VBOX_HARDDISK_VBf3c0d5ba-e6881c52)
Installing for i386-pc platform.
Installation finished. No error reported.

# exit
logout

13. unmount chroot and shutdown

# umount /mnt/boot/grub
# umount /mnt/dev
# umount /mnt/proc
# umount /mnt/sys
# zfs umount -a
# zpool export rpool
# init 0

14. Cleanup and finish
** Create snapshot
** bootup
** install ssh and configure root to login in sshd_config, restart ssh

15. udev rule for grub bug

# cat /etc/udev/rules.d/70-zfs-grub-fix.rules
ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*", ENV{ID_FS_TYPE}=="zfs_member", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL} $env{ID_BUS}-$env{ID_SERIAL}-part%n"

# /etc/init.d/udev restart

16. install desktop software

# apt-get install ubuntu-desktop