Systemctl With Docker and ZFS

admin May 20, 2020

I previously wrote about Ubuntu 20.04 as a rpool(boot volume) on OCI (Oracle Cloud Infrastucture). If using a ZFS rpool you probably wont have this silly race condition I am writing about here.

So for this POC I was using Docker and an isci mounted disk for the Docker root folder. Unfortunately there are a couple issues. The first one not related to Docker just booting up and the zpool not being imported. Fix A is for that. The second issue is that Docker may not wait for the zpool to be ready before it starts and just automatically lay down its docker folder you specified in daemon.json. And of course then zfs will not mount even if it was imported with fix A.

Fix A

If you don't know yet please create your zpool with the by-id device name not for example /dev/sdb. If this zpool was already created you can fix this after the fact with export and import and updating the cache.

You can look at systemctl status zfs-import-cache.service to see what happened at boot with this zpool. There are many opinions on how to fix this; suffice to say this is what I used and it works reliably for me so far.

Create service

# cat /etc/systemd/system/tank01-pool.service
[Unit]
Description=Zpool start service
After=dev-disk-by\x2did-wwn\x2d0x6081a22b818449d287b13b59a47bc407.device

[Service]
Type=simple
ExecStart=/usr/sbin/zpool import tank01
ExecStartPost=/usr/bin/logger "started ZFS pool tank01"

[Install]
WantedBy=dev-disk-by\x2did-wwn\x2d0x6081a22b818449d287b13b59a47bc407.device

# systemctl daemon-reload
# systemctl enable tank01-pool.service

# systemctl status tank01-pool.service
● tank01-pool.service - Zpool start service
     Loaded: loaded (/etc/systemd/system/tank01-pool.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Tue 2020-05-19 02:18:05 UTC; 5min ago
   Main PID: 1018 (code=exited, status=0/SUCCESS)

May 19 02:18:01 usph-vmli-do01 systemd[1]: Starting Zpool start service...
May 19 02:18:01 usph-vmli-do01 root[1019]: started ZFS pool tank01
May 19 02:18:01 usph-vmli-do01 systemd[1]: Started Zpool start service.
May 19 02:18:05 usph-vmli-do01 systemd[1]: tank01-pool.service: Succeeded.

To find your exact device

# systemctl list-units --all --full | grep disk | grep tank01
      dev-disk-by\x2did-scsi\x2d36081a22b818449d287b13b59a47bc407\x2dpart1.device                                                                                       loaded    active   plugged   BlockVolume tank01                                                           
      dev-disk-by\x2did-wwn\x2d0x6081a22b818449d287b13b59a47bc407\x2dpart1.device                                                                                       loaded    active   plugged   BlockVolume tank01                                                           
      dev-disk-by\x2dlabel-tank01.device                                                                                                                                loaded    active   plugged   BlockVolume tank01                                                           
      dev-disk-by\x2dpartlabel-zfs\x2d9eb05ecca4da97f6.device                                                                                                           loaded    active   plugged   BlockVolume tank01                                                           
      dev-disk-by\x2dpartuuid-d7d69ee0\x2d4e45\x2d3148\x2daa7a\x2d7cf375782813.device                                                                                   loaded    active   plugged   BlockVolume tank01                                                           
      dev-disk-by\x2dpath-ip\x2d169.254.2.2:3260\x2discsi\x2diqn.2015\x2d12.com.oracleiaas:16bca793\x2dc861\x2d49e8\x2da903\x2dd6b3809fe694\x2dlun\x2d1\x2dpart1.device loaded    active   plugged   BlockVolume tank01                                                           
      dev-disk-by\x2duuid-9554707573611221628.device                                                                                                                    loaded    active   plugged   BlockVolume tank01                                                           

# ls -l /dev/disk/by-id/ | grep sdb
    lrwxrwxrwx 1 root root  9 May 18 22:32 scsi-36081a22b818449d287b13b59a47bc407 -> ../../sdb
    lrwxrwxrwx 1 root root 10 May 18 22:32 scsi-36081a22b818449d287b13b59a47bc407-part1 -> ../../sdb1
    lrwxrwxrwx 1 root root 10 May 18 22:33 scsi-36081a22b818449d287b13b59a47bc407-part9 -> ../../sdb9
    lrwxrwxrwx 1 root root  9 May 18 22:32 wwn-0x6081a22b818449d287b13b59a47bc407 -> ../../sdb
    lrwxrwxrwx 1 root root 10 May 18 22:32 wwn-0x6081a22b818449d287b13b59a47bc407-part1 -> ../../sdb1
    lrwxrwxrwx 1 root root 10 May 18 22:33 wwn-0x6081a22b818449d287b13b59a47bc407-part9 -> ../../sdb9

Fix B

This was done before and just showing for reference how you enable the docker zfs storage.

# cat /etc/docker/daemon.json
{ 
  "storage-driver": "zfs",
  "data-root": "/tank01/docker"
}

For the timing issue you have many options in systemctl and probably better than this. For me just delaying a little until isci and zpool import/mount is done works OK.

# grep sleep /etc/systemd/system/multi-user.target.wants/docker.service 
ExecStartPre=/bin/sleep 60

# systemctl daemon-reload