Author: admin

Recording a quick test of Vault.

## hashicorp vault: https://www.vaultproject.io
## download vault executable and move to /usr/sbin so we have it in the path for this test. should rather be in /usr/local/bin

$ vault -autocomplete-install
$ exec $SHELL

$ vault server -dev
==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.3.4

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
...

## new terminal 
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ export VAULT_DEV_ROOT_TOKEN_ID="<...>"

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.3.4
Cluster Name    vault-cluster-f802bf67
Cluster ID      aa5c7006-9c7c-c394-f1f4-1a9dafc17688
HA Enabled      false

$ vault kv put secret/awscreds-iqonda {AWS_SECRET_ACCESS_KEY=<...>,AWS_ACCESS_KEY_ID=<...>}
Key              Value
---              -----
created_time     2020-03-20T18:58:57.461120823Z
deletion_time    n/a
destroyed        false
version          4

$ vault kv get -format=json secret/awscreds-iqonda | jq -r '.data["data"]'
{
  "AWS_ACCESS_KEY_ID": "<...>",
  "AWS_SECRET_ACCESS_KEY": "<...>"
}

$ vault kv get -format=json secret/awscreds-iqonda | jq -r '.data["data"] | .AWS_ACCESS_KEY_ID'
<...>

$ vault kv get -format=json secret/awscreds-iqonda | jq -r '.data["data"] | .AWS_SECRET_ACCESS_KEY'
<...>

Recording my test running AWS CLI in a docker image.

## get a base ubuntu image

# docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
...

## install the Aws Cli and commit to a image

# docker run -it --name awscli ubuntu /bin/bash
root@25b777958aad:/# apt update
root@25b777958aad:/# apt upgrade
root@25b777958aad:/# apt install awscli
root@25b777958aad:/# exit

# docker commit 25b777958aad awscli
sha256:9e1f0fef4051c86c3e1b9beecd20b29a3f11f86b5a63f1d03fefc41111f8fb47

## alias to run a docker image with cli commands

# alias awscli="docker run -it --name aws-iqonda --rm -e AWS_DEFAULT_REGION='us-east-1' -e AWS_ACCESS_KEY_ID='<...>' -e AWS_SECRET_ACCESS_KEY='<...>' --entrypoint aws awscli"

# awscli s3 ls | grep ls-al
2016-02-17 15:43:57 j.ls-al.com

# awscli ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId,Tags[?Key==`Name`].Value|[0],State.Name,PrivateIpAddress,PublicIpAddress]' --output text
i-0e38cd17dfed16658	ec2server	running	172.31.48.7	xxx.xxx.xxx.xxx

## one way to hide key variables with pass/gpg https://blog.gruntwork.io/authenticating-to-aws-with-environment-variables-e793d6f6d02e

$ pass init <email@addr.ess>
$ pass insert awscreds-iqonda/aws-access-key-id
$ pass insert awscreds-iqonda/aws-secret-access-key

$ pass
Password Store
└── awscreds-iqonda
    ├── aws-access-key-id
    └── aws-secret-access-key

$ pass awscreds-iqonda/aws-access-key-id
<...>
$ pass awscreds-iqonda/aws-secret-access-key
<...>

$ export AWS_ACCESS_KEY_ID=$(pass awscreds-iqonda/aws-access-key-id)
$ export AWS_SECRET_ACCESS_KEY=$(pass awscreds-iqonda/aws-secret-access-key)

** TODO: how to batch this? this is fine for desktop use but I do not want a gpg keyring password prompt either text or graphic in a server scripting situation. Maybe look at hashicorp vault?

$ env | grep AWS
AWS_SECRET_ACCESS_KEY=<...>
AWS_ACCESS_KEY_ID=<...>

## for convenience use an alias
$ alias awscli="sudo docker run -it --name aws-iqonda --rm -e AWS_DEFAULT_REGION='us-east-1' -e AWS_ACCESS_KEY_ID='$AWS_ACCESS_KEY_ID' -e AWS_SECRET_ACCESS_KEY='$AWS_SECRET_ACCESS_KEY' --entrypoint aws awscli"

$ awscli s3 ls 

Some useful References:

• https://www.tecmint.com/install-run-and-delete-applications-inside-docker-containers/
• https://blog.gruntwork.io/authenticating-to-aws-with-environment-variables-e793d6f6d02e
• https://aws.amazon.com/blogs/aws/aws-secrets-manager-store-distribute-and-rotate-credentials-securely/
• https://lostechies.com/gabrielschenker/2016/09/21/easing-the-use-of-the-aws-cli/
• https://medium.com/@hudsonmendes/docker-have-a-ubuntu-development-machine-within-seconds-from-windows-or-mac-fd2f30a338e4
• https://unix.stackexchange.com/questions/60213/gpg-asks-for-password-even-with-passphrase

My experiment to:

• create an Ubuntu 20.04 (not final release as of Feb 20) server in Virtualbox
• setup server with a ZFS root disk
• enable serial console
• Virtualbox export to OCI

As you probably know newer desktop versions of Ubuntu will offer ZFS for root volume during installation. I am not sure if that is true in Ubuntu 20.04 server installs and when I looked at the 19.10 ZFS installation I did not necessarily want to use the ZFS layout they did. My experiment is my custom case and also tested on Ubuntu 16.04 and 18.04.

Note this is an experiment and ZFS layout, boot partition type, LUKS, EFI, multiple boot disks(mirrored) and netplan are all debatable configurations. Mine may not be ideal but it works for my use case.

Also my goal here was to export a bootable/usable OCI (Oracle Cloud Infrastructure) compute instance.

Start by booting a recent desktop live CD. Since I am testing 20.04 (focal) I used that. In the live cd environment open a terminal, sudo and apt install ssh. Start the ssh service and set ubuntu user password.

$ ssh ubuntu@192.168.1.142
$ sudo -i

##
apt-add-repository universe
apt update
apt install --yes debootstrap gdisk zfs-initramfs

## find correct device name for below
DISK=/dev/disk/by-id/ata-VBOX_HARDDISK_VB26c080f2-2bd16227
USER=ubuntu
HOST=server
POOL=ubuntu

##
sgdisk --zap-all $DISK
sgdisk --zap-all $DISK
sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK
sgdisk     -n2:1M:+512M   -t2:EF00 $DISK
sgdisk     -n3:0:+1G      -t3:BF01 $DISK
sgdisk     -n4:0:0        -t4:BF01 $DISK
sgdisk --print $DISK

##
zpool create -o ashift=12 -d \
    -o feature@async_destroy=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@extensible_dataset=enabled \
    -o feature@filesystem_limits=enabled \
    -o feature@hole_birth=enabled \
    -o feature@large_blocks=enabled \
    -o feature@lz4_compress=enabled \
    -o feature@spacemap_histogram=enabled \
    -o feature@userobj_accounting=enabled \
    -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \
    -O normalization=formD -O relatime=on -O xattr=sa \
    -O mountpoint=/ -R /mnt bpool ${DISK}-part3

zpool create -o ashift=12 \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
    -O mountpoint=/ -R /mnt rpool ${DISK}-part4

zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=off -o mountpoint=none bpool/BOOT

zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu
zfs mount rpool/ROOT/ubuntu

zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/ubuntu
zfs mount bpool/BOOT/ubuntu

## Note: I skipped creating datasets for home, root, var/lib/ /var/log etc etc

##
debootstrap focal /mnt
zfs set devices=off rpool

## 
cat > /mnt/etc/netplan/01-netcfg.yaml<< EOF
network:
  version: 2
  ethernets:
    enp0s3:
      dhcp4: true
EOF

##
cat > /mnt/etc/apt/sources.list<< EOF
deb http://archive.ubuntu.com/ubuntu focal main universe
EOF

##
mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login

##
locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
echo "US/Central" > /etc/timezone    
dpkg-reconfigure -f noninteractive tzdata

##
passwd

##
apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs
apt install --yes grub-pc
grub-probe /boot

##update-initramfs -u -k all  <- this does not work. try below 
KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

# edit /etc/default/grub
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=5
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/ubuntu console=tty1 console=ttyS0,115200"
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

##
update-grub
grub-install $DISK

##
cat > /etc/systemd/system/zfs-import-bpool.service<< EOF
[Unit]
  DefaultDependencies=no
  Before=zfs-import-scan.service
  Before=zfs-import-cache.service
    
[Service]
  Type=oneshot
  RemainAfterExit=yes
  ExecStart=/sbin/zpool import -N -o cachefile=none bpool
    
[Install]
  WantedBy=zfs-import.target
EOF

systemctl enable zfs-import-bpool.service

##
zfs set mountpoint=legacy bpool/BOOT/ubuntu
echo bpool/BOOT/ubuntu /boot zfs \
    nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab
zfs snapshot bpool/BOOT/ubuntu@install
zfs snapshot rpool/ROOT/ubuntu@install

##
systemctl enable serial-getty@ttyS0
apt install ssh
systemctl enable ssh

##
exit
##
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

##
reboot

** detach live cd

NOTE: grub had a prompt on reboot but no options. try below

KERNEL=ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'
update-initramfs -u -k $KERNEL
update-grub

REF: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS

As you know bash is not ideal with multi arrays. Frequently I find myself wanting to read something like json into bash and loop over it. There are many ways to do this including readarray etc. I found this to work best for me. Note json can have lists so I collapse those with jq’s join. Example:

# cat restic-loop-snaps.sh 
#!/bin/bash

function loopOverArray(){

   restic snapshots --json | jq -r '.' | jq -c '.[]'| while read i; do
	id=$(echo "$i" | jq -r '.| .short_id')
	ctime=$(echo "$i" | jq -r '.| .time')
	hostname=$(echo "$i" | jq -r '.| .hostname')
	paths=$(echo "$i" | jq -r '. | .paths | join(",")')
	tagss=$(echo "$i" | jq -r '. | .tags | join(",")')
	printf "%-10s - %-40s - %-20s - %-30s - %-20s\n" $id $ctime $hostname $paths $tags
    done
}

loopOverArray

# ./restic-loop-snaps.sh 
0a71b5d4   - 2019-05-31T05:03:20.655922639-05:00      - pop-os               - /DATA/MyWorkDocs        -                     
...