Bash Read Json Config File

Couple of things here. I wanted to do some restic scripts but at the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.

Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.

You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can’t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use “shopt -s latpipe” to get around this. Apparently this is not a problem with ksh.

This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.

Example json config file.

$ cat restic-jobs.json 
{ "Jobs":
  [
   {
    "jobname": "aws-s3",
    "repo": "sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO",
    "sets":
      [
       {
        "folders": [ "/DATA" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "data","biz" ]
       },
       {
        "folders": [ "/ARCHIVE" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "archive","biz" ]
       }
      ],
      "quiet": "true"
    },
    {
     "jobname": "azure-onedrive",
     "repo":  "rclone:azure-onedrive:restic-backups",
     "sets":
       [
       {
        "folders": [ "/DATA" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "data","biz" ]
       },
       {
        "folders": [ "/ARCHIVE" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "archive","biz" ]
       }
      ],
     "quiet": "true"
    }
  ]
} 

Script details.

cat restic-jobs.sh 
#!/bin/bash
#v0.9.1

JOB="aws-s3"
eval "$(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + "=\"" + .value + "\""' restic-jobs.json)"
if [[ "$jobname" == "" ]]; then
  echo "no job found in config: " $JOB
  exit
fi

echo "found: $jobname"

#sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )

echo

sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)

backup_jobs=()
## need this for bash issue with variables and pipe subshell
shopt -s lastpipe

echo $sets | jq -rc '.sets[]' | while IFS='' read set;do
    cmd_line="restic backup -q --json "

    folders=$(echo "$set" | jq -r '.folders | .[]')
    for st in $folders; do cmd_line+=" $st"; done
    excludes=$(echo "$set" | jq -r '.excludes | .[]')
    for st in $excludes; do cmd_line+=" --exclude $st"; done
    tags=$(echo "$set" | jq -r '.tags | .[]')
    for st in $tags; do cmd_line+=" --tag $st"; done

    backup_jobs+=("$cmd_line")
done

for i in "${backup_jobs[@]}"; do
  echo "cmd_line: $i"
done

Script run example. Note I am not passing the job name just hard code at the top for my test.

./restic-jobs.sh 
found: iqonda-aws-s3

cmd_line: restic backup -q --json  /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz
cmd_line: restic backup -q --json  /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz

Restic create backup and set tag with date logic

Also see previous post https://blog.ls-al.com/bash-date-usage-for-naming if you are interested. This post is similar but more specific to restic tagging.

Below is a test script and a test run. At the time of restic backup I create a tag in order to do snapshot forget based on tags.

root@pop-os:/tmp# cat backup-tags.sh 
#!/bin/bash

create_tag () {
  tag="daily"
  if [ $(date +%a) == "Sun" ]; then tag="weekly" ; fi
  if [ $(date +%d) == "01" ]; then 
   tag="monthly"
   if [ $(date +%b) == "Jan" ]; then
     tag="yearly"
   fi
  fi
}
create_tag
echo "backup policy: " $tag

create_tag_unit_test () {
  for i in {1..95}
  do 
      tdate=$(date -d "+$i day")
      tag="daily"
      if [ $(date -d "+$i day" +%a) == "Sun" ]; then tag="weekly" ; fi
      if [ $(date -d "+$i day" +%d) == "01" ]; then
      tag="monthly"
        if [ $(date -d "+$i day" +%b) == "Jan" ]; then
          tag="yearly"
        fi
      fi
  printf "%s - %s - %s | " "$(date -d "+$i day" +%d)" "$(date -d "+$i day" +%a)" "$tag" 
  if [ $(( $i %5 )) -eq 0 ]; then printf "\n"; fi
  done
}
create_tag_unit_test

root@pop-os:/tmp# ./backup-tags.sh 
backup policy:  daily
22 - Fri - daily      | 23 - Sat - daily      | 24 - Sun - weekly     | 25 - Mon - daily      | 26 - Tue - daily      | 
27 - Wed - daily      | 28 - Thu - daily      | 29 - Fri - daily      | 30 - Sat - daily      | 01 - Sun - monthly    | 
02 - Mon - daily      | 03 - Tue - daily      | 04 - Wed - daily      | 05 - Thu - daily      | 06 - Fri - daily      | 
07 - Sat - daily      | 08 - Sun - weekly     | 09 - Mon - daily      | 10 - Tue - daily      | 11 - Wed - daily      | 
12 - Thu - daily      | 13 - Fri - daily      | 14 - Sat - daily      | 15 - Sun - weekly     | 16 - Mon - daily      | 
17 - Tue - daily      | 18 - Wed - daily      | 19 - Thu - daily      | 20 - Fri - daily      | 21 - Sat - daily      | 
22 - Sun - weekly     | 23 - Mon - daily      | 24 - Tue - daily      | 25 - Wed - daily      | 26 - Thu - daily      | 
27 - Fri - daily      | 28 - Sat - daily      | 29 - Sun - weekly     | 30 - Mon - daily      | 31 - Tue - daily      | 
01 - Wed - yearly     | 02 - Thu - daily      | 03 - Fri - daily      | 04 - Sat - daily      | 05 - Sun - weekly     | 
06 - Mon - daily      | 07 - Tue - daily      | 08 - Wed - daily      | 09 - Thu - daily      | 10 - Fri - daily      | 
11 - Sat - daily      | 12 - Sun - weekly     | 13 - Mon - daily      | 14 - Tue - daily      | 15 - Wed - daily      | 
16 - Thu - daily      | 17 - Fri - daily      | 18 - Sat - daily      | 19 - Sun - weekly     | 20 - Mon - daily      | 

Below is the restic backup script setting a tag and then snapshot forget based on the tag.

As always this is NOT tested use at your own risk.

My “policy” is:

  • weekly on Sunday
  • 01 of every month is a monthly except if 01 is also a new year which makes it a yearly
  • everything else is a daily
root@pop-os:~/scripts# cat desktop-restic.sh 
#!/bin/bash
### wake up backup server and restic backup to 3TB ZFS mirror
cd /root/scripts
./wake-backup-server.sh

source /root/.restic.env

## Quick and dirty logic for snapshot tagging
create_tag () {
  tag="daily"
  if [ $(date +%a) == "Sun" ]; then tag="weekly" ; fi
  if [ $(date +%d) == "01" ]; then
   tag="monthly"
   if [ $(date +%b) == "Jan" ]; then
     tag="yearly"
   fi
  fi
}

create_tag
restic backup -q /DATA /ARCHIVE --tag "$tag" --exclude *.vdi --exclude *.iso --exclude *.ova --exclude *.img --exclude *.vmdk

restic forget -q --tag daily --keep-last 7
restic forget -q --tag weekly --keep-last 4
restic forget -q --tag monthly --keep-last 12

if [ "$tag" == "weekly" ]; then
  restic -q prune
fi

sleep 1m
ssh user@192.168.1.250 sudo shutdown now

Bash Read Array From Config File

I was recently needing to read values from a configuration file into bash and had some success with reading json with jq into bash array(s). However I resorted to a non json version which worked well. Something like this.

Config File

$cat array-simple.cfg
[bucket1]
name=bucket name 1
exclude=folder1 folder 2

[bucket2]
name=bucket name 2
exclude=folder5

Code

$ cat array-simple.sh
#!/bin/bash
while read line; do
    if [[ $line =~ ^"["(.+)"]"$ ]]; then
        arrname=${BASH_REMATCH[1]}
        declare -A $arrname
    elif [[ $line =~ ^([_[:alpha:]][_[:alnum:]]*)"="(.*) ]]; then
        declare ${arrname}[${BASH_REMATCH[1]}]="${BASH_REMATCH[2]}"
    fi
done < array-simple.cfg

echo ${bucket1[name]}
echo ${bucket1[exclude]}

echo ${bucket2[name]}
echo ${bucket2[exclude]}

for i in "${!bucket1[@]}"; do echo "$i => ${bucket1[$i]}"; done

for i in "${!bucket2[@]}"; do echo "$i => ${bucket2[$i]}"; done

Run

$ ./array-simple.sh 
bucket name 1
folder1 folder 2
bucket name 2
folder5
exclude => folder1 folder 2
name => bucket name 1
exclude => folder5
name => bucket name 2

Bash History Plus Comment

If you like using Control-R in bash to find previous commands here is a useful tip. You can add a comment to a command and then when you use Control-R searching by typing you can find it by your comment. Example I use apt update. Run command including your comment (shell will ignore the comment of course). Then when Control-R searching type your string you used in the comment.

 # apt update ; apt upgrade #quickupdate

Now hit Control-R and type to search “quick”.

(reverse-i-search)`quick': apt update ; apt upgrade #quickupdate

Test Tcp Open Port

If you don’t have telnet or nc installed and want to quickly test firewall traffic to a server and specific port you can try this. It needs a new enough bash but still pretty quick and handy.

Good test port is open

$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/172.18.10.66/1521'
$ echo $?
0

Port not open

$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/172.18.10.66/15'
$ echo $?
124

Good test port is open to google FQDN

$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/google.com/80'
[opc@ocilxeasdbt02 ~]$ echo $?
0

Bash Date Usage For Naming

I am recording some scripting I used to create backup classification/retention naming. It can be simplified into one function easily but I kept it like this so I can copy and paste easier which function I need. Script is pretty self explanatory but basically it takes today’s date and name my eventual backup file name based on some logic.

# cat test_class.sh 
HOSTNAME=$(hostname -s)
BACKUP_CLASSIFICATION="UNCLASSIFIED"

function retention_date() {
  MM=`date -d ${1} +%m`
  DD=`date -d ${1} +%d`
  DAY=`date -d ${1} +%u`

  if [ $DD == 01 ]
  then
     if [ $MM == 01 ]
     then
       BACKUP_CLASSIFICATION="YEARLY"
     else
       BACKUP_CLASSIFICATION="MONTHLY"
     fi
  else
    if (($DAY == 7)); then
     BACKUP_CLASSIFICATION="WEEKLY"
    else
     BACKUP_CLASSIFICATION="DAILY"
    fi
  fi

}

function retention_today() {
  MM=`date '+%m'`
  DD=`date '+%d'`
  DAY=`date +%u`
  
  if [ $DD == 01 ]
  then
     if [ $MM == 01 ]
     then
       BACKUP_CLASSIFICATION="YEARLY"
     else
       BACKUP_CLASSIFICATION="MONTHLY"
     fi
  else
    if (($DAY == 7)); then
     BACKUP_CLASSIFICATION="WEEKLY"
    else
     BACKUP_CLASSIFICATION="DAILY"
    fi
  fi

}

echo "TEST TODAY"
DATE=`date +%Y-%m-%d`
retention_today
echo $HOSTNAME-$BACKUP_CLASSIFICATION-$DATE
  
echo 
echo "TEST SPECIFIC DATES"
testD=(
 '2018-01-01'
 '2018-02-02'
 '2018-03-01'
 '2018-02-06'
 '2018-07-14'
 '2018-07-15'
)

for D in "${testD[@]}"
do
  DATE=`date -d ${D} +%Y-%m-%d`
  retention_date $D
  echo $HOSTNAME-$BACKUP_CLASSIFICATION-$DATE
done

Run and output.

# ./test_class.sh 
TEST TODAY
oci04-DAILY-2018-07-20

TEST SPECIFIC DATES
oci04-YEARLY-2018-01-01
oci04-DAILY-2018-02-02
oci04-MONTHLY-2018-03-01
oci04-DAILY-2018-02-06
oci04-DAILY-2018-07-14
oci04-WEEKLY-2018-07-15

Expect and bash

Quick example of spawning a problem that have a password prompt and can’t accept a documented parameter for a password. I used another bash script to simulate a password prompt but in my automation challenge it was an executable that prompted.

Main script just to take a password for passing to expect. Could also be hard coded.

$ cat expect_example_main.sh
#!/bin/bash
echo “Enter the password: ”
read -s -e password
./expect_example.exp $password ;

Here is the expect script that will be interacting with the real program with the password prompt.

$ cat expect_example.exp
#!/usr/bin/expect -f

# Set variables
set password [lindex $argv 0]
set date [exec date +%F]

# Log results
log_file -a expect-$date.log

# Announce device & time
send_user “\n”
send_user “>>>>> Working @ [exec date] <<<<<\n" send_user "\n" spawn ./expect_example_prompt.sh expect "*assword:" {send "$password\r"} interact This is the simulated executable with the prompt. Expect will be spawning this one. $ cat expect_example_prompt.sh #!/bin/bash echo "Enter the password: " read -s -e pwd if [ $pwd == 'fool' ]; then echo "password correct" else echo "password NOT correct!" fi Showing run time with correct password. $ ./expect_example_main.sh Enter the password: >>>>> Working @ Thu Jan 19 14:54:00 CST 2017 <<<<< spawn ./expect_example_prompt.sh Enter the password: password correct Showing run time with incorrect password. $ ./expect_example_main.sh Enter the password: >>>>> Working @ Thu Jan 19 14:54:28 CST 2017 <<<<< spawn ./expect_example_prompt.sh Enter the password: password NOT correct!

Powerline for your terminal on Ubuntu

I noticed on Fedora it is really easy to enable powerline in your terminal. This article explains it well: http://fedoramagazine.org/add-power-terminal-powerline/

On Ubuntu 15.04 it looks like powerline is pretty easy to enable although it did not appear like the tmux or vim powerline plugins are available from regular repos.

Enable powerline in your terminal like this:

# apt-get install powerline fonts-powerline

~ cat .bashrc
[..]
if [ -f `which powerline-daemon` ]; then
  powerline-daemon -q
  POWERLINE_BASH_CONTINUATION=1
  POWERLINE_BASH_SELECT=1
  . /usr/share/powerline/bindings/bash/powerline.sh
fi

ZFSSA List Snapshots Script

Quick script to illustrate interacting with the ZFS Storage Appliance. In this example I am listing ZFSSA snapshots containing a search string.  Note I edited this for the article without re-testing it still works.

#!/bin/sh

Usage() {
 echo "$1 -u <Appliance user> -h <appliance> -j <project> -p <pool> -s <containsString>"
 exit 1
}

PROG=$0
while getopts u:h:s:j:p flag
do
  case "$flag" in
  p) pool="$OPTARG";;
  j) project="$OPTARG";;
  s) string="$OPTARG";;
  u) user="$OPTARG";;
  h) appliance="$OPTARG";;
  \?) Usage $PROG ;;
  esac
done

[ -z "$pool" -o -z "$project" -o -z "$appliance" -o -z "$user" ] && Usage $PROG

ssh -T $user@$appliance << EOF
script
var MyArguments = {
  pool: '$pool',
  project: '$project',
  string: '$string'
}

function ListSnapshotsbyS (Arg) {
  run('cd /');                          // Make sure we are at root child context level
  run('shares');
  try {
      run('set pool=' + Arg.pool);
  } catch (err) {
      printf ('ERROR: %s\n',err);
      return (err);
  }

  var allSnaps=[];
  try {
      run('select ' + Arg.project + ' snapshots');
      snapshots=list();
      for(i=0; i < snapshots.length; i++) {
          allSnaps.push(snapshots[i]);
      }
        run('done');
  } catch (err) {
      printf ('ERROR: %s\n',err);
            return(err);
  }

  for(i=0; i < allSnaps.length; i++) {
   if (Arg.string !="") {
    var idx=allSnaps[i].indexOf(Arg.string);
    if (idx>0) {
      printf('#%i: %s contained search string %s \n',i,allSnaps[i], Arg.string);
    }
   } else {
      printf('#%i: %s \n',i,allSnaps[i]);
   }
  }
  return(0);
}
ListSnapshotsbyS(MyArguments);
.
EOF

Watch Process Id

Sometimes you want to keep tabs on a long running process and get notified by email when it is done. This is an example of just that.

#!/bin/bash
pid=$1
me="$(basename $0)($$):"
if [ -z "$pid" ]
then
    echo "$me a PID is required as an argument" >&2
    exit 2
fi

name=$(ps -p $pid -o comm=)
if [ $? -eq 0 ]
then
    echo "$me waiting for PID $pid to finish ($name)"
    while ps -p $pid > /dev/null; do sleep 1; done;
else
    echo "$me failed to find process with PID $pid" >&2
    exit 1
fi
## I used a python mailer but mostlikely this will be mail or mailx.
python pymail.py $pid