Bash Read Json Config File

Couple of things here. I wanted to do some restic scripts but at the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.

Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.

You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can’t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use “shopt -s latpipe” to get around this. Apparently this is not a problem with ksh.

This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.

Example json config file.

$ cat restic-jobs.json 
{ "Jobs":
  [
   {
    "jobname": "aws-s3",
    "repo": "sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO",
    "sets":
      [
       {
        "folders": [ "/DATA" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "data","biz" ]
       },
       {
        "folders": [ "/ARCHIVE" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "archive","biz" ]
       }
      ],
      "quiet": "true"
    },
    {
     "jobname": "azure-onedrive",
     "repo":  "rclone:azure-onedrive:restic-backups",
     "sets":
       [
       {
        "folders": [ "/DATA" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "data","biz" ]
       },
       {
        "folders": [ "/ARCHIVE" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "archive","biz" ]
       }
      ],
     "quiet": "true"
    }
  ]
} 

Script details.

cat restic-jobs.sh 
#!/bin/bash
#v0.9.1

JOB="aws-s3"
eval "$(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + "=\"" + .value + "\""' restic-jobs.json)"
if [[ "$jobname" == "" ]]; then
  echo "no job found in config: " $JOB
  exit
fi

echo "found: $jobname"

#sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )

echo

sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)

backup_jobs=()
## need this for bash issue with variables and pipe subshell
shopt -s lastpipe

echo $sets | jq -rc '.sets[]' | while IFS='' read set;do
    cmd_line="restic backup -q --json "

    folders=$(echo "$set" | jq -r '.folders | .[]')
    for st in $folders; do cmd_line+=" $st"; done
    excludes=$(echo "$set" | jq -r '.excludes | .[]')
    for st in $excludes; do cmd_line+=" --exclude $st"; done
    tags=$(echo "$set" | jq -r '.tags | .[]')
    for st in $tags; do cmd_line+=" --tag $st"; done

    backup_jobs+=("$cmd_line")
done

for i in "${backup_jobs[@]}"; do
  echo "cmd_line: $i"
done

Script run example. Note I am not passing the job name just hard code at the top for my test.

./restic-jobs.sh 
found: iqonda-aws-s3

cmd_line: restic backup -q --json  /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz
cmd_line: restic backup -q --json  /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz

Oracle OCI CLI Query

Some bash snippets of using –query, jq and interacting with Bash to manipulate into variables.

Collect boot volume’s id

SRCBOOTVOLID=$(oci --profile $profile bv boot-volume list --compartment-id "$source_compartment" --availability-domain "$source_ad" --query "data [?\"display-name\" == '$instance_name (Boot Volume)'].{id:id}" | jq -r '.[] | .id')

Collect instance ocid

INSTANCEID=$(oci --profile $profile compute instance launch --availability-domain $target_ad --compartment-id $sandbox_compartment --shape VM.Standard1.1 --display-name "burner-$instance_name-instance-for-custom-image" --source-boot-volume-id $BOOTVOLID --wait-for-state RUNNING --subnet-id $sandbox_subnetid --query "data .{id:id}" | jq -r '. | .id')

Stop instance and collect the id (or whatever you need from the json)

STOPPEDID=$(oci --profile $profile compute instance action --action STOP --instance-id $INSTANCEID --wait-for-state STOPPED --query "data .{id:id}" | jq -r '. | .id')

Collect the work-request-id to monitor in a loop after I export a custom image to object storage. Note in the query the field I need is NOT in the data section.

WORKREQUESTID=$(oci --profile $profile compute image export to-object --image-id $IMAGEID --namespace faketenancy --bucket-name DR-Images --name $today-$instance_name-custom-image-object --query '"opc-work-request-id"' --raw-output)

while [ "$RESULT" != "SUCCEEDED" ]
do
  RESULT=$(oci --profile myprofile work-requests work-request get --work-request-id $WORKREQUESTID --query "data .{status:status}" | jq -r '. | .status')
  echo "running export job and $RESULT checking every 2 mins"
  sleep 2m
done

Restic scripting plus jq and minio client

I am jotting down some recent work on scripting restic and also using restic’s json output with jq and mc (minio client).

NOTE this is not production just example. Use at your own risk. These are edited by hand from real working scripts but since they are edited they will probably have typos etc in them. Again just examples!

Example backup script. Plus uploading json output to an object storage bucket for analysis later.

# cat restic-backup.sh
#!/bin/bash
source /root/.restic-keys
resticprog=/usr/local/bin/restic-custom
#rcloneargs="serve restic --stdio --b2-hard-delete --cache-workers 64 --transfers 64 --retries 21"
region="s3_phx"
rundate=$(date +"%Y-%m-%d-%H%M")
logtop=/reports
logyear=$(date +"%Y")
logmonth=$(date +"%m")
logname=$logtop/$logyear/$logmonth/restic/$rundate-restic-backup
jsonspool=/tmp/restic-fss-jobs

## Backing up some OCI FSS (same as AWS EFS) NFS folders
FSS=(
"fs-oracle-apps|fs-oracle-apps|.snapshot"           ## backup all exclude .snapshot tree
"fs-app1|fs-app1|.snapshot"                         ## backup all exclude .snapshot tree
"fs-sw|fs-sw/oracle_sw,fs-sw/restic_pkg|.snapshot"  ## backup two folders exclude .snapshot tree
"fs-tifs|fs-tifs|.snapshot,.tif"                  ## backup all exclude .snapshot tree and *.tif files
)

## test commands especially before kicking off large backups
function verify_cmds
{
  f=$1
  restic_cmd=$2
  printf "\n$rundate and cmd: $restic_cmd\n"
}

function backup
{
 f=$1
 restic_cmd=$2

 jobstart=$(date +"%Y-%m-%d-%H%M")

 mkdir $jsonspool/$f
 jsonfile=$jsonspool/$f/$jobstart-restic-backup.json
 printf "$jobstart with cmd: $restic_cmd\n"

 mkdir /mnt/$f
 mount -o ro xx.xx.xx.xx:/$f /mnt/$f

 ## TODO: shell issue with passing exclude from variable. verify exclude .snapshot is working
 ## TODO: not passing *.tif exclude fail?  howto pass *?
 $restic_cmd > $jsonfile

 #cat $jsonfile >> $logname-$f.log
 umount /mnt/$f
 rmdir /mnt/$f

## Using rclone to copy to OCI object storage bucket.
## Note the extra level folder so rclone can simulate 
## a server/20190711-restic.log style.
## Very useful with using minio client to analyze logs.
 rclone copy $jsonspool s3_ash:restic-backup-logs

 rm $jsonfile
 rmdir $jsonspool/$f

 jobfinish=$(date +"%Y-%m-%d-%H%M")
 printf "jobfinish $jobfinish\n"
}

for fss in "${FSS[@]}"; do
 arrFSS=(${fss//|/ })

 folders=""
 f=${arrFSS[0]}
 IFS=',' read -ra folderarr <<< ${arrFSS[1]}
 for folder in ${folderarr[@]};do folders+="/mnt/${folder} "; done

 excludearg=""
 IFS=',' read -ra excludearr <<< ${arrFSS[2]}
 for exclude in ${excludearr[@]};do excludearg+=" --exclude ${exclude}"; done

 backup_cmd="$resticprog -r rclone:$region:restic-$f backup ${folders} $excludearg --json"

## play with verify_cmds first before actual backups
 verify_cmds "$f" "$backup_cmd"
 #backup "$f" "$backup_cmd"
done

Since we have json logs in object storage lets check some of then with minio client.

# cat restic-check-logs.sh
#!/bin/bash

fss=(
 fs-oracle-apps
)

#checkdate="2019-07-11"
checkdate=$(date +"%Y-%m-%d")

for f in ${fss[@]}; do
  echo
  echo
  printf "$f:  "
  name=$(mc find s3-ash/restic-backup-logs/$f -name "*$checkdate*" | head -1)
  if [ -n "$name" ]
  then
    echo $name
    # play with sql --query later
    #mc sql --query "select * from S3Object"  --json-input .message_type=summary s3-ash/restic-backup-logs/$f/2019-07-09-1827-restic-backup.json
    mc cat $name  | jq -r 'select(.message_type=="summary")'
  else
    echo "Fail - no file found"
  fi
done

Example run of minio client against json

# ./restic-check-logs.sh

fs-oracle-apps:  s3-ash/restic-backup-logs/fs-oracle-apps/2019-07-12-0928-restic-backup.json
{
  "message_type": "summary",
  "files_new": 291,
  "files_changed": 1,
  "files_unmodified": 678976,
  "dirs_new": 0,
  "dirs_changed": 1,
  "dirs_unmodified": 0,
  "data_blobs": 171,
  "tree_blobs": 2,
  "data_added": 2244824,
  "total_files_processed": 679268,
  "total_bytes_processed": 38808398197,
  "total_duration": 1708.162522559,
  "snapshot_id": "f3e4dc06"
}

Note all of this was done with Oracle Cloud Infrastructure (OCI) object storage. Here are some observations around the OCI S3 compatible object storage.

  1. restic can not reach both us-ashburn-1 and us-phoenix-1 regions natively. s3:<tenant>.compat.objectstorage.us-ashburn-1.oraclecloud.com works but s3:<tenant>.compat.objectstorage.us-phoenix-1.oraclecloud.com does NOT work. Since restic can use rclone I am using rclone to access OCI object storage and rclone can reach both regions.
  2. rclone can reach both regions.
  3. minio command line client (mc) have the same issue as restic. Can reach us-ashburn-1 but not us-phoenix-1.
  4. minio python API can connect to us-ashburn-1 but shows an empty bucket list.