json – Riaan's SysAdmin Blog

Couple of things here. I wanted to do some restic scripts but at the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.

Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.

You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can’t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use “shopt -s latpipe” to get around this. Apparently this is not a problem with ksh.

This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.

Example json config file.

$ cat restic-jobs.json 
{ "Jobs":
  [
   {
    "jobname": "aws-s3",
    "repo": "sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO",
    "sets":
      [
       {
        "folders": [ "/DATA" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "data","biz" ]
       },
       {
        "folders": [ "/ARCHIVE" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "archive","biz" ]
       }
      ],
      "quiet": "true"
    },
    {
     "jobname": "azure-onedrive",
     "repo":  "rclone:azure-onedrive:restic-backups",
     "sets":
       [
       {
        "folders": [ "/DATA" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "data","biz" ]
       },
       {
        "folders": [ "/ARCHIVE" ],
        "excludes": [ ".snapshots","temp"],
        "tags": [ "archive","biz" ]
       }
      ],
     "quiet": "true"
    }
  ]
}

Script details.

cat restic-jobs.sh 
#!/bin/bash
#v0.9.1

JOB="aws-s3"
eval "$(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + "=\"" + .value + "\""' restic-jobs.json)"
if [[ "$jobname" == "" ]]; then
  echo "no job found in config: " $JOB
  exit
fi

echo "found: $jobname"

#sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )

echo

sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)

backup_jobs=()
## need this for bash issue with variables and pipe subshell
shopt -s lastpipe

echo $sets | jq -rc '.sets[]' | while IFS='' read set;do
    cmd_line="restic backup -q --json "

    folders=$(echo "$set" | jq -r '.folders | .[]')
    for st in $folders; do cmd_line+=" $st"; done
    excludes=$(echo "$set" | jq -r '.excludes | .[]')
    for st in $excludes; do cmd_line+=" --exclude $st"; done
    tags=$(echo "$set" | jq -r '.tags | .[]')
    for st in $tags; do cmd_line+=" --tag $st"; done

    backup_jobs+=("$cmd_line")
done

for i in "${backup_jobs[@]}"; do
  echo "cmd_line: $i"
done

Script run example. Note I am not passing the job name just hard code at the top for my test.

./restic-jobs.sh 
found: iqonda-aws-s3

cmd_line: restic backup -q --json  /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz
cmd_line: restic backup -q --json  /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz

Restic has the ability to show output in json. Here is how I used it for some CSV type reporting I needed on backup jobs.

Example json output.

# restic -r rclone:s3_phx:/restic-backup backup /root --json | jq -r 'select(.message_type=="summary")'
{
  "message_type": "summary",
  "files_new": 0,
  "files_changed": 0,
  "files_unmodified": 761,
  "dirs_new": 0,
  "dirs_changed": 0,
  "dirs_unmodified": 0,
  "data_blobs": 0,
  "tree_blobs": 0,
  "data_added": 0,
  "total_files_processed": 761,
  "total_bytes_processed": 251861194,
  "total_duration": 1.118076434,
  "snapshot_id": "09d1a6b9"
}

With @csv filter.

# restic -r rclone:s3_phx:/restic-backup backup /root --json | jq -r 'select(.message_type=="summary") | [.files_new,.files_changed,.files_unmodified,.dirs_new,.dirs_changed,.dirs_unmodified,.data_blobs,.tree_blobs,.data_added,.total_files_processed,.total_bytes_processed,.total_duration,.snapshot_id] | @csv'
0,0,764,0,0,0,0,0,0,764,251918381,1.037043765,"3a55b3b3"

I needed double quotes and could not figure out how to tell @csv filter to quote so below workaround for now. This was then usable in my bash script.

# restic -r rclone:s3_phx:/restic-backup backup /root --json | jq -r 'select(.message_type=="summary") | "\"\(.files_new)\",\"\(.files_changed)\",\"\(.files_unmodified)\",\"\(.dirs_new)\",\"\(.dirs_changed)\",\"\(.dirs_unmodified)\",\"\(.data_blobs)\",\"\(.tree_blobs)\",\"\(.data_added)\",\"\(.total_files_processed)\",\"\(.total_bytes_processed)\",\"\(.total_duration)\",\"\(.snapshot_id)\""'
"0","3","761","0","0","0","3","1","2901845","764","251920790","2.035211002","fb9d780b"

Category: json

Bash Read Json Config File

Restic json output and jq