Bash Read Json Config File

Couple of things here:

• I wanted to do some restic scripts
• At the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.

Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.

You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can\'t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use shopt -s latpipe to get around this. Apparently this is not a problem with ksh.

This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.

Example json config file.

$ cat restic-jobs.json 
{ Jobs:
  [
   {
    jobname: aws-s3,
    repo: sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO,
    sets:
      [
       {
        folders: [ /DATA ],
        excludes: [ .snapshots,temp],
        tags: [ data,biz ]
       },
       {
        folders: [ /ARCHIVE ],
        excludes: [ .snapshots,temp],
        tags: [ archive,biz ]
       }
      ],
      quiet: true
    },
    {
     jobname: azure-onedrive,
     repo:  rclone:azure-onedrive:restic-backups,
     sets:
       [
       {
        folders: [ /DATA ],
        excludes: [ .snapshots,temp],
        tags: [ data,biz ]
       },
       {
        folders: [ /ARCHIVE ],
        excludes: [ .snapshots,temp],
        tags: [ archive,biz ]
       }
      ],
     quiet: true
    }
  ]
} 

Script details.

$ cat restic-jobs.sh 
#!/bin/bash
#v0.9.1

JOB=aws-s3
eval $(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + =\ + .value + \' restic-jobs.json)
if [[ $jobname ==  ]]; then
  echo no job found in config:  $JOB
  exit
fi

echo found: $jobname

#sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )

echo

sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)

backup_jobs=()
## need this for bash issue with variables and pipe subshell
shopt -s lastpipe

echo $sets | jq -rc '.sets[]' | while IFS='' read set;do
    cmd_line=restic backup -q --json 

    folders=$(echo $set | jq -r '.folders | .[]')
    for st in $folders; do cmd_line+= $st; done
    excludes=$(echo $set | jq -r '.excludes | .[]')
    for st in $excludes; do cmd_line+= --exclude $st; done
    tags=$(echo $set | jq -r '.tags | .[]')
    for st in $tags; do cmd_line+= --tag $st; done

    backup_jobs+=($cmd_line)
done

for i in ${backup_jobs[@]}; do
  echo cmd_line: $i
done

Script run example. Note I am not passing the job name just hard code at the top for my test.

$ ./restic-jobs.sh 
found: iqonda-aws-s3

cmd_line: restic backup -q --json  /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz
cmd_line: restic backup -q --json  /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz