Couple of things here. I wanted to do some restic scripts but at the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.
Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.
You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can’t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use “shopt -s latpipe” to get around this. Apparently this is not a problem with ksh.
This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.
Example json config file.
$ cat restic-jobs.json
{ "Jobs":
[
{
"jobname": "aws-s3",
"repo": "sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO",
"sets":
[
{
"folders": [ "/DATA" ],
"excludes": [ ".snapshots","temp"],
"tags": [ "data","biz" ]
},
{
"folders": [ "/ARCHIVE" ],
"excludes": [ ".snapshots","temp"],
"tags": [ "archive","biz" ]
}
],
"quiet": "true"
},
{
"jobname": "azure-onedrive",
"repo": "rclone:azure-onedrive:restic-backups",
"sets":
[
{
"folders": [ "/DATA" ],
"excludes": [ ".snapshots","temp"],
"tags": [ "data","biz" ]
},
{
"folders": [ "/ARCHIVE" ],
"excludes": [ ".snapshots","temp"],
"tags": [ "archive","biz" ]
}
],
"quiet": "true"
}
]
}
Script details.
cat restic-jobs.sh
#!/bin/bash
#v0.9.1
JOB="aws-s3"
eval "$(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + "=\"" + .value + "\""' restic-jobs.json)"
if [[ "$jobname" == "" ]]; then
echo "no job found in config: " $JOB
exit
fi
echo "found: $jobname"
#sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )
echo
sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)
backup_jobs=()
## need this for bash issue with variables and pipe subshell
shopt -s lastpipe
echo $sets | jq -rc '.sets[]' | while IFS='' read set;do
cmd_line="restic backup -q --json "
folders=$(echo "$set" | jq -r '.folders | .[]')
for st in $folders; do cmd_line+=" $st"; done
excludes=$(echo "$set" | jq -r '.excludes | .[]')
for st in $excludes; do cmd_line+=" --exclude $st"; done
tags=$(echo "$set" | jq -r '.tags | .[]')
for st in $tags; do cmd_line+=" --tag $st"; done
backup_jobs+=("$cmd_line")
done
for i in "${backup_jobs[@]}"; do
echo "cmd_line: $i"
done
Script run example. Note I am not passing the job name just hard code at the top for my test.
./restic-jobs.sh found: iqonda-aws-s3 cmd_line: restic backup -q --json /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz cmd_line: restic backup -q --json /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz