{"id":1437,"date":"2019-12-14T11:31:44","date_gmt":"2019-12-14T17:31:44","guid":{"rendered":"http:\/\/blog.ls-al.com\/?p=1437"},"modified":"2020-04-12T12:07:49","modified_gmt":"2020-04-12T17:07:49","slug":"bash-read-json-config-file","status":"publish","type":"post","link":"https:\/\/blog.ls-al.com\/bash-read-json-config-file\/","title":{"rendered":"Bash Read Json Config File"},"content":{"rendered":"<p>Couple of things here:<\/p>\n<ul>\n<li>I wanted to do some restic scripts <\/li>\n<li>At the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.<\/li>\n<\/ul>\n<p>Meanwhile I was trying json since I can definitely use bash\/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.<\/p>\n<p>You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can\\'t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use shopt -s latpipe to get around this. Apparently this is not a problem with ksh.<\/p>\n<p>This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.<\/p>\n<h3>Example json config file.<\/h3>\n<pre><code class=\"language-bash\">$\u00a0cat restic-jobs.json \n{ Jobs:\n  [\n   {\n    jobname: aws-s3,\n    repo: sftp:myuser@192.168.1.112:\/TANK\/RESTIC-REPO,\n    sets:\n      [\n       {\n        folders: [ \/DATA ],\n        excludes: [ .snapshots,temp],\n        tags: [ data,biz ]\n       },\n       {\n        folders: [ \/ARCHIVE ],\n        excludes: [ .snapshots,temp],\n        tags: [ archive,biz ]\n       }\n      ],\n      quiet: true\n    },\n    {\n     jobname: azure-onedrive,\n     repo:  rclone:azure-onedrive:restic-backups,\n     sets:\n       [\n       {\n        folders: [ \/DATA ],\n        excludes: [ .snapshots,temp],\n        tags: [ data,biz ]\n       },\n       {\n        folders: [ \/ARCHIVE ],\n        excludes: [ .snapshots,temp],\n        tags: [ archive,biz ]\n       }\n      ],\n     quiet: true\n    }\n  ]\n} <\/code><\/pre>\n<h3>Script details.<\/h3>\n<pre><code class=\"language-bash\">$ cat restic-jobs.sh \n#!\/bin\/bash\n#v0.9.1\n\nJOB=aws-s3\neval $(jq --arg JOB ${JOB} -r &#039;.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + =\\ + .value + \\&#039; restic-jobs.json)\nif [[ $jobname ==  ]]; then\n  echo no job found in config:  $JOB\n  exit\nfi\n\necho found: $jobname\n\n#sets=$(jq --arg JOB ${JOB} -r &#039;.Jobs[] | select (.jobname==$JOB) | .sets | .[]&#039; restic-jobs.json )\n\necho\n\nsets=$(jq --arg JOB ${JOB} -r &#039;.Jobs[] | select (.jobname==$JOB)&#039; restic-jobs.json)\n\nbackup_jobs=()\n## need this for bash issue with variables and pipe subshell\nshopt -s lastpipe\n\necho $sets | jq -rc &#039;.sets[]&#039; | while IFS=&#039;&#039; read set;do\n    cmd_line=restic backup -q --json \n\n    folders=$(echo $set | jq -r &#039;.folders | .[]&#039;)\n    for st in $folders; do cmd_line+= $st; done\n    excludes=$(echo $set | jq -r &#039;.excludes | .[]&#039;)\n    for st in $excludes; do cmd_line+= --exclude $st; done\n    tags=$(echo $set | jq -r &#039;.tags | .[]&#039;)\n    for st in $tags; do cmd_line+= --tag $st; done\n\n    backup_jobs+=($cmd_line)\ndone\n\nfor i in ${backup_jobs[@]}; do\n  echo cmd_line: $i\ndone<\/code><\/pre>\n<blockquote>\n<p>Script run example. Note I am not passing the job name just hard code at the top for my test.<\/p>\n<\/blockquote>\n<pre><code class=\"language-bash\">$ .\/restic-jobs.sh \nfound: iqonda-aws-s3\n\ncmd_line: restic backup -q --json  \/DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz\ncmd_line: restic backup -q --json  \/ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Couple of things here: I wanted to do some restic scripts At the same time use a configuration file. The<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,107,106,99,1],"tags":[],"class_list":["post-1437","post","type-post","status-publish","format-standard","hentry","category-bash","category-jq","category-json","category-restic","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1437","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1437"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1437\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}