{"id":1368,"date":"2019-07-12T10:48:56","date_gmt":"2019-07-12T15:48:56","guid":{"rendered":"http:\/\/blog.ls-al.com\/?p=1368"},"modified":"2019-07-12T10:51:54","modified_gmt":"2019-07-12T15:51:54","slug":"restic-scripting-plus-jq-and-minio-client","status":"publish","type":"post","link":"https:\/\/blog.ls-al.com\/restic-scripting-plus-jq-and-minio-client\/","title":{"rendered":"Restic scripting plus jq and minio client"},"content":{"rendered":"\n

I am jotting down some recent work on scripting restic and also using restic's json output with jq and mc (minio client).<\/p>\n\n\n\n

NOTE this is not production just example. Use at your own risk. These are edited by hand from real working scripts but since they are edited they will probably have typos etc in them. Again just examples!<\/p>\n\n\n\n

Example backup script. Plus uploading json output to an object storage bucket for analysis later.<\/strong><\/p>\n\n\n

\n# cat restic-backup.sh\n#!\/bin\/bash\nsource \/root\/.restic-keys\nresticprog=\/usr\/local\/bin\/restic-custom\n#rcloneargs="serve restic --stdio --b2-hard-delete --cache-workers 64 --transfers 64 --retries 21"\nregion="s3_phx"\nrundate=$(date +"%Y-%m-%d-%H%M")\nlogtop=\/reports\nlogyear=$(date +"%Y")\nlogmonth=$(date +"%m")\nlogname=$logtop\/$logyear\/$logmonth\/restic\/$rundate-restic-backup\njsonspool=\/tmp\/restic-fss-jobs\n\n## Backing up some OCI FSS (same as AWS EFS) NFS folders\nFSS=(\n"fs-oracle-apps|fs-oracle-apps|.snapshot"           ## backup all exclude .snapshot tree\n"fs-app1|fs-app1|.snapshot"                         ## backup all exclude .snapshot tree\n"fs-sw|fs-sw\/oracle_sw,fs-sw\/restic_pkg|.snapshot"  ## backup two folders exclude .snapshot tree\n"fs-tifs|fs-tifs|.snapshot,.tif"                  ## backup all exclude .snapshot tree and *.tif files\n)\n\n## test commands especially before kicking off large backups\nfunction verify_cmds\n{\n  f=$1\n  restic_cmd=$2\n  printf "\\n$rundate and cmd: $restic_cmd\\n"\n}\n\nfunction backup\n{\n f=$1\n restic_cmd=$2\n\n jobstart=$(date +"%Y-%m-%d-%H%M")\n\n mkdir $jsonspool\/$f\n jsonfile=$jsonspool\/$f\/$jobstart-restic-backup.json\n printf "$jobstart with cmd: $restic_cmd\\n"\n\n mkdir \/mnt\/$f\n mount -o ro xx.xx.xx.xx:\/$f \/mnt\/$f\n\n ## TODO: shell issue with passing exclude from variable. verify exclude .snapshot is working\n ## TODO: not passing *.tif exclude fail?  howto pass *?\n $restic_cmd > $jsonfile\n\n #cat $jsonfile >> $logname-$f.log\n umount \/mnt\/$f\n rmdir \/mnt\/$f\n\n## Using rclone to copy to OCI object storage bucket.\n## Note the extra level folder so rclone can simulate \n## a server\/20190711-restic.log style.\n## Very useful with using minio client to analyze logs.\n rclone copy $jsonspool s3_ash:restic-backup-logs\n\n rm $jsonfile\n rmdir $jsonspool\/$f\n\n jobfinish=$(date +"%Y-%m-%d-%H%M")\n printf "jobfinish $jobfinish\\n"\n}\n\nfor fss in "${FSS[@]}"; do\n arrFSS=(${fss\/\/|\/ })\n\n folders=""\n f=${arrFSS[0]}\n IFS=',' read -ra folderarr <<< ${arrFSS[1]}\n for folder in ${folderarr[@]};do folders+="\/mnt\/${folder} "; done\n\n excludearg=""\n IFS=',' read -ra excludearr <<< ${arrFSS[2]}\n for exclude in ${excludearr[@]};do excludearg+=" --exclude ${exclude}"; done\n\n backup_cmd="$resticprog -r rclone:$region:restic-$f backup ${folders} $excludearg --json"\n\n## play with verify_cmds first before actual backups\n verify_cmds "$f" "$backup_cmd"\n #backup "$f" "$backup_cmd"\ndone\n<\/pre><\/div>\n\n\n
Since we have json logs in object storage lets check some of then with minio client.<\/strong><\/p>\n\n\n
\n# cat restic-check-logs.sh\n#!\/bin\/bash\n\nfss=(\n fs-oracle-apps\n)\n\n#checkdate="2019-07-11"\ncheckdate=$(date +"%Y-%m-%d")\n\nfor f in ${fss[@]}; do\n  echo\n  echo\n  printf "$f:  "\n  name=$(mc find s3-ash\/restic-backup-logs\/$f -name "*$checkdate*" | head -1)\n  if [ -n "$name" ]\n  then\n    echo $name\n    # play with sql --query later\n    #mc sql --query "select * from S3Object"  --json-input .message_type=summary s3-ash\/restic-backup-logs\/$f\/2019-07-09-1827-restic-backup.json\n    mc cat $name  | jq -r 'select(.message_type=="summary")'\n  else\n    echo "Fail - no file found"\n  fi\ndone\n<\/pre><\/div>\n\n\n
Example run of minio client against json<\/strong><\/p>\n\n\n
\n# .\/restic-check-logs.sh\n\nfs-oracle-apps:  s3-ash\/restic-backup-logs\/fs-oracle-apps\/2019-07-12-0928-restic-backup.json\n{\n  "message_type": "summary",\n  "files_new": 291,\n  "files_changed": 1,\n  "files_unmodified": 678976,\n  "dirs_new": 0,\n  "dirs_changed": 1,\n  "dirs_unmodified": 0,\n  "data_blobs": 171,\n  "tree_blobs": 2,\n  "data_added": 2244824,\n  "total_files_processed": 679268,\n  "total_bytes_processed": 38808398197,\n  "total_duration": 1708.162522559,\n  "snapshot_id": "f3e4dc06"\n}\n<\/pre><\/div>\n\n\n
Note all of this was done with Oracle Cloud Infrastructure (OCI) object storage.  Here are some observations around the OCI<\/strong> S3 compatible<\/strong> object storage.<\/p>\n\n\n\n
  1. restic can not reach both us-ashburn-1 and us-phoenix-1 regions natively.  s3<\/strong>:<tenant>.compat.objectstorage.us-ashburn-1.oraclecloud.com works but s3<\/strong>:<tenant>.compat.objectstorage.us-phoenix-1.oraclecloud.com does NOT work.  Since restic can use rclone I am using rclone to access OCI object storage and rclone can reach both regions.<\/li>
  2. rclone can reach both regions.<\/li>
  3. minio command line client (mc) have the same issue as restic.  Can reach us-ashburn-1 but not us-phoenix-1.<\/li>
  4. minio python API can connect to us-ashburn-1 but shows an empty bucket list.<\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"
    I am jotting down some recent work on scripting restic and also using restic’s json output with jq and mc<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[107,108,87,96,99],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1368"}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1368"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1368\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}