Using ansi colors can be very helpful to see script out[put. Example of cleaning ansi color codes from output before putting it in your log. You may want to keep them since cat would still handle it. However if you don't like your logs with unreadable codes this is an example of cleaning it first with sed.
# crontab -l
00 22 * * * /root/scripts/backup.sh -w 1 -t 192.168.1.112 | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" > "/logs//bin/date +\%Y-\%m-\%d-backup-192.168.1.112.log"
Comments Off on sed remove ansi colors from output
comments
If you need to use an alias inside a script you need this:
shopt -s expand_aliases
source ~/.bash_aliasesI recently started using the docker OCI client instead of trying to install it local. For some reason it is just not working. So now I use the docker image but as you can see you dont want to be using this command everytime.
docker run --rm -it -v "$HOME/.oci:/oracle/.oci" ociSo an alias is helpful but as mentioned wont just work in you script. Example how I use the command in a script and this works. My alias is ocicli.
CREATED_APPLY_JOB_ID=$(ocicli resource-manager job create-apply-job --stack-id $CREATED_STACK_ID --execution-plan-strategy FROM_PLAN_JOB_ID --execution-plan-job-id "$CREATED_PLAN_JOB_ID" --wait-for-state SUCCEEDED --query 'data.id' --raw-output)
Comments Off on Bash alias inside a script
comments
You probably use (and rely) on command search a lot already but thought I would add something about fish. If you are from the csh or ksh days you will recall the big adoption for bash came because of up and down scroll through command and maybe to a lesser extent tab for showing completion options.
I recently started using fish in some places. I know zsh has become popular as a bash alternative and zsh has the same powerful history search Control-R functionality.
Unfortunately with fish it is handled different. Many people prefer the way fish handles it and I admit I really like the type ahead and showing it in lighter color like shown below.
There are an ongoing push to also add Control-R search like bash and zsh handles it. As far as I can tell in v3.3.1 which is in the latest Ubuntu 22.04 beta this is not working as I wanted.
With fish you start typing and you can still then use up and down keys to get through a list of related searches. I like and dislike the way fish does it. For the moment I am still mostly going back to bash since zsh also annoys me(for other reasons). But I could end up with fish if they leave the type ahead like it is and add exact Control-R incremental search like bash and zsh.
Modern shells save previous commands in a command history. You can view earlier commands by using the up and down arrows. Fish extends this concept by integrating the history search functionality. To search the history, simply type in the search string, and press the up arrow. By using the up and down arrow, you can search for older and newer matches. The fish history automatically removes duplicate matches and the matching substring is highlighted. These features make searching and reusing previous commands much faster.
Using up and down arrows it's not a good alternative to Ctrl+R, because if the phrase you are looking for it is really deep in the history, you'll have to hit up/down keys a lot! I just use history | grep -i [phrase]
Comments Off on Linux Shell Incremental Search
comments
I have a few scripts that I only periodically execute. For example a backup to a USB not always plugged in or to a computer not always online. For convenience I made some shortcuts on my GNOME desktop. I can easily just run these in a terminal of course but sometimes I just want to quickly click an icon and finish.
There are a few idiosyncracies around executing like this. In general you may have run into passing processes variables and redirection of output issues. In addition to those also "Exec" inside a desktop shortcut adds a few more issues. Read here for the Desktop Entry Specification
For my purposes here are example entries from .desktop files:
Rsync to USB
Exec=gnome-terminal --profile=job-output -e 'sudo /home/rrosso/scripts/rsync-TANKs-2TBUSB.sh'Update IP address in a remote firewall
Exec=gnome-terminal --profile=job-output -- /bin/sh -c 'cd /TANK/DATA/MySrc ; python3 ip-add-iqonda-aws.py ; sleep 60'Executing using bash with date in log file:
Exec=bash -c "sudo /root/scripts/zfs-replication.sh -w 1 -t 192.168.1.79 | tee /TANK/backups/logs//bin/date +%%Y-%%m-%%d-desktop-zfs-replicate-192.168.1.79.log"I also prefer using gnome-terminal so I can format the output on the screen better:
Exec=gnome-terminal --profile=job-output -- bash -c "sudo /root/scripts/zfs-replication.sh -w 1 -t 192.168.1.79 | tee /TANK/backups/logs//bin/date +%%Y-%%m-%%d-desktop-zfs-replicate-192.168.1.79.log ; sleep 60"
Comments Off on Gnome Desktop Shortcut Exec
comments
start string and reverse search up to next string. then replacing a line inside the block
$ cat listener-update.sh
#!/bin/bash
#v0.9.1
file="listener_test.yml"
dir="./unregister"
variable="bar"
codeuri_num=$(grep -n "CodeUri: $dir" $file | awk '{print $1}' FS=":")
function_num=$(grep -n "Type: 'AWS::Serverless::Function'" $file | awk '{print $1}' FS=":")
block=$(sed "${function_num},${codeuri_num}!d" $file)
echo
if [[ "$block" == *"AutoPublishCodeSha256"* ]];then
echo "found AutoPublishCodeSha256 so update"
line=$(echo "${block}" | grep -n "AutoPublishCodeSha256")
line_num=$(awk "/Auto/ && NR >= ${function_num} && NR <= ${codeuri_num} {print NR}" $file)
newline="AutoPublishCodeSha256: $var"
sed -i "${line_num}s/.*/ \ \ \ \ \ AutoPublishCodeSha256: $variable/" $file
else
echo "AutoPublishCodeSha256 not found so insert"
#codeuri_num=$((codeuri_num+1))
sed -i "${codeuri_num} i \ \ \ \ \ \ AutoPublishCodeSha256: $variable" $file
fi
Comments Off on bash-scan-text-block-reverse
comments
Couple of things here:
Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.
You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can\'t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use shopt -s latpipe to get around this. Apparently this is not a problem with ksh.
This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.
$ cat restic-jobs.json
{ Jobs:
[
{
jobname: aws-s3,
repo: sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO,
sets:
[
{
folders: [ /DATA ],
excludes: [ .snapshots,temp],
tags: [ data,biz ]
},
{
folders: [ /ARCHIVE ],
excludes: [ .snapshots,temp],
tags: [ archive,biz ]
}
],
quiet: true
},
{
jobname: azure-onedrive,
repo: rclone:azure-onedrive:restic-backups,
sets:
[
{
folders: [ /DATA ],
excludes: [ .snapshots,temp],
tags: [ data,biz ]
},
{
folders: [ /ARCHIVE ],
excludes: [ .snapshots,temp],
tags: [ archive,biz ]
}
],
quiet: true
}
]
} $ cat restic-jobs.sh
#!/bin/bash
#v0.9.1
JOB=aws-s3
eval $(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + =\ + .value + \' restic-jobs.json)
if [[ $jobname == ]]; then
echo no job found in config: $JOB
exit
fi
echo found: $jobname
#sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )
echo
sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)
backup_jobs=()
## need this for bash issue with variables and pipe subshell
shopt -s lastpipe
echo $sets | jq -rc '.sets[]' | while IFS='' read set;do
cmd_line=restic backup -q --json
folders=$(echo $set | jq -r '.folders | .[]')
for st in $folders; do cmd_line+= $st; done
excludes=$(echo $set | jq -r '.excludes | .[]')
for st in $excludes; do cmd_line+= --exclude $st; done
tags=$(echo $set | jq -r '.tags | .[]')
for st in $tags; do cmd_line+= --tag $st; done
backup_jobs+=($cmd_line)
done
for i in ${backup_jobs[@]}; do
echo cmd_line: $i
doneScript run example. Note I am not passing the job name just hard code at the top for my test.
$ ./restic-jobs.sh
found: iqonda-aws-s3
cmd_line: restic backup -q --json /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz
cmd_line: restic backup -q --json /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz
Comments Off on Bash Read Json Config File
comments
Also see previous post https://blog.ls-al.com/bash-date-usage-for-naming if you are interested. This post is similar but more specific to restic tagging.
Below is a test script and a test run. At the time of restic backup I create a tag in order to do snapshot forget based on tags.
# cat backup-tags.sh
#!/bin/bash
create_tag () {
tag=daily
if [ $(date +%a) == Sun ]; then tag=weekly ; fi
if [ $(date +%d) == 01 ]; then
tag=monthly
if [ $(date +%b) == Jan ]; then
tag=yearly
fi
fi
}
create_tag
echo backup policy: $tag
create_tag_unit_test () {
for i in {1..95}
do
tdate=$(date -d +$i day)
tag=daily
if [ $(date -d +$i day +%a) == Sun ]; then tag=weekly ; fi
if [ $(date -d +$i day +%d) == 01 ]; then
tag=monthly
if [ $(date -d +$i day +%b) == Jan ]; then
tag=yearly
fi
fi
printf %s - %s - %s | $(date -d +$i day +%d) $(date -d +$i day +%a) $tag
if [ $(( $i %5 )) -eq 0 ]; then printf \n; fi
done
}
create_tag_unit_test# ./backup-tags.sh
backup policy: daily
22 - Fri - daily | 23 - Sat - daily | 24 - Sun - weekly | 25 - Mon - daily | 26 - Tue - daily |
27 - Wed - daily | 28 - Thu - daily | 29 - Fri - daily | 30 - Sat - daily | 01 - Sun - monthly |
02 - Mon - daily | 03 - Tue - daily | 04 - Wed - daily | 05 - Thu - daily | 06 - Fri - daily |
07 - Sat - daily | 08 - Sun - weekly | 09 - Mon - daily | 10 - Tue - daily | 11 - Wed - daily |
12 - Thu - daily | 13 - Fri - daily | 14 - Sat - daily | 15 - Sun - weekly | 16 - Mon - daily |
17 - Tue - daily | 18 - Wed - daily | 19 - Thu - daily | 20 - Fri - daily | 21 - Sat - daily |
22 - Sun - weekly | 23 - Mon - daily | 24 - Tue - daily | 25 - Wed - daily | 26 - Thu - daily |
27 - Fri - daily | 28 - Sat - daily | 29 - Sun - weekly | 30 - Mon - daily | 31 - Tue - daily |
01 - Wed - yearly | 02 - Thu - daily | 03 - Fri - daily | 04 - Sat - daily | 05 - Sun - weekly |
06 - Mon - daily | 07 - Tue - daily | 08 - Wed - daily | 09 - Thu - daily | 10 - Fri - daily |
11 - Sat - daily | 12 - Sun - weekly | 13 - Mon - daily | 14 - Tue - daily | 15 - Wed - daily |
16 - Thu - daily | 17 - Fri - daily | 18 - Sat - daily | 19 - Sun - weekly | 20 - Mon - daily | Below is the restic backup script setting a tag and then snapshot forget based on the tag.
As always this is NOT tested use at your own risk.
My policy is:
# cat desktop-restic.sh
#!/bin/bash
### wake up backup server and restic backup to 3TB ZFS mirror
cd /root/scripts
./wake-backup-server.sh
source /root/.restic.env
## Quick and dirty logic for snapshot tagging
create_tag () {
tag=daily
if [ $(date +%a) == Sun ]; then tag=weekly ; fi
if [ $(date +%d) == 01 ]; then
tag=monthly
if [ $(date +%b) == Jan ]; then
tag=yearly
fi
fi
}
create_tag
restic backup -q /DATA /ARCHIVE --tag $tag --exclude *.vdi --exclude *.iso --exclude *.ova --exclude *.img --exclude *.vmdk
restic forget -q --tag daily --keep-last 7
restic forget -q --tag weekly --keep-last 4
restic forget -q --tag monthly --keep-last 12
if [ $tag == weekly ]; then
restic -q prune
fi
sleep 1m
ssh user@192.168.1.250 sudo shutdown now
Comments Off on Restic create backup and set tag with date logic
comments
I was recently needing to read values from a configuration file into bash and had some success with reading json with jq into bash array(s). However I resorted to a non json version which worked well. Something like this.
Config File
$cat array-simple.cfg
[bucket1]
name=bucket name 1
exclude=folder1 folder 2
[bucket2]
name=bucket name 2
exclude=folder5
Code
$ cat array-simple.sh
#!/bin/bash
while read line; do
if [[ $line =~ ^"["(.+)"]"$ ]]; then
arrname=${BASH_REMATCH[1]}
declare -A $arrname
elif [[ $line =~ ^([_[:alpha:]][_[:alnum:]]*)"="(.*) ]]; then
declare ${arrname}[${BASH_REMATCH[1]}]="${BASH_REMATCH[2]}"
fi
done < array-simple.cfg
echo ${bucket1[name]}
echo ${bucket1[exclude]}
echo ${bucket2[name]}
echo ${bucket2[exclude]}
for i in "${!bucket1[@]}"; do echo "$i => ${bucket1[$i]}"; done
for i in "${!bucket2[@]}"; do echo "$i => ${bucket2[$i]}"; done
Run
$ ./array-simple.sh
bucket name 1
folder1 folder 2
bucket name 2
folder5
exclude => folder1 folder 2
name => bucket name 1
exclude => folder5
name => bucket name 2
Comments Off on Bash Read Array From Config File
comments
If you like using Control-R in bash to find previous commands here is a useful tip. You can add a comment to a command and then when you use Control-R searching by typing you can find it by your comment. Example I use apt update. Run command including your comment (shell will ignore the comment of course). Then when Control-R searching type your string you used in the comment.
# apt update ; apt upgrade #quickupdate
Now hit Control-R and type to search "quick".
(reverse-i-search)`quick': apt update ; apt upgrade #quickupdate
Comments Off on Bash History Plus Comment
comments
If you don't have telnet or nc installed and want to quickly test firewall traffic to a server and specific port you can try this. It needs a new enough bash but still pretty quick and handy.
Good test port is open
$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/172.18.10.66/1521' $ echo $? 0
Port not open
$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/172.18.10.66/15' $ echo $? 124
Good test port is open to google FQDN
$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/google.com/80' [opc@ocilxeasdbt02 ~]$ echo $? 0
Comments Off on Test Tcp Open Port
comments