I recently used Borg for protecting some critical files and jotting down some notes here.

Borg exist in many distribution repos so easy to install. When not in a repo they have pre-compiled binaries that can easily be added to your Linux OS.

Pick a server to act like your backup server (repository). Pretty much any Linux server where you can direct your client to send their backups to. You want to make your backup folder big enough of course.

https://thisiscasperslife.wordpress.com/2017/11/28/using-borg-backup-across-ssh-with-sshkeys/
https://opensource.com/article/17/10/backing-your-machines-borg

# yum install borgbackup
# useradd borg
# passwd borg
# sudo su - borg 
$ mkdir /mnt/backups
$ cat /home/borg/.ssh/authorized_keys
ssh-rsa AAAAB3N[..]6N/Yw== root@server01
$ borg init /mnt/backups/repo1 -e none

 **** CLIENT server01 with single binary(no repo for borgbackup on this server)

$ sudo su - root
# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/borg_key
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/borg_key.
Your public key has been saved in /root/.ssh/borg_key.pub.

# ./backup.sh 
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
Synchronizing chunks cache...
Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
Done.
------------------------------------------------------------------------------
Archive name: server01-2018-03-29
Archive fingerprint: 79f91d82291db36be7de90c421c082d7ee4333d11ac77cd5d543a4fe568431e3
Time (start): Thu, 2018-03-29 19:32:45
Time (end):   Thu, 2018-03-29 19:32:47
Duration: 1.36 seconds
Number of files: 1069
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               42.29 MB             15.41 MB             11.84 MB
All archives:               42.29 MB             15.41 MB             11.84 MB

                       Unique chunks         Total chunks
Chunk index:                    1023                 1059
------------------------------------------------------------------------------
Keeping archive: server01-2018-03-29                     Thu, 2018-03-29 19:32:45 [79f91d82291db36be7de90c421c082d7ee4333d11ac77cd5d543a4fe568431e3]

*** RECOVER test. Done on BORG server directly but will test from client directly also. May need BORG_RSH variable.

$ borg list repo1
server01-2018-03-29                     Thu, 2018-03-29 19:32:45 [79f91d82291db36be7de90c421c082d7ee4333d11ac77cd5d543a4fe568431e3]

$ borg list repo1::server01-2018-03-29 | less

$ cd /tmp
$ borg extract /mnt/backups/repo1::server01-2018-03-29  etc/hosts

$ ls -l etc/hosts 
-rw-r--r--. 1 borg borg 389 Mar 26 15:50 etc/hosts

APPENDIX: client backup.sh cron and source

# crontab -l
0 0 * * * /root/scripts/backup.sh > /dev/null 2>&1

# sudo su - root
# cd scripts/
# cat backup.sh 
#!/usr/bin/env bash

##
## Set environment variables
##

## if you don't use the standard SSH key,
## you have to specify the path to the key like this
export BORG_RSH='ssh -i /root/.ssh/borg_key'

## You can save your borg passphrase in an environment
## variable, so you don't need to type it in when using borg
# export BORG_PASSPHRASE="top_secret_passphrase"

##
## Set some variables
##

LOG="/var/log/borg/backup.log"
BACKUP_USER="borg"
REPOSITORY="ssh://${BACKUP_USER}@10.1.1.2/mnt/backups/repo1"

#export BORG_PASSCOMMAND=''

#Bail if borg is already running, maybe previous run didn't finish
if pidof -x borg >/dev/null; then
    echo "Backup already running"
    exit
fi

##
## Output to a logfile
##

exec > >(tee -i ${LOG})
exec 2>&1

echo "###### Backup started: $(date) ######"

##
## At this place you could perform different tasks
## that will take place before the backup, e.g.
##
## - Create a list of installed software
## - Create a database dump
##

##
## Transfer the files into the repository.
## In this example the folders root, etc,
## var/www and home will be saved.
## In addition you find a list of excludes that should not
## be in a backup and are excluded by default.
##

echo "Transfer files ..."
/usr/local/bin/borg create -v --stats                   \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d}'    \
    /root                                \
    /etc                                 \
    /u01                                 \
    /home                                \
    --exclude /dev                       \
    --exclude /proc                      \
    --exclude /sys                       \
    --exclude /var/run                   \
    --exclude /run                       \
    --exclude /lost+found                \
    --exclude /mnt                       \
    --exclude /var/lib/lxcfs


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.
/usr/local/bin/borg prune -v --list $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6

echo "###### Backup ended: $(date) ######"

In addition to using Borg this test was also about pushing backups to Oracle OCI object storage so below is some steps I followed. I had to use the newest rclone because v1.36 had weird issues with the Oracle OCI S3 interface.

# curl https://rclone.org/install.sh | sudo bash

# df -h | grep borg
/dev/mapper/vg01-vg01--lv01  980G  7.3G  973G   1% /mnt/backups

# sudo su - borg

[$ cat ~/.config/rclone/rclone.conf 
[s3_backups]
type = s3
env_auth = false
access_key_id = ocid1.credential.oc1..aaaa[snipped]
secret_access_key = KJFevw6s=
region = us-ashburn-1
endpoint = [snipped].compat.objectstorage.us-ashburn-1.oraclecloud.com
location_constraint = 
acl = private
server_side_encryption = 
storage_class = 

$ rclone  lsd s3_backups: 
          -1 2018-03-27 21:07:11        -1 backups
          -1 2018-03-29 13:39:42        -1 repo1
          -1 2018-03-26 22:23:35        -1 terraform
          -1 2018-03-27 14:34:55        -1 terraform-src

Initial sync. Note I am using sync but be warned you need to figure out if you want to use copy or sync. As far as I know sync may delete not only on target but also on source when syncing.

$ /usr/bin/rclone -v sync /mnt/borg/repo1 s3_backups:repo1
2018/03/29 22:37:00 INFO  : S3 bucket repo1: Modify window is 1ns
2018/03/29 22:37:00 INFO  : README: Copied (replaced existing)
2018/03/29 22:37:00 INFO  : hints.38: Copied (new)
2018/03/29 22:37:00 INFO  : integrity.38: Copied (new)
2018/03/29 22:37:00 INFO  : data/0/17: Copied (new)
2018/03/29 22:37:00 INFO  : config: Copied (replaced existing)
2018/03/29 22:37:00 INFO  : data/0/18: Copied (new)
2018/03/29 22:37:00 INFO  : index.38: Copied (new)
2018/03/29 22:37:59 INFO  : data/0/24: Copied (new)
2018/03/29 22:38:00 INFO  : 
Transferred:   1.955 GBytes (33.361 MBytes/s)
Errors:                 0
Checks:                 2
Transferred:            8
Elapsed time:        1m0s
Transferring:
 *                                     data/0/21: 100% /501.284M, 16.383M/s, 0s
 *                                     data/0/22: 98% /500.855M, 18.072M/s, 0s
 *                                     data/0/23: 100% /500.951M, 14.231M/s, 0s
 *                                     data/0/25:  0% /501.379M, 0/s, -

2018/03/29 22:38:00 INFO  : data/0/22: Copied (new)
2018/03/29 22:38:00 INFO  : data/0/23: Copied (new)
2018/03/29 22:38:01 INFO  : data/0/21: Copied (new)
2018/03/29 22:38:57 INFO  : data/0/25: Copied (new)
2018/03/29 22:38:58 INFO  : data/0/27: Copied (new)
2018/03/29 22:38:59 INFO  : data/0/26: Copied (new)
2018/03/29 22:38:59 INFO  : data/0/28: Copied (new)
2018/03/29 22:39:00 INFO  : 
Transferred:   3.919 GBytes (33.438 MBytes/s)
Errors:                 0
Checks:                 2
Transferred:           15
Elapsed time:        2m0s
Transferring:
 *                                     data/0/29:  0% /500.335M, 0/s, -
 *                                     data/0/30:  0% /500.294M, 0/s, -
 *                                     data/0/31:  0% /500.393M, 0/s, -
 *                                     data/0/32:  0% /500.264M, 0/s, -

2018/03/29 22:39:45 INFO  : data/0/29: Copied (new)
2018/03/29 22:39:52 INFO  : data/0/30: Copied (new)
2018/03/29 22:39:52 INFO  : S3 bucket repo1: Waiting for checks to finish
2018/03/29 22:39:55 INFO  : data/0/32: Copied (new)
2018/03/29 22:39:55 INFO  : S3 bucket repo1: Waiting for transfers to finish
2018/03/29 22:39:56 INFO  : data/0/31: Copied (new)
2018/03/29 22:39:57 INFO  : data/0/36: Copied (new)
2018/03/29 22:39:57 INFO  : data/0/37: Copied (new)
2018/03/29 22:39:57 INFO  : data/0/38: Copied (new)
2018/03/29 22:39:58 INFO  : data/0/1: Copied (replaced existing)
2018/03/29 22:40:00 INFO  : 
Transferred:   5.874 GBytes (33.413 MBytes/s)
Errors:                 0
Checks:                 3
Transferred:           23
Elapsed time:        3m0s
Transferring:
 *                                     data/0/33:  0% /500.895M, 0/s, -
 *                                     data/0/34:  0% /501.276M, 0/s, -
 *                                     data/0/35:  0% /346.645M, 0/s, -

2018/03/29 22:40:25 INFO  : data/0/35: Copied (new)
2018/03/29 22:40:28 INFO  : data/0/33: Copied (new)
2018/03/29 22:40:30 INFO  : data/0/34: Copied (new)
2018/03/29 22:40:30 INFO  : Waiting for deletions to finish
2018/03/29 22:40:30 INFO  : data/0/3: Deleted
2018/03/29 22:40:30 INFO  : index.3: Deleted
2018/03/29 22:40:30 INFO  : hints.3: Deleted
2018/03/29 22:40:30 INFO  : 
Transferred:   7.191 GBytes (34.943 MBytes/s)
Errors:                 0
Checks:                 6
Transferred:           26
Elapsed time:     3m30.7s

Run another sync showing nothing to do.

$ /usr/bin/rclone -v sync /mnt/borg/repo1 s3_backups:repo1
2018/03/29 22:43:13 INFO  : S3 bucket repo1: Modify window is 1ns
2018/03/29 22:43:13 INFO  : S3 bucket repo1: Waiting for checks to finish
2018/03/29 22:43:13 INFO  : S3 bucket repo1: Waiting for transfers to finish
2018/03/29 22:43:13 INFO  : Waiting for deletions to finish
2018/03/29 22:43:13 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                26
Transferred:            0
Elapsed time:       100ms

Test script and check log

$ cd scripts/
$ ./s3_backup.sh 
$ more ../s3_backups.log 
2018/03/29 22:43:56 INFO  : S3 bucket repo1: Modify window is 1ns
2018/03/29 22:43:56 INFO  : S3 bucket repo1: Waiting for checks to finish
2018/03/29 22:43:56 INFO  : S3 bucket repo1: Waiting for transfers to finish
2018/03/29 22:43:56 INFO  : Waiting for deletions to finish
2018/03/29 22:43:56 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                26
Transferred:            0
Elapsed time:       100ms

Check size used on object storage.

$ rclone size s3_backups:repo1
Total objects: 26
Total size: 7.191 GBytes (7721115523 Bytes)

APPENDIX: s3_backup.sh crontab and source

$ crontab -l
50 23 * * * /home/borg/scripts/s3_backup.sh

$ cat s3_backup.sh 
#!/bin/bash
set -e

#repos=( repo1 repo2 repo3 )
repos=( repo1 )

#Bail if rclone is already running, maybe previous run didn't finish
if pidof -x rclone >/dev/null; then
    echo "Process already running"
    exit
fi

for i in "${repos[@]}"
do
    #Lets see how much space is used by directory to back up
    #if directory is gone, or has gotten small, we will exit
    space=`du -s /mnt/backups/$i|awk '{print $1}'`

    if (( $space < 3450000 )); then echo "EXITING - not enough space used in $i" exit fi /usr/bin/rclone -v sync /mnt/backups/$i s3_backups:$i >> /home/borg/s3_backups.log 2>&1
done