[somewhat] byzantine backup setup

below – a description of somewhat convoluted backup system that so far have been serving me pretty well.

i need to take backups of [mostly] linux servers located in a few locations. after a bit of research done ~5 years ago i’ve decided to use rdiff-backup as a file-level incremental backup tool and backupninja as a backup-orchestration framework.

initially i was running rdiff-backup on each of the backed up servers, it was invoked by backupninja executed from the dedicated backup machines. but that was not very efficient. rdiff-backup with destination available via ssh was quite slow so i replaced it with this steps invoked from the backup machine:

• invoke rsync that copies relevant files from the backed up machine to local spool folder on the backup machine
• run mysqldump of the backed up machine and store the backup content in the same spool folder
• execute rdiff-backup using local spool folder as a source and local rdiff repository as a destination

similar steps are repeated for all of the backed up servers from given site.

why backupninja not plain shell scripts? because i prefer having single source of notifications about backup failures and a single backup log.

sample script – one of plenty .sh files in my /etc/backup.d/ [note that the file does not need shebang and does not have to be executable; .sh extension is enough and backupninja will execute it correctly]:

# if any of the commands fails - i'd like the whole script to terminate
# and backup ninja to report error for this backup job
function handle {
        echo Error
        error problem occured
}
set -e
trap handle ERR

host=some.name.com

# touch a file that will be used later to monitor freshness of the backups
ssh root@$host "date > /xyz/touched"

# replicate files - this varies depending on the host
/usr/bin/rsync -Ravz --delete --include ".*" root@$host:/var/spool :/etc :/xyz :/boot :/usr/local :/var/backups :/var/www :/root /mnt/backup/spool/$host/

# get dump of the mysql database
# if it's a large database - -c arcfour reduces overhead of ssh
ssh -c arcfour root@$host "mysqldump --defaults-file=/etc/mysql/debian.cnf --skip-lock-tables --single-transaction  --flush-logs --hex-blob  --master-data=2  -A --skip-extended-insert " > /mnt/backup/spool/$host/mysqldump.sql
# and stored procedures
ssh root@$host "mysqldump --defaults-file=/etc/mysql/debian.cnf  --no-data --no-create-info --all-databases --routines  " > /mnt/backup/spool/$host/procs.sql

# update the rdiff-backup archive and remove revisions older than 2 weeks
cd /mnt/backup/spool/$host/
rdiff-backup --force --print-statistics  . /mnt/backup/rdiff/$host/
rdiff-backup --force --force --remove-older-than 14D /mnt/backup/rdiff/$host/

once all the local backup jobs are done replication to the remote backup location is initiated. i use rsync for that. rsync wrapped in a bit of bash code to make it run in parallel and more efficiently use available bandwidth and iops. it’s also wrapped in set of re-tries to have higher chance of succeeding even if there’s short network outage during it.

backups created in this way arrive to a central location. in theory all should arrive until ~6am so every morning at 7am a verifying batch job is started. it uses rdiff’s –verify option to verify consistency of each of repositories [usually one per server]. only if the verification of all the repositories succeeded data is mirrored to an external hard drive. and then the verification is run again… to be sure. currently the backup size is ~900GB, with ~500GB of the last backups and 400GB of rdiff’s increments that can be used to recover files from the last 14 days. with such data size the process of verification + replication to the external disk + another verification takes ~5-6 hours.

external drives are rotated once per week with 6 drives kept offline and one attached to the central backup server. the rotated disks are also encrypted using truecrypt. we use regular sata hard drives and e-sata caddies. sounds crude but works pretty well and is much simpler and faster to deal with than backup tapes.

monitoring

i think paranoia is pretty healthy symptom when it comes to the backups. backupninja has mail notification mechanism that will send details about any failed job. the scripts i use have custom error handlers configured by trap handle ERR which means that any failed command – be it rsync, mysqldump, rdiff or anything else – will terminate the particular script and generate an error, other scripts will still be executed.

i also have simple nagios monitoring of the /var/log/backupninja.log – i just check for presence of lines not containing words Info|Debug; if there are any – those are signs of failed jobs. so.. i’ll get a mail about failed job even in case local mail delivery from the backup server was not working.

before actual backup starts my scripts ‘touch’ small flag file – usually located in /xyz/ folder. freshness of the corresponding files on the backup drive is checked with yet another nagios script. i also monitor size and freshness of the mysqldumps; i have different size tolerance level for different groups of servers. the nagios script which checks mysqldump files also makes sure that backup has at the end something like:

-- Dump completed on 2013-05-03 22:51:19

nagios will also notify me about failure of backup verification on the central backup server [verification can fail for the local content – most often because transfer over the internet did not finish at the expected time, but it can also fail when verifying content of the external drive. i even once caught whole sector – 512B – not written correctly to the disk. spooky.].

hardware

servers are a bit dated, but with enough spare machines with nearly identical configuration – it’s ok. currently i use:

• in one of the offices – refurbished Dell Vostro 430 desktop with 8GB ram, i7-860 CPU, 2x 3TB, 2x 4TB SATA drives mirrored every 24h
• in a data cetenter – Dell PE 1950 III with 16GB ram, 2x E5430 CPUs, two RAID1s of 2x 1TB raptors and 2x 2TB wd green drives.
• in central backup location – another Dell PE 1950 III with 8GB ram, X5460 CPU, esata card and few esata-sata caddies

overkill? maybe but [relatively] fast cpus help a lot. pbzip2 compression of vmware backups and some parts of the rdiff-compression are cpu bound. first already utilizes all available cores, second – just one, but few of my backups scripts run multiple rdiffs in parallel. some of the servers use truecrypt for the storage drives, extra computing power is useful there as well.

problems

some issues that pop up from time to time:

• backup jobs taking longer than expected – most often due to temporary drop in available network bandwidth between sites, less often due to ‘collision’ with another io-intensive tasks
• rsync run on samba shares containing opened word/excel documents gives.. strange results. sometimes content of files changes without time-stamp change. so backups of those parts are run using -c options of rsync.
• people happily dumping 10s of GBs of their private data on the network shares

more

yes – there’s more:

• there are vmware esxi backups taken using ghettovcb; they arrive via nfs to the backup servers in each of the locations, get compressed with pbzip2 and then rdiffed like regular files. due to the size of vmware snapshots those backups are taken once per week.
• windows machines have scheduled jobs written by my colleague. first shadow copy is created, then windows rdiff run locally saves transfers data to the backup server
• using another script created by my colleague ms exchange’s mail boxes are first dumped to pst files and then transferred to a linux server where they get pbzip2’ed and rdiff’ed
• yet another script created by my work mate takes xtrabackup‘s of most critical mysql databases. just in case those mysqldumps were corrupted.
• mysql’s bin-logs are shuffled around every 15 minutes – this is invoked from cron but also executed by backupninja to re-use existing log monitoring infrastructure
• some data files are kept ’till forever’, i call it retention; to make sure they dont ‘rot’ over time md5 check-sums are calculated and checked every few days
• use of TCP BBR significantly improves utilization of available bandwidth when transfering data over the internet, between the datacenters

 

2022-05 notes:

• rdiff-backup has been replaced with borg backup ; restic & kopia are worth considering. all three provide compressed, de-duplicated storage which is well suited for keeping multiple backups and allow random access to the previous snapshots,
• speed / compression benchmark – https://github.com/deajan/backup-bench
• i no longer use vmware esxi, it was replaced by KVM; ghettovcb is replaced by custom scripts taking snapshots of all VMs disks and rsync’ing them for backups,
• mysqldump has been problematic for larger drives – it randomly leads to write-query pile-ups during backups. more and more we have to take backups from replicas or use physical backup approach [ xtrabackup, mariabackup ],
• where compression is needed – gzip, bzip2 was replaced by zstd ; if it goes to borg later – i use it with –rsyncable switch,
• more and more often backups are done to SSDs or even NVMes to allow more parallelization of backup jobs and verifications and use all those cores that would otherwise sit idle