Back up your server

Set up your own off-server backups with rsync or BorgBackup, include your databases, run them nightly, and prove you can restore them.

Your server's disk is one copy of everything. A mistyped rm, an upgrade gone wrong, a break-in or a cancelled server can each take it away. A backup only counts if it lives on another machine and you've restored from it at least once.

The panel's snapshots are good for undoing a change, but they sit on the same machine as the server and are deleted with it, so they aren't backups.

We'll copy backups to a second machine you control, called backup.example.com here. It can be another server, a box at home, or a storage service that accepts SSH.

If it's another of your Rack4 servers in the same location, you can copy to it over your private network, which doesn't count towards your monthly transfer. Keep a copy somewhere else as well: off-site backups covers why.

What to include

Usually:

  • /etc, your configuration;
  • /home and /root;
  • /var/www, or wherever your sites and apps live;
  • /srv, /opt, and Docker volumes under /var/lib/docker/volumes if you use them;
  • database dumps. Not the raw files of a running database.

You can leave out /usr, /var/cache and /tmp. A reinstall brings the system back; it's your data you can't replace.

Dump the databases first

Copying a live database's files can give you a copy that won't open. Write a dump before each backup instead. For MySQL or MariaDB:

bash
install -d -m 700 /var/backups/db
mysqldump --all-databases --single-transaction --routines | gzip > /var/backups/db/mysql.sql.gz

For PostgreSQL:

bash
sudo -u postgres pg_dumpall | gzip > /var/backups/db/postgres.sql.gz

Then make sure /var/backups/db is in the backup.

The simple way: rsync

rsync copies only what changed since last time. Give the server its own SSH key for backups and put the public half on the backup machine:

bash
ssh-keygen -t ed25519 -N '' -f /root/.ssh/backup -C backup@web-1
ssh-copy-id -i /root/.ssh/backup.pub backup@backup.example.com

Then:

bash
rsync -a --delete -e 'ssh -i /root/.ssh/backup' /etc /home /var/www /var/backups/db backup@backup.example.com:web-1/

That gives you a mirror. Its weakness is that it mirrors mistakes too: if a file gets deleted or encrypted on the server, the next run faithfully copies the damage across. If that worries you, and it should a bit, use Borg.

The better way: BorgBackup

Borg keeps compressed, encrypted, deduplicated snapshots, so you can go back to last Tuesday. Install it on both machines with apt install -y borgbackup. Then, on the server:

bash
export BORG_RSH='ssh -i /root/.ssh/backup'
borg init --encryption=repokey-blake2 backup@backup.example.com:web-1.borg
borg key export backup@backup.example.com:web-1.borg /root/borg-key.txt

borg init asks you for a passphrase. Put that passphrase and the exported key file somewhere that isn't this server, like your password manager. Lose both and the backup is unreadable, by you included.

Make a backup and trim the old ones:

bash
borg create --stats --compression zstd backup@backup.example.com:web-1.borg::'{hostname}-{now}' /etc /home /root /var/www /var/backups/db
borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 backup@backup.example.com:web-1.borg
borg compact backup@backup.example.com:web-1.borg

That keeps a week of dailies, a month of weeklies and six monthlies.

Run it every night

Put your commands in /usr/local/sbin/backup, starting with #!/bin/sh and set -e. For Borg, add the export BORG_RSH=... line from above, and export BORG_PASSCOMMAND='cat /root/.borg-passphrase' with that file set to mode 600. Make the script executable with chmod 700, then add this with crontab -e as root:

text
30 3 * * * /usr/local/sbin/backup >> /var/log/backup.log 2>&1

Look at the log the next morning. Then look again now and then. Backups have a way of failing quietly.

Prove you can restore

Restore into a scratch folder, never over live files:

bash
mkdir /tmp/restore-test && cd /tmp/restore-test
borg list backup@backup.example.com:web-1.borg
borg extract backup@backup.example.com:web-1.borg::ARCHIVE-NAME etc/hostname
cat etc/hostname

Use an archive name from borg list in place of ARCHIVE-NAME. With rsync, copy a file back the other way. And load a database dump into a test database at least once. Check that your backup can be restored walks through a full drill: files, a database, checking the app works, and timing it.

Going further

Ready when you are

Order a server