Backup Strategies That Actually Work

I have lost count of how many times a client has called me in a panic because a server died and they had no usable backup. Not no backup at all — they had a backup. It just was not worth the drive it was stored on. Stale copies, partial exports, corrupted archives, tapes nobody tested in six months. Backup strategies that look good on paper but fall apart the second the disk fails. I have been running production servers for over a decade, Trinidad and abroad, and I have made most of these mistakes myself. Here is what actually works.

The 3-2-1 Rule Is Not Optional

If you take nothing else from this article, take this. The 3-2-1 rule is the oldest principle in data protection because it works. Three copies of your data, on two different media types, with one copy off-site. Three copies means your live production data plus at least two backups. Two different media means you are not putting everything on spinning disks and calling it a day — pair local NAS with cold storage or tape. One copy off-site means if the building burns down or someone spills coffee into the rack, you are not starting from scratch.

I run a combination of BorgBackup to a local NAS and a nightly encrypted push to Backblaze B2. The NAS handles hourly snapshots for quick restores; B2 is my insurance policy. The encrypted part matters. If you are sending backups to cloud storage without client-side encryption, you are effectively handing someone your server room keys. Use Borg's built-in encryption or wrap everything in GPG before it leaves your network. Do not rely on the provider's server-side encryption alone.

Test Your Restores or Your Backup Is Worthless

This is the one that stings. Most people set up a backup script, verify the files landed somewhere, and call it done. That is not a backup — that is a file transfer. A backup is only real when you have successfully restored from it. I test restores on a schedule, and I do it on isolated hardware or a throwaway VM, not on the production box. Restoring on top of a broken system tells you nothing about whether the data itself is sound.

Every quarter I spin up a clean Ubuntu VM, install Borg, pull the latest archive from B2, and run a full restore to a scratch directory. Then I spot-check database dumps by importing them into a fresh Postgres instance and running a few queries. It takes about an hour. That hour has saved me twice when I discovered archive corruption that had been silently propagating for weeks. Automated backup scripts send success emails by default. Train yours to send failure alerts instead — and make sure somebody actually reads them.

Database Backups Need Special Treatment

Rsyncing a MySQL data directory while the server is running is not a backup. I see this constantly. You get a snapshot of files that may or may not be consistent, and if the database crashes mid-copy, that snapshot is garbage. Databases need application-consistent backups. For Postgres, use pg_dump or pg_basebackup. For MySQL, use mysqldump or Percona XtraBackup for hot backups. For MongoDB, mongodump with the oplog flag. These tools understand the internal state of the database and give you a point-in-time consistent snapshot.

I take a full database dump nightly and stream transaction logs every five minutes. That gives me point-in-time recovery down to the minute. If someone runs a destructive query at 2:47 PM on a Tuesday, I can restore to 2:46 PM and lose almost nothing. Wal-G for Postgres and mysqlbinlog for MySQL handle this well. Pair that with a retention policy that keeps daily backups for thirty days, weekly for six months, and monthly for a year. Anything older than that goes into deep archive on cold storage. You are unlikely to need a backup from fourteen months ago, but when you do, you will be glad it is there.

Automation Must Include Alerts and Monitoring

A cron job that silently fails is worse than no backup at all. At least with no backup you know you are unprotected. A broken backup script gives you false confidence, and false confidence is what gets you a 3 AM phone call. I have an alerting chain for every backup job: a health check that verifies the archive was created, a size check that flags archives under the expected threshold, and a completion timestamp that triggers a warning if the job runs longer than its window.

I pipe everything through a monitoring tool. Uptime Kuma works for simple endpoint checks, but for backup-specific monitoring I prefer something that understands cron schedules. Dead Man's Snitch or Healthchecks.io both work well — they expect a ping on schedule and alert you when it does not arrive. That catches the case where the server itself is gone and cannot email you. I also set up a second monitor that checks the age of the latest backup file on the remote storage endpoint. If the newest archive is older than 26 hours, it fires a warning. Redundancy in monitoring is cheap; data loss is not.

Disaster Recovery Is a Playbook, Not a Hope

Knowing you have backups is one thing. Knowing how to rebuild your entire infrastructure from scratch is another. I maintain a runbook for every service I manage. It documents the exact steps to provision a new server, install dependencies, restore databases, reconfigure the web server, and validate that everything is working. I store a copy of the runbook outside the infrastructure it describes — on a personal Git repository, on a USB drive in a fireproof safe, and printed in a binder. When the primary server is unreachable, you cannot read the runbook off of it.

I run a full disaster recovery drill once every six months. I pick a service, shut it down, and see how fast I can bring it back on fresh hardware from backup alone. No shortcuts, no skipping steps because I remember how it is configured. The first time I did this, it took me four hours to recover a service I thought would take thirty minutes. I found missing SSH keys, a database user that was not included in the dump, and a config file that only lived in /tmp. Every gap I found went into the runbook, and the next drill took forty-five minutes. That is the difference between theory and practice.

Set a recovery time objective and a recovery point objective for each service. Your email server can probably be down for an hour. Your e-commerce database cannot. Allocate your backup infrastructure budget accordingly. Not everything needs B2 and a hot spare in another region. But the things that matter — customer data, financial records, authentication databases — should have multiple layers of protection and a tested path back to production.

This is Allan Ali for Sylt.ing.