VPS backup and restore: encrypted, off-site, and actually tested

Snapshots are not backups — and you want both

A snapshot is a point-in-time copy of a volume that lives on the same storage fabric as the volume it copied. That is not a criticism, it is a description, and it explains precisely what a snapshot is excellent at and what it cannot do. Every plan here includes hourly snapshots with seven days of retention, restoring one takes about thirty seconds, and they are the correct answer to the single most common incident in server operations: you edited a configuration file at four in the afternoon and the service has not started since. For that failure mode nothing else is close — it is faster than any backup tool, it is already running, and it costs nothing extra.

What a snapshot cannot do is survive the thing it is stored on. It shares a fate with the volume, the host, the hypervisor and in most designs the account. If the machine is seized, if the account is closed, if the datacenter has a very bad day, the snapshots go with it — they are a copy of the data, not an independent one. The distinction is not academic on a provider that deliberately cannot identify you: there is no support path that reconstructs your server from somebody else's records, no recovery flow keyed to a phone number, and no billing history with your legal name attached. What you took is what you have.

Nor does a seven-day window help with the two slow failures. The file you deleted five weeks ago is gone from every snapshot long before you notice. Silent corruption — a failing application writing subtly wrong data, a bad migration, a sync client faithfully propagating a mistake — is often discovered a month later, by which point every retained copy contains the corruption. Backups with longer retention and a genuine history are what cover those, and they cover them precisely because they are not continuous mirrors of current state.

The framework worth internalising here is old and still correct. The 3-2-1 rule asks for three copies of the data, on two different kinds of storage, with one of them off-site. The modern addendum adds two more digits — 3-2-1-1-0 — where the extra one is a copy that is immutable or offline, and the zero is the number of errors in your last verification. The digits are a mnemonic rather than a standard, but they encode the three properties that actually matter: independence, immutability, and evidence that the thing works. Hourly snapshots give you one of the three.

Read that table as an argument for running both rather than choosing between them. Snapshots handle frequency and speed; backups handle independence and history. They cost different things, they fail in disjoint ways, and the combination is far stronger than either doubled.

The two numbers that decide everything else

Before any tool is installed, write down two figures. The recovery point objective is how much data you can afford to lose, measured in time — if the last usable copy is six hours old, your RPO is six hours and the six hours of work in between are gone. The recovery time objective is how long you can afford to be unavailable while you rebuild. Everything downstream — schedule, method, where the second copy lives, how much you spend — falls out of those two numbers, and choosing software before choosing them is how people end up with an elaborate answer to a question nobody asked.

Asked casually, most people say zero and zero. That is not a budget, it is a wish, and it has a price attached that is worth stating plainly: an RPO near zero means continuous replication rather than scheduled copies, and an RTO near zero means a warm second machine already running. Both are buildable and both roughly double the operating cost of the thing you are protecting. For the overwhelming majority of self-hosted workloads the honest answers are somewhere between one hour and one day for the recovery point, and a few hours for the recovery time, which is comfortably served by a scheduled encrypted backup and a documented rebuild.

The numbers also differ per dataset on the same machine, which is the detail that makes them useful. Mail is the sharpest example: a message that arrived twenty minutes ago exists nowhere else, cannot be regenerated, and its sender has no idea it needs resending — so a mail server wants a recovery point measured in minutes even though the software itself reinstalls in an afternoon. A Bitcoin node inverts that completely: the chain data is several hundred gigabytes that every peer on the network will happily send you again, so backing it up is largely a waste of disk, while the wallet file next to it is unrecoverable and worth protecting like a private key, because that is exactly what it is.

That last row deserves saying out loud, because a guide about backups has an obvious incentive not to. Some machines do not need one. A build agent whose entire state comes from a git repository, a throwaway sandbox, a stateless reverse proxy whose configuration lives in version control — for these the hourly snapshots included with the plan are a complete and defensible answer, and the correct amount of backup engineering is none. Knowing which of your servers are in that category is worth more than backing all of them up badly.

What to copy — and the much larger pile you should skip

The instinct is to image the whole disk, and on a rented virtual machine that instinct is wrong. The operating system is disposable: a fresh instance provisions here in a median of forty-one seconds, and reinstalling packages from a distribution mirror is faster and more reliable than restoring them from your own copy. What is not disposable is a surprisingly small set, and the discipline of naming that set explicitly is most of the work — a backup you can describe in a paragraph is one you can verify, while a whole-disk image is a thing you hope about.

The irreplaceable set is usually four things. First, application data: the maildir, the data directory, the media store, the uploads folder. Second, databases, which need their own treatment and get the next section. Third, secrets — and this is the category people forget until it is expensive. TLS private keys and the ACME account key, SSH host keys if you would rather not re-trust the fingerprint on every client, the DKIM signing key without which your mail starts failing authentication, WireGuard private keys, and above all the onion service private key, which is the .onion address: lose it and the address is gone permanently, with no registry anywhere to appeal to. Fourth, the handful of files under /etc you actually changed, plus the systemd units, cron entries and firewall ruleset that make the machine behave the way it does.

The pile to skip is much larger and mostly obvious once stated. Pseudo-filesystems like /proc, /sys and /dev are kernel views rather than data. Package caches, container images, virtual environments and dependency directories are all re-derivable from a lockfile in seconds. Logs beyond your retention policy are noise that deduplicates badly and inflates every subsequent run. And large re-syncable datasets — blockchain state, a public mirror, a media library you can re-rip — deserve an explicit decision rather than a default: backing up three hundred gigabytes of chain data costs real money every month to avoid a re-sync you could run for free while the service is degraded but functional.

The part almost everyone under-invests in is the answer to "restore into what?". Data without a target is not a recovery. If the way your server was built exists only in your memory and a shell history you have since lost, your recovery time is unbounded no matter how good the repository is. The fix is cheap and boring: keep the provisioning steps as a script or a plain text runbook, put it inside the backup repository so it is restored alongside the data, and update it whenever you change the machine. Even a hundred lines of notes — packages, versions, config file locations, DNS records, the order services must start in — converts a bad week into a bad afternoon.

One nuance worth flagging for anyone running full-disk encryption: the LUKS header and its key slots are part of your irreplaceable set too. A corrupted header turns a perfectly intact disk into random noise, the header is a few megabytes, and backing it up costs nothing. Store it as carefully as the passphrase — anyone holding both holds the disk.

Databases: the file you copy is not the database

This is the single most common way an otherwise competent backup turns out to be worthless. A database engine keeps state in memory, in a write-ahead or redo log, and in data files that are being modified continuously. Copying those files while the engine is running captures them at different instants — page four of a table from before a transaction, page five from after — and the result is a set of files that are individually intact and collectively inconsistent. The cruel part is that such a copy usually restores. It comes up, it answers queries, and the damage surfaces weeks later as a corrupt index or a row that violates a constraint the engine swears it enforced. A backup that fails loudly is far better than one that fails quietly.

The correct approach depends on the engine and is well documented in each case. For MariaDB and MySQL, a logical dump taken inside a single consistent transaction is right for anything up to a few tens of gigabytes; past that, Mariabackup performs a physical hot copy while the server runs. For PostgreSQL, pg_dump gives you a portable logical snapshot, and pg_basebackup combined with archived write-ahead logs gives you point-in-time recovery — the ability to restore to 14:32 rather than to whenever the last dump happened to run. For SQLite, use the built-in online backup or VACUUM INTO; cp on a live database file is exactly the mistake described above, and SQLite is the engine where people make it most often because it looks like an ordinary file.

Consistency between the database and the files it describes is the second half of the problem, and it is the half that specifically bites file-sync and forum software. The database says a file exists at a path; the filesystem is where the bytes live; if you dump the database at 02:00 and copy the data directory at 02:40, everything created in between exists as bytes with no row, or as a row with no bytes. Applications that offer a maintenance mode — Nextcloud is the obvious one — solve this by briefly refusing writes while both are captured. Where that is unacceptable, a filesystem or volume snapshot taken in a single instant gives you a consistent pair to copy from at leisure, which is the same trick the hourly snapshots use.

One small operational detail pays for itself immediately: write dumps uncompressed and let the backup tool handle compression. Both restic and borg deduplicate by content-defined chunking, so two dumps taken a day apart share the overwhelming majority of their chunks and the second one costs almost nothing to store. Compress the dump first and a single changed byte near the beginning cascades through the entire compressed stream, no chunk matches anything, and every run stores a full copy. Teams routinely find their repository growing by the full database size daily for exactly this reason — the fix is deleting one pipe from a shell script.

Finally, decide deliberately whether you need point-in-time recovery, because it is the answer to a failure the daily dump cannot address. If somebody runs a destructive query at 14:32 and you discover it at 17:00, restoring last night's dump loses a working day. Continuous archiving replays the log up to 14:31 instead. It costs more storage and meaningfully more operational complexity, so it is not a default — but for anything where a human with write access can destroy data faster than you can notice, it is the difference between an incident and a catastrophe.

Encrypt before it leaves: restic, borg, and the key you must not lose

The property that makes everything else in this guide possible is client-side encryption: the data is encrypted on the machine that produced it, before a single byte crosses the network, using a key the destination never sees. That one design decision turns the backup target into untrusted storage, and untrusted storage can live anywhere — a cheap server in another country, an object store, a friend's NAS — without any of them being in a position to read your data. Providers who encrypt server-side are protecting you from a different and much narrower set of threats, and the difference matters most in exactly the situations you are buying offshore hosting to survive.

Restic and Borg both do this correctly and the choice between them is genuinely close. Restic is a single static binary with no dependencies, speaks a long list of backends natively — SFTP, S3-compatible object storage, plain local paths — and is the easier of the two to run from a container or a minimal image. Borg is the older and in some respects more surgical tool, with stronger compression options, excellent append-only support on the server side, and a repository format that many people find easier to reason about; it wants borg installed on both ends when going over SSH, which is a small constraint that occasionally decides the matter.

Whichever you pick, the passphrase is now the whole ballgame, and it deserves to be treated the way you would treat a wallet seed. It must not live only on the machine it protects — a passphrase stored beside the data it encrypts protects you against precisely nothing, because every scenario that takes the server takes the key with it. Put it in a password manager on a different device, or on paper in a different building, and ideally both. Borg lets you export the repository key to a separate file, restic lets a repository carry several independent keys, and either mechanism gives you a second way in that does not depend on remembering a string. There is no recovery, no reset link and no support ticket that reverses this: an encrypted repository without its key is indistinguishable from random data, which is the entire point.

It is worth being explicit about why plain rsync to another server is not a substitute, since it is the thing people reach for first. Rsync produces a mirror, and a mirror has no history: delete a file today and tonight's run faithfully deletes it there too. It has no deduplication, so keeping thirty days means thirty copies unless you get clever with hard links. It has no built-in encryption at rest, so the destination reads everything. And it has no verification story beyond re-reading the source. It is an excellent transport and a poor backup, and the distinction shows up on the day you need the version from three weeks ago.

Append-only, or it is a copy rather than a backup

Here is the scenario that decides the architecture. Someone gets root on your server — via an unpatched application, a leaked token, a dependency that turned hostile. They are now inside a machine that holds valid credentials to its own backup target, because that is how scheduled backups work. Anything that machine can delete, they can delete, and deleting the backups is not an afterthought in a modern intrusion, it is step one. The same applies without an adversary: a well-meaning script with a stray variable, run as root, is perfectly capable of pruning a repository to nothing.

The control that closes this is append-only enforcement on the receiving side. Configured this way, the client can create new snapshots and cannot remove or rewrite old ones — the restriction is enforced by the process on the backup host, not by the goodwill of the client. Borg implements it with borg serve --append-only, pinned to the key in the target's authorized_keys so the client cannot ask for anything else. Restic has the same shape via rest-server with --append-only. On object storage the equivalent is versioning plus an object-lock retention policy, which achieves the same result through a different mechanism.

The SSH configuration that carries this is worth getting exactly right, because it is the load-bearing line. In the backup user's authorized_keys, prefix the client's key with a forced command and the restrictions the sshd manual documents: command="borg serve --append-only --restrict-to-path /srv/backups/web01",restrict. That single line means a key stolen from the client cannot open a shell, cannot forward a port, cannot touch another host's repository, and cannot delete anything. Give every client its own key and its own path.

Which raises the obvious question: if the client can never delete, what removes old snapshots? Pruning happens elsewhere, on a schedule, with a credential the protected machine has never held. In practice that means the backup host prunes its own repositories from a local cron, or a third small machine holds the privileged key and runs retention weekly. Borg has a known wrinkle here — an append-only repository needs its compaction run on the server side rather than by the client — and treating the backup host as the owner of retention resolves that cleanly. The principle generalises: the machine that writes backups should never be the machine that can destroy them.

Retention itself is a policy question with a conventional answer that works: keep the last several dailies, four or five weeklies, and six to twelve monthlies. Both tools express this declaratively, so you state the shape you want and let the tool decide which snapshots satisfy it. The monthlies are what catch slow corruption, and they are also the ones people cut first when the repository grows — which is exactly backwards, since a compressed deduplicated monthly of a dataset that barely changes costs almost nothing.

Where the second copy goes when nobody can identify you

Off-site is doing a lot of work in the 3-2-1 rule, and on a no-KYC provider it means something more specific than "another building". It means a copy whose survival does not depend on the same company, the same account, the same payment relationship or the same legal jurisdiction as the original. Two servers in one account are two servers with one fate, however far apart the datacenters are: an account closed for any reason takes both, and a legal instrument served on one entity reaches everything that entity holds. Independence is a property of the relationship, not only of the geography.

The practical version of that is easier than it sounds, because the same panel already offers four jurisdictions. A production instance in the Netherlands with its repository pushed to a Starter tier in Iceland costs $8.50 a month and puts your recovery outside the reach of any single national process — and because the jurisdictions differ in what they actually resist, the pairing is worth a moment's thought rather than a coin flip. Since the repository is encrypted before it leaves the source, the backup host is untrusted by design: what you are buying with the second location is availability and legal distance, not confidentiality. Confidentiality you already have.

For a genuinely independent third copy, the strongest architecture inverts the direction of the connection. Instead of the server pushing to a target it holds credentials for, a machine you control — a box at home, a NAS, a laptop that wakes on a schedule — pulls from the server. The production machine then holds no credential to any backup destination at all, which makes the compromise scenario from the previous section structurally impossible rather than merely mitigated. It costs you a machine that has to be reachable or awake on schedule, which is why it complements a pushed off-site copy rather than replacing it.

Two constraints specific to this kind of hosting are worth stating without decoration. There is no account recovery: no identity document to present, no phone number to receive a code, no representative who can confirm you are you. That is the same property you are paying for at signup, and it applies symmetrically — which makes the passphrase and the runbook load-bearing in a way they simply are not on a mainstream provider. And if the backup target needs paying for, it needs paying for anonymously too, or the second copy quietly reintroduces the identity link the first one was designed to avoid. Paying for both machines from the same crypto wallet is fine; paying for the backup with a card is a decision, not an oversight.

The drill, and how you find out it stopped working

Backup jobs do not usually fail dramatically. They fail by degrees: an exclude pattern broadened during a cleanup, a credential rotated on one side only, a disk filling on the target, a cron entry lost in a distribution upgrade. In every one of those cases the machine keeps running, nothing alerts, and the repository quietly stops growing. The design that catches this is not "alert on failure" — a job that no longer runs cannot report its own failure — but alert on the absence of success. Have each successful run ping a dead-man's-switch endpoint, and have that endpoint shout when the ping does not arrive on time. It is ten minutes of setup and it is the difference between noticing in a day and noticing during a restore.

Integrity checking is the second half. Both tools can verify that the repository's metadata is coherent, and more usefully that the stored chunks actually decrypt and match their hashes. Reading everything is expensive in bandwidth, so both offer a partial form — restic checks a percentage of the data per run, borg verifies data on demand — and a sensible cadence is a full metadata check weekly and a partial data check monthly, sized so a complete pass happens over a quarter. Storage does rot, and the point of checking is to learn about it while you still have another copy.

None of which is evidence that you can restore. That evidence has exactly one source, which is restoring. Deploy a fresh instance — the smallest tier is $8.50 and you will destroy it within the hour — restore the repository into it, and then do the part people skip: start the application and look at real data. Log in. Open a document from last month. Send a test message through the mail server. Query a table that should have yesterday's rows in it. Files existing is not the test; the service working on top of them is. Do this the first time you set the backup up, and then on a calendar interval you actually honour, because the drill also validates the runbook, and the runbook decays faster than the data.

The runbook deserves its own paragraph because it is the cheapest item here and the most often missing. Write down, in order: where the repository lives and how to reach it, where the passphrase is kept, how to provision a replacement machine, which packages and versions to install, what to restore and in what order, which DNS records to change, and how to confirm it worked. Keep it outside the machine it describes — in the repository itself, in a password manager, on paper — and assume the person reading it is tired, working at an unpleasant hour, and possibly not you. That last assumption is what turns a set of notes into something a colleague or a family member can execute.

Put together, the whole design is unremarkable and that is its virtue: the hourly snapshots your plan already includes for the accidents of the last few days, an encrypted repository pushed nightly to a second jurisdiction where it cannot be deleted, a passphrase somewhere the server has never seen, a ping that complains when the job goes quiet, and a rehearsal on the calendar. Nothing on that list is difficult, most of it is an afternoon, and the whole thing costs less per month than the coffee you would buy while rebuilding from nothing. If you are setting this up now, the second machine takes about a minute to provision — start with the restore you have never tested, because it is the only part of this that tells you the truth.