We restored a server into rescue on our own hardware: what happened

Our off-site backups can restore a copy of a server into a new one. The risky part is that a copy has the live server's jobs on it: booted normally, its cron lines, timers, containers and workers start sending email, charging cards or taking work from queues, from a second machine. So the order form has a box, Start it in rescue mode: the copy waits in the rescue system with its disk unmounted until you've turned those jobs off.

Until this week that box had only passed our host software's automated tests, and our guides said so. On 13 November 2026 we ran it on our New York hardware, the way a customer would, and this is the record.

The setup#

We ordered a Budget server (1 vCPU, 2 GB) with Ubuntu 24.04 through the API, paid like any order, and put five "canary" jobs on it. Each does one harmless thing: it writes a line to a log with the time, the boot it ran in and the server's address.

• a systemd service that runs at every boot;
• an @reboot line in root's crontab;
• a systemd timer with OnBootSec=20s and Persistent=true;
• a Docker container with --restart=always;
• a user's own service, kept running by loginctl enable-linger.

Those are the ways a copied server starts doing things by itself. We also wrote a 50 MB file of random data and a small SQLite database, and recorded their checksums. Then we rebooted the original once: all five canaries wrote their line. A drill where nothing fires proves nothing unless the jobs are known to fire.

Backup and restore#

The backup (2.6 GB of data on the 40 GB disk) took about a minute. We then ordered a new server from it with Start it in rescue mode ticked (restoreRescue: true in the API). From the order to the copy waiting in rescue took 4 minutes 14 seconds.

The question that matters is whether the restored disk ever booted. The hypervisor's own log for the new server answers it: the server started once on its fresh system while it was being built, shut down, and the next start, after the backup was written, was the rescue system, with the restored disk attached as a second drive. The restored system never started.

In rescue#

The rescue system had the copy's disk as /dev/vdb, not mounted. We mounted it and looked at the canary log: only the original's lines, no new boot. Both checksums matched.

Then we ran the commands from Test a backup restore exactly as written: mask cron, anacron and Docker on the copy's disk, disable our two custom units, remove the linger file. Every one ran without changes.

Leaving rescue#

We chose leave rescue. The copy booted its own disk for the first time, on its new address, and we waited past the 20 seconds the slowest canary waits. Not one line from the new boot. Cron and Docker were masked, no canary timer was scheduled, and the database read back all 1,000 rows.

What we didn't test#

• A restore when rescue can't start. The order is refused or the restore isn't started, so nothing from the backup boots; that's covered by our automated tests, not this drill.
• An interrupted restore.
• Rescue running out after 24 hours. We didn't wait a day.
• A big disk. This was 2.6 GB of data. A 23.4 GB backup we restored on 31 October took about 25 minutes, without rescue.

What it found#

Two things in our own pages, both fixed the same night:

• On a staff account, a backup's new server… link opened the staff create form, which ignores the backup and would have built a blank server. Staff can't place orders, which a restore into a new server needs, so the link is now hidden for staff, and the form says why.
• While testing the WireGuard tutorial on the same night, we found it said a Rack4 Ubuntu server's network interface is eth0. It's enp1s0. With the wrong name the tunnel connects but your devices get no internet through it. The tutorial now uses the right name, and we checked that a client's IPv4 and IPv6 traffic leaves through the server.

Neither showed up in our automated tests. That's the point of running things on the real hardware, and why we'll keep doing it and writing down what we find.

The guides that describe this path now say what we tested and what we didn't: Off-site backups and Test a backup restore.