Automate snapshots before risky changes with the CLI

To take a snapshot automatically before a risky change, run the change through a short script that calls rack4 snapshots <server> create, waits until the server's pendingAction is empty and the snapshot is ready, and only then runs your command. If the change goes wrong, rack4 snapshots <server> revert <name> puts the disk back.

What we tested: the script below passes shellcheck, and we ran it on Ubuntu 24.04 with the rack4 CLI against a local copy of the Rack4 API: with a read-only token (it stopped before changing anything), four times in a row on one server (the fourth run deleted its own oldest snapshot to make room), and with no command (it printed its usage). A revert of the newest snapshot was queued afterwards.

How snapshots behave#

A few facts the script is built around (Snapshots has the full story):

• A server keeps 3 snapshots you take yourself. Automatic ones from a schedule don't count toward that; the API refuses a fourth manual one with limit_reached.
• A running server shuts down for a snapshot and starts again, usually within a minute. Plan for that minute.
• Taking a snapshot answers at once and finishes in the background. The server's pendingAction is snapshot until it's done, then null, and the snapshot's state goes from creating to ready.
• A snapshot lives on the same host as the server. It's for undoing a change, not a backup against losing the host; keep real backups too.

By hand first#

With the CLI installed and logged in:

rack4 snapshots web-1 create before-upgrade
rack4 snapshots web-1

taking snapshot "before-upgrade" of web-1; a running server shuts down for it and starts again
ID                     NAME            KIND    STATE  TAKEN
snap_amyimqrlomo5zj3t  before-upgrade  manual  ready  2026-10-21T23:13:42.998Z

That's fine for one change. The script below adds the waiting, the checking and the rotation, so you can put it in front of any command.

The snapshot-then script#

Save this as ~/.local/bin/snapshot-then (or anywhere on your PATH) and make it executable with chmod +x ~/.local/bin/snapshot-then:

#!/usr/bin/env bash
# snapshot-then SERVER COMMAND...: snapshot SERVER with rack4, wait until the snapshot is ready, then run COMMAND.
set -euo pipefail
server="$1"; shift
[ $# -gt 0 ] || { echo "usage: snapshot-then SERVER COMMAND..." >&2; exit 2; }

idle() {
  for _ in $(seq 1 60); do
    [ "$(rack4 --json servers show "$server" | jq -r .pendingAction)" = "null" ] && return 0
    sleep 5
  done
  echo "$server is still busy after 5 minutes" >&2
  exit 1
}

# 3 snapshots of your own per server: delete the oldest one this script took to make room.
snaps="$(rack4 --json snapshots "$server")"
if [ "$(jq '[.[] | select(.automatic | not)] | length' <<<"$snaps")" -ge 3 ]; then
  old="$(jq -r '[.[] | select((.automatic | not) and (.label | startswith("pre-")))][0].id // empty' <<<"$snaps")"
  [ -n "$old" ] || { echo "$server has 3 snapshots of its own already; delete one first" >&2; exit 1; }
  rack4 --yes snapshots "$server" rm "$old"
  idle
fi

label="pre-$(date -u +%Y%m%d-%H%M%S)"
id="$(rack4 --json snapshots "$server" create "$label" | jq -r .snapshot.id)"
idle
state="$(rack4 --json snapshots "$server" | jq -r --arg id "$id" '.[] | select(.id == $id) | .state')"
[ "$state" = "ready" ] || { echo "snapshot $label ended as ${state:-missing}: not running the command" >&2; exit 1; }
echo "snapshot $label is ready. To undo: rack4 snapshots $server revert $label" >&2
exec "$@"

What it does, in order:

1. If the server already has three snapshots of its own, it deletes the oldest one whose name starts with pre-, so it only ever rotates its own. If all three are yours, it stops and asks you to delete one.
2. It takes a snapshot called pre-<date>-<time> in UTC.
3. It waits, checking every 5 seconds for up to 5 minutes, until the server has no action running.
4. It checks the snapshot is ready, and stops if it isn't.
5. It prints the command to undo, then replaces itself with your command (exec), so your command's exit code is the script's.

Use it#

Put it in front of the command that makes the change. With rack4 ssh, the change runs on the server:

snapshot-then web-1 rack4 ssh web-1 -- 'apt-get update && apt-get -y full-upgrade'

snapshot pre-20261022-010036 is ready. To undo: rack4 snapshots web-1 revert pre-20261022-010036

Anything works after the server name: an Ansible run, a deploy script, a database migration:

snapshot-then db-1 ansible-playbook site.yml --limit db-1
snapshot-then web-1 ./deploy.sh

The script reads RACK4_TOKEN or the saved login, like every rack4 command, so it runs the same from a cron job or a CI step with the token in the environment.

Check it worked#

rack4 snapshots web-1

ID                     NAME                 KIND    STATE  TAKEN
snap_usz26wzf7jyb4cux  pre-20261022-010043  manual  ready  2026-10-22T01:00:43.333Z
snap_6gh6rc3iykglvwm7  pre-20261022-010049  manual  ready  2026-10-22T01:00:50.175Z
snap_urcctcb6vbjtuler  pre-20261022-010101  manual  ready  2026-10-22T01:01:01.890Z

After many runs there are never more than three, all ready, the newest from your latest change.

Undo a change#

If the change broke something, revert to the snapshot the script named:

rack4 snapshots web-1 revert pre-20261022-010101

It asks you to type the hostname, then puts the disk back as it was and starts the server. Everything written since the snapshot is lost, including logs and uploads, so copy off anything you need first. Add --yes to skip the question in a script.

Keep a nightly one as well#

For changes you don't plan, add an automatic schedule. These don't use up the three manual slots:

rack4 snapshots web-1 schedule daily --at 3 --tz Europe/Berlin --keep 3
rack4 snapshots web-1 schedule

web-1: automatic snapshots daily at 03:00 Europe/Berlin, keeping the newest 3
next: 2026-10-22T01:00:44.000Z
each automatic snapshot pauses the server for about a minute

Troubleshooting#

rack4: This token does not have the write scope. The CLI is logged in with a read-only token. Log in with a read-and-write one, or set RACK4_TOKEN for this command.

web-1 has 3 snapshots of its own already; delete one first. All three manual snapshots were taken some other way. Delete one you no longer need with rack4 snapshots web-1 rm <name>.

web-1 is still busy after 5 minutes. Another action was running, or the snapshot is taking unusually long. Check the server's events with rack4 servers show web-1, and open a ticket with the request id if a snapshot doesn't finish.

jq: command not found. Install it: sudo apt install -y jq, sudo dnf install -y jq or brew install jq.

Next steps#

• rack4 CLI cheat sheet, every command on one page.
• Deploy on push with GitHub Actions, which takes a snapshot before each deploy.
• Backups you should run, for copies that live somewhere else.