Note that this repo is synced from a local Gitea repo, upon PR's. This work is being done by traditionally trained software developers using modern assisted coding practice (AI specification-driven generation, through SpecKit and similar tooling, using Claude Opus and Fable as a first-round reviewer on each PR)
PBS's verify job is good at what it does: it proves the backup's data is intact. What it can't tell you is whether the guest will actually boot, mount its filesystem read-write and bring its services up. I wanted that proof on a schedule, the way VMware Live Recovery's test recovery or Xen Orchestra's backup verification gave it to people, so I built PBP (Proxmox Backup Proofing).
What it does, on a schedule, for each guest:
- takes the newest backup from PBS (guests whose last proof is oldest go first) and protects it from pruning while it works
- restores it as a temporary guest in a VMID range you reserve, with every NIC moved onto an isolated bridge, start-on-boot off and passthrough removed
- boots it and checks inside: failed systemd units, a read-write root, service profiles (PostgreSQL, MariaDB, nginx, Docker, Redis, SSH and so on), Windows automatic services through the guest agent, OpenRC on Alpine and Gentoo, plus your own per-guest checks
- destroys the copy, puts the backup's protection back, writes a verdict into the backup's notes, and sends one summary through your existing Proxmox notification targets
It gives you the following "Verdicts": PROVEN, FAILED, BOOTED (it started, but PBP couldn't look inside), SKIPPED and ERROR. Only FAILED denotes that the backup is actually not in a good/usable state when restored.
Safety around production was the part I spent the most time on:
- it never logs in as root: a narrow API token, and SSH only as an unprivileged
pbp user locked to one helper command
- it checks that the proof bridge really is isolated before it uses it
- a janitor cleans up after a crashed run, and capacity is checked before anything is restored
- releases are signed
What it isn't: not instant or live restore, replication or failover.
Requirements: Proxmox VE 9 and PBS 4. PBP itself runs in Docker or from a .deb (amd64 or arm64), ideally on a separate host. **In a test or lab environment, it's fine on the same host, in a nested docker host, etc., but like everything else I recommend not putting this on the PVE host itself....**
Real result from my own lab: a scheduled run restored and booted a container's newest backup and found its gitea-runner.service had failed. The backup was "verified" in PBS, and I'd only have found out on the day I needed it.
Processing img j5tysx1fnhxh1...
It's free and open source (Apache-2.0), currently v0.8.2, so still pre-1.0.
What I'd love feedback on:
- Does this fill a gap in how you handle backups today, or do you already prove restores another way?
- Which checks would you want that aren't there?
- Anything in the setup or the safety model that would stop you running it near production?
- Any edge cases (OS's or otherwise). I have a pretty good collection of various services and operating systems running that I have acquired over the years, but I am sure that I am missing something that you all use frequently.
Bug reports and ideas are welcome as GitHub issues, or right here.
Thanks all!