Disaster Recovery

This is the operator playbook for backing up patinaDB and recovering from the failure modes a production deployment actually hits: one node down, the leader down, the whole cluster/host gone, and “someone ran a bad write.” It is deliberately concrete — exact endpoints, exact commands, exact caveats — not a restatement of the architecture chapters.

Qualification status (#1261): the later local 1M-ticket HEAD/PITR restore drills pass after the generic compound-index build fix; see docs/combined-rollout-1261-2026-09-10.md and the first-customer chapter. Those measurements do not qualify the new asynchronous backup-job path at 1M. The older timeout evidence remains in docs/backup-nightly-2026-09-08.md.

For asynchronous server backups, use the maintenance-job API. It produces the same portable HEAD/PITR format described here, but separates submission, preparation and download. ready is a local artifact, not an off-host recovery copy. The synchronous /mgmt/snapshot endpoint remains available.

Scope: this covers the clustered server. Everything here also applies, minus the Raft-specific sections, to a standalone embedded database directory via the CLI backup/restore commands.

1. What each backup mechanism gives you

There are three distinct mechanisms. They are not interchangeable — know which one covers which failure.

The shared whole-history importer stops the process if a storage error occurs after replacement writes have begun (#1443); it must not return control with a partially replaced timeline. Reopening recovers the prior durable history. An enclosing deferred transaction is rejected before import begins. This does not make the preceding chunked graph restore atomic. Clustered history restore has the separate protocol and publication requirements below.

What none of these carry

Be honest about the gaps before you rely on any of them:

• PITR restore needs all members on peer protocol 21. The leader sends the history in blocks of at most 1 MiB and each replica verifies its complete length and SHA-256 before publishing the history and release markers together. An older or unconfirmed cluster receives 409 before graph writes; there is no leader-local fallback. Apply can lag on followers: wait for their applied frontier before verifying history there. New learners obtain pending restore state through snapshots and older timeline through normal history backfill; wait for history readiness before directing time-travel queries to them. Temporary disk usage includes the durable encoded blocks and private verified input. Decode memory remains proportional to the history; the 1 MiB block size is a transport bound, not a total-memory limit. The existing restore upload cap still applies. Graph restore remains chunked and its documented rollback is graph-only; this is not an atomic transaction over the entire restore endpoint.
• Tags lose their original engram id in a HEAD-only backup. A tag name survives restore and still resolves — but it now points at the restored HEAD, not the historical engram it originally named, because a HEAD-only backup carries no history to re-point it at. Only a ?history=true restore preserves what a tag actually pointed to.
• Restore is additive, not a wipe. /mgmt/restore (and the CLI without --force) is meant to populate a fresh/empty registry. It creates each database (idempotent) and replays graph ops on top of whatever is already there — restoring into a cluster that already has data in the same database will not give you a clean replacement, it will merge/duplicate. Always restore onto a freshly bootstrapped, empty cluster. The REST endpoint’s ?force=true flag does not change this: it merely lifts the non-empty-target refusal so the restore is allowed to merge — it never deletes existing data. And for a PITR backup the guard is absolute: installing the engram history onto a database that has data or a timeline of its own is refused, with or without ?force=true.
• A restore starts a new database identity. Once every member runs protocol 17, /mgmt/restore finishes by giving each restored database a new incarnation (also when the name already existed and ?force=true merged into it), resets its history generation to 0 and records where the content came from (restored_from: the source incarnation and history generation at backup time). Backups taken by older servers carry no source identity; they still restore, with no recorded lineage. Below protocol 17 the restore keeps the target’s existing identity, as before. A failed restore is rolled back before this step, so it never changes the identity.
• The Raft snapshot is not off-site. It lives in each node’s own data directory. If you lose the disks (or the whole host/rack/region), the Raft snapshot is gone with everything else. Only the downloaded portable backup file is a real disaster-recovery artifact — store it somewhere the cluster’s own failure can’t take with it.
• Community-mode 30-day retention limits how far back a live AS OF can reach, independent of backups entirely: engrams older than the history-retention window (30 days on Community, unlimited on Pro/Enterprise — see Editions & Limits) are periodically compacted (squashed) on the running cluster. This governs recovery from a bad write via live time-travel (§4d), not the portable backup’s own timeline — a ?history=true backup taken before a squash boundary still carries what existed at backup time.
• A backup is a snapshot, not a continuous log. Between two backups, anything written and then lost is gone unless it’s still inside the live cluster’s own retention window (§4d covers that path). Backups are for “the cluster itself is gone,” not a substitute for the built-in time-travel retention.
• Backups taken by older releases may not carry every definition family. The backup format has grown over time (constraints, index families, WASM function defs were added field by field). Every field is optional, so an older backup still restores with a current binary — but the CLI restore prints a loud warning naming the def families the old file cannot have carried. Re-create those definitions manually, or re-take the backup with a current binary.

2. Recommended backup cadence & retention

There’s no one right cadence — it’s a trade-off between RPO (§3) and storage/operational cost. As a starting point for a production deployment:

Practical notes:

• Since 0.15.1, portable restore decodes the request incrementally without retaining a separate full raw-JSON buffer. The default upload limit remains 1 GiB. Set PATINADB_RESTORE_MAX_BODY_BYTES on the restore target to a positive byte cap when an artifact is larger (0/invalid values retain the default). The qualification profile explicitly selects 16 GiB, not a universal production recommendation. Decoded graph/history objects still occupy RAM; size the target and cap accordingly. Only one portable restore per node is admitted; concurrent requests receive retryable HTTP 503. Uploads have a 30-second between-chunk idle deadline and a 30-minute decode/upload deadline. Existing pristine-target and non-atomic chunked-apply rules still apply.

• The root operator password is target-local startup configuration and is not replaced by portable user restoration. Reconnect application users using their restored credentials, but keep using the target’s operator credentials for management. A completed download alone is not a verified restore.

• ?history=true costs more (it streams the whole engram delta/snapshot timeline, not just current HEAD) — measure the file size and export time against your actual write volume before committing to an hourly cadence.

• The export takes a leader linearizability barrier first (/mgmt/snapshot is leader-only — a follower returns a 503 leader hint) and is a consistent cut: every database’s HEAD label, its graph, and its engram history are read from one pinned point in time, so you do not need to quiesce writes while a backup streams. Writes are briefly serialized only while that point is pinned, never for the duration of the stream — measured under issue #1105, a 212 MB PITR backup taking 1.73 s moved a concurrent writer’s median latency from 3116 µs to 3333 µs (+7%), with no stall resembling the backup’s own duration. See Backup & disaster recovery for the full measurement and the caveat for older builds.

• Store backups off the cluster’s own infrastructure (different host, different failure domain, ideally a different region/AZ) — a backup that lives on the same disks/rack as the cluster is not a disaster-recovery artifact, it’s a second copy of the same single point of failure.

• Verify every backup after taking it (§5) — an untested backup is a guess, not a plan.

• Alert on backup age. A node stamps the Unix time of every successfully completed /mgmt/snapshot stream (a fully delivered download; an aborted download never stamps) and exports it as the Prometheus gauge patinadb_backup_last_success_timestamp_seconds — persisted across restarts, and ABSENT (no series, not a 0) on a node that never completed a backup. The shipped alert rules PatinaDBBackupStale (cluster-wide newest backup older than your RPO + buffer — default 26 h, tune to your cadence) and PatinaDBBackupNeverRecorded (no backup ever completed for 25 h — catches “the backup cron never existed”) turn “the backup cron silently stopped” from a restore-day surprise into a page.

3. RPO / RTO

RPO (Recovery Point Objective) — how much data you can afford to lose, measured as time since the last durable capture point. RTO (Recovery Time Objective) — how long recovery is allowed to take.

3a. Single node / follower lost

3b. Leader lost (quorum intact)

3c. Whole cluster / host / region lost — restore from portable backup

3c-bis. Majority permanently lost, a minority survives — the case that surprises people

This is the gap between 3a and 3c, and it has the worst RPO of the three, which is counter-intuitive because a node with your data is still running.

Concretely: 2 of 3 voters are permanently destroyed (host loss, disk loss, a region gone) and one node is still up, holding committed writes made since your last backup.

Why the surviving node’s newer data is not recoverable. Raft requires a quorum for every membership change, so the survivor cannot shrink the cluster to itself: POST /mgmt/evict-voter deliberately refuses with WouldBreakQuorum rather than hand you a configuration that cannot commit. That refusal is correct — it is the same guard that stops you accidentally stranding a healthy cluster.

patinaDB deliberately ships no “force new cluster” escape hatch. Some systems offer one (take the survivor’s data directory, force it into a single-member cluster). It is a genuine footgun: if the other nodes are not actually dead — a network partition that looks like host loss, a host that comes back — you have created two clusters that both believe they are authoritative, and the resulting split-brain loses acknowledged writes silently. We judged the blast radius of the recovery tool worse than the scenario it recovers.

What this means for you, practically: for the majority-loss scenario your backup cadence IS your RPO — the running survivor buys you nothing. If that RPO is unacceptable for your workload, the lever is more frequent backups (§2) and/or spreading voters across failure domains so that losing a majority at once is not plausible in the first place. Size the cadence against this row, not against 3a’s zero.

3d. Bad write / logical corruption

3e. Physical (on-disk) corruption on one node — a different failure from 3d

§3d is a logical corruption: the database faithfully stored what it was told, and what it was told was wrong. This one is the opposite — the bytes on disk no longer decode. A bit flip, a failing drive, a filesystem that lost a page.

What you will see. patinaDB is deliberately fail-loud here. A record that is present-but-undecodable is never treated as absent and never silently skipped: it is classified as a storage fault and it halts that node, rather than letting it keep serving answers computed from data it could not read. That is the intended behaviour and it is what keeps a corrupt replica from silently diverging from its peers — but it does mean one bad record takes a whole node out, which is a larger blast radius than a sharded system would have for the same fault.

The important consequence: in a replicated cluster this is a routine event with no data loss — the whole point of running three nodes. On a single node it is a restore-from-backup event, and your RPO is your backup cadence. That asymmetry is the strongest operational argument for replication, stronger than the failover argument, because a bit flip needs no operator error and no outage to happen.

4. Recovery procedures

All curl examples assume -u neo4j:"$PW" (or the appropriate RBAC user) unless the endpoint is explicitly auth-exempt.

4a. Single node lost — rejoin / learner catch-up

1. Confirm the cluster is otherwise healthy: GET /mgmt/cluster on any surviving node — check current leader + voter/learner list.
2. Bring the failed node back (same disk/data dir if recoverable, or a fresh empty data dir if the disk is gone):
   • Same disk recoverable: just restart the server with the same --id/--addr/--db. It resumes from its own persisted Raft log/state and catches up on the tail.
   • Disk lost / fresh replacement: start it fresh, then have an existing member add it as a learner:

     curl -u neo4j:"$PW" -X POST http://<leader>:21001/mgmt/add-learner \
       -H 'content-type: application/json' \
       -d '{"id": <new-id>, "addr": "<new-node-host:21001>"}'
     

     It bootstraps via the streamed Raft snapshot (memory-bounded, not proportional to total graph size in RAM) and then replays the log tail.
3. Watch it catch up: GET /ready reports 503 {"reason":"installing"} while installing, then {"reason":"lagging"} while catching up the tail, then 200 (see Health vs. readiness probes).
4. Once caught up, promote it to a voter if it isn’t already (only needed if it’s a fresh id, not a resumed one):

   curl -u neo4j:"$PW" -X POST http://<leader>:21001/mgmt/change-membership \
     -H 'content-type: application/json' -d '[1,2,3]'
   

No backup is needed for this path — it’s normal Raft self-healing.

This same fresh-empty-data-dir + rejoin procedure is also the zero-downtime way to reclaim disk from a .redb main file that’s grown far larger than the live graph — see Reclaiming disk from a bloated .redb file, which covers both this rolling-rebuild option and the offline patinadb <db> compact CLI command for a node you can afford to take fully down.

4b. Leader lost — auto-failover

Automatic: the remaining voters elect a new leader within one election round (sub-second to a few seconds); writes acknowledged before the crash are never lost (§3b). Clients following the 503 leader-hint (or the Bolt Neo.ClientError.Cluster.NotALeader forward) or using neo4j:// routing re-route with no operator action.

Manual, only if the old leader doesn’t come back:

1. Confirm the new leader: GET /mgmt/cluster on any node.
2. If the dead node was a voter you don’t intend to bring back, evict it so it stops counting toward quorum math (see §4e) — otherwise a second failure can cost you quorum.
3. If it will come back, just let it rejoin as a follower (§4a) — no membership change needed, it resumes as the id it already was.

4c. Whole cluster lost — bare-metal restore from a portable backup

1. Provision fresh host(s) for the new cluster. Start node 1 with --bootstrap on an empty data directory (this is important — restore is additive, not a wipe; see §1). For a ?history=true (PITR) backup a pristine target is not just important but required: installing the history onto a database that has data or a timeline of its own is refused outright.
2. Get the backup file onto the new node (however you stored it — S3, scp, etc.).
3. Restore it:

   curl -u neo4j:"$PW" -X POST http://<new-leader>:21001/mgmt/restore \
     -H 'content-type: application/json' \
     --data-binary @patinadb-backup.json
   

   No ?force=true — the target is empty, so the non-empty-target guard does not fire, and force would only ever permit a merge, never a wipe. (For an embedded/CLI-managed database instead of a server cluster: patinadb ./fresh.db restore ./backup.bak.)
4. For a PITR backup, restore once through the leader after every cluster member confirms peer protocol 21. Wait for each member to apply the restore, then check AS OF, original tag targets and patinadb.diff on each node.
5. Add remaining voters as learners and promote them (§4a step 2/4). They receive graph state through Raft snapshots and history through backfill. Wait for history readiness as well as graph catch-up; they do not need a separate portable restore or a private copy of the backup file.
6. Verify before declaring done (§5) — node/edge counts, a known query, and (if PITR) an AS OF probe.
7. Repoint clients/DNS/neo4j:// routing at the new cluster.

4d. Bad write / logical corruption — restore to a point-in-time

Prefer this over a full rebuild whenever the mistake is caught within the live retention window — it’s faster (§3d) and needs no backup file at all.

1. Find the last-known-good engram, e.g.:

   CALL patinadb.engrams() YIELD id, parent, timestamp, message
   RETURN id, timestamp, message ORDER BY timestamp DESC LIMIT 20
   

   or diff a suspect engram to see exactly what it changed:

   CALL patinadb.diff('<engram-id>')
   

2. Confirm the state at that point looks right with a read-only time-travel query (no risk — time travel never writes):

   curl -u neo4j:"$PW" -X POST http://<leader>:21001/cypher -H 'content-type: application/json' \
     -d '{"query":"MATCH (n) RETURN count(n)","at":"<good-engram-id>"}'
   

3. Promote that historical state back to HEAD as a new, append-only engram (nothing is rewritten or deleted from history):

   CALL patinadb.restore('<good-engram-id>')
   

   This replicates through Raft like any other write (see Engrams → Restore).
4. If the bad write is older than the live retention window (30 days Community / unlimited Pro/Enterprise — see Editions & Limits), it has already been compacted out of the live cluster’s history and this path is unavailable; fall back to §4c using the most recent ?history=true backup that predates it.

4e. Permanently-dead voter

A dead voter that isn’t coming back should be evicted — otherwise it continues to count toward quorum and one more failure can cost you write availability.

curl -u neo4j:"$PW" -X POST http://<leader>:21001/mgmt/evict-voter \
     -H 'content-type: application/json' -d '{"id": <dead-node-id>}'

• Must be run on the leader (a follower returns a 503 leader hint).
• Live-quorum guard: refused with 409 if the resulting voter set couldn’t still form a majority — it will not hand you a cluster that can’t commit. Evicting an unknown id or the last remaining voter is 400.
• Check the result with GET /mgmt/cluster. Then add a replacement via /mgmt/add-learner + /mgmt/change-membership (§4a) to get back to full voter count. See also Removing a dead voter.

4f. Physical corruption on one node — wipe and re-sync from its peers

For the failure described in §3e: a node halts with a storage/decode error (the log will name the fault; it is not a query error and not a crash loop you should try to ride out).

Do not try to repair the file. There is no record-level repair tool, and deliberately so: the node’s peers already hold the same committed state, which makes a rebuild both faster and more trustworthy than any surgery on a file whose integrity is already in question.

In a cluster (no data loss, no client downtime):

1. Confirm the rest of the cluster is healthy first — GET /mgmt/cluster on a surviving node, and check a write still commits. If the corrupt node was a voter and you are now at bare quorum, do step 2 one node at a time.
2. Stop the corrupt node if it has not already halted itself.
3. Move its data directory aside — do not delete it yet. Keep it until the rebuild is verified; it is the only evidence if this turns out to be a failing drive rather than a one-off flip.
4. Start it on a fresh, empty directory, same --id and --addr. It will catch up from the leader, by log replay if it is recent enough and by a streamed snapshot otherwise. No membership change is needed: the node id is unchanged, so it is still the same cluster member.
5. Verify with §5 — and specifically confirm the rebuilt node reports /ready as serviceable, since that is the check that will not pass until it has genuinely caught up.

This is the same “rolling rebuild onto a fresh directory” procedure documented under Reclaiming disk from a bloated .redb file; it is written up there for that use case, but the mechanism is identical and this is the other reason to reach for it.

Then look at the disk. One corrupted record is a symptom, not an incident. Check SMART/dmesg on that host before putting it back into rotation — if the drive is failing, the rebuild will corrupt again, and the second occurrence is much more expensive to diagnose than the first.

On a single node (embedded, CLI, or one-node server) there is no peer, so this is a restore-from-backup event — §4c, with §3c’s RPO. This asymmetry is the strongest operational argument for running three nodes.

5. Verifying a restore actually worked

Never consider a restore (or a backup) trustworthy until you’ve checked it. At minimum, after any restore in §4c or §4d:

1. Counts match expectations. Compare node/edge counts against what you expect (from GET /version’s usage block, or a Cypher count query) — not necessarily bit-for-bit against the pre-disaster source (writes may have landed between backup and disaster), but in the right ballpark and never suspiciously zero/empty.

   MATCH (n) RETURN count(n)
   

2. A known query returns the expected shape. Pick something business-meaningful and stable (a specific node by a unique key, a relationship count for a well-known entity) and check it by hand.
3. Schema/constraints came back. SHOW CONSTRAINTS, SHOW INDEXES, SHOW EDGE SORTED INDEXES, SHOW FUNCTIONS — confirm the defs you expect are present (they should be, per §1’s table), not silently dropped. For a backup taken by an older release, expect the CLI restore warning naming the def families the old file could not carry (§1).
4. If PITR (--history/?history=true): confirm AS OF actually reaches history, not just HEAD — pick a known older engram id (if you captured one before the disaster) and confirm it reconstructs the expected historical state, on the specific node you restored the timeline onto (§4c step 4):

   curl -u neo4j:"$PW" -X POST http://<restored-node>:21001/cypher \
     -H 'content-type: application/json' \
     -d '{"query":"MATCH (n) RETURN count(n)","at":"<known-old-engram-id>"}'
   

5. Cluster health. GET /mgmt/cluster shows the expected voters, GET /ready is 200 on every node, and each node’s applied log index has converged on the leader’s.

Drill this periodically (quarterly is a reasonable default for most deployments; more often if write volume or business criticality is high) — restore a real backup into a scratch cluster, run the checks above, and time it against your RTO target (§3c). An untested restore procedure is not a disaster-recovery plan.

The shipped scripts/verify-restore.sh (see Backup & disaster recovery) automates steps 1–2 for a quick post-restore sanity check; it does not replace a full drill.

See also

• Production Deployment — the 3-node bring-up and the §6 backup/restore quick reference.
• REST API — the /mgmt/snapshot, /mgmt/restore, and /mgmt/export endpoint reference.
• Editions & Limits — the PITR (?history=true) license gate and the history-retention window.
• Engrams — CALL patinadb.restore(...), tags, AS OF.
• Day-2 Operations — readiness semantics, the rolling-rebuild disk-reclaim procedure.
• High Availability — failover behaviour, membership changes, snapshot transfer bounds.

Refused snapshot repair: re-seed required

A returning replica may have applied more log entries than an incoming snapshot contains while still missing the snapshot’s graph history. Repair needs the full log tail above that snapshot’s cut. If those entries have already been purged, patinaDB refuses the repair before replacing the graph or changing its persisted applied position. The node records a durable re-seed required fence: reads and writes stay fenced, further snapshot installs cannot clear it, and startup reports the recovery instructions.

Stop the affected node and preserve its data directory for diagnosis. Verify the healthy cluster and recovery source, then replace/rejoin the replica using a fresh data directory through the supported disaster-recovery procedure. Do not remove just the fence, reset last_applied, or downgrade to bypass the check: those actions do not restore the missing data.

When the required tail exists, a forced repair retains its own checked copy before graph replacement. Boot recovery can therefore finish after a later Raft log purge. It validates the retained copy before modifying the graph and publishes a checkpoint covering the replayed tail before removing the install intent. These recovery descriptors use local format version 2; version 1 is still read, with its original live log tail checked before restore. This does not change the Raft snapshot wire format or existing stored property encodings. An interrupted version-2 install must be recovered by a binary that understands that descriptor.