Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Large backups and offline restore

The versioned portable archive is the streaming alternative to the legacy JSON backup. It carries the registry’s graph, schema and users; history mode also preserves its Engrams and tag targets. Reconstructible history snapshot caches are omitted, but indispensable history bases and import/restore-marker snapshots remain, including markers that have older history as a parent.

Archive version 2 also preserves single-property registrations, compound index definitions and names used by DROP INDEX name. Online archives capture these from the same pinned view as the graph. Restore rebuilds compound index entries before publishing the destination. Use the matching server/admin revision: older readers reject version 2 rather than silently dropping its metadata. The new reader still accepts version 1, but cannot reconstruct index declarations that those archives never recorded (#1327). Recreate missing declarations from your schema migrations, then take a new backup; do not assume a legacy archive is a complete schema backup.

Online backup job

Submit on the leader using your existing global Admin credentials:

POST /mgmt/backups?history=true&format=archive&request_id=<UUID>
GET  /mgmt/backups/<UUID>
GET  /mgmt/backups/<UUID>/file

HTTP 202 means admitted, not complete. Download only when the job is ready; save the binary response as backup.pdb. Reusing the UUID with another history mode or format conflicts. History backup requires the existing PITR entitlement. Job status, retention and deletion otherwise follow maintenance jobs.

When PATINADB_BACKUP_JOB_MAX_BYTES is unset, this archive format has no fixed total-size cap. Setting it imposes an explicit byte budget. The disk reserve PATINADB_BACKUP_JOB_MIN_FREE_BYTES still applies; running out of capacity is an error, never a successful partial backup. Legacy JSON behavior is unchanged.

Required admin binary

Build both tools from the same revision:

cargo build --release -p patinadb-raft --bins

For a stopped registry, an offline backup avoids network transfer entirely:

patinadb-admin backup --from /data/patina --output /backups/backup.pdb --history
patinadb-admin verify /backups/backup.pdb
patinadb-admin restore /backups/backup.pdb --into /data/patina-restored

The server image also includes patinadb-admin. With Docker, override the server entrypoint using --entrypoint /usr/local/bin/patinadb-admin, mount the archive and target parent directory, and grant the container user access to those paths. Stop the source server before an offline backup; never restore over its mounted registry.

The output archive and restore destination must not already exist. There is deliberately no --force: the tool must never overwrite a running or mistaken target. Restore needs an existing parent directory on Linux and enough space for staging, index rebuilds and its free-space reserve. It does not contact a server and is not limited by an HTTP/query timeout. Run it as the intended database OS user, or deliberately adjust ownership before starting that user’s server; archive files and staging are private.

Both backup and restore accept --min-free-bytes (default 1 GiB). Restore also accepts an optional --max-archive-bytes policy limit. These do not provision space; monitor allocated disk and leave additional headroom for rebuilding indexes. The full restore’s memory/index-build behavior still needs qualification at your scale. The chunk payload is bounded to 1 MiB and manifest to 64 MiB; there is no graph-sized JSON snapshot in the new archive path.

verify checks the manifest and complete chunk/checksum chain. It is not a substitute for a restore drill: restore also validates graph/history records and rebuilds indexes. Truncated, corrupt or budget-exceeding input never publishes its staging registry. A crash may leave a private .patinadb-restore-* sibling; do not start a server on an unfinished stage.

Archives record each source database’s incarnation and history generation when it has one. An offline restore stores that as the restored database’s restored_from lineage; the database itself gets a new incarnation from the leader once the fresh cluster runs protocol 17. Archives from older servers restore without lineage. An archive carrying lineage is rejected by older patinadb-admin binaries (unknown field) rather than restored partially.

After restore, start a fresh cluster using the new registry directory and your normal license/configuration. Old Raft membership, logs, causal bookmarks and deduplication receipts are not restored. Do not drop this directory under an existing cluster identity. Users/password hashes and grants do survive: protect the archive like the original database. Checksums are not encryption or proof that the archive came from a trusted party.

Restored cluster bootstrap (#1325, development correction): older servers can let a new learner reach the causal log index without receiving the restored graph. Use the corrected server for the restored registry’s first start and initialization, not only when adding replicas later. A registry already initialized by an older server without the new bootstrap marker must be restored into a fresh registry for this procedure. Upgrading an already inconsistent cluster is not an automatic repair.

Initialize the restored registry with one local voter first (--bootstrap or a single-member POST /mgmt/init). The corrected server rejects direct multi-voter initialization of restored state. Before admitting the first new learner it durably checkpoints that state and retires the incomplete log prefix. An interrupted checkpoint remains pending across restart. A retryable DatabaseUnavailable response means admission has not completed: retry the same membership intent, rather than clearing bookmarks or manually copying live database files. The checkpoint can outlive an individual request. Its required baseline frontier is pinned durably on the first attempt: later writes remain a replayable tail rather than moving the retry target. A checkpoint from an obsolete causal domain is rebuilt before admission can complete. The server also waits for Raft’s snapshot-publication acknowledgement before requesting the corresponding log purge; a file appearing on disk is not enough.

Add the other nodes as learners, verify their graph and indexes, then promote the two additional local voters together. Keep regional replicas non-voting. Switch backend endpoints and trust configuration only after verifying the new cluster and imported grants. Old causal bookmarks are invalid in the fresh cluster. Do not silently discard them and retry a request; explicitly start a new recovery session after reconciling the acknowledged backup cut. Write-token receipts are also not restored: replaying an old write token can execute its mutation again. Reconcile uncertain pre-disaster writes before resubmission; tokens generated in the new cluster retain normal deduplication.

The development correction passed same-host SATA recovery at 64 and 4,096 tickets into three voters plus one regional learner, with three Rust/neo4rs backends repointed and fresh causal writes verified (#1261/#1325). Four real process-abort/restart boundaries passed at 64 tickets. Historical-cut checks cover the initially restored node, not immediate history availability on every replica. These are not independent-host, power-loss, WAN or 1M multi-node recovery results; keep deployment-specific restore drills.

Offline backup captures a stopped node’s durable local state; it does not prove that this node was current with its former leader. Recover incomplete installs and finish required history/publication recovery before taking a PITR backup.

Compatibility and qualification

format=archive is explicit. Existing JSON clients and POST /mgmt/restore remain unchanged; send the binary archive to patinadb-admin, not the JSON restore endpoint. The older embedded patinadb backup/restore format is also distinct. Unsupported magic/version is rejected rather than guessed.

Legacy JSON POST /mgmt/restore?force=true can create _restore_pre_* tags for the pre-restore state of existing databases. Treat those names as opaque. The #1323 development fix binds new names to their exact Engram instead of seconds, preventing repeated force restores from moving an earlier generated marker. Existing tags are not rewritten; this does not change archive restore or make force-merge restore a pristine disaster-recovery operation.

The previous #1261 JSON PITR run exceeded its configured 32-GiB budget and did not reach restore. That failed run remains failed. A new archive/admin drill on 2026-09-23 restored a one-million-ticket PITR archive under the current workflow: 3,896,529,031 transfer bytes, 259.682 seconds offline restore and 1,483,657,216 bytes sampled restore peak RSS (#1261). Boot-to-ready afterward was 0.202 seconds; do not confuse that with full recovery time. Mixed read p90s were 78–169 ms; several exceeded three times idle despite passing the existing 300-ms floor. No latency threshold was relaxed for this run.

A separate two-hour local five-node run recovered four HEAD/PITR archives into fresh isolated registries and checked graph, indexes, grants, constraints and subsequent writes (#1261). The source database directories were inaccessible during those restores. The large run selected PITR restore only; a HEAD export served as a contention control. These are single-host results, not off-host DR or a guarantee for your history size. See the qualification report for exact binary identity, failures, resource reserves and remaining gates.