Maintenance jobs

For large portable backups use the explicit streamed archive and offline admin restore. Legacy JSON remains available; the new binary format avoids exporting all reconstructible history snapshot caches.

Long-running work and squash

Background execution does not remove the need for resource limits or query deadlines. Interactive queries remain synchronous and bounded; large maintenance needs explicit admission, status and completion semantics. Losing a connection is not proof that an admitted job was cancelled.

Prepared squash jobs are experimental and disabled by default. The explicit REST interface described below separates Raft admission, replica preparation, publication and completion. It does not change CALL patinadb.squash: that command remains synchronous, including its existing apply work and possible request timeout. Planning shares the maintenance artifact lane and streams the canonical identity, but this is not a production or performance qualification. A database with at most 4096 live vertices and edges plans on a separate small-database lane instead, so it does not wait behind a long tag, backup or squash job for a larger database. At most one such small squash runs beside one heavy job per node; heavy jobs remain serialized. The bound counts live records, not file size: a database file keeps its high-water allocation after history churn. The bound is a heuristic, not a measured memory limit.

The #1261 1M maintenance run6 still failed at the synchronous squash HTTP timeout. Post-squash PITR export/restore was not reached. Successful historical-read tests or component preparation tests do not supersede that failure; a new complete maintenance measurement is required. A subsequent #1261 component run on the 1M-ticket copy passed (222.090 s preparation, 5.145 ms local publication, 2.096 GiB maximum RSS; fresh process with warmed OS cache). Raw evidence: async-squash-1261-20260913-component-run2/summary.json and benchmark.log; repository report: docs/async-squash-1261-2026-09-13.md. A small real cluster also passes history-inclusive backup and pristine PITR restore after squash. Neither result substitutes for the complete mixed-load 1M maintenance gate. The repository qualification driver selects this experimental path with history_qualify.py --maintenance-only --async-squash; it requires committed completion before PITR export and checks pinned history on a pristine restore.

Experimental prepared squash

Keep PATINADB_EXPERIMENTAL_SQUASH_JOBS disabled until the deployment’s qualification gates pass. For an explicit experiment, enable it consistently on every participating node, including learners. All members must support protocol 16; do not downgrade a cluster containing durable squash-job records.

The endpoints require global Admin and access to the target database:

POST /mgmt/squashes
GET  /mgmt/squashes/<job_id>
POST /mgmt/squashes/<job_id>/cancel

Submit JSON on the leader, for example:

{"database":"default","keep_recent":10,"request_id":"baf2012c-205f-41b7-8bf5-612c10518826"}

request_id is an optional client-generated UUID. Reusing it with identical database/options returns the existing replicated record while it remains; different options conflict. Omitting it creates a new request. Optional before is a Unix-seconds cutoff. keep_recent must be at least one. A job handles one selected prefix of at most 4096 Engrams: completion does not guarantee that the whole database has reached the requested retention target.

HTTP 202 acknowledges durable admission, not completion. The record identifies the fixed prefix; ordinary graph appends do not move that job’s cutoff. One job per database can be active, with limits of 32 active jobs and 256 retained records cluster-wide. These are admission limits, not throughput measurements. Terminal-record deletion/expiry is not exposed yet; the retained-record ceiling can refuse additional submissions. Dropping a database retains terminal records: unfinished work becomes Superseded, successful work becomes Retired. This preserves request deduplication and the protocol-16 minimum across drop/recreate; dropping databases is not a way to reset that compatibility requirement or cap.

These are logical state names; the detail endpoint returns the serialized record. Preparation and publication receipts are local to each node. Both readiness barriers include every current member, including non-voting learners, not just a voter quorum. An unavailable learner can therefore delay completion. Membership changes require receipts from the new current membership.

Normal HEAD reads and graph writes can proceed during preparation; CPU, disk, MVCC retention and the artifact lane remain shared. During committed publication, affected history queries and retention metadata are fenced rather than exposing different old/new history boundaries. History backup is refused until publication is safe; retry after completion. Busy publication retries; neither preparation nor publication has a qualified latency guarantee.

Cancel explicitly while the record is Queued, including while a worker is preparing. A racing commit can win: cancellation then returns a conflict and recovery must finish the decision. Disconnecting a client never cancels a job. Do not use feature-flag changes, node shutdown or manual staging-file deletion as cancellation mechanisms.

Job decisions survive restart and ride Raft snapshots. A recovering node must verify its published boundary or recover the exact decided artifact; absence of a local staging manifest is not proof of cancellation. A node without the needed history remains fenced until recovery. Successful logical publication does not itself prove reclaimed disk, an off-host backup, or a successful PITR restore. The complete crash, learner, mixed-load and post-squash restore gates remain part of experimental qualification.

Disabling new admission does not stop recovery/finalization of committed jobs. Existing status/cancellation endpoints remain available; query job details on the leader for a fresh status barrier. Durable squash records automatically keep history backfill available for recovery, even without its experimental environment setting after restart. Cleanup may run after publication, but its progress is not a replicated job-completion guarantee and staging-artifact expiry is not provided.

Tag snapshot jobs

The Raft server runs tag-snapshot preparation outside serialized Raft apply. CREATE TAG returns after its name, exact Engram and retention pin are durably committed; it does not wait for the full snapshot. The tag is immediately usable through existing historical reconstruction where that history is available.

Waiting tag jobs retain their durable history reference, but do not hold a native database read version (#1485). A worker acquires its immutable view when preparation is admitted. If the tag is no longer at that view’s HEAD, the worker uses the retained historical reconstruction path. This can require more replay than capturing at submission; no preparation-latency guarantee is made. An active snapshot or query can still retain old disk pages while writes continue. Releasing a native reader permits reuse of eligible pages; it does not promise an immediate decrease in the database file’s allocated disk size. The durable history pin remains until the tag is removed. No new setting is required. A causal bookmark covers the tag commit, not background snapshot completion.

Historical fallback can reuse one private primary-data snapshot base between jobs, without constructing query indexes on that base. Each target replays its own deltas in an isolated overlay. This export-only base is separate from the fully indexed stores used for historical queries and is invalidated with history. Reuse can cross newer snapshot boundaries only with a checked ancestry chain, unchanged snapshot definitions and replayable deltas. Missing optional history, a floor boundary, snapshot-only transitions or the bounded replay-work ceiling retain the nearest-snapshot rebuild. No tag is declared ready before publication. Snapshot I/O still competes with live traffic; there is no background latency or total-memory guarantee.

Inspect the queue

As an administrator, use Bolt or the existing Cypher HTTP endpoint:

SHOW JOBS;
USE default SHOW JOBS;

Tag rows are local to the receiving node and selected database; backup rows are node-local full-registry jobs (database = '*') included in every database’s listing. The result is not a cluster-wide view; a remote learner may be at a different stage. HTTP returns the rows under jobs. Bolt returns these columns:

For tag rows, succeeded means this node has published the target snapshot. It does not mean all replicas have done so, nor that an off-host backup exists. There is no percentage estimate. SHOW JOBS currently has no WHERE, YIELD, ORDER BY, manual retry, cancellation or priority syntax. It requires global Admin.

Squash rows show the receiving node’s applied replicated state for the selected database, not a linearizable cluster-wide status poll. Their lowercase states are queued, committed, succeeded, cancelled and superseded; retired records also appear as succeeded here. Use the detail endpoint to distinguish retirement. Squash attempts and sanitized errors describe only the receiving node; waiting members and percentage estimates are not exposed. Cancel through the explicit REST endpoint, not SHOW JOBS syntax.

Recovery and resource behavior

Live tags are the persistent work list. Multiple tags pointing to one Engram share one job. Pending work is rediscovered after restart or history backfill; preparation can repeat, but publication is checked and idempotent. Failures retry with bounded backoff. Error/attempt history is a bounded in-memory diagnostic, not an audit log. Rows disappear when their last tag reference disappears.

One tag job executes per node at a time and shares the existing process-wide snapshot-build admission limit. A waiting job does not retain a read snapshot. An active job retains its immutable view and history dependencies. Normal graph writes can continue; destructive history operations may wait for that lease. Disk, CPU and retained MVCC pages are still shared resources, not isolated quotas. Historical targets may require on-disk reconstruction before snapshot streaming.

DROP TAG removes demand; moving a tag changes its target. An already-running serializer may finish its pass, but obsolete work is discarded before publication. Database replacement and snapshot installation also invalidate old prepared work. Do not explicitly unpin live tag targets whose history you need: the existing administrative unpin override still removes their retention protection. Dropping the final tag retains the usual unpin behavior.

This first increment covers server-side CREATE TAG. Embedded synchronous tagging and tagging inside existing bulk/upsert APIs retain their previous behavior. User queries and the legacy synchronous squash command have not been moved into the tag queue; experimental squash jobs use their own replicated lifecycle. The existing synchronous backup endpoint keeps its completion contract. During a rolling upgrade an older node may still build tag snapshots synchronously.

Asynchronous backups

All backup-job endpoints require global Admin and use the same authentication as /mgmt/snapshot. Archives contain every database and the user directory, including password hashes: protect downloads and server storage accordingly.

Submit on the leader, using HTTP POST:

POST /mgmt/backups?history=false&request_id=<client-generated-UUID>
POST /mgmt/backups?history=true&request_id=<client-generated-UUID>

The response is HTTP 202 with a persistent job_id. Repeating the same request ID on the same node, while its record remains, returns the existing job; changing its history option returns 409. Omitting the ID creates a new job each time. IDs are not cluster-wide retry tokens: after a lost response, inspect the original node rather than blindly resubmitting on another leader. PITR/history still requires its license entitlement and complete local history. These checks are repeated when execution begins.

GET    /mgmt/backups/<job_id>
GET    /mgmt/backups/<job_id>/file
DELETE /mgmt/backups/<job_id>

SHOW JOBS exposes queued, running, ready, failed, and expired for backups. Backup rows have an empty engram and no tags: the archive’s heads map names the per-database capture points. The detail endpoint also provides the history option, byte length, creation/expiry timestamps (Unix seconds), attempts, and a sanitized error. Capture occurs when the worker executes, not at enqueue time; it takes a fresh leader barrier. Queued jobs hold no database read views.

ready means the complete archive has been flushed, synced and published locally, not that an external backup exists. Download the file, persist it outside the cluster, and verify it using a restore drill. Completed HTTP streaming updates the existing backup-success stamp; this still cannot prove client-side persistence. Interrupted downloads retain the archive and can be restarted from byte zero. There is no Range/resume support in this increment.

Jobs and files belong to their original node and are not Raft-replicated. On restart, queued jobs remain queued, ready archives remain downloadable, and interrupted running jobs become failed. They are not silently retried against a different data cut. Submit a new ID to retry a failed job. Leadership loss before capture can fail the job; no automatic cross-node failover is promised.

One backup/tag serialization job runs per node through shared admission. Up to four backup downloads can be active. A node keeps at most 16 job records; delete terminal jobs to free slots (active jobs cannot be deleted). Ready archives expire after 24 hours; queued jobs expire 24 hours after submission. Expired files are reclaimed by maintenance, while status records remain until deletion. These are configured limits, not performance measurements.

PATINADB_BACKUP_JOB_MAX_BYTES limits each asynchronous archive when explicitly set. The legacy JSON format defaults to 16 GiB; format=archive has no default total-byte cap and uses the offline admin restore tool described above. PATINADB_BACKUP_JOB_MIN_FREE_BYTES sets the filesystem free-space floor (default 1 GiB). Both accept positive integer byte counts. Serialization checks available space periodically and fails rather than publishing a partial archive; this is not an exclusive disk reservation against other processes. Budget MVCC retention, Raft snapshots, downloads and normal writes separately. The old synchronous JSON endpoint is not governed by these job-specific limits; explicitly choosing format=archive applies the disk reserve and any configured byte budget there too.

The JSON backup/restore format is unchanged. Missing referenced history snapshot bodies now fail export rather than silently produce an incomplete PITR archive (#1289). Backup capture also opens the referenced snapshot files under a short, nonblocking retention lease (#1291). The backup retains those handles through serialization: later pruning or atomic replacement cannot remove its captured bodies. The lease is released before serialization, rather than held across the export. Active retention can refuse capture; submit a new job after it finishes. Each captured snapshot consumes a file descriptor until the view is dropped; descriptor exhaustion or a missing body fails capture, never produces a partial successful archive. Unlinked bodies still consume disk while handles remain open. Avoid concurrent database/schema/RBAC changes during preparation: the existing consistent-cut guarantee covers graph, HEAD and history, not an atomic snapshot of every live schema/access-control definition. Ordinary graph updates are covered by the pinned views. Ready-file checks detect missing or size-mismatched files, not arbitrary same-length corruption; retain independent integrity checks and restore drills for the downloaded recovery copy.

The legacy synchronous endpoint also retains cleanup ownership during preparation and the handoff to its response stream (#1292). Cancelling the request does not leave an unowned completed archive behind once the blocking preparation finishes. This is distinct from async jobs: disconnecting a client does not cancel a durable job, and its ready artifact remains available until expiry or deletion.