Day-2 Operations

Once a cluster is up, three things make it operable: a probe that tells a load balancer or Kubernetes when a node can actually serve, request correlation so a slow query is traceable end-to-end, and a way to remove a permanently-dead voter so it stops blocking membership changes and quorum math.

Backups, RPO/RTO planning, and step-by-step recovery for a lost node, a lost leader, or a whole-cluster restore build on the /mgmt/snapshot / /mgmt/restore mechanics in Production Deployment → Backup & disaster recovery and the removal procedure below — ask your patinaDB contact for the full disaster-recovery runbook if you don’t already have it.

Health vs. readiness probes

The server exposes two unauthenticated probe endpoints. Both are auth-exempt (like /version) so a load balancer or orchestrator can reach them without credentials, and neither is ever shed by the concurrency limiter. /metrics is not in that set: it needs a credential and the global Admin role unless --insecure-open-metrics is passed (see the manual’s Security chapter).

/ready returns 200 with {"ready": true, …} only when all of the following hold, and otherwise 503 with a short machine-readable reason (checked in priority order):

Example bodies:

// GET /ready  → 200
{ "ready": true, "current_leader": 1, "last_applied": 42, "apply_lag": 0 }

// GET /ready  → 503 (fresh node, no leader yet)
{ "ready": false, "reason": "no_leader", "current_leader": null, "last_applied": 0, "apply_lag": 0 }

Readiness is not a guarantee that the next write finishes within a particular deadline. After a whole-cluster restart, a leader can have its local state fully applied and fresh quorum contact while rebuilding volatile follower replication positions. Retained large log entries can make that recovery significant (#1261, repository docs/customer-blackbox-million-1261-2026-09-16.md). For a planned restart/admission check, also inspect the leader’s patinadb_raft_follower_lag for the replicas that your deployment expects to have caught up; absent series do not prove zero lag. A zero-valued self entry may also be present. This diagnostic is not a replacement for quorum rules or causal bookmarks, and waiting for an intentionally unavailable member would unnecessarily block a healthy surviving quorum.

Because /ready fails when a node is leaderless, lagging, mid-install, or degraded, pointing a load balancer at it removes known-unready nodes from the rotation. It does not replace bookmark waits for causal reads. Keep /health as the liveness probe so a slow-but-catching-up node is not restarted while it recovers.

The distroless image ships without a shell, so the compose healthcheck can’t curl from inside the container — probe /health and /ready from the orchestrator / an external monitor instead.

Kubernetes example

livenessProbe:
  httpGet: { path: /health, port: 8080 }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5

Request correlation & tracing

Startup snapshot recovery

When Raft must reconstruct a local checkpoint after log purging, it preserves an existing causal domain before peer-version discovery has completed (#1494). That local recovery does not confirm peer capabilities: serving the cached checkpoint remains gated on compatible, confirmed member versions. Recovery can therefore finish locally while replication is still waiting for discovery.

During local Raft initialization, a recovered current-format checkpoint in the same causal domain can be retained even if its frontier is older than the already-applied state. This avoids rebuilding it solely because peer discovery has not started. The permission is limited to the initializing task and store; replication workers still check peer compatibility. Missing or invalid local checkpoints can still require reconstruction; large-file recovery latency and the current four-GiB pressure profile remain under qualification.

After initialization, a completed peer-version poll can still leave unreachable members unconfirmed. In that case local causal checkpoints use logical format7 and preserve pending history restores and replay receipts (#1498). They remain unavailable for replication until compatible versions are confirmed. Known pre21 peers retain the existing refusal. This avoids treating discovery alone as a fatal storage error; it does not make incompatible members compatible or remove the disk work needed to create a checkpoint. State-only snapshots still require the separate history backfill for older timeline data.

Persistent-store durable commits retain redb’s quick-repair metadata, including the terminal durability barrier after a deferred multi-tree write (#1307). This avoids the full allocator/checksum reconstruction normally needed after an unclean shutdown. Deferred writes still roll back unless their durability barrier completed. This adds allocator metadata and two-phase commit work to durable writes; it is not free and does not make all node startup instantaneous. Older files without this metadata may still need a full repair on their first unclean reopen. Never skip integrity checks or repeatedly kill a repairing node to meet a readiness timeout. Snapshot reconstruction, log replay, index recovery and cluster catch-up are separate startup costs.

Before starting Raft or pinning boot snapshot views, the server performs an empty durable preparation commit on each opened graph, after recovery of a pending snapshot install. redb can then reclaim pages whose old read versions became unpinned by the restart before the first replicated write arrives. Startup logs report startup storage preparation started and startup storage preparation complete per database. A storage error prevents startup instead of admitting a node with failed preparation. Budget this work in startup probes: it may involve disk I/O and has no fixed latency bound. This does not change graph contents, history HEAD, replication positions or normal write durability, and does not replace cluster catch-up or compaction. Embedded callers can request the same boundary with Dataset::prepare_for_serving() before admitting work; calls inside an active deferred transaction are rejected.

If a Raft log prefix has already been purged and its cached snapshot is missing or incompatible with the current causal domain, startup may rebuild a snapshot before the listener becomes available. Budget this separately from ordinary process restart and inspect the snapshot-build spans; it can involve substantial disk I/O. Startup recovery does not wait for the peer-version publisher, which starts after Raft construction. Once the application is initialized, normal snapshot builds still use the bounded first-version-publication gate.

Correlation and export

Every HTTP request runs inside a tracing span carrying a request id:

• If the caller sends an X-Request-Id header it is honored (used verbatim); otherwise a fresh UUID is minted.
• The id is echoed on the response X-Request-Id header, and it appears on every log line emitted while handling the request — so a slow or failing query is traceable from the client through the server logs.
• A valid inbound W3C traceparent supplies both correlation and remote span parenting through the same validator. Invalid or duplicate parent headers are ignored. Vendor tracestate and baggage are not imported in this increment.

Setting --otel-endpoint <url> (or PATINADB_OTEL_ENDPOINT, e.g. http://otel-collector:4318) installs a real OTLP span exporter (OTLP/HTTP protobuf, /v1/traces is appended automatically). Only explicitly selected diagnostic spans are eligible for export, not ordinary log events. Outbound /raft/* peer RPCs propagate the current OTel context when present; this alone does not guarantee propagation through every internal Raft task or batch. This is independent of PATINADB_LOG=json (structured JSON logging) — set either, both, or neither; X-Request-Id correlation is always on regardless of this flag. Exporter initialization or sampling configuration failure logs a warning at startup and the node runs normally with export disabled (initialization failure is never fatal; once initialized, network failures during actual export are async/best-effort and never block a request). With no endpoint set, no OTLP provider or exporter is installed; existing logs/spans and header validation still have overhead.

RFC0033’s initial implementation samples 1% of trace IDs by default, with a node-wide ceiling of 128 admitted spans per one-second window. Configure these using PATINADB_TRACE_SAMPLE_PERCENT (0–100) and PATINADB_TRACE_SPANS_PER_SECOND (1–1024). These are configuration limits, not measured performance figures. Remote sampled flags do not bypass either limit. Invalid sampling configuration disables export with a startup warning, not the database. Saturation may produce incomplete traces; policy rejection is counted in patinadb_trace_dropped_total by bounded reason (rate_limit, policy_busy).

The SDK exporter queue holds at most 512 spans, with batches of at most 64 and a two-second HTTP export timeout. Exported spans contain at most 16 allowlisted scalar attributes, with strings capped at 128 bytes; events and links are not exported yet. Unknown/raw paths are folded to <unmatched>. Request IDs unsafe for diagnostic attributes are redacted in the span while response echo behavior remains compatible. Existing ordinary application logs are not covered by this export allowlist. Export goes only to the operator’s endpoint, never to licensing heartbeat telemetry.

Current coverage is deliberately narrow: HTTP handler spans, Bolt RUN dispatch, bookmark admission, snapshot build and its blocking write phase, and blocking snapshot restore, plus local fulltext/vector search phases. RUN dispatch is not the full query/result lifecycle, and HTTP handler completion is not streamed-body completion. Snapshot workers retain their parent span across the blocking-pool handoff and cancellation.

Search diagnostics expose two logical levels below the search operation:

Streaming Bolt graph queries also expose db.query.stream with preparation, planning, eligible result-cache lookup, fast-path selection, and execution phases. Execution measures iterator construction and eager operators; lazy row consumption and client backpressure remain inside the enclosing stream. Aggregation separates input consumption/folding from finalization. Parent and child durations overlap; do not add them. These spans contain no query text or parameters and are subject to the same sampling/export limits below.

db.search.fulltext
  .lookup / .parse
  .index
    .statistics / .evaluate / .rank
db.search.vector
  .lookup
  .index
    .centroids / .candidates / .rank

Fulltext evaluation includes posting reads/decoding and recursive query scoring; vector candidates include posting reads/decoding, scoring and top-k maintenance. These are fused phases, not independent I/O and CPU timers. Rank includes result collection and ordering, not entity hydration or Bolt delivery. Missing indexes, invalid dimensions and empty vector requests terminate before later phases. There are no per-document, per-term or per-centroid spans. Index names, query text, vectors and entity IDs are not recorded. These spans do not change search or bookmark semantics; disabled-export overhead is not claimed to be zero.

The OTLP HTTP client runs in blocking mode on the SDK’s dedicated batch worker, not on request executor threads. A local 1M diagnostic run captured nested spans (#1261, docs/latest-onboarding-traced-1261-2026-09-15.md); this is not a paired overhead qualification or full RFC0033 acceptance.

Snapshot diagnostics now include logical subphases: checkpoint table copy, durable commit, close and sync; packaging copy/hash; installation copy/hash, sync, open-probe and swap. Verification separates blocking-pool admission from read/hash work. Detached receive finalization retains its parent context. These support operation → phase → subphase drill-down, not per-record tracing. Durations are wall time, including waits; nested or overlapping spans must not be summed. Copy/hash and receive/write remain fused phases. Missing children can also mean sampling/rate/queue drops, not absent work. Cross-Raft parentage and background history coverage are not complete; no overhead qualification yet.

Still pending from RFC0033: complete transaction/PULL/body lifecycles, targeted capture, Bolt transaction-metadata propagation, queue byte/drop accounting, global live-span/per-operation budgets, bounded provider shutdown, richer phase coverage and overhead qualification. The initial queue/attribute bounds are not a demonstrated whole-process memory ceiling. No public trace endpoint or automatic slow-trace retention is provided.

Diagnosing causal waits and Raft apply

For a local incident capture, enable RUST_LOG=info,patinadb_raft::progress=debug,patinadb::apply_progress=debug,patinadb::squash_progress=debug.

Legacy squash emits legacy_squash_attempt events before each phase. Differences between cumulative elapsed_us values belong to the preceding phase. Correlate progress_id and rust_thread with the enclosing Raft apply; IDs are local to each event family. The phases distinguish reconstruction-slot admission, chain selection, reconstruction, operation materialization/hash, genesis writes, write and retention locks, revalidation, reparenting, prefix deletion and terminal sync. An end marked unfinished is not success; ok, noop and retry are explicit. Floor refusal precedes the attempt and emits no squash event. These events do not alter durability or prove that a sync request incurred physical I/O. PATINADB_LOG=json makes these events JSON lines. The target is disabled at ordinary info level; it uses the existing logging filter and does not change query budgets, replication, durability or retry policy.

Progress scopes have a process-local progress_id, optional first/last Raft indexes, elapsed microseconds, a phase and an outcome. Raft events on Linux include the native thread ID (native_tid); other platforms report zero. Core apply events include rust_thread; match it to the enclosing synchronous Raft write event to associate the native thread. Core and Raft scope IDs use separate counters; keep the event target as well as process generation in the correlation key. A dispatch scope can move from an async worker into a blocking worker, so correlate by scope ID and index as well as thread. IDs restart with the process; keep node and process generation when correlating logs across a restart. Nested graph-barrier and metadata events run on the apply worker. Durations overlap and include waits; do not add nested durations.

The events distinguish receiving an AppendEntries range and advertised commit index, entering the state-machine apply, acquiring its cut lock, queueing and starting blocking work, individual entry handling, graph buffer flush/sync, background-snapshot admission and state-machine metadata persistence. A causal wait records its required index and whether it completed, timed out or shut down. Conditional writes also distinguish replay detection, constraint fingerprint, preimage checks and the main write. Core apply phases separate prevalidation, graph/index updates, fulltext, vectors, history, deferred scope exit, the terminal sync request and snapshot admission. Phase events precede the named operation; the next event bounds the interval. An outer group can defer the core sync request: only its later group barrier provides the durable batch boundary. Receiving or acknowledging an append does not prove graph application. A scope ending with unfinished means the caller did not reach its explicit completion marker; it can be an error, cancellation or unwinding, not a diagnosis of the underlying failure. An entry’s processing result is separate from the final durable batch completion and OpenRaft’s published applied position.

These events record protocol positions and fixed phase names, not query text, properties, credentials or error bodies. They are ordinary local log events, not additional OTLP spans. Log volume and synchronous log writes can affect the incident being measured; enable this target for a bounded capture and retain the matching source/binary identity. No performance or latency guarantee follows from enabling diagnostics. The configured three-hour memory-pressure soak passed; broader latency and capacity bounds remain workload-dependent (#1409). That run qualifies the stabilization fixes, not the RFC0036 admission governor. The memory budget defaults are active on the server; their performance acceptance remains open.

Slow maintenance warnings

Four WARN events are logged at the default level when maintenance work is slow. They name a database and fixed phase names, never query text or values.

Phase durations are in milliseconds and are sequential within one event. A tag job and a squash preparation of a large database share one artifact lane per node, so a long tag build explains a long squash lane_wait_ms on the same node. Squashes of databases with at most 4096 live vertices and edges use the separate small lane. These events identify where time went, not why; they do not change any timeout, admission or retry policy.

Telemetry-degrade observability and break-glass

A community-mode node that can’t reach the telemetry endpoint for the whole grace window (default 72h) degrades: client writes are refused until a heartbeat succeeds again (see Licensing & Telemetry). Two things make that freeze operable instead of a surprise:

• Alert before it happens. /metrics exposes patinadb_telemetry_degraded (0/1, always present) and patinadb_telemetry_seconds_until_degrade (present only while community mode is armed and not yet degraded) — wire a Prometheus alert on the latter dropping below, say, one hour so you learn about a telemetry-endpoint outage well before writes actually stop.

• A time-boxed break-glass override, for the rare case where you need writes to keep flowing through a telemetry outage you can’t fix immediately:

  patinadb-raft --id 1 --addr 0.0.0.0:21001 --db ./data --bootstrap \
    --auth-password "$PW" \
    --telemetry-degrade-override-until 72h
  

  Flag --telemetry-degrade-override-until <value> / env PATINADB_TELEMETRY_DEGRADE_OVERRIDE. The value is always resolved to an absolute deadline, so the override must expire — accepted forms:

  Form	Example	Resolves to
  Duration from startup	72h, 3d, 30m, 90s	now + duration
  RFC3339 UTC timestamp	2026-07-12T09:00:00Z	that instant
  Bare unix-second integer	1799999999	that instant

  While the override is active, /metrics also exposes patinadb_telemetry_override_until_seconds (seconds remaining before the override itself expires) so you don’t lose track of it. The override is logged loudly at startup — it is a deliberate, visible escape hatch, not a silent bypass. Once it expires, the normal grace-window degrade behavior resumes exactly as if it had never been set.

Tenant isolation and read-proc guards

Two related operational knobs, covered in full elsewhere but worth knowing about when running a shared/multi-tenant cluster:

• --rbac-closed (database-level deny) stops a non-admin’s global role from reaching every database by default — see Authentication & TLS: database-level deny.
• Expensive read procedures are bounded, not unlimited. CALL patinadb.algo.betweenness/closeness (O(V·E), Reader-callable) are guarded by a cooperative deadline under --query-timeout-secs (the algorithm checks in periodically and bails cleanly instead of running past the timeout to completion) plus a static work budget, PATINADB_MAX_ALGO_WORK (env, default ~1e9 — see Configuration Reference), so a Reader can’t pin a blocking thread indefinitely even with no --query-timeout-secs set. Not yet implemented: a bounded blocking-thread pool and a per-user concurrent-read-procedure cap — today’s guards stop a single expensive call from running forever, but a burst of many concurrent expensive calls from different users is not yet rate-limited.

Capacity assumption: bounded label / property-name cardinality

patinaDB interns every label, relationship type, and property name into a process-global arena (Label = internment::Intern<String>, patinadb::models::Label) that never frees for the lifetime of the process. This is a deliberate design choice: identifiers are expected to be bounded-cardinality (a schema’s worth of distinct names), so a small, ever-growing table is fine.

The important consequence: distinct names come from query text, not just from DDL. Every parsed query interns the identifiers it mentions, so a workload that generates fresh property names or labels per request (e.g. SET n.{timestamp} = … with a varying name, or generated Cypher with embedded ids in identifier position) grows the arena — and with it resident memory — for as long as the process runs. A restart resets it.

Operating guidance:

• Treat identifier cardinality as a schema property. If your application never mints new labels/property names at runtime, no action is needed — this is the configuration the design assumes.
• Observe it. label_intern_count() (patinadb::models) reports the number of distinct strings interned so far in the process. It is currently a library-level observable (used by the test-suite), not yet exported on /metrics or a /mgmt endpoint — wiring it into metrics and alerting on sustained growth (a monotonic climb after schema-stable warm-up means somebody is minting identifiers at runtime) is the recommended follow-up (unverified until wired — today, watch process RSS as the proxy).
• Multi-tenant caveat. Do not expose raw query text to untrusted tenants: a tenant who varies identifier strings per request can grow the arena without bound (a slow memory-exhaustion vector). Map/parameter keys and string values are safe — they are EcoString, not interned — it is only identifier positions (labels, relationship types, property names, variables) that intern.

Crash-loop quarantine (deterministic fail-stop faults)

patinaDB’s write path fail-stops (aborts the process) when a partially landed write cannot be rolled back in-process — by design: a loud crash with clean recovery beats silently fossilizing torn state. When the underlying fault is transient (a real I/O hiccup), the restart simply works. When it is deterministic — durable on-disk corruption that the Raft replay re-reads on every boot — the node would crash-loop forever.

The server detects that loop empirically (never by guessing from the error’s type — an error type cannot prove determinism): each fail-stop abort appends one line to <data-dir>/torn_aborts.tsv, and at boot the node checks it. Three aborts with the identical signature within 15 minutes put the node into quarantine instead of booting:

• the process stays up and stops looping; Raft and Bolt are not started, the database directory is not opened;
• every HTTP route answers 503 with {"quarantined": true, "reason": …} (plaintext, health-probe friendly);
• the metric patinadb_quarantined is 1, and the boot log states the reason and this runbook;
• nothing is deleted and nothing is auto-repaired.

That last point is deliberate. The cluster usually holds the state redundantly, so the obvious “fix” — wipe this node and re-fetch a snapshot — is usually right. But if the corruption travelled through the replicated log itself, every replica has it, and auto-wiping would destroy the last locally inspectable artifact while re-importing the damage. A human decides:

1. Inspect: keep a copy of the data directory (and torn_aborts.tsv — it names the failing operation and error) if you want a post-mortem.
2. Check the peers: if the other members are healthy and serving, this is single-node damage. Remove the node’s data directory, delete torn_aborts.tsv, and start it again — it re-joins and receives a fresh snapshot from the leader.
3. If peers show the same aborts (replicated damage): restore the cluster from a backup (restore), accepting the documented history consequences; engram-history loss vs. full rebuild is an explicit operator trade-off, not something the node may decide alone.
4. Supervised override: PATINADB_QUARANTINE_OVERRIDE=1 boots normally despite the verdict (e.g. after fixing the disk underneath) — the override is logged loudly.

Stale markers are harmless: only aborts younger than 15 minutes count, so a node that aborted long ago (and has run fine since) boots normally even if the file was never cleaned up.

Removing a dead voter

A permanently-unreachable voter stays in the quorum set until you remove it, and it can block further membership changes. In a 3-voter cluster, one dead voter plus one more failure is a quorum loss — so evict a node you don’t intend to bring back.

curl -u admin:… -X POST http://<leader>/mgmt/evict-voter \
     -H 'content-type: application/json' -d '{"id": 3}'
# → 200 {"ok": true, "evicted": 3, "voters": [1, 2]}

• Admin-only (under /mgmt/), and must be issued on the leader — a follower returns a 503 leader hint (like a misrouted write).
• It performs a demote-then-remove in one joint-consensus membership change.
• Quorum guard: the request is refused (409) if the resulting voter set’s live members could no longer form a majority. “Live” requires both reachability over GET /health and genuine Raft replication progress (a voter that answers /health but has fallen behind — or dropped out of — the leader’s replication view is not counted live), so the endpoint won’t hand you a cluster that can’t actually commit. Evicting an unknown id, or the last remaining voter, is a 400.

Automatic dead-voter eviction

A leader-only background failure detector can auto-evict a voter that stays dead past a configurable window, reusing the exact same quorum guard as the manual endpoint above — it can never strand the cluster.

patinadb-raft --id 1 --addr 0.0.0.0:21001 --db ./data --bootstrap \
  --auth-password "$PW" \
  --auto-evict-after-secs 300

• Off by default (unset or 0) — no background task runs, behavior is byte-for-byte the manual-only path above.
• A voter must be continuously dead for the whole window before it’s evicted (a single bad probe never triggers it — the timer resets the instant the voter is seen alive again).
• If eviction would break quorum, the detector refuses and backs off — it never forces the removal. Every attempt (allowed or refused) is recorded in the audit log.

Learner→voter auto-promotion is a follow-on. Use GET /mgmt/cluster to see the live voter/learner topology before and after any eviction.

Reclaiming disk from a bloated .redb file

What each file and table holds is described in How Storage Works.

patinaDB’s B-tree storage engine reuses freed pages internally. Native commits or close may shrink unused tail space, but the file need not return to the size of its live graph. After bulk loading, deletes, history maintenance or a period of long-lived readers, the per-db .redb file can retain a large high-water allocation.

• Snapshot-count retention manages separate _engram_snapshots/*.snap files; it is not compaction of the main graph file.
• Deleting graph/history records can free internal pages for later writes without returning all of that space to the filesystem.
• A pinned read version preserves older pages while writes continue. Slower paced snapshots can extend this retention window. Track patinadb_oldest_pinned_read_seconds and patinadb_pinned_reads alongside allocated file bytes and free disk; those gauges cover owned read-snapshot handles, not every internal redb reader.

The #1409 owned-copy checks demonstrated native reuse and partial offline reclamation with unchanged logical content. They did not establish bounded long-term disk growth. Measure allocated blocks, not just apparent file length, and distinguish main files from snapshot sidecars and temporary build files.

If GET /mgmt/dbsizes shows a database’s .redb file itself is the bloat (not its snapshot sidecars), reclaim it with one of the two paths below.

Offline: patinadb <db> compact

The patinadb maintenance tool (shipped alongside the server) provides a subcommand that runs the storage engine’s built-in compaction directly on a node’s .redb file:

patinadb ./data/mydb compact
# compacted ./data/mydb: 46141440 -> 6295552 allocated bytes (39845888 freed, 86.4%)

This is OFFLINE ONLY — the target database must not be open anywhere else (a running patinadb-raft node, or another patinadb process holding the same file open). Compaction needs exclusive access — no concurrent readers or writers at all (an inherent property of the single-file B-tree) — so it cannot safely run against a live server. There is deliberately no /mgmt/compact REST endpoint or live/online compaction path — attempting to open a database that’s still in use fails loud with a clear “database already open” error and touches nothing (never corruption, never a partial compaction). To use it against a clustered node: stop that one node, run compact, then start it back up (a follower rejoins and catches up normally; see the rolling rebuild below for a way to do this with zero downtime across the whole cluster).

Compaction relocates and frees pages, then shrinks the file — it changes only the physical file, never any logical content, so every graph, engram-history record, and index reads back byte-identically afterward; AS OF time-travel, constraints, and compound/fulltext/vector indexes are all unaffected.

Measured (this box; a 50,000-node graph, each node with 3 string properties, then 49,500 of them DETACH DELETEd — the allocated-block figures below are blocks() * 512, never metadata().len(), per this repo’s measurement discipline, since the storage engine’s sparse pre-allocated tail makes len() over-report):

A 7.3× reduction (86.4% of the allocated bytes reclaimed) for this shape; the exact ratio depends on how much of the file’s high-water mark is genuinely dead versus reused-in-place, so re-measure on your own data rather than assuming this number.

Zero-downtime: rolling rebuild onto a fresh directory

For a clustered node you don’t want to take fully offline, the same effect is achieved by rebuilding a follower from scratch via the existing streamed- snapshot bootstrap (Disaster Recovery covers the mechanics for a lost node; this is the identical procedure used deliberately, as a space-reclamation step, on a node that isn’t lost):

1. Stop the bloated follower and remove (or move aside) its --db data directory — a fresh, empty directory in its place.
2. Start it back up with the same --id/--addr; it rejoins as before and add-learner/normal catch-up streams a snapshot into it, which writes a compact file (no bloat — it’s built fresh from the current graph, not accumulated by however many past deletes/updates the old file absorbed).
3. Once it’s caught up, fail over onto it (or just leave it as a healthy follower) and repeat for the next node — one at a time, so the cluster never loses quorum.
4. Finally rebuild the former leader the same way once it’s no longer serving writes for that database.

This needs no new tooling — it’s the same rejoin path a genuinely lost node uses — and, unlike the offline compact command, it never takes the cluster down (only the one node being rebuilt, which the remaining voters cover for).

An allocated-vs-live-bytes gauge (so bloat is visible before it becomes a disk alert) is a documented follow-up, not implemented yet — for now, GET /mgmt/dbsizes (the on-disk size) alongside a rough live-graph estimate from MATCH (n) RETURN count(n) / GET /version’s usage block is the manual signal to watch for a growing gap between the two.

Development-only transaction memory refusal

Explicit admitted Bolt COMMIT also reserves its new CAS token containers, fresh value copies, label arrays and sorting scratch; local request/entry clones share those data owners. Refusal occurs before proposal and reports no effect. Constraint catalog source buffers, native pages and history decoding remain outside that accounting, as do fallback preimages and other write paths. A large old property can therefore make COMMIT refuse even when the small update itself fitted RUN. The structural OCC tables and retained constraint values now share the same allowance, including intervening Engrams checked since BEGIN. The existing accumulator keeps charged maps and additive UNIQUE tuples across history pages. Final constraint keys and their temporary construction storage also share this allowance. A small input can still need many numeric composite-key variants; refusal during that expansion occurs before proposal with no effect. Existing vertex/edge properties needed to complete a key now decode under the same allowance. A small update can therefore be refused while reading an untouched, large key column. The source value stays charged until the key construction using it ends; errors are not converted into an absent property. This also applies to intervening history checks and the pinned CAS view. Label source arrays and temporary registry IDs now reserve before allocation as well; a vertex with many secondary labels can therefore exhaust the allowance even when the changed property is small. Read/corruption errors cannot become an empty successful CAS label preimage. A small write may therefore refuse because its conflict check meets a large foreign commit or retains the IDs of many small intervening commits. The ID list now reserves capacity before growth; metadata strings are borrowed during the walk. This does not yet cover decoded Engram bodies. A new transaction after RESET starts from a fresh base; it does not replay an unknown write. The legacy wire schema and server defaults are unchanged; full qualification is still pending. A refused unproposed COMMIT leaves the graph unchanged and moves the Bolt connection into its ordinary failed state; RESET permits new work. Reducing only the new value’s size may not help if the captured old value is what exceeds the allowance. The proposed RFC 0036 ENV settings remain unavailable in the server; the current coverage is exercised through the explicit Rust governor API.

In that development profile, projection DISTINCT accounts its retained key set. Explicit-column DISTINCT also accounts buffered output slots, consumes all input before output and preserves the existing cap on total input rows, including duplicates. A covered allocation refusal during private resolve returns a typed resource error without publishing a partial transaction or Engram. Original row payloads and other operator allocations still need coverage; see the current limitations.

When an embedded host supplies the development governor, outbound AppendEntries and Vote bodies also use its protected Apply allowance. A body reservation failure is a peer transport failure and uses ordinary Raft recovery/retry; it must not be interpreted as patinadb_effect=none for a proposed write. Existing RPC deadlines and receiver limits remain unchanged. The default CLI still leaves the development governor disabled; this adapter alone does not qualify automatic sizing, complete apply capacity or a node-wide memory bound.

Parallel query tuning

PATINADB_QUERY_THREADS=1 remains the default. For large eligible read queries, raise it gradually and measure latency with identical query text, data, cache settings and result checks. It caps the caller plus helpers for each query. PATINADB_QUERY_POOL_THREADS caps persistent background query workers across concurrent queries. Callers always work on their own query, even when the pool is occupied. Eligible dispatches now decline to serial before parallel preparation when all background workers are busy. With partial availability, the cap becomes min(query cap, idle workers + 1). This sampled capacity is not reserved; later queries can compete for it. Raising the cap is reasonable to evaluate for large eligible reads when the host has spare CPU capacity and measured latency improves. The load check does not establish saturated-workload throughput or protect Raft latency by itself. Leave CPU capacity for Raft and I/O; the pool does not reserve it automatically. These settings do not parallelize every query shape. Raising the cap keeps factorized path counts, grouped path counts, covering aggregation and deferred-prefix filters available. An applicable serial specialization takes priority over parallel work. EXPLAIN shows aggregate access choices and deferred-prefix candidates; PROFILE records the specialization actually used (and the runtime prefix cut). This priority rule does not guarantee that every other admitted parallel query is faster: compare latency and throughput under your workload before raising the cap.

PATINADB_QUERY_MORSELS_PER_WORKER controls the target scan partition count (default 1 times the per-query worker cap, clamped to 1–64). Plain scans split the UUID key space without a preliminary label scan; traversal drivers retain key-count boundaries. More partitions may improve work distribution but also increase partial-result and merge overhead. Measure both single-query latency and concurrent throughput before changing it. At cap 4, up to three helpers and the caller execute work: the same four executing threads as the former four scoped workers with a waiting caller. Cap 1 uses the serial executor without constructing a pool job. Use --check-config to validate the integer settings before restarting. Restart the process when changing pool size or the per-query cap. Worker memory is charged to the same query budget, and memory limits remain in force.

PROFILE reports parallel.load with idle capacity and the admitted cap or decline. EXPLAIN does not sample runtime load. The existing /metrics endpoint exposes patinadb_query_pool_busy_workers and patinadb_query_pool_size (background workers only), plus patinadb_query_pool_jobs_total, patinadb_query_pool_morsels_total, and patinadb_query_pool_load_declined_total. Jobs and morsels include merge work and errors; load declines count operator attempts rather than distinct queries. An unused pool reports size zero.

Parallel ordered pagination

PATINADB_QUERY_THREADS remains 1 by default. With a larger cap, eligible read-only fixed-length traversals can use parallel top-K for ORDER BY with LIMIT, including SKIP. Each partial retains up to SKIP + LIMIT rows, so deep pagination increases the working set. Seeded MATCH pipelines can partition their materialized outer rows. Results retain serial tie order; projections and workers read one pinned snapshot. Outer preparation still runs serially. A single selected start can instead expand a fixed-length prefix serially and partition its frontier when sampled work and actual frontier size justify workers. This currently covers one-direction paths from the first endpoint, with distinct binding names and one label per node. Ordered drivers and unsupported selected paths retain serial execution, as do Apply outers with unestimated cardinality (including DISTINCT, aggregation and UNWIND). Admission uses estimated outer rows and bounded fan-out samples before materializing the outer; unavailable estimates retain serial execution. Frontier and per-morsel continuation buffers also consume query memory and may cause serial fallback.

Grouped consumers may run serially over ordered parallel Apply output, retaining serial aggregation and collection order.

Use EXPLAIN’s parallel.topk / parallel.apply lines to inspect structural eligibility and cost declines, and PROFILE to check actual engagement or fallback. A candidate may remain serial because of the row threshold, unsupported inner shape, memory admission or snapshot preparation. Existing specialized index paths can take precedence. Keys outside the proven scalar ordering domains (including large integers and temporal/compound keys) also use serial replay. Worker failure or refusal replays through the serial executor before publishing any rows. Variable-length inner traversals, CALL-import Apply and parallel unbounded sorting are outside this extension. More threads are not a demonstrated throughput improvement; test representative queries and memory budgets before changing the cap. See RFC 0041 for the generated-graph measurement limits.

Inline secondary labels

Physical snapshots carrying schema-8 inline labels require protocol 22 (snapshot format 8). During rolling upgrades, protocol-21 clusters use logical format 7, including history-restore state and causal identity. Older domainless clusters use their existing logical formats; scoped snapshots require at least protocol 21. History-restore stages or receipts also require protocol 21 for transfer.

RFC 0040 inline secondary-label readiness runs by default at database open, including fresh empty stores. PATINADB_INLINE_LABELS defaults on and is read once per process; restart to change it. Boolean values follow the normal runtime-knob grammar. Opening a store without READY rebuilds its derived adjacency values before publishing READY. Interrupted rebuilds leave READY absent and can be retried. Rebuild stages each adjacency tree separately and atomically swaps it into place; the original edge keys survive an interrupted staging pass. Temporary disk space is required until the old pages can be reclaimed. Offline compact can reclaim those pages after the database closes. One private SF1-copy measurement is recorded in the RFC 0040 storage evidence (#1442): first open plus packed rebuild took about 95 s; allocated space after offline compaction grew by 0.68%. These SF1 figures are not a general capacity or migration-time guarantee.

This switch controls automatic open-time ensure, not live writer maintenance or whether readers honor an already-present READY marker. Setting it back to 0 therefore does not undo a rebuild or disable trusted reads in an existing READY store. Use separate original store copies for before/after experiments. Bulk imports and snapshot installs write final inline values. Successful bulk updates preserve readiness with a fresh cache token when the prior graph was ready or empty. Updating an already stale nonempty graph does not certify its untouched entries; reads continue to use secondary metadata until rebuild. Live label changes maintain inline values and revoke older cache trust. The deferred insert and bounded label-change costs are accepted in ADR 0018; see RFC 0040 “Increment 1 measurements” (#1442).

Independent backend: transaction admission

For the independent Rust search backend, SEARCH_DB_TIMEOUT_SECONDS retains its 8-second default and bounds transaction admission as well as the existing response wait. A server Neo.TransientError.General.DatabaseUnavailable refusal of explicit BEGIN is retried after a 25 ms policy delay within that original budget. Every BEGIN retains the configured database and causal bookmarks. There is no new environment toggle. This handling never retries RUN or COMMIT, never retries a transport failure, and never starts a write from a BEGIN response that arrives after the admission deadline; it explicitly rolls that late transaction back. Subsequent HTTP503/timeout write outcomes remain ambiguous and require exact readback, not blind replay (#1483).

Diagnosing a write refused before proposal

For non-chunked HTTP upserts and REST autocommit data writes, READ_PUBLICATION_IN_PROGRESS together with write_outcome: "not_submitted" certifies refusal before this request entered Raft. Count it as a rejected operation, not a committed write. Other 503 responses, transport failures and timeouts remain governed by the existing ambiguous-write policy. See the client contract. The receipt adds no automatic retry or timeout change (#1483).