High Availability

A single --bootstrap node is already a working server, but a quorum of one has no redundancy. Add nodes to get automatic failover: if the leader dies, the survivors elect a new one and writes continue.

Cluster sizing

Raft tolerates failures up to a quorum. Use an odd number of voters:

An even voter set is strictly worse than the odd set one smaller: 4 voters tolerate 1 failure — the same as 3 — but every election and write needs a quorum of 3 of the 4 alive, so the extra voter buys nothing and only adds another way to lose quorum. The extreme case is 2 voters: like 1 voter it tolerates zero failures, yet the cluster dies when either node fails. POST /mgmt/init and POST /mgmt/change-membership therefore refuse an even voter set by default with a 409 explaining the math:

{
  "error": "refusing an even voter set of 2: …",
  "hint": "pass \"allow_even\": true to force"
}

If you genuinely want an even set (a throwaway test cluster, say), wrap the member list and opt in explicitly:

curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/change-membership \
  -H 'content-type: application/json' \
  -d '{"members": [1,2], "allow_even": true}'

(The same {"members": {…}, "allow_even": true} wrapper works for /mgmt/init, whose bare form is an {id: addr} map.)

Single-statement write-size limit

A replicated write crosses the peer wire inside one Raft log entry, and every follower’s /raft/append receiver bounds that body (PATINADB_RAFT_MAX_BODY_BYTES, default 64 MiB). An entry over the cap is not “slow” — it is not replicable at all. To keep that from ever reaching the wire, the leader measures every proposal against a single-entry limit (PATINADB_MAX_ENTRY_BYTES, default 48 MiB) and rejects an oversized statement up front with a non-retryable client error (Neo.ClientError.Statement.ArgumentError on Bolt, HTTP 400 on REST) that names the limit. Retrying the identical statement can never succeed — split the write into smaller batches instead:

• Cypher: CALL { … } IN TRANSACTIONS OF <n> ROWS,
• driver-side: smaller UNWIND batches per statement,
• bulk loading: the chunk_rows option of the bulk-upsert endpoints.

Both knobs are documented under Configuration. If you raise them for a bulk workload, set PATINADB_RAFT_MAX_BODY_BYTES identically on every node and keep PATINADB_MAX_ENTRY_BYTES comfortably below it.

Raise both, never only the entry cap. The two are an ordered pair: above the receiver’s body cap the proposer-side limit cannot protect anything — every entry in the gap is proposed and then rejected by every follower with 413, forever, which is the exact cluster-wide wedge this limit exists to prevent. patinaDB now enforces the ordering for you: --check-config fails on a misordered pair, and a node that boots with one logs a loud warning and clamps its effective entry cap down to the body cap (it does not refuse to start, so upgrading a cluster that already runs a misordered pair is never an outage). Nothing node-local can detect a body cap raised on some nodes only — that part is still on you.

Batched appends

That single-entry limit does not bound a batch. When a follower falls behind — it was restarted, it is a fresh learner catching up, or a write burst outran replication — Raft ships up to 300 pending entries in one append body, and a batch of individually-legal entries can still exceed the receiver cap. Nothing about that is your fault or your configuration’s, so patinaDB handles it for you: the sender byte-budgets every append body against PATINADB_RAFT_MAX_BODY_BYTES and splits an oversized batch into budget-sized bodies, at entry granularity, delivered in order. You do not need to configure anything, and there is no visible effect beyond the catch-up taking a few more round trips. patinadb_raft_append_chunked_batches_total counts how often the oversized-batch path was entered.

After leader changes, a follower can already hold entries whose matching position the new leader has not yet confirmed. Before sending an oversized body, the sender may try up to two ordinary, authenticated empty AppendEntries requests at exact LogIds from that batch. A successful Raft log-match response proves only that prefix; its payload need not be resent. A conflicting or unavailable probe falls back to ordinary replication, and a higher vote still fences the sender. The probes share the original RPC time budget, not an extra unbounded timeout. Small appends are unchanged. Counters patinadb_raft_append_prefix_probes_total{target} and patinadb_raft_append_prefix_proven_entries_total{target} distinguish attempted checks from confirmed entries. These counters are not throughput or readiness guarantees; continue monitoring applied positions and follower lag.

One caveat if your cluster runs on a slow interconnect: a chunk still has to cross the wire inside Raft’s 250 ms RPC deadline, and a 56 MiB chunk (the budget at the default 64 MiB cap) needs roughly 1.9 Gbps of sustained goodput to make that. On a 1 Gbps link, lower PATINADB_RAFT_MAX_BODY_BYTES — it sets the receiver cap and the sender’s chunk budget together — until a full chunk transfers comfortably within 250 ms; 16 MiB is a reasonable starting point for 1 Gbps. This only affects catch-up after real lag; steady-state replication sends far smaller batches and never chunks.

Growing a cluster

Start one node with --bootstrap, then add peers as learners (they catch up without voting) and promote them to voters:

# Node 1 (bootstrap leader)
patinadb-raft --id 1 --addr 127.0.0.1:21001 --db ./n1 --bootstrap \
  --bolt-addr 127.0.0.1:7687 --auth-password secret --cluster-secret cluster-s3cr3t

# Nodes 2 and 3 (no bootstrap — they join)
patinadb-raft --id 2 --addr 127.0.0.1:21002 --db ./n2 \
  --bolt-addr 127.0.0.1:7688 --auth-password secret --cluster-secret cluster-s3cr3t
patinadb-raft --id 3 --addr 127.0.0.1:21003 --db ./n3 \
  --bolt-addr 127.0.0.1:7689 --auth-password secret --cluster-secret cluster-s3cr3t

Then, against the leader’s --addr:

# Register the new nodes as learners (id → its HTTP addr)
curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/add-learner \
  -H 'content-type: application/json' -d '{"id":2,"addr":"127.0.0.1:21002"}'
curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/add-learner \
  -H 'content-type: application/json' -d '{"id":3,"addr":"127.0.0.1:21003"}'

# Promote to a 3-voter membership
curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/change-membership \
  -H 'content-type: application/json' -d '[1,2,3]'

Cluster secret is mandatory for joining nodes. Peer /raft/* RPCs authenticate with a dedicated --cluster-secret (PATINADB_CLUSTER_SECRET) carried in the X-Cluster-Secret header — not with the Basic credentials a client uses. All nodes must share the same value. If it is left unset it falls back to --auth-password, but a clustered (non-bootstrap) node with auth enabled refuses to start without a dedicated --cluster-secret — so nodes 2 and 3 above (and every --join node below) need it set before first launch. A single --bootstrap node may start without it (it has no peers) but logs a warning; set it before adding peers. You may set it to the same value as the admin password to keep the effective secret unchanged while decoupling the two, so they can rotate independently (see Production Deployment → Internal cluster traffic). The --auth-password still has to match across nodes for operator /mgmt/* calls like the ones below.

Auto-join with --join

Instead of running /mgmt/add-learner by hand, start a fresh node with --join <member> (the HTTP address of any current member). It registers itself as a learner (read replica) on startup, following a leader hint if --join points at a follower:

# Node 1 leads; nodes 2 and 3 auto-join as learners. All three need the same
# --cluster-secret (see the note above) — a joining node refuses to start without it.
patinadb-raft --id 1 --addr 127.0.0.1:21001 --db ./n1 --bootstrap --cluster-secret cluster-s3cr3t ...
patinadb-raft --id 2 --addr 127.0.0.1:21002 --db ./n2 --join 127.0.0.1:21001 --cluster-secret cluster-s3cr3t ...
patinadb-raft --id 3 --addr 127.0.0.1:21003 --db ./n3 --join 127.0.0.1:21001 --cluster-secret cluster-s3cr3t ...

--join is mutually exclusive with --bootstrap. The nodes join as learners; promote them to voters when you want them to count toward quorum:

curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/change-membership \
  -H 'content-type: application/json' -d '[1,2,3]'

Failover behaviour

• Election timeout is 750–1500 ms with a 250 ms heartbeat. When the leader stops heartbeating, a survivor wins an election and takes over.
• Clients connected to the dead node reconnect to a survivor (neo4j:// routing drivers do this automatically — see Bolt on --advertised-addr).
• Committed writes are durable on a quorum and survive the failover; in-flight uncommitted writes to the dead leader may need to be retried.

The failover test suite exercises exactly this: a 3-node cluster, a write through the leader, kill the leader, and assert the survivors elect a new leader, writes continue, and the cluster retains all records — including the multi-database case. A companion test drives sequential writes plus linearizable reads across a single leader kill and checks that no acknowledged write is lost across the failover and that the linearizable-read count is monotonic (never decreases). This is a solid failover + acked-write-durability smoke test.

Scope of the durability test. The linearizable-read test issues writes sequentially, uses idempotent MERGE (which masks any accidental double-apply), and kills the leader once — there are no concurrent clients, no network partitions, and no operation reordering. It verifies that acked writes survive a single failover and that read counts stay monotonic; it is not a linearizability check in the Jepsen sense. True concurrent-history linearizability testing (Knossos/Elle-style history checking under concurrency, partitions, and reordering) is future work.

A healing partition can briefly cost you the leader

Worth knowing before you see it in a graph and go looking for a fault that isn’t there.

patinaDB’s consensus layer does not implement pre-vote. The practical consequence: a voter that gets cut off from the cluster — a switch reload, a security-group change, a hypervisor pause — does not sit quietly. It notices the missing heartbeats, campaigns, and raises its term, even though it can never win while it is in the minority. When connectivity comes back, that higher term is what the rest of the cluster now sees, and the healthy, still-serving leader steps down and an election runs.

So the disruption arrives on heal, not on the partition — which is the counter-intuitive part. Nothing is lost: no committed write is affected, and the isolated node cannot have committed anything of its own. What you get is a brief write-unavailability blip, on the order of one election (~1–2 s), caused by a network recovering.

A leader lease (derived from the election timeout, roughly 2–3 s in the default configuration) suppresses this for short interruptions — a blip shorter than that window costs nothing. It delays the campaign; it does not prevent it. A partition that outlives the lease will produce the effect above.

What to do about it: treat it as normal. Your client should already be retrying writes — see the client retry contract — and a retry rides this out the same way it rides out an ordinary failover. It is not a reason to avoid healing a partition, and it is not a reason to page anyone. It is a reason not to interpret a leader change immediately after a network repair as a second fault: it is the first one finishing.

Removing a dead voter

A permanently-unreachable voter stays in the quorum set and blocks further membership changes. Evict it on the leader with a live-quorum guard:

curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/evict-voter \
     -H 'content-type: application/json' -d '{"id": 3}'

The endpoint demotes-then-removes the voter in one membership change and refuses (409) if doing so would leave the surviving voters unable to form a quorum. See Day-2 Operations for the full contract, plus the /health vs /ready probes and request-id tracing.

Learners as read replicas

A learner replicates the log and applies it locally but does not vote. Since any node serves reads from its own applied state, learners act as asynchronous read replicas. The single-node, learner, and voter cases are all the same binary and the same apply path.

Snapshot transfer: bounds and resume

When a follower or learner falls too far behind, the leader brings it up to date by streaming a whole snapshot to its POST /raft/snapshot receiver (an internal peer RPC, authenticated by the cluster secret — not a client endpoint). The transfer is file-backed and streamed in 64 KiB chunks on both sides (O(chunk) memory, no whole-graph blob), and hardened against the failure modes that matter at scale:

• Interrupted transfers resume mid-stream (issue #607). The receiver keeps the partial receive file; a retrying sender first probes (x-patinadb-snapshot-probe → {"received": N}) and then streams only the remaining bytes from offset N. A length mismatch answers 409 and the sender restarts from byte 0 (once). A pre-#607 peer never answers the probe, so a mixed-version cluster degrades to the old from-scratch behavior.
• Receive-side bounds (issue #605, per-attempt semantics since #1246). PATINADB_MAX_SNAPSHOT_BYTES (env-only, default 8 GiB, 0 = unlimited) caps a single upload attempt — over the cap the receiver aborts that attempt with 413 and KEEPS the partial file, so the probe/resume chain above continues from that offset and a snapshot larger than the cap assembles over several attempts. (Before #1246 the cap counted cumulatively and the abort deleted the partial: every retry restarted at byte 0 and died at the same offset — a snapshot past the cap could never install. Measured in the #1145 rejoin drill on a 71 GiB graph: 166 aborts, ~1.45 TB retransmitted, the follower never caught up.) A sender-declared total size that exceeds the destination filesystem’s free space is rejected with 507 before a byte is written, and the snapshot header prefix is capped at 4 MiB. The whole transfer is additionally bounded by install_snapshot_timeout (300 s, applied as a hard TTL, issue #592).

Tests: tests/snapshot_transfer_limits.rs (probe/resume, over-cap 413 with the partial kept, a larger-than-cap snapshot installing over the resume chain, insufficient-disk 507) and store.rs::decode_snapshot_header_rejects_prefix_above_the_hard_cap.

A slow (e.g. WAN-linked) learner can structurally never catch up (#1068)

The bullet list above resumes an INTERRUPTED transfer without losing progress — that is not the risk this section is about. The actual race is between two independently-sized things: how OFTEN the leader starts a new snapshot build, and how LONG one transfer takes to reach a given learner.

network.rs’s sender drops an in-flight transfer the moment a newer snapshot supersedes the one it is sending (the leader always serves its current snapshot, never a stale one). If the leader’s snapshot-build trigger fires again before a slow learner finishes downloading the current snapshot, that transfer is aborted out from under it and restarted against the newer id — the #607 resume above only helps a retry of the SAME snapshot id, not this case. If the transfer reliably takes longer than the trigger interval, this repeats forever: the learner is not simply slow, it can never finish, for as long as the write rate holds.

Measured starting point for this issue (10 sustained writes/s, the openraft default snapshot_logs_since_last of 5000): the leader triggers a new snapshot build every 5000 ÷ 10 = 500 s (~8.3 min). A snapshot transfer observed on a real WAN link for the deployment this issue was raised against took 13–65 min. 65 min (or even the low end, 13 min) is longer than the 8.3-minute trigger interval, so under those conditions a learner behind that link cannot complete a catch-up at all — not probabilistically, structurally. Change any one of those three numbers (a slower write rate, a wider trigger threshold, a faster link) and the same arithmetic can give the opposite answer; there is nothing universally broken about snapshot transfer, only this specific combination of rate/threshold/bandwidth.

What to do about it today (the structural fix — making the snapshot trigger aware of a slow transfer already in flight — is very likely an openraft-level change and is tracked separately, not shipped as part of #1068):

• Raise PATINADB_SNAPSHOT_LOGS_SINCE_LAST (issue #1099) so the trigger interval comfortably exceeds your worst observed transfer time. This trades a longer Raft log (slower catch-up start, more log disk) for a wider window a transfer can complete inside.
• Improve the link (shortens the transfer) or reduce the sustained write rate a learner must survive a full catch-up under.
• Check the arithmetic yourself before deploying across a WAN link, don’t guess: set both PATINADB_RAFT_EXPECTED_WRITE_RATE_PER_SEC (your sustained write rate, writes/s) and PATINADB_RAFT_EXPECTED_SNAPSHOT_TRANSFER_SECS (your own measured or estimated whole-transfer time on the slowest WAN link in the deployment — roughly snapshot size ÷ link bandwidth) and run --check-config. It reports the trigger interval implied by your current PATINADB_SNAPSHOT_LOGS_SINCE_LAST next to your declared transfer time and flags the combination as a problem when the transfer cannot structurally finish. Both env vars are opt-in declarations — patinaDB cannot measure either at boot — so leaving them unset changes nothing about a node’s behaviour or its --check-config output.

The per-attempt install_snapshot_timeout (300 s default, PATINADB_RAFT_INSTALL_SNAPSHOT_TIMEOUT_MS, issue #1066) is a different knob: since issue #1068 the streamed upload itself is bounded by PROGRESS (PATINADB_RAFT_SNAPSHOT_PROGRESS_TIMEOUT_MS, 30 s default — the transfer is aborted only after that long with zero bytes flowing, so its total duration is unbounded on any link that keeps delivering), and install_snapshot_timeout covers only the resume probe and the post-upload install/response wait. Neither knob helps or hurts this race — the failure mode here is a newer snapshot superseding the old one, not a request expiring.

Gate: app::wan_snapshot_budget::tests pins the calculation itself (not a specific number) — it reproduces this section’s own measured scenario (10 writes/s, threshold 5000, a 13–65 min transfer) and fails if the defaults ever drift into a regime where that scenario reads as feasible, plus a counter-proof that the same formula reads a slower workload or a raised threshold as feasible.

Cluster-aware Bolt routing (reads → followers, writes → leader)

A Neo4j routing driver (neo4j:// scheme) asks the cluster for a routing table and then load-balances: it sends writes to a WRITE server and spreads reads over the READ servers. patinaDB answers that request — over both the native Bolt ROUTE message and the dbms.routing.getRoutingTable procedure — with the real cluster topology:

The follower-side self-exclusion measures an apply backlog (last_log_index − last_applied on the node itself), not distance from the leader’s log. A bandwidth-limited WAN learner — log delivery slow, apply fast — may accumulate no local backlog. The optional --max-leader-contact-age-secs gate additionally requires recent leader contact and bounds advertised leader-commit versus local-applied lag. Configure it explicitly; it is disabled by default. Routing still does not replace causal bookmarks or a linearizable read mode. See Multi-Region Topology.

Running nodes across regions? See Multi-Region Topology for opt-in regional tiers and scoped bookmarks. Regional refresh is coordinated by the leader; the legacy table described here has no region preference.

The effect: neo4j:// clients automatically offload reads to the followers and send writes to the leader — read scaling with no application changes. On a single-node --bootstrap cluster the leader is the only member, so all three roles resolve to that one node (no change from a standalone server).

Each node learns its peers’ advertised Bolt addresses from a background poll of every peer’s GET /version (which now reports advertised_bolt_addr) — the same poller that tracks peer protocol versions. Because the addresses returned to drivers are the --advertised-addr values, set that to each node’s public host:port when running behind a TLS-terminating proxy, so routing stays reachable (see Bolt). A follower is never advertised as a writer merely because the leader’s Bolt address has not been discovered yet. A plain routing request can probe the known leader on demand, sharing one discovery operation per node and bounded by the existing two-second version-probe budget. If the leader or its address remains unavailable, WRITE is empty and that incomplete table has a one-second TTL rather than the normal 300 seconds. Eligible READ and ROUTE entries remain available. Drivers must refresh/retry when no writer is available; retain bookmarks and use the documented write-token contract to reconcile unknown outcomes, not blind mutation retries.

A follower read is eventually consistent (it serves that node’s local applied state, which may lag the leader). That is exactly what you want for analytics and browsing. When you need a read that is both lag-immune and free to load-balance across followers, use the clean-tag pattern below.

Read consistency

By default a read serves the local applied state of whichever node you hit:

• Causally consistent within a database — the Raft log is a total order, so a single node never sees writes out of order.
• Eventually consistent across replicas — a follower or learner that lags the leader’s commit index may not yet reflect a write that has already committed elsewhere. Reads are fast (no cluster round-trip).

When you need a read to reflect every write committed before it began, opt in to a linearizable read on the REST /cypher endpoint:

curl -u neo4j:secret -X POST 127.0.0.1:21001/cypher \
  -H 'content-type: application/json' \
  -d '{"query":"MATCH (n:Person) RETURN n","consistency":"linearizable"}'

This routes through a leader read-index barrier: the leader confirms it is still leader (a heartbeat to a quorum), waits until it has applied the current commit index, then runs the read. Only the leader can serve a linearizable read — sending one to a follower returns a 503 whose body names the leader (leader_id / leader_addr), exactly like a misrouted write. It costs one intra-cluster round-trip; the default "local" read skips it.

consistency is a REST-only knob. The Bolt path always serves the default local read, but a standard Neo4j driver session gets read-your-writes automatically there too, via causal bookmarks — no consistency field to set. See Bolt → Causal consistency for the mechanism.

Summary — which surface gives which guarantee:

Lag-immune reads across followers: the clean-tag pattern

A linearizable read pins you to the leader, which defeats read scaling. When you want a consistent, lag-immune read that can still load-balance freely across followers, read as of a named tag instead:

USE mydb AS OF TAG 'nightly-2026-07-04'
MATCH (n:Person) RETURN n

An engram tag names a specific point in history. Tags are replicated, snapshotted, and deterministic, so reading AS OF TAG '<name>' returns bit-identical results on every node regardless of replication lag — a follower that is behind on new writes still reconstructs the tagged state exactly. This gives you a stable, reproducible read that any follower can serve:

• Point routing-driver reads at the followers (automatic — see above).
• Tag a known-good state (e.g. after a nightly load) and have reporting/analytics read AS OF TAG that tag.
• Every replica agrees on the answer, and no read has to touch the leader.

See Time Travel and Engrams for creating and managing tags. Use consistency: linearizable (above) only when you specifically need read-your-latest-write against the live HEAD.

A node that is still backfilling its engram history

A node that joins or rejoins a cluster receives the current state first and its engram history — the timeline behind AS OF, AS OF TAG and diff — afterwards, in the background. In between it is a fully useful cluster member: it holds quorum, applies writes, and answers reads at the current state. What it cannot do is answer a time-travel read that reaches further back than it has got, and it will not pretend otherwise.

Voting is not election eligibility. Such a member may vote, but cannot become leader until every local database has complete history. Before relying on a replacement voter for failover, check /version reports history_incomplete: false; current-state row counts, membership and apply-lag checks are insufficient. Do not disable this safety rule to make a failover test pass.

How to see it. /ready reports the condition as its own field, and the node stays ready: true while it does:

{ "ready": true, "history": "history_backfilling", "current_leader": 2, "apply_lag": 0 }

There are three values, and the difference between the first two matters more than anything else in this section:

The middle two look similar and are opposites. A history_backfilling node is making progress; a history_backfill_fenced node is not and never will be, and an operator who reads it as “still catching up” waits forever.

The #1261 cluster experiment encounters this with an old bulk-benchmark fixture seeded below the normal Engram write path: both replacement voters have current data but fenced history. This is not a fixture of ordinary Raft write history. The experimental root-snapshot history carrier is being qualified separately; it must not be assumed enabled or qualified by a current-state bootstrap succeeding.

How long it takes, and how to slow it down. The backfill pulls the history back in segments, each its own short request whose result is saved before the next one starts. That means it can be throttled or paused at any point without losing progress and without any risk of it having to start over:

PATINADB_BACKFILL_MAX_BYTES_PER_SEC=1048576   # unset (default) = unthrottled

Set it if the node is behind a link you also need for other traffic.

Do you need to? On a LAN, measured: no. A backfill running unthrottled alongside a snapshot transfer changed that transfer’s wall time by 0.99x-1.02x over four runs — i.e. not at all — while 166-207 backfill segments were verifiably in flight during the measurement window (#1145, 2026-08-30; loopback, --release, 200 engrams of history, a shared 16-core box at loadavg 14-16; harness scripts/backfill-contention-rehearsal.sh).

That result does not carry to a WAN link, and we do not claim it does. Both legs ran over loopback, so what was measured is CPU, disk and lock contention between the two transfers — not shared bandwidth. If your backfilling node sits behind a rate-limited link that a snapshot transfer also needs, throttle it; that case is unmeasured.

Three things change while any member is in this state, all of them deliberate and all of them temporary:

1. That node will not become leader. Only a node with a complete history can be leader, because the leader is what resolves restore, fork and diff and then replicates the answer. The node still votes and still replicates — it simply does not stand for election. In the rare case where every remaining node is backfilling, the cluster deliberately holds no election rather than electing a node that would answer those commands from a history it does not have.
2. History retention is paused for that database. A squash rewrites history into a genesis whose body is the state at the cutoff, which a backfilling node cannot produce. The leader defers it and logs history retention: deferred — a cluster member is backfilling the engram history of THIS database. The same deferral applies to CREATE TAG … AS OF and FORK … AS OF pointing anywhere below HEAD; tagging and forking at HEAD are unaffected, and so are your other databases — the deferral is scoped to the database that is actually catching up. If a backfill never finishes, retention for that database never runs — so history grows for a reason you can see in the log, rather than silently.
3. GET /mgmt/snapshot?history=true is refused on such a node (409), because the archive it produced would be silently short. A plain HEAD-state backup (without ?history=true) is unaffected — it claims no timeline.

Telling “catching up” from “never will”. Two metrics separate them, and this is the distinction that used to be impossible to make from outside:

The last two are the ones worth an alert (the shipped alerts.yml covers them as PatinaDBSnapshotTransferDiscarded). Without them a livelocking rejoin and an ordinary slow one look identical, which is why that failure had to be found by arithmetic rather than by looking at a dashboard.

What your clients see is documented in Client retry contract — a re-route to the leader, which a contract-following client already handles, and which carries "reason": "history_below_local_floor" so you can tell it apart from an ordinary failover in a log.

Rolling upgrades

Nodes in a cluster exchange a versioned wire/disk protocol: the Raft log entry payload (AppRequest), its response, and the streamed-snapshot record format. That format is append-only and versioned, which is what makes a rolling upgrade (upgrade one node at a time, no full-cluster downtime) safe as long as you upgrade in the right order.

What the format guarantees

• Append-only log-entry variants. New kinds of replicated command are only ever appended to the AppRequest enum, never inserted or reordered. Log entries are stored positionally (postcard, a non-self-describing codec), so reordering would silently re-map already-persisted entries; a build-time test pins the exact order and count to prevent it. A newer node can therefore always decode an older node’s entries.
• Observable protocol version. Each node reports a protocol_version on its GET /version endpoint. It bumps whenever a new variant (or other wire/disk change) lands, so you can confirm what every node speaks before and during an upgrade.
• Automatic capability gate. The leader will not propose a command that some cluster member is too old to apply. Each node polls every peer’s GET /version in the background and tracks the cluster-wide minimum protocol version. Before a command is appended to the Raft log, the leader checks the version that command requires against that minimum: if any member is older — or its version hasn’t been confirmed yet (conservative: unknown is treated as too old) — the proposal is rejected without being written (409 on REST, a failure with the same message on Bolt) telling you to finish the upgrade first. This gate is live and load-bearing today, not just future-proofing — most commands (a plain graph write) only need the baseline version, but several newer features (per-label/relationship-type RBAC grants, CREATE FUNCTION … LANGUAGE wasm) are pinned to a higher minimum protocol version and are correctly blocked from proposing until every member has upgraded past it.
• Loud rejection of an unknown variant. If an older node receives a peer RPC carrying a command it does not know how to decode (because a newer leader emitted it), it logs a clear error — “unknown AppRequest variant — this node is older than the leader; upgrade it” — and returns 422, instead of a silent or opaque failure. A mixed-version wedge is diagnosable from the logs.
• Snapshot format guard. The streamed snapshot carries a format_version for its record stream. On install, a node refuses an unknown (newer) version before clearing its graph, so a version-skewed snapshot can never tear a half-restored store — the node keeps its existing data and can retry once upgraded.

Upgrade order

Upgrade the nodes one at a time (each catches back up before you move on); the leader can be upgraded last or stepped down first. You no longer have to time it perfectly: the capability gate blocks any command a not-yet-upgraded member couldn’t apply, so exercising a new feature too early fails cleanly (409 / Bolt failure, “upgrade all nodes first”) instead of wedging a node. Once every node reports the new protocol_version on GET /version, the gate opens on its own and the new functionality just works.

The mixed-version window pauses the newest safety checks — finish upgrades promptly

Ordinary writes never block during an upgrade — that is deliberate. To keep it that way, the proposer falls back to older wire shapes while any member is below the current protocol, and that fallback quietly pauses the newest apply-time safety checks for the duration:

• below v9: the label-membership conflict check (concurrent SET n:X / SET n:Y against the same base can silently lose one update again);
• below v10: the constraint-catalog check (a write racing a CREATE CONSTRAINT across a leader change can land unchecked again).

This is a bounded-time trade-off: acceptable for the minutes a normal rolling upgrade takes, not for a stalled one (one member never upgraded = those checks stay off forever). The window is therefore measured and loud:

• gauge patinadb_cluster_min_protocol_version — the confirmed cluster minimum (0 = some member not yet confirmed);
• gauge patinadb_mixed_version_window_seconds — how long the cluster has been below current (0 = fully current);
• a WARN in the log once the window exceeds 60 s, repeated every 5 minutes while it persists, and an INFO when it closes.

Alert on patinadb_mixed_version_window_seconds exceeding your expected upgrade duration. The node deliberately never escalates beyond observability (no write refusal on a long window) — whether a stalled upgrade should eventually gate anything is an operator policy decision, not one the node makes implicitly.

Check GET /version on every node to see when they all agree on protocol_version.

Expect a short write blip when the leader restarts

openraft 0.9 has no leadership transfer: a planned leader restart cannot hand off first, so the remaining voters only notice the missing heartbeats and elect a new leader after the configured election timeout — 750–1500 ms (heartbeat_interval 250 ms; app.rs election_bounds). SIGTERM is handled: the HTTP server drains in-flight requests and the Raft state flushes cleanly, so the blip is a write-availability gap, not data loss — writes that arrive during the gap fail fast as retryable, and a client following the retry contract (retry writes for at least ~10 s — measured: a full-cluster cold start refuses writes for ~3.5 s, see scripts/deployment-rehearsal.sh) rides through it without an error surfacing. Reads on the surviving followers keep working (subject to their freshness).

Recommended order for a 3-node cluster — followers first, leader last:

1. Pick a follower (GET /mgmt/cluster on any node shows members[].is_leader and current_leader; GET /health on the node itself also reports current_leader).
2. Restart it onto the new version. Wait until it is back and caught up before touching the next node:
   • GET /ready returns 200 (not 503 with reason: "no_leader" / "installing" / "lagging");
   • GET /mgmt/cluster shows it in members with a last_applied index that has converged with the leader’s;
   • GET /version shows the new protocol_version (the capability gate keeps commands the mixed cluster can’t yet apply from being proposed — mixed-version operation is safe, see above).
3. Repeat for the second follower.
4. Restart the leader last. Expect the 750–1500 ms re-election blip here, plus the new leader’s startup; verify the same way (/mgmt/cluster shows a new current_leader, /ready 200 on every node, protocol_version equal everywhere).

Because of the protocol-version capability gate, a mixed-version cluster in the middle of this procedure is safe: nothing is proposed that an un-upgraded member cannot apply, so there is no order dependency to get wrong — the leader-last ordering only minimises the number of re-election blips to one.

Scope of the guarantees

The capability gate is the leader-side safeguard: it refuses to emit a command until it has confirmed every member can apply it (unconfirmed peers block conservatively). The append-only variant discipline and the loud 422 on an unknown variant remain the defence in depth if a command ever does reach an older node, and a version-skewed snapshot is still refused before it can clear a graph. Finer-grained, per-feature negotiation (beyond a single monotonic protocol version) is possible future work.

Causal bookmarks during a mixed-version rollout. An old-version node ignores an inbound Bolt session bookmark and emits a meaningless stub instead of a real one. So while a rolling upgrade is in progress, a session that lands a write on an already-upgraded node and then a read on a not-yet-upgraded one can still observe a stale result — this is transient (it clears once that node is upgraded) and does not affect data safety, but it is worth knowing about if you rely on read-your-writes during the upgrade window itself.