High Availability

A single --bootstrap node is already a working server, but a quorum of one has no redundancy. Add nodes to get automatic failover: if the leader dies, the survivors elect a new one and writes continue.

Cluster sizing

Raft tolerates failures up to a quorum. Use an odd number of voters:

Growing a cluster

Start one node with --bootstrap, then add peers as learners (they catch up without voting) and promote them to voters:

# Node 1 (bootstrap leader)
patinadb-raft --id 1 --addr 127.0.0.1:21001 --db ./n1 --bootstrap \
  --bolt-addr 127.0.0.1:7687 --auth-password secret

# Nodes 2 and 3 (no bootstrap — they join)
patinadb-raft --id 2 --addr 127.0.0.1:21002 --db ./n2 \
  --bolt-addr 127.0.0.1:7688 --auth-password secret
patinadb-raft --id 3 --addr 127.0.0.1:21003 --db ./n3 \
  --bolt-addr 127.0.0.1:7689 --auth-password secret

Then, against the leader’s --addr:

# Register the new nodes as learners (id → its HTTP addr)
curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/add-learner \
  -H 'content-type: application/json' -d '{"id":2,"addr":"127.0.0.1:21002"}'
curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/add-learner \
  -H 'content-type: application/json' -d '{"id":3,"addr":"127.0.0.1:21003"}'

# Promote to a 3-voter membership
curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/change-membership \
  -H 'content-type: application/json' -d '[1,2,3]'

All nodes must share the same --auth-password — peer RPCs carry the same Basic credentials.

Auto-join with --join

Instead of running /mgmt/add-learner by hand, start a fresh node with --join <member> (the HTTP address of any current member). It registers itself as a learner (read replica) on startup, following a leader hint if --join points at a follower:

# Node 1 leads; nodes 2 and 3 auto-join as learners.
patinadb-raft --id 1 --addr 127.0.0.1:21001 --db ./n1 --bootstrap ...
patinadb-raft --id 2 --addr 127.0.0.1:21002 --db ./n2 --join 127.0.0.1:21001 ...
patinadb-raft --id 3 --addr 127.0.0.1:21003 --db ./n3 --join 127.0.0.1:21001 ...

--join is mutually exclusive with --bootstrap. The nodes join as learners; promote them to voters when you want them to count toward quorum:

curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/change-membership \
  -H 'content-type: application/json' -d '[1,2,3]'

Failover behaviour

• Election timeout is 750–1500 ms with a 250 ms heartbeat. When the leader stops heartbeating, a survivor wins an election and takes over.
• Clients connected to the dead node reconnect to a survivor (neo4j:// routing drivers do this automatically — see Bolt on --advertised-addr).
• Committed writes are durable on a quorum and survive the failover; in-flight uncommitted writes to the dead leader may need to be retried.

The failover test suite exercises exactly this: a 3-node cluster, a write through the leader, kill the leader, and assert the survivors elect a new leader, writes continue, and the cluster retains all records — including the multi-database case. A companion test drives sequential writes plus linearizable reads across a single leader kill and checks that no acknowledged write is lost across the failover and that the linearizable-read count is monotonic (never decreases). This is a solid failover + acked-write-durability smoke test.

Scope of the durability test. The linearizable-read test issues writes sequentially, uses idempotent MERGE (which masks any accidental double-apply), and kills the leader once — there are no concurrent clients, no network partitions, and no operation reordering. It verifies that acked writes survive a single failover and that read counts stay monotonic; it is not a linearizability check in the Jepsen sense. True concurrent-history linearizability testing (Knossos/Elle-style history checking under concurrency, partitions, and reordering) is future work.

Removing a dead voter

A permanently-unreachable voter stays in the quorum set and blocks further membership changes. Evict it on the leader with a live-quorum guard:

curl -u neo4j:secret -X POST 127.0.0.1:21001/mgmt/evict-voter \
     -H 'content-type: application/json' -d '{"id": 3}'

The endpoint demotes-then-removes the voter in one membership change and refuses (409) if doing so would leave the surviving voters unable to form a quorum. See Day-2 Operations for the full contract, plus the /health vs /ready probes and request-id tracing.

Learners as read replicas

A learner replicates the log and applies it locally but does not vote. Since any node serves reads from its own applied state, learners act as asynchronous read replicas. The single-node, learner, and voter cases are all the same binary and the same apply path.

Cluster-aware Bolt routing (reads → followers, writes → leader)

A Neo4j routing driver (neo4j:// scheme) asks the cluster for a routing table and then load-balances: it sends writes to a WRITE server and spreads reads over the READ servers. patinaDB answers that request — over both the native Bolt ROUTE message and the dbms.routing.getRoutingTable procedure — with the real cluster topology:

The effect: neo4j:// clients automatically offload reads to the followers and send writes to the leader — read scaling with no application changes. On a single-node --bootstrap cluster the leader is the only member, so all three roles resolve to that one node (no change from a standalone server).

Each node learns its peers’ advertised Bolt addresses from a background poll of every peer’s GET /version (which now reports advertised_bolt_addr) — the same poller that tracks peer protocol versions. Because the addresses returned to drivers are the --advertised-addr values, set that to each node’s public host:port when running behind a TLS-terminating proxy, so routing stays reachable (see Bolt). If the leader is momentarily unknown (an election is in flight) WRITE falls back to the local node — the client then gets the leader-hint 503 on the misrouted write and retries.

A follower read is eventually consistent (it serves that node’s local applied state, which may lag the leader). That is exactly what you want for analytics and browsing. When you need a read that is both lag-immune and free to load-balance across followers, use the clean-tag pattern below.

Read consistency

By default a read serves the local applied state of whichever node you hit:

• Causally consistent within a database — the Raft log is a total order, so a single node never sees writes out of order.
• Eventually consistent across replicas — a follower or learner that lags the leader’s commit index may not yet reflect a write that has already committed elsewhere. Reads are fast (no cluster round-trip).

When you need a read to reflect every write committed before it began, opt in to a linearizable read on the REST /cypher endpoint:

curl -u neo4j:secret -X POST 127.0.0.1:21001/cypher \
  -H 'content-type: application/json' \
  -d '{"query":"MATCH (n:Person) RETURN n","consistency":"linearizable"}'

This routes through a leader read-index barrier: the leader confirms it is still leader (a heartbeat to a quorum), waits until it has applied the current commit index, then runs the read. Only the leader can serve a linearizable read — sending one to a follower returns a 503 whose body names the leader (leader_id / leader_addr), exactly like a misrouted write. It costs one intra-cluster round-trip; the default "local" read skips it.

consistency is a REST-only knob today. The Bolt path uses the default local read; driver-level causal consistency (bookmarks) is not yet implemented.

Lag-immune reads across followers: the clean-tag pattern

A linearizable read pins you to the leader, which defeats read scaling. When you want a consistent, lag-immune read that can still load-balance freely across followers, read as of a named tag instead:

USE mydb AS OF TAG 'nightly-2026-07-04'
MATCH (n:Person) RETURN n

An engram tag names a specific point in history. Tags are replicated, snapshotted, and deterministic, so reading AS OF TAG '<name>' returns bit-identical results on every node regardless of replication lag — a follower that is behind on new writes still reconstructs the tagged state exactly. This gives you a stable, reproducible read that any follower can serve:

• Point routing-driver reads at the followers (automatic — see above).
• Tag a known-good state (e.g. after a nightly load) and have reporting/analytics read AS OF TAG that tag.
• Every replica agrees on the answer, and no read has to touch the leader.

See Time Travel and Engrams for creating and managing tags. Use consistency: linearizable (above) only when you specifically need read-your-latest-write against the live HEAD.

Rolling upgrades

Nodes in a cluster exchange a versioned wire/disk protocol: the Raft log entry payload (AppRequest), its response, and the streamed-snapshot record format. That format is append-only and versioned, which is what makes a rolling upgrade (upgrade one node at a time, no full-cluster downtime) safe as long as you upgrade in the right order.

What the format guarantees

• Append-only log-entry variants. New kinds of replicated command are only ever appended to the AppRequest enum, never inserted or reordered. Log entries are stored positionally (bincode), so reordering would silently re-map already-persisted entries; a build-time test pins the exact order and count to prevent it. A newer node can therefore always decode an older node’s entries.
• Observable protocol version. Each node reports a protocol_version on its GET /version endpoint. It bumps whenever a new variant (or other wire/disk change) lands, so you can confirm what every node speaks before and during an upgrade.
• Automatic capability gate. The leader will not propose a command that some cluster member is too old to apply. Each node polls every peer’s GET /version in the background and tracks the cluster-wide minimum protocol version. Before a command is appended to the Raft log, the leader checks the version that command requires against that minimum: if any member is older — or its version hasn’t been confirmed yet (conservative: unknown is treated as too old) — the proposal is rejected without being written (409 on REST, a failure with the same message on Bolt) telling you to finish the upgrade first. Today every command is protocol v1, so the gate always passes; it is future-proofing that makes the next new command safe automatically, in any node-upgrade order.
• Loud rejection of an unknown variant. If an older node receives a peer RPC carrying a command it does not know how to decode (because a newer leader emitted it), it logs a clear error — “unknown AppRequest variant — this node is older than the leader; upgrade it” — and returns 422, instead of a silent or opaque failure. A mixed-version wedge is diagnosable from the logs.
• Snapshot format guard. The streamed snapshot carries a format_version for its record stream. On install, a node refuses an unknown (newer) version before clearing its graph, so a version-skewed snapshot can never tear a half-restored store — the node keeps its existing data and can retry once upgraded.

Upgrade order

Upgrade the nodes one at a time (each catches back up before you move on); the leader can be upgraded last or stepped down first. You no longer have to time it perfectly: the capability gate blocks any command a not-yet-upgraded member couldn’t apply, so exercising a new feature too early fails cleanly (409 / Bolt failure, “upgrade all nodes first”) instead of wedging a node. Once every node reports the new protocol_version on GET /version, the gate opens on its own and the new functionality just works.

Check GET /version on every node to see when they all agree on protocol_version.

Scope of the guarantees

The capability gate is the leader-side safeguard: it refuses to emit a command until it has confirmed every member can apply it (unconfirmed peers block conservatively). The append-only variant discipline and the loud 422 on an unknown variant remain the defence in depth if a command ever does reach an older node, and a version-skewed snapshot is still refused before it can clear a graph. Finer-grained, per-feature negotiation (beyond a single monotonic protocol version) is possible future work.