Multi-Region Topology

patinaDB supports serving reads from a region close to your users while keeping writes in a single, authoritative region — one Raft cluster, every voter in the write region, every remote region running as learners. This chapter describes that topology (RFC-0023), what it guarantees, what it costs, and opt-in regional routing with scoped causal bookmarks (RFC-0027/0028).

This is a topology decision, not an operating procedure — see Production Deployment for the day-to-day runbook (rotation, backup, bind & expose) that applies regardless of how many regions you run.

The shape: one cluster, one write region, N read regions

   Write region (e.g. eu-west)              Read region (e.g. us-east)
  ┌─────────────────────────────┐          ┌──────────────────────────┐
  │  node 1 (voter)              │          │  node 4 (learner)        │
  │  node 2 (voter)   <───Raft───┼──────────┼─→ node 5 (learner)       │
  │  node 3 (voter)              │  WAN     │  node 6 (learner)        │
  └─────────────────────────────┘          └──────────────────────────┘
        ▲ writes                                  ▲ reads (local)
        │                                          │
   backend, EU traffic                     backend, US traffic

• Voters live only in the write region. They form the Raft quorum and elect the leader. The current leader coordinates writes; being a voter does not make every member an independent writer. See High Availability → Cluster sizing for how many you need there.
• Every remote region runs as learners (High Availability → Learners as read replicas): they replicate the log and apply it locally, exactly like an in-region learner, just over a longer link. Learners never vote and never become leader — growing or shrinking a read region never touches quorum size or failover behavior in the write region.
• This is one cluster, not N clusters. There is a single Raft log, a single monotonic index, a single set of causal bookmarks. A write in the EU region and a bookmarked read in the US region compose correctly today, because there is only one index space to compose against — see High Availability → Read consistency for the bookmark mechanism itself.

Node roles: membership is not a region label

All nodes run the same server binary. There is no separate reader executable or node_type: reader setting: Raft membership determines voting rights.

“Follower” is often used informally for any replica. For a remote reader that must not vote, configure a learner, not another voting follower. node_name is a display label and node_region selects routing locality; neither grants voting rights nor restricts which voter can win an election. There is no permanently assigned writer among the three voters.

Setup recipe: three EU voters and remote learners

Example inventory (names and region labels are operator choices):

Place the three voters in independent failure domains within the main region, not on one host. A three-voter majority is two: one voter may fail without losing write quorum. Adding remote learners does not change that arithmetic. One learner per read region is not local read HA; add another independent learner there if local-reader failure must be tolerated without remote fallback. Every replica holds the data, not just tickets associated with its region.

1. Configure authentication, private peer connectivity, verified HTTP/Bolt TLS, the shared cluster secret and suitable license entitlements on every node. Give each node a unique numeric ID, its own data directory, and a client-reachable advertised Bolt address. Use the complete production setup, not the fragment below alone.

2. Bootstrap only node 1 with --bootstrap. Start nodes 2 and 3 with --join <existing-member-http-address>; they initially join as learners.

3. Check catch-up/readiness and peer-version confirmation, then call POST /mgmt/change-membership on the leader with JSON [1,2,3]. This is the complete desired voter set, not an incremental list. The membership procedure shows the calls.

4. Start nodes 4 and 5 using --join, without --bootstrap. Leave them as learners: do not add their IDs to the voter set. Joining and catching up do not automatically promote them. Never bootstrap a separate cluster in a read region and expect it to join this replication stream.

5. Enable regional routing and scoped bookmarks on the upgraded members. This YAML fragment is for node 4; change name/region for each other node:

   node_name: us-reader-a
   node_region: us-east
   regional_routing_enabled: true
   causal_bookmark_mode: compatible
   bookmark_wait_timeout_secs: 10
   

   These are routing settings, not the membership declaration. The timeout is a starting policy, not a measured WAN latency promise. Configure the leader-contact freshness guard and routing coordination budgets for your actual links as described below. Validate each complete configuration with patinadb-raft --config /path/to/node.yaml --check-config before starting it.

6. Admit a learner to client traffic only when current-state readiness and a bookmarked read pass. History backfill can finish separately; current-state readiness is not proof that historical queries are ready. Verify that the voter membership remains exactly [1,2,3] after adding remote readers.

Use the regional driver policy below for each backend deployment. Losing the entire EU voting majority stops writes; remote learners are not an automatic disaster-recovery quorum, regardless of how many remain online.

Why one cluster instead of one cluster per region

The alternative — an independent cluster per region, kept in sync by some replication layer — was evaluated and rejected (RFC-0023 §3.3): patinaDB has no applier that can play a foreign engram stream into a second cluster today, and a design that replicates across cluster boundaries reopens exactly the read-your-writes hazard this topology is built to avoid (bookmarks and the CDC resume cursor are correct within one cluster’s index space; scoped causal bookmarks reject another domain, but do not replicate between clusters). A stretched-quorum design (voters split across regions) was also evaluated and rejected for now: it needs Raft timing headroom that must be measured and configured for the actual links, and a leader that fails over into the wrong region would pay a WAN round-trip on every write until it fails over again. Learners-in-remote-regions has neither problem, at the cost described below.

Operating it: a routing driver per regional policy

Enable regional_routing_enabled: true, causal_bookmark_mode: compatible (or strict after legacy clients have migrated), and a positive bookmark_wait_timeout_secs on the upgraded nodes. Set node_region and optionally node_name per node; numeric Raft IDs remain the membership identity.

A US backend can use one routing driver seeded at an upgraded local learner:

neo4j+s://us-reader.example:7687?region=us-east&policy=prefer-local

Use a driver resolver or appropriate seed DNS to reach another upgraded router when the initial seed is unavailable. A routing driver manages multiple underlying Bolt connections; an individual Bolt connection still terminates at one server. Keep region/policy constant per driver instance, not per transaction.

Mark reads as reads: use session.execute_read(...), or a session/transaction with READ_ACCESS. Ordinary session.run(...) defaults to write access in the Python driver and can therefore reach the leader even for a read-only Cypher statement. A regional URI does not override the driver’s chosen access mode.

For the official Python driver 6.3.0, explicitly select the database in each session, for example driver.session(database="default", bookmarks=bookmarks). This is the supported learned-router seed-failover path exercised by tests/bolt-driver/test_regional_routing.py: the same driver survives its original seed being killed, using discovered routers rather than a replacement seed supplied by the test.

An unspecified database has a different discovery requirement in that driver. With patinaDB’s server-side routing (SSR) capability disabled, each new unnamed session resolves its home database through the original URI seed. The routing table cached for the returned default database does not supply the routers for that unnamed lookup; increasing the table TTL does not fix this. If intentionally using unnamed sessions, supply a resolver or seed DNS that still yields a live router. The driver gate separately exercises failure with a dead sole seed and recovery through a resolver yielding another local learner. patinaDB does not advertise unimplemented SSR support to bypass this distinction. These are local driver-contract checks, not WAN qualification.

Use managed read transactions (session.execute_read(...)) when relying on the driver to retry a connection lost during failover. Raw autocommit session.run(...) and manually opened transactions do not automatically replay failed work. The driver gate uses managed reads for failure/recovery legs, preserving bookmarks and the same driver instance; its autocommit and explicit-BEGIN causal checks remain unmanaged so their timeout and invalid-bookmark errors are observable.

• WRITE targets the current leader in the write region.
• READ, prefer-local: the first nonempty eligible tier wins: local nonleaders, local leader, remote nonleaders, remote leader. Tiers are not mixed, because address ordering does not make a standard driver prefer the first region.
• READ, local-only: only the first two tiers are allowed. No eligible local reader means a retryable routing failure, not silent remote fallback.
• ROUTE lists capable current members. A learner forwards regional table computation to the known leader with bounded private coordination.
• No region context: legacy nonregional selection remains unchanged.
• Direct bolt+s://: still selects one endpoint, not automatic discovery.

Profile probes share a bounded coordination budget across concurrency-limited waves, so a stalled earlier member does not monopolize discovery of later readers. Overload or insufficient deadlines can still exclude healthy peers; tune the coordination deadline for the actual network, not loopback timings.

With local-only, losing every local reader can also prevent a standard driver from acquiring a write connection: the driver expects a usable routing table, including readers. Use a separate policy/driver for a write-availability escape hatch; local-only is not a promise to allow cross-region reads during failure.

For a backend that must retain writes during a regional reader outage, use a global neo4j+s:// driver for writes and a separate regional local-only driver for reads. Pass the opaque bookmarks between their sessions explicitly. Do not replay an unknown write through the other pool. The docs-only application suite exercises this separation in scripts/customer-blackbox (#1261).

Configure redundant bootstrap addresses too: learned routers can be discarded after failed discovery, leaving a driver dependent on its original seed again. If using a custom resolver with a symbolic seed alias, expand only that alias to trusted router endpoints. Pass discovered physical addresses through unchanged; rewriting a selected regional reader to arbitrary voters breaks the routing policy.

Carry the acknowledged write’s bookmarks into the next read. A selected reader waits for its applied prefix before opening the read snapshot, or returns a bounded retryable error. The driver is not promised to switch automatically to the writer. An explicit writer fallback starts a new transaction with the same bookmarks and remaining request deadline; it must respect the selected regional policy.

What IS guaranteed: read-your-writes, everywhere, via bookmarks

A single Raft index space means the causal-bookmark mechanism (High Availability → Read consistency) composes correctly across regions with no extra code: write in the EU region, carry the bookmark to the US region, read there — the local learner waits (bounded, by the honest cost of replication lag) until its own applied index reaches the bookmarked one, then serves the read. This is read-your-writes by construction, not a special multi-region feature — it is the same mechanism a single-region deployment already relies on, and it is the reason this topology was chosen over the alternatives (RFC-0023 §3.1): the single-backend client that reads back its own writes is the sharpest detector of a read-your-writes violation, and this topology passes it without new machinery.

Legacy freshness filtering and its limits

Regional routing obtains fresh authenticated profiles on the leader and filters applied progress relative to its own frontier. A changed membership invalidates the response. History backfill alone does not exclude a current-state-ready learner. Profiles are not quorum leases; queries still enforce their bookmarks. Regional refresh requires contact with the write-region leader. The following describes legacy routing without a region context.

The routing table’s READ list drops a lagging replica when it is computed on the leader — the leader has a full per-follower replication view, and excludes anyone lagging beyond --readiness-max-lag (see Configuration Reference). But when the table is computed on a follower or learner — which, in this topology, is every node a remote-region client is likely to ask — there is no such view to filter with: openraft only populates per-peer replication metrics on the leader. The follower therefore omits unknown peers, rather than advertising them as fresh. A learner cannot infer sibling lag from this local metrics view (verified directly against the vendored openraft 0.9.24 source: RaftMetrics:: replication is Some(..) only on the leader).

What issue #1070 does fix: a learner always knows its own apply lag — the same signal /ready and the evict-voter guard already compute, against the same --readiness-max-lag knob — and now excludes itself from its own READ recommendation when it knows it is stale, rather than recommending a read target it can see, from its own state, is behind.

Apply backlog alone does not detect slow log delivery. Configure --max-leader-contact-age-secs for the additional fail-closed local freshness gate: it checks recent leader contact and the advertised leader commit index against the local applied index. Without this optional setting, that additional gate is disabled. See Configuration Reference. Neither local freshness nor routing makes an arbitrary read linearizable. Use causal bookmarks for read-your-writes; request the documented linearizable read mode when that stronger contract is needed.

It still cannot vouch for any other member. If you need a hard bound on how stale a read target can be, bolt:// direct pinning (above) to a specific node you monitor independently is the honest answer — the routing table’s filtering is a best-effort improvement on top of that, not a substitute for it.

What is NOT built: automatic write failover across regions

If the write region is unreachable, writes stop — learners never campaign for leadership, so there is no automatic promotion of a remote-region learner into a voter. Ordinary membership changes still require the existing quorum; after losing it, simply promoting a learner cannot repair it. Recover enough original voters, or use a tested disaster-recovery procedure to restore into a fresh cluster — see Day-2 Operations. This is the honest price of the topology (RFC-0023 §3.1): it buys read locality and a correct RYW story with comparatively little new machinery, at the cost of write availability being tied to one region’s health, same as a single-region deployment always was.

Linearizable reads are leader-only and REST-only — by design, not a gap

consistency: "linearizable" (see High Availability → Read consistency) only ever runs on the leader, and only over REST — there is no Bolt path and no proxying to a remote leader. Do not read this as a missing feature to work around. It is the property that makes this topology viable at all: every write is linearizable in the write region (that is what “leader” means), and every read region gets a bounded-wait, bookmark-consistent read instead of paying a WAN round-trip per query. A client in a read region that occasionally needs a genuinely linearizable read pays for it explicitly — one WAN hop, on the rare call that asks for it — rather than every read silently paying that cost, or every read silently NOT getting that guarantee. Keeping linearizable reads leader-only and REST-only is a deliberate constraint of this topology, not scope this project intends to expand.

Summary

See RFC-0023 for the full decision record, including the alternatives considered and why they were set aside.

Leader-contact freshness

Set max_leader_contact_age_secs (or --max-leader-contact-age-secs) on read replicas to reject local Bolt and HTTP reads after that many seconds without an accepted leader heartbeat. The default is disabled for compatibility. A node that currently believes it is leader is exempt, so this follower/learner guard is not a quorum lease or a linearizability guarantee; use REST consistency=linearizable when that stronger leader-read guarantee is required. The guard is an admission check before query execution. Explicit Bolt read transactions are checked before their repeatable-read snapshot is pinned and again when a statement runs.