Client Retry Contract

patinaDB is a replicated database with optimistic concurrency. That means a correct client must retry certain errors — not because the database is unreliable, but because “retry me” is a normal, expected outcome of two writers touching the same data at the same time, or of a leader changing mid-request.

A backend that does not implement this contract will surface routine concurrency and failover events to its users as hard errors. This is the single most common integration mistake, so this chapter is deliberately concrete.

The rule

Every error response carries a machine-readable retryable field. Key on it.

{
  "error": "write rejected: base state changed",
  "neo4j_code": "Neo.TransientError.General.TransientError",
  "retryable": true
}

Do not key on the HTTP status code, and never on the message text. The message is for humans and may change. The status is genuinely ambiguous — see the 409 trap below.

Some responses carry additional fields — leader_id, leader_addr, and reason. The first two are contract: they tell a client where to retry. reason is not. It names why this node could not serve the request, for your logs and your operators; it is free to gain new values, and a client that branches on it will eventually branch wrong. Log it, alert on it, never if reason == ….

resp = session.post(url, json=payload)
if resp.ok:
    return resp.json()

body = resp.json()
if body.get("retryable"):
    # back off and retry — see "How to retry" below
    ...
else:
    raise ApplicationError(body["error"], code=body["neo4j_code"])

Explicit refusal before a write is submitted

A non-chunked /mgmt/upsert, /mgmt/upsert-edges or /mgmt/upsert-graph, or an autocommit data write through /cypher, may answer HTTP 503 with code: "READ_PUBLICATION_IN_PROGRESS" and write_outcome: "not_submitted". This combination means native snapshot admission failed during private resolve or preparation, before submitting this request to Raft. It made no graph write; a refused upsert also created no pre-load tag. A client may report the refusal, or submit a new attempt with backoff within its own deadline.

Non-chunked bulk endpoints and pipelined autocommit data writes through /cypher can also return code: "WRITE_ADMISSION_TIMEOUT", neo4j_code: "Neo.TransientError.General.DatabaseUnavailable", retryable: true and write_outcome: "not_submitted" when bulk initial leader/applied admission or pipelined ordering/read-index admission expires before any proposal. These waits use the smaller of the remaining query budget and bookmark_wait_timeout_secs (default 10 seconds); a disabled budget is omitted. The original absolute query deadline is retained across resolution and OCC attempts. This does not bound or certify a mutation after proposal.

Bolt pipelined autocommit writes have no such receipt, so the bookmark-wait budget does not apply to them. Their ordering/read-index wait is bounded only by the remaining query timeout. When none is configured they wait as long as needed, for example behind a slow apply. When the query timeout expires first, they fail with Neo.TransientError.General.DatabaseUnavailable / “write admission timed out before proposal”, without this HTTP non-submission receipt. Configure client timeouts to exceed admission waiting plus transport overhead: a client that disconnects earlier still has an unknown outcome. Chunked loads receive no non-submission receipt, even at this initial barrier. The receipt concerns graph submission, not absence of all server activity: governance permits are acquired before the barrier and released on refusal, and the denied attempt is audited. No load marker or graph mutation is created by this refusal.

The field describes this request’s outcome; retryable still describes whether a new attempt can help. Both pipelined and serialized modes emit the receipt only at that pre-proposal boundary. Reads, schema/management commands, chunked loads (including CALL { ... } IN TRANSACTIONS) and errors after proposal do not gain this receipt. A chunked load may have committed earlier chunks before an error. Endpoint or query text alone never authorizes a retry.

A missing receipt, an HTTP503 alone, a socket error or a timeout does not prove that a write was refused. Retain the ambiguous outcome and use the documented idempotency/readback policy. Do not infer rollback from an absent value or blindly replay a mutation. No client or server retry is added by this field (#1483).

The 409 trap

409 CONFLICT is returned for three different situations — two of them opposite:

A client that retries on a non-retryable 409 loops forever. A client that fails on a retryable 409 drops a perfectly recoverable write (or, for the “not yet confirmed” case, gives up on a race that was already over by the time it read the response). This is exactly why the retryable field exists — never infer it from the fact that the status is 409, and never from the message text, only two of these three cases share the SAME neo4j_code too (see the table below) — retryable is the only field that tells them apart in every case.

The two capability-gate rows apply identically to a control command, not just a Cypher write. CREATE DATABASE, CREATE USER, and every other administrative/DDL command go through the SAME capability gate as a graph write, and (as of issue #866) return the exact same 409 body shape and the exact same two neo4j_codes for the two situations above — a client does not need a separate code path for “did my write get gated” vs “did my CREATE DATABASE get gated”. Before #866 the control-command surface returned a bare, un-annotated 409 for this condition (no retryable, no neo4j_code at all) — worse than a wrong value, since a missing field cannot even be measured and disproven; it silently looks like a response that never made any retry promise.

The failover case (easy to get wrong)

Neo.ClientError.Cluster.NotALeader is classified as a client error by the Neo4j taxonomy, but it is retryable — a client that only checks for the Neo.TransientError. prefix will treat a routine leader election as fatal.

It is retryable by re-routing, not by repeating the request against the same node. The response tells you where to go:

{
  "error": "not the leader",
  "neo4j_code": "Neo.ClientError.Cluster.NotALeader",
  "retryable": true,
  "leader_id": 2,
  "leader_addr": "patinadb-node2.internal:21001"
}

Retry against leader_addr. If it is null, an election is in progress — back off and retry the whole discovery.

If you use the Bolt protocol with an official Neo4j driver and a neo4j:// URL, the driver’s routing table handles this for you. The REST API has no such layer: your client owns it.

The same response, for a different reason: an incomplete history

A node that has just joined (or rejoined) a cluster serves reads immediately, but its engram history — the timeline behind AS OF, AS OF TAG and diff — arrives afterwards, in the background. While that is still running, a time-travel read that reaches further back than the node has got answers with the same re-route response as above, plus "reason": "history_below_local_floor":

{
  "error": "engram 0e5b… is not in this node's local history, and this node's history is INCOMPLETE …",
  "neo4j_code": "Neo.ClientError.Cluster.NotALeader",
  "retryable": true,
  "leader_id": 2,
  "leader_addr": "patinadb-node2.internal:21001",
  "reason": "history_below_local_floor"
}

If you already implement the retry contract, you do not need to do anything — this is the response you already handle. It is listed here because two things about it are worth knowing before you hit it in production:

• It is temporary and it is visible. The node reports the condition on /ready as a history field (history_backfilling) without going un-ready — it is serving, and pulling it out of a load balancer would turn a non-outage into one. If that field instead reads history_backfill_fenced, the node will not recover on its own and needs an operator; see High availability.
• Ordinary reads and writes are unaffected. Only a time-travel read that reaches below what the node has so far is re-routed. A read at the current state is answered locally, as always.

The cost, stated plainly: with a Bolt driver, this evicts the node from your routing table. Official Neo4j drivers treat Neo.ClientError.Cluster.NotALeader as a routing error and drop the server from their table before retrying. So an application that runs AS OF queries in a loop against a node that is still backfilling will repeatedly lose that node from its read pool and re-fetch routing. It is self-correcting and it costs you no wrong answers — but if your workload is time-travel-heavy and you are rejoining nodes often, prefer pinning those queries to a node whose /ready shows no history field.

Do not parse the human-readable text. On some code paths the error / message fields carry raw internal routing text instead of a clean sentence — e.g. has to forward request to: Some(2), Some(BasicNode { addr: … }). That is the same not-the-leader condition in workmanlike dress. Branch on the machine-readable fields only — neo4j_code, retryable, leader_id, leader_addr — and treat error/message as log text, never as a contract.

What each code means

Retryable:

Note (fixed): OCC write-write conflicts and a COMMIT losing leadership used to be surfaced as Neo.TransientError.Transaction.LockClientStopped. The official Neo4j drivers’ error-classification logic excludes that specific code from automatic retry by name (the Python driver’s error-rewrite map turns it into a non-retryable ClientError; the Java/JS 4.4-line drivers exclude it too) — so a routine cluster event surfaced as a hard application error under a driver’s managed-transaction function. Both cases now emit Neo.TransientError.General.TransientError, which every official driver variant retries. If you see the old code from an older patinaDB deployment, treat it exactly like the row above (re-read, re-resolve, retry) — the semantics never changed, only which code named them.

Not retryable — fix the request, not the timing:

Writes are at-least-once

patinaDB’s write contract is at-least-once, not exactly-once. (An idempotency token narrows this to recognisably at-least-once for the writes that carry one — see that section; everything here stays true for any write that carries no token.) Inside the cluster, re-application of the same Raft log entry is deduplicated by the store — internal redelivery is safe. But a client-level retry is a new execution, not a redelivery: the statement is resolved against current state again, and CREATE mints fresh vertex UUIDs at resolve time (Uuid::new_v4 in the DML resolve path). So:

If you sent a write, the ack was lost (a 503, a timeout, a connection reset — anything where you did not see a definitive response), and the write had in fact committed, resending it applies it a second time. With CREATE, that means duplicate nodes/relationships that differ only in their UUIDs.

This is measured, not assumed (#959)

Until 2026-08 the paragraph above was an argument from how the system is built. It is now a measurement. Deterministic simulation testing (RFC-0019) gained the ability to lose a response after the request had already been delivered and taken effect — the “the write happened, the acknowledgement did not” case, which the simulator previously could not produce at all, because a fault was only ever decided before delivery.

Under that fault the ambiguous outcome is not a corner case. It is the normal one:

Measured 2026-08-11 (#959, crates/patinadb-raft/tests/dst_response_drop.rs). Conditions: 3-node cluster under the deterministic simulator, one node partitioned for the workload window, 55% of responses dropped after delivery, 6 single-shot writes per run with a 400 ms client budget, seeds 0..16.

The class occurred in 10 of 16 runs. In every one of those 10, the set of writes the client saw time out was identical to the set of writes that were nonetheless committed and present on every node afterwards (unacked == unacked-but-persisted). Under this fault, every single client timeout was a false negative: the write was already on a quorum when the client gave up on it.

Read the consequence carefully, because it is the whole reason the rules below are not optional. A client that blindly resends after a timeout is, in these runs, resending something that already committed — in 10 runs out of 16. Timeouts are not a rare unlucky path where duplicates might appear; they are a path where, in this fault regime, the duplicate is what you get unless the write was idempotent by construction. Constraints, MERGE, and the conditional-apply work (RFC-0022, preimage/CAS) are what stand between “the ack was lost” and “the data forked”.

The honest scope of the number: it characterises this fault regime (an aggressive response-loss schedule on a degraded cluster), not steady-state production, where most writes are acknowledged normally. It does not say “10/16 of your writes duplicate”. It says that when you do hit the ambiguous outcome, the write having succeeded is the overwhelmingly likely explanation — so “it timed out, so it probably didn’t apply” is exactly backwards.

Rules for living with this:

• Declare UNIQUE / NODE KEY constraints on business keys. The duplicate retry then fails cleanly with Neo.ClientError.Schema.ConstraintValidationFailed instead of silently forking your data — the constraint turns “applied twice” into “loud error on the second attempt”.
• Prefer MERGE over CREATE for anything with a business key; a retried MERGE matches the already-committed row and is naturally idempotent.
• Retry-once discipline for ambiguous outcomes. A response with retryable: false (or a non-retryable Bolt failure) means the write definitively did not apply — safe to resend. No response at all is the ambiguous case: resend at most once, and only for writes that are idempotent by construction (constraint- or MERGE-guarded) — otherwise reconcile first (query for the row the write would have created).
• Bolt: the same ambiguity surfaces to drivers as a Neo.TransientError.* failure (e.g. Neo.TransientError.General.TransientError on a write-write conflict or a lost-leadership COMMIT, or Neo.TransientError.Transaction. Terminated when an operator or the idle reaper killed the transaction), as Neo.ClientError.Cluster.NotALeader when the write reached a non-leader / a leaderless window, or as a plain connection error when the socket drops mid-commit. A driver executing a transaction function may auto-retry transient failures; the application owns idempotency — use constraints / MERGE there too.

Idempotency tokens

The rules above make a retry survivable. An idempotency token makes it recognisable: you attach a unique token to a write, and if that exact request already committed — even under a different leader — the retry applies nothing and tells you where the original landed.

Where the token goes. The token is wired end to end on both client surfaces:

• REST: send it as the Idempotency-Key request header on POST /cypher (HTTP header names are case-insensitive). The write’s JSON response carries dedup_active and, when dedup fired, duplicate_of — see Two fields, two different questions.
• Bolt: put it in the transaction metadata under the reserved key idempotency_key — in BEGIN’s tx_metadata for an explicit transaction (see below), or in the RUN’s own metadata for an autocommit statement.

A token must be printable ASCII (0x21–0x7E, no spaces or control characters), non-empty, and at most 128 bytes. An invalid token fails the request loudly with a client error — it is never silently treated as “no token”, so you can never believe a write was protected when it was not. Omitting the token means “no token”; explicitly supplying an empty header or empty token value is an error.

What it does, and what it is not

Transactions mean all-or-nothing; tokens prevent applying the same request twice across transaction attempts. Consider a lost commit acknowledgement:

Backend                         Database
   ---- COMMIT ------------------->
                                   commits durably
   <--- acknowledgement --- X connection lost

The backend cannot tell whether the commit succeeded. Closing the connection or sending an abort cannot undo an already committed transaction. Retrying “increment this counter” in a new transaction can therefore increment it twice, even though both transactions individually satisfy atomicity.

• Confirmed rollback: the transaction did not commit; retry as appropriate.
• Confirmed commit: the write succeeded.
• Unknown commit outcome: retry the same logical request with the same token within its protection window, or use an inherently idempotent operation.

Generate the token once per logical request, outside any retried transaction callback. A new logical request needs a new token. The official Neo4j driver does not automatically generate Patina’s idempotency_key; the backend must supply and preserve it. Recovery across a backend restart also requires retaining the request identity. Bookmarks are separate: they make a subsequent reader wait for the committed write, not deduplicate another write.

The repository’s scripts/customer-blackbox/backend is an executable application example: its optional HTTP Idempotency-Key is scoped to a stable backend principal, with a durable request ledger committed in the same transaction as the ticket mutation. The ledger stores the canonical request fingerprint and original status/body; a conflicting reuse returns409. The backend reconciles against the writer before acknowledging an uncertain outcome, rather than trusting a speculative result produced before COMMIT. A replay may return a newer bookmark while preserving the original application result.

This application ledger is distinct from the database’s bounded native token cache. Its example has no automatic expiry: plan retention and storage before using that pattern in production. Preserve the backend’s stable principal IDs and shared secret across restarts; rotating them without migration breaks lookup of old request identities. Headerless example requests retain their original unknown-outcome behavior. See the example’s README and CONTRACT for configuration; this is not automatic indefinite deduplication of every database transaction.

A token answers a different question from everything else in this chapter. Constraints and MERGE make a duplicate application harmless. Conditional apply (RFC-0022) asks “is the value this write assumed still current?” — and a retry after a lost response passes that check, because nothing else changed the state; that is exactly what made the timeout ambiguous. A token asks the one question neither of those asks:

“Has this exact request already been committed?”

That is an existence check over request identity, independent of the data. It runs at apply time on every node, not on the leader, which is what makes it work in the case it exists for: after a failover your retry reaches a different node, and that node must reach the same conclusion.

Explicit transactions: the token goes on BEGIN, not each statement

session.execute_write(...) — the API official Neo4j drivers steer you toward — runs over an explicit transaction, not autocommit: one BEGIN, one or more RUNs, one COMMIT. Set the token in BEGIN’s tx_metadata (the same place a driver already sets provenance metadata via begin_transaction(metadata=…)); it is validated at BEGIN and applies to the whole transaction’s single Raft write at COMMIT. A RUN’s own tx_metadata inside an already-open explicit transaction is not read for this — the token is a property of the transaction, matching how the same message already carries provenance metadata. COMMIT’s success carries dedup_active/duplicate_of exactly like an autocommit write’s terminal response (issue #1136). An invalid token at BEGIN fails BEGIN itself, loudly, for the same reason an invalid token fails autocommit’s RUN: it must never silently become “no token” and let you believe the whole transaction was protected when BEGIN quietly ignored it.

Reading the dedup fields back over Bolt is the catch. For an explicit transaction — which is what the recommended driver API, session.execute_write(...), runs — the dedup_active/duplicate_of metadata rides on the COMMIT message’s SUCCESS, and the official Python driver does not expose that message’s metadata to your code: the ResultSummary you can reach comes from each statement’s RUN/PULL SUCCESS, and those statements were buffered, not committed, so they honestly report dedup_active: false. Over an explicit Bolt transaction the dedup fields are therefore not readable from application code — the protection is active (the token on BEGIN is honoured at COMMIT), but you cannot observe whether it fired. What you can rely on: a recognised duplicate commits as a success that changes nothing, so the driver’s auto-retry of a lost commit response is safe. If you need to observe the dedup decision — e.g. to distinguish “applied” from “already applied” — issue the write as a single autocommit statement (whose terminal SUCCESS you do get via ResultSummary) or over REST, where both fields are plain JSON response members.

The token contract — this part is yours, and we cannot check it

You choose the token, and it must be unique per logical write. A UUID per request is the obvious choice.

The server can neither enforce this nor detect a violation. If you use the same token for two genuinely different writes, the second is recognised as a duplicate of the first: it applies nothing, and you are told it succeeded. You lose that write, silently.

This is not an oversight we intend to close, and the reason is worth understanding, because it tells you how carefully to generate tokens. The obvious guard would be to bind the token to a fingerprint of the write itself and reject a mismatch. That is impossible here: a retried statement is re-resolved on the leader, and CREATE mints fresh UUIDs at resolve time (the same fact this chapter opened with), so a retry’s resolved write legitimately differs from the original’s. A fingerprint would make every honest CREATE retry look like a different request and dedup nothing at all.

So: derive the token from your notion of the request — an inbound request id, a job id, a (business-key, operation) pair — and never from a value that changes between attempts (a timestamp, a random draw taken per attempt, a retry counter).

The guarantee is a capacity, not a promise

A token is honoured for a bounded window. Read this as a capacity:

A token is honoured for 10 minutes of cluster time, or until 100 000 newer writes have landed in the same database — whichever comes first.

Both bounds are deliberately far outside a real retry. The measured cold-start window after a total cluster outage is ~4.3 s at worst (#1150), and this chapter’s own advice is to retry for at least ~10 seconds; ten minutes is two orders of magnitude above that. The count bound is what stops a burst from growing the map without limit — at the ~10 writes/s profile these bounds are written for, the time window holds roughly 6 000 entries, so the count ceiling is ~16× headroom.

The window is measured in leader-stamped time, not in any node’s local clock. That is a correctness requirement, not a detail: every node must reach the same expiry decision for the same log position, or two replicas could disagree about whether your retry is a duplicate — one applying it, one not.

A forward clock jump breaks this structurally, and the database cannot fix it for you. If a leader’s clock runs more than the retention window ahead of real time, one write it stamps is enough to move the retention watermark permanently into the future: every correctly-stamped token after it is born expired, so the very next retry duplicates — and the watermark is monotone, so no amount of correct clocks afterwards moves it back (issue #1137). The in-database signal is a WARN in the leader log, which nobody reads at 3am. Synchronised clocks with alerting are therefore a hard precondition for relying on tokens, not a recommendation — see Production Deployment. A cluster whose clocks are all equally wrong is unaffected; it is skew between leaders larger than the retention window that hurts.

Measured cost (2026-08-23, RFC-0024 / #1080, on a 62 GB / NVMe box, release build, allocated disk blocks): ~293 bytes per live token at 10 000 entries; a database sitting at the 100 000 ceiling holds ~25 MB of map (25 309 184 B measured at the real ceiling, #1151, 2026-09-01), and the 10-minute window at ~10 writes/s holds ~1.8 MB. A write that carries a token costs about 9% more on the apply path than the same write without one (1462 → 1342 single-entry applies/s in the same harness); a write that carries no token is unaffected.

When the map is full, writes are let through — not blocked

If a burst fills the map to its ceiling, patinaDB evicts the oldest tokens early and accepts the write. It does not refuse writes to protect the map.

That is a deliberate trade, and it has a real price on both sides:

• Refusing writes would introduce a failure mode that does not exist today, and it would appear during a failover retry storm — precisely when the cluster is already degraded and every client is already retrying. It turns a duplicate into an outage.
• Letting writes through means a retry whose token was just evicted is treated as new, and duplicates after all. That is the honest price: a gap in exactly the scenario the feature exists for.

Because that gap would otherwise be invisible, it is counted and logged: the metric patinadb_raft_idempotency_capacity_evictions_total and a WARN on the leader fire whenever a token is evicted early. If you see either, your guarantee is the capacity bound, not the time bound — size your write rate or your expectations accordingly.

This has been measured against the real ceiling (#1151, 2026-09-01, bench_idempotency_retry_storm): a burst of 150 000 token records — 1.5× the ceiling — accepts every record, evicts the oldest 50 000 tokens with every eviction counted and logged. The benchmark then looks up every token before recording any retry tokens: 50 000 lookups miss (33%). It subsequently records those missing tokens, which evicts another 50 000 still-valid entries. This measures the native token table; it does not execute the application mutations or an end-to-end concurrent client retry storm.

The measured 33% is not an upper bound on duplicated writes. In a live workload, an unrecognised retry can apply again and insert a fresh token before another retry arrives. That insertion can evict a token which was still protected at the start of the retry wave. Retry order and concurrent writes therefore affect which requests remain protected. Capacity-eviction counters and warnings indicate lost protection, not a safe percentage of retries. Once a token is absent, neither reuse of that token nor a fresh token establishes that the original mutation did not commit; reconcile through the application ledger/business identity or apply an explicit application retry policy.

Two fields, two different questions

The answer to a write carries two token-related fields. They are not two views of the same thing, and reading one as the other is the mistake to avoid:

dedup_active is not the duplicate signal. It is true for a perfectly ordinary, first-time, brand-new write — that is the normal case, and it means “you are protected”, not “this already happened”. The duplicate signal is duplicate_of alone. A client that treats dedup_active: true as “this was a duplicate” concludes that every protected write already happened, and may discard results that are real.

Read them in that order — first whether the protection was on, then whether it caught something:

• dedup_active: false — the mechanism was not active for this write, which therefore has the ordinary at-least-once contract described earlier in this chapter. Two causes: you sent no token, or — the one to watch — a rolling upgrade is in progress. While any cluster member is still on an older protocol version, patinaDB falls back to the older wire format so writes keep flowing, and the token is dropped along the way. Writes never stop; the protection pauses.
• dedup_active: true + no duplicate_of — protection was on, and this write is new. This is the normal, healthy answer, and it is what the large majority of your writes will return. The write applied.
• dedup_active: true + duplicate_of: <index> — protection was on and it fired: this exact request had already committed. The write applied nothing, and <index> is where the original landed.

The same write, twice — side by side

The two answers below are the same request, sent twice. Reading them together is the fastest way to see which field carries which meaning:

// First delivery — an ordinary, successful, brand-new write.
{ "dedup_active": true,          // the protection was on ...
  // "duplicate_of" is ABSENT    // ... and had nothing to catch
  "log_index": 4711,
  "stats": { "nodes_created": 1 } }

// The retry, after the first response was lost.
{ "dedup_active": true,          // the protection was on ...
  "duplicate_of": 4711,          // ... and it fired: already committed, at 4711
  "log_index": 4711,             // the ORIGINAL's index — usable as a bookmark
  // no "stats" at all — same as any write that changed nothing
}

dedup_active is true in both. It is not what tells them apart — duplicate_of is.

A client that sends tokens and never reads dedup_active will believe it is protected throughout an upgrade window when it is not. That is the one way to be worse off with tokens than without them, and reading one field prevents it.

What a recognised duplicate looks like

• It is a success, not an error. It is never a 409, never a conflict, and never something to retry.
• It reports no write statistics — the stats map is omitted entirely, exactly as it is for any write that changed nothing. They describe this call, and this call changed nothing. They are deliberately not the original’s numbers: those cannot be reconstructed (a MERGE that created on the first attempt merely matches on the retry), and reporting a re-derived figure as history would be a fabrication.
• Read-your-writes still works. The position it reports is the original’s, so a bookmark or min_index taken from a deduplicated retry waits for the entry that actually carries your effect.

So a duplicate reads as: this call did nothing, because it already happened — here is where.

Not covered

Bulk and chunked loads (CALL {} IN TRANSACTIONS, bulk upsert) are many log entries, not one request; a single token cannot describe them, and they carry none. Schema and administrative commands (CREATE DATABASE, CREATE INDEX, …) are already idempotent by name and need no token.

How to retry

Re-read, don’t replay. The important part of retrying a Neo.TransientError.General.TransientError write-write conflict is not resending the same bytes — it is recomputing the write against current state. A read-modify-write must re-read:

for attempt in range(5):
    current = read_counter(key)                      # re-read every attempt
    resp = write_counter(key, current + 1)           # recompute from it
    if resp.ok:
        break
    if not resp.json().get("retryable"):
        raise ApplicationError(...)
    sleep(backoff(attempt))
else:
    raise RetriesExhausted(...)

Resending a value computed from a stale read will simply be rejected again — or, worse, silently overwrite the concurrent write you were told about.

Bound your retries. Use exponential backoff with jitter (e.g. 50 ms doubling to ~2 s, ±25 % random) and a hard attempt cap. Unbounded retry loops turn a brief contention spike into a self-sustaining stampede.

Make writes idempotent where you can. MERGE on a business key, rather than CREATE, means a retry after an ambiguous failure (a timeout where you never learned the outcome) cannot create a duplicate.

Size the retry budget for a cluster restart, not just for contention. The longest window your client has to ride out is not a contended key — it is the cluster coming back after a total outage (a datacentre power event, a node-pool restart). While the cluster re-forms, a node can already report a current_leader read from persisted Raft state, naming a node that is not yet serving; writes in that window are refused as retryable.

Measured (#1150, 2026-08-30): median 4.13 s, worst observed 4.32 s. Three independent runs of 20 simultaneous full-cluster restarts each (n = 60), on a 16-core Linux box at loadavg 13–22, --release, 3 voters on loopback, polled at 50 ms, timed from “all three processes restarted” to “a write returns 200”. The median reproduced to within 0.7 % across the three runs; a minority of restarts complete in ~0.3 s, which is the lucky case and not what a budget is sized for.

Retry writes for at least ~10 seconds before declaring the database down. That leaves roughly 2.3× headroom over the worst window observed. It is short, but far longer than a single-shot write or a two-attempt budget will survive — a client that gives up after one try will report an outage the cluster does not have.

This number is from a small database, and the ceiling is not. These nodes held a handful of vertices. A cold start also has to open the storage file, load the registry and replay the Raft log, all of which grow with the data — so at customer scale the window is longer by an unmeasured amount. The 10-second advice is calibrated against the small-database window with the headroom above; if your database is large, measure your own restart (scripts/cold-start-window.sh) rather than inheriting this figure.

Keep transactions short

An open explicit transaction (BEGIN … COMMIT) pins a read version for its whole lifetime. Long-lived transactions:

• hold back storage reclamation, so the data directory grows;
• widen the window for a write–write conflict at COMMIT, so they retry more.

Read the data you need, commit, and do your application-side thinking outside the transaction. Do not hold a transaction open across a user interaction or a network call to another service.

Reading your own writes

Following a write, a read routed to a different replica may not have applied it yet. Two options:

• Bolt drivers: use the same session. Bookmarks are handled for you, and a read waits for the replica to catch up (or reports BookmarkTimeout — retryable).
• REST: pass the write response’s log_index back as ?min_index=<log_index> on the read — any node (followers included) waits until it has applied the write, then serves (a timeout is a retryable BookmarkTimeout). See REST API → Read-your-writes over REST. Alternatively send the read to the leader, or pass "consistency": "linearizable" (leader-only; it waits for a read barrier).

See High Availability for the routing details.

Checklist

Before going to production, verify your client:

• keys retry decisions on retryable, not on the HTTP status or message
• treats NotALeader as re-route, using leader_addr
• re-reads before recomputing a retried read-modify-write
• bounds retries (backoff + jitter + attempt cap)
• uses MERGE-on-business-key so an ambiguous retry cannot duplicate — the measured evidence says a write that timed out has most likely already committed
• keeps explicit transactions short and never open across external calls
• surfaces a non-retryable error to the caller instead of looping
• sends an idempotency token on retried writes (Idempotency-Key header / idempotency_key metadata), derives each token from its own request identity, never from a per-attempt value; reads dedup_active to know whether the protection was active at all; and keys “this was a duplicate” on duplicate_of, never on dedup_active — noting that over an explicit Bolt transaction these fields are not readable from application code (they ride the COMMIT success, which drivers do not surface)

The deployment rehearsal exercises this contract against a real cluster under a forced leader failover — run it before you launch.