Production Deployment (3-Node Bolt Cluster)

This is an operator walkthrough for one specific, common deployment: a 3-node patinadb-raft Raft cluster with automatic failover, fronting a single trusted backend service that connects over Bolt with a Neo4j driver, and using graph CRUD + Cypher, versioning / time-travel (AS OF, engrams, diffs, tags) and search (full-text + vector indexes).

The threat model is a trusted, single-tenant backend — the only client is your own application, not arbitrary internet users. So this is a deploy + secure

• operate guide, not an untrusted-input hardening guide.

It ties together the reference chapters — High Availability, Authentication & TLS, Day-2 Operations, Editions & Limits, Configuration, Bolt, REST API and Time Travel — into one go-live sequence. Read those for depth; this chapter is the checklist.

1. Prerequisites & licensing

A production 3-node cluster requires a commercial license. Read this before anything else — it changes how you bring the cluster up.

The Community edition (the default with no valid license) caps max_voters at 1 (Editions & Limits). That has two concrete consequences for a 3-node cluster:

• POST /mgmt/change-membership refuses any voter set larger than 1 with 409 VoterCapExceeded. That endpoint is the documented way to grow a --bootstrap node into a 3-voter cluster and to replace a dead voter later (evict-voter → add-learner → change-membership) — a routine Day-2 op. In Community mode both are blocked.
• Community mode also enforces mandatory telemetry: if the node cannot phone home within the grace window (default 72 h), it degrades to read-only (writes return 503, reads keep working). --disable-telemetry is refused at startup without a valid license (fail-closed), so an isolated / air-gapped deployment cannot opt out. See Licensing & Telemetry.
• Community additionally caps ~5,000,000 nodes + edges (writes degrade to read-only past the cap) and 30-day rolling history retention (older time-travel history is squashed away — see §8).

A valid license lifts every one of these: max_voters becomes unlimited (or your tier’s cap), telemetry becomes best-effort and disableable, the scale cap is raised or removed, and retention defaults to unlimited. Install a license on every node before forming the cluster — see Installing a license and Getting a license. Precedence: --license / PATINADB_LICENSE (a file path or an inline token) → <db-root>/license.key.

Confirm the tier on each node after boot with the auth-exempt GET /version:

curl -s http://<node>:21001/version | jq '{tier, entitlements, usage}'
# tier must read "licensed" — "community" means the node found no valid license.

Closed (issue #1075). A single-shot POST /mgmt/init {"1":addr1,"2":addr2,"3":addr3} used to call Raft initialize directly and skip the voter-cap check entirely — a Community node could bootstrap a fixed multi-voter set in one call and never touch change-membership again. /mgmt/init now enforces the SAME 409 VoterCapExceeded check change-membership does, refused before initialize ever runs (so a refused attempt leaves the node cleanly uninitialized, free to retry with a valid set). It is still guarded against an even voter count too — 409 unless you pass {"members": {…}, "allow_even": true} (that override affects only the even-voter guard, never the voter cap; see Cluster sizing). Community remains subject to the telemetry / scale / retention caps regardless — it is not a viable production configuration — get a license.

Clocks: a hard precondition for idempotency tokens

If your clients use idempotency tokens (Client retry contract), synchronised clocks are not a nice-to-have. A leader whose clock jumps forward by more than the token retention window moves the dedup watermark permanently into the future — every token stamped afterwards is born expired and the next retry duplicates, with nothing but a WARN in the leader log to say so (issue #1137). Monotony of the watermark means no in-database recovery exists.

Therefore, on every node: run NTP/chrony, and alert on clock skew (your monitoring, not the database’s — the database cannot see skew between machines, only its own stamp). A cluster whose clocks are all equally wrong is unaffected; the dangerous case is skew between leaders larger than the retention window (default 10 minutes).

Sizing

Disk sizing

Each node stores the whole graph (every voter has a full replica), its Raft log, periodic snapshots, and the engram history (the versioning timeline). With unlimited retention (the licensed default) the engram history and snapshots grow without bound over the deployment’s lifetime — budget for it, and read §8 to decide between unlimited history and a bounded retention window. Back up the entire --db directory as a unit (graph, Raft log, snapshots, engram log); each database is a subdirectory of --db (On-disk layout).

Monitor free space with the patinadb_data_dir_available_bytes gauge and the shipped PatinaDBDiskSpaceLow / PatinaDBDiskSpaceCritical alerts (§7).

2. Bring up the 3-node cluster

A cluster forms by bootstrapping node 1, joining nodes 2 and 3 as learners, then promoting all three to voters. The promotion (change-membership) is the step that requires a license (§1).

With Docker Compose

The repository ships a 3-node stack at deploy/docker-compose.yml (plus Prometheus + Grafana). Treat it as a frictionless local demo, not a production template: it runs --insecure-disable-auth with no license, and its cluster-init step promotes to a 3-voter set via change-membership [1,2,3] — which an unlicensed node refuses with 409, so a plain unlicensed docker compose up leaves node 1 leading with nodes 2 and 3 as learners, not a 3-voter HA set. For production, harden it: add a license, turn auth on, and mount secrets as files.

# production 3-node compose (adapted from deploy/docker-compose.yml).
name: patinadb-prod

x-node: &node
  image: your-registry/patinadb-raft:<version>   # build from the repo Dockerfile
  restart: unless-stopped
  networks: [cl]
  environment:
    # Secrets mounted as files (see §3 "Secrets"); never bake them into the image.
    PATINADB_AUTH_PASSWORD_FILE: /run/secrets/auth_password
    PATINADB_CLUSTER_SECRET_FILE: /run/secrets/cluster_secret
    PATINADB_LICENSE_FILE: /run/secrets/license   # REQUIRED for a 3-voter cluster
  healthcheck:
    # distroless has no shell/curl; the binary is its own probe.
    test: ["CMD", "/usr/local/bin/patinadb-raft", "--healthcheck", "http://127.0.0.1:21001/ready"]
    interval: 30s
    timeout: 5s
    start_period: 20s
    retries: 3

services:
  node1:
    <<: *node
    container_name: patinadb-node1
    command:
      - --id=1
      - --addr=patinadb-node1:21001
      - --bolt-addr=0.0.0.0:7687
      - --advertised-addr=patinadb-node1.internal:7687   # a reachable host:port
      - --db=/data
      - --bootstrap
    ports: ["21001:21001", "7687:7687"]
    volumes: ["node1-data:/data", "./secrets:/run/secrets:ro"]

  node2:
    <<: *node
    container_name: patinadb-node2
    command:
      - --id=2
      - --addr=patinadb-node2:21001
      - --bolt-addr=0.0.0.0:7687
      - --advertised-addr=patinadb-node2.internal:7687
      - --db=/data
      - --join=patinadb-node1:21001
    ports: ["21002:21001", "7688:7687"]
    volumes: ["node2-data:/data", "./secrets:/run/secrets:ro"]

  node3:
    <<: *node
    container_name: patinadb-node3
    command:
      - --id=3
      - --addr=patinadb-node3:21001
      - --bolt-addr=0.0.0.0:7687
      - --advertised-addr=patinadb-node3.internal:7687
      - --db=/data
      - --join=patinadb-node1:21001
    ports: ["21003:21001", "7689:7687"]
    volumes: ["node3-data:/data", "./secrets:/run/secrets:ro"]

networks: { cl: { driver: bridge } }
volumes: { node1-data: , node2-data: , node3-data: }

--join registers nodes 2 and 3 as learners automatically. Once all three report their peer versions, promote them to voters against the leader (substitute your password):

curl -u neo4j:"$PW" -X POST http://patinadb-node1:21001/mgmt/change-membership \
  -H 'content-type: application/json' -d '[1,2,3]'
# → {"ok": true}   (a 409 here means Community mode / no license — see §1)

Notes. The demo --advertised-addr values are 127.0.0.1:7687/7688/7689 because everything runs on one host. On a real multi-host cluster set --advertised-addr to each node’s reachable host:port so neo4j:// routing sends drivers somewhere they can connect (§4). The distroless image runs as uid 65532; a fresh bind-mounted host volume is root-owned — chown -R 65532:65532 it, use a Docker named volume, or set the Kubernetes securityContext.fsGroup: 65532.

Without Docker (bare processes / systemd)

Equivalent to the compose above, one process per host:

# Node 1 — bootstrap leader
patinadb-raft --id 1 --addr node1.internal:21001 --db /var/lib/patinadb --bootstrap \
  --bolt-addr 0.0.0.0:7687 --advertised-addr node1.internal:7687 \
  --license /etc/patinadb/license.key
  # PATINADB_AUTH_PASSWORD, PATINADB_CLUSTER_SECRET set in the unit's environment

# Nodes 2 and 3 — auto-join as learners
patinadb-raft --id 2 --addr node2.internal:21001 --db /var/lib/patinadb --join node1.internal:21001 \
  --bolt-addr 0.0.0.0:7687 --advertised-addr node2.internal:7687 --license /etc/patinadb/license.key
patinadb-raft --id 3 --addr node3.internal:21001 --db /var/lib/patinadb --join node1.internal:21001 \
  --bolt-addr 0.0.0.0:7687 --advertised-addr node3.internal:7687 --license /etc/patinadb/license.key

Then promote to a 3-voter set (as above). If you prefer to skip the bootstrap → learner → promote sequence, a single POST /mgmt/init forms the set in one shot — but re-read the §1 nuance about the voter cap:

curl -u neo4j:"$PW" -X POST http://node1.internal:21001/mgmt/init \
  -H 'content-type: application/json' \
  -d '{"1":"node1.internal:21001","2":"node2.internal:21001","3":"node3.internal:21001"}'

Verify the cluster formed

# One leader + three voters, from any node:
curl -u neo4j:"$PW" -s http://node1.internal:21001/mgmt/cluster | jq

# Every node is ready (auth-exempt):
for n in node1 node2 node3; do curl -s http://$n.internal:21001/ready | jq -c; done

For growing/shrinking membership, failover behaviour, and rolling upgrades see High Availability.

3. Secure it

The default posture is fail-closed: an empty --auth-password refuses to start unless you also pass --insecure-disable-auth (Authentication). Work through the security deployment checklist; the essentials for this profile:

Backend ↔ database link (Bolt auth + TLS)

• Auth. Set a long, random --auth-password, at least 16 characters (the root-admin credential; REST Basic + Bolt LOGON) — issue #1065 exempts this exact username from the failed-attempt rate limiter, so unlike a directory-user password, this one has no per-minute guess cap backing it up; see Authentication for why. Your backend authenticates over Bolt at LOGON. For least privilege, create a dedicated user with only the roles it needs rather than logging in as neo4j (Users & roles) — unlike root, that user IS subject to the limiter (correctly: it goes through argon2 and carries real guessable entropy, so it needs the protection root doesn’t). That means it CAN get locked out (idle-timeout churn, a stale cached credential during your own credential rotation — the same shape issue #1065 fixed for root). The recovery path is admin-mediated, not self-service (a locked-out account cannot free itself): have root run POST /mgmt/unlock-user {"username": "..."} to clear the lock without touching the password, or ALTER USER … SET PASSWORD to do both at once (issue #1071 review follow-up). Root can always reach either call, since root is exempt from the limiter it is unlocking someone else out of.
• TLS. Native --tls-cert / --tls-key (PEM) secures the HTTP plane (REST + management + peer RPC) and the Bolt endpoint (bolt+s:// / wss://) with the same certificate — connect the driver with the +s scheme. It is opt-in; without it everything is plaintext, so a plaintext node logs a loud cleartext-credentials warning at startup. If you terminate TLS at a reverse proxy instead, set --advertised-addr to the public endpoint. See Native TLS for the HTTP plane and Reverse-proxy termination.

Internal cluster traffic (cluster secret + peer TLS)

• Cluster secret. Peer /raft/* RPCs authenticate with a dedicated --cluster-secret (PATINADB_CLUSTER_SECRET) carried in the X-Cluster-Secret header. All nodes must share the same value. If left empty it falls back to --auth-password, but a dedicated secret lets you rotate the two independently.
• Peer TLS. For a self-signed / private-CA cluster, give every node --tls-ca <ca.pem> so peers verify each other; the cert’s SANs must cover the peer --addr hosts.

Secrets (files & managers)

Do not pass secrets as plain env vars (they leak into docker inspect / /proc/<pid>/environ). Use the file or command conventions (Secrets management) for PATINADB_AUTH_PASSWORD, PATINADB_CLUSTER_SECRET, and PATINADB_LICENSE:

• <VAR>_FILE — mount the secret as a file: PATINADB_AUTH_PASSWORD_FILE=/run/secrets/auth_password.
• <VAR>_COMMAND — fetch from a secrets manager: PATINADB_AUTH_PASSWORD_COMMAND="vault kv get -field=pw secret/patinadb".

Precedence: direct env/flag > <VAR>_FILE > <VAR>_COMMAND > config file > default. All three are fail-closed — an unreadable file or a failing command is a loud startup error, never a silent “no secret”.

Online rotation

Rotate without a full restart (admin-only, per node): stage a new cluster secret on every node with POST /mgmt/rotate-cluster-secret {"secret":…} (both accepted during the grace window), then POST /mgmt/retire-cluster-secret on every node; rotate the root password with POST /mgmt/rotate-admin-password {"password":…}.

Admin-password rotation runbook (issue #1065 — read this before rotating in a single-backend deployment). A per-principal failed-attempt rate limiter is a total-outage switch in this profile: one backend client means one username, effectively one source, so an idle-timeout reconnect burst that races a naive password swap can exhaust the shared budget and lock the account out, including the new password it was just given. Three things prevent that, layered:

1. The root-admin username is exempt from the failed-attempt limiter entirely (App::authenticate_throttled) — however many wrong guesses land against it, the response is always a plain 401, never a 429. This is the primary guarantee: the account cannot be locked out, so recovery (including calling /mgmt/rotate-admin-password itself) is always possible. This exemption applies ONLY to the configured root username — directory (CREATE USER) accounts are unaffected and stay fully throttled.
2. POST /mgmt/rotate-admin-password is dual-accept: the outgoing password stays valid alongside the new one for a grace window, so any client attempt still carrying the old password (a config-propagation lag, or a reconnect that started before the rotation) keeps succeeding instead of getting a 401 mid-propagation.
3. The same call also clears the admin username’s rate-limiter window (belt-and-braces alongside #1 — harmless, and keeps the invariant true even if the exemption’s scope ever narrows).

A restart during the window is not a special case — read this before you rely on the window at all (#1065 review follow-up R4). Every rotatable secret (Secrets in auth.rs — the current password AND the dual-accept alt_password) lives ONLY in the process’s own memory. Nothing about a rotation is written to disk anywhere. So the moment this process restarts, for ANY reason — a deliberate restart, but just as much a crash, an OOM-kill, a redeploy, a Kubernetes pod reschedule — the node reconstructs its credentials from scratch, from whatever --auth-password / PATINADB_AUTH_PASSWORD / config file currently says. If you called rotate-admin-password but have not yet updated that external source to match, the node silently reverts to the pre-rotation password the instant it restarts: not merely “the new password stops working” but the OLD one starts working again too, with nothing in the process’s own state to show a rotation ever happened. A client that had already switched to the new password gets a wall of 401s that looks like an unrelated outage; a client still on the old one, which you may have been actively trying to retire, quietly starts succeeding again.

This is not something a future release can safely fix by persisting the rotation state. Writing the rotated password (even as an argon2 hash, the directory-user pattern) into the on-disk store would create the ONE thing that has never existed for the root credential: an offline-crackable artifact on disk for the exact secret whose failed-attempt-limiter exemption (mechanism 1 above) is safe only because no such artifact exists. It would also make root’s credential dependent on the database’s own state for the first time — a restore from an older backup would silently reset the password too, the same surprise class moved somewhere harder to see. Given that, the right fix is procedural: make a restart during the window harmless instead of trying to make the window durable.

Runbook — update the config FIRST, then rotate to match. This order makes a mid-window restart a no-op for any client that has already switched, instead of a silent regression:

1. Update your backend’s configured password to <new> — before calling the rotate endpoint. The running node still only accepts the OLD password at this point; nothing changes for it yet.
2. POST /mgmt/rotate-admin-password {"password": "<new>"}, authenticated with the OLD password. Response has "dual_accept": true — both passwords now work. The config on disk and the node’s new primary password are now the SAME value, so a restart from here on reconstructs exactly the state you intended, not the pre-rotation one.
3. Let/force the backend to reconnect, then confirm it is actually using the new password — e.g. a successful authenticated request from the backend’s own logs/metrics, not just “the deploy finished”. This is the step a runbook that only says “rotate, then retire” skips, and skipping it is what turns retiring into a self-inflicted lockout of a backend that never actually picked up the new password.
4. Only then, POST /mgmt/retire-admin-password to drop the old password. There is no rush — retiring is optional, since the next rotation drops the old password anyway. If you suspect the OLD password was compromised, retire as soon as step 3 confirms, not before.

With this order, a restart between step 2 and step 4 is harmless for any client that already picked up the new password (config and in-memory state agree — nothing to revert to) and unchanged risk for one that has not (the same window step 3 exists to close). The reordering does not make the window durable — it removes the specific danger of an UNPLANNED restart silently undoing a rotation the operator believed had already taken effect.

Skipping straight to step 4 before step 3 reopens exactly the failure mode this issue is about, for any client that hasn’t yet reconnected — the only difference from the pre-#1065 bug is that the account itself can no longer be rate-limiter-locked (mechanism 1 above); an old, still-configured backend would simply get a hard 401 instead.

If you have NOT set a dedicated --cluster-secret (issue #1065 review follow-up R5): step 3 covers your PEERS too, not just your client. Without --cluster-secret, the effective peer secret for /raft/* RPCs falls back to the admin password — a documented single-credential convenience. During the dual-accept window this fallback now ALSO dual-accepts (both the outgoing and the new admin password work as the peer secret), so a rotation no longer tears a coupled peer straight out of quorum the instant you rotate. But the window still closes at step 4 — retiring drops the OLD admin password everywhere, on the peer path exactly as on the client path. If any peer is still presenting the outgoing password when you retire (because you only confirmed your BACKEND client in step 3, not your other nodes), that peer is refused just as hard as an unconfirmed client would be — except a lost peer looks like a partition or a quorum problem, not an auth problem, and is correspondingly harder to diagnose under pressure. Extend step 3 to cover every peer: confirm each node’s own outbound /raft/* traffic is authenticating cleanly (GET /mgmt/metrics on each node reporting a healthy current_leader, or check each node’s log for peer-auth denials) before retiring on any of them. The dual-accept window moved the danger from “the moment you rotate” to “the moment you retire” — it did not remove it. A deployment with its own dedicated --cluster-secret is unaffected either way: rotate it independently with POST /mgmt/rotate-cluster-secret / POST /mgmt/retire-cluster-secret, on its own schedule, as documented above.

Bind & expose deliberately

Bind --addr and --bolt-addr to the private interfaces the backend and peers reach — not 0.0.0.0 on a public interface. /metrics is authenticated and global-Admin-only by default (its db=<name> labels would otherwise let any authenticated per-db Reader enumerate other tenants’ database names — issue #693), so point your Prometheus scrape at an Admin credential; keep it that way, or open it only on a private monitoring network with --insecure-open-metrics (which bypasses both the credential and the role floor). /health, /ready and /version are always open probes. With authentication enabled, the public /version response never contains database names; peers presenting X-Cluster-Secret additionally receive the database-scoped history-floor map used by retention coordination. An explicitly auth-disabled node retains the legacy map because all of its databases are public already.

Multi-tenant note (usually N/A here). This profile is single-tenant, so the default open RBAC is fine. If you later host multiple isolated tenants, enable --rbac-closed (closed-mode tenant isolation).

4. Connect the backend

Point your Neo4j driver at the cluster. Prefer the routing scheme so reads spread across followers and writes go to the leader automatically (Cluster-aware routing):

neo4j://<any-node-host>:7687      # or neo4j+s:// with TLS

• neo4j:// vs bolt://. neo4j:// asks the cluster for a routing table and load-balances: writes → leader, reads → followers/learners, with automatic reconnection on failover. bolt:// pins to one node (no routing). Use neo4j:// (or neo4j+s:// with TLS). Routing hands drivers each node’s --advertised-addr, so those must be reachable from the backend.
• Read consistency. A default read serves the node’s local applied state (causally consistent within a database, eventually consistent across replicas — a follower may lag). The Bolt path always reads locally; the "consistency":"linearizable" opt-in is REST-only and leader-only. When you need a lag-immune read that any follower can serve, read AS OF TAG '<name>' (a replicated, deterministic point — the “clean-tag” pattern in Read consistency).
• Transactions are snapshot isolation, not serializable. BEGIN … COMMIT gets repeatable reads and first-committer-wins on write-write conflicts (a conflict raises the driver-retryable Neo.TransientError.General.TransientError — wrap writes in a retry loop). SI does not prevent write skew; for an invariant spanning rows one tx reads and another writes, use a single autocommit statement or an app-level guard. See Transactions.
• Pool & limits. Size the driver’s connection pool below the server’s --max-bolt-connections (default 1024) — a cap on authenticated connections; an unauthenticated socket (silent, stalled mid-handshake, or idling after HELLO) never spends this budget (issue #440), instead bounded in time by --bolt-handshake-timeout-secs (default 10) and --bolt-idle-timeout-secs (default 300). Tune the tx guards for your workload: --max-tx-ops (default 5,000,000 buffered ops per explicit tx) and --idle-tx-timeout-secs (default 300 — an abandoned BEGIN is rolled back and its connection closed). A client that RUNs a streaming read then never PULLs is reaped after --stream-pull-timeout-secs (default 300). See Connection resource limits and Transaction resource limits.

Concurrency & retries

If your backend runs multiple replicas, read this first. A horizontally scaled backend — say 3 app replicas behind a load balancer — means two replicas can issue a write to the same node or edge at the same time. That is a supported, everyday topology, and patinaDB does not lose an update or create a duplicate under it: the server detects the collision and lets exactly one writer through. But it handles the collision by rejecting the other writer with a retryable error, not by making it wait — so the one thing your backend MUST do is retry that error, on every replica, for every write. The mechanism below is how, and it is a requirement for a multi-writer deployment, not a recommendation. A backend that treats the retryable conflict as a hard failure will see write failures under contention (never lost or corrupted data — but avoidable failures). This is verified end-to-end: 16 concurrent clients hammering one counter over autocommit land every increment exactly once, and two replicas creating the same UNIQUE-constrained key collide so that exactly one succeeds (pipelined_bolt_autocommit_contended_counter_never_loses_an_update, bolt_occ_duplicate_unique_create_conflicts).

You do not lock the database. patinaDB has no pessimistic-lock primitive (no SELECT … FOR UPDATE) and does not need one. Concurrency control is optimistic and works out of the box:

• Ordinary write–write conflicts (two requests updating the same node/edge) are detected at COMMIT: the first committer wins, the second gets the driver-retryable Neo.TransientError.General.TransientError. Use your driver’s managed transactions — session.execute_write(tx_fn) (Python) / session.executeWrite(...) (JS/Java) — which retry a transient automatically. That is the whole mechanism: concurrent updates to the same data are serialized and retried with no locking and no lost updates.

  # REQUIRED for a multi-writer backend: the driver retries the transient for you.
  def bump(tx, node_id):
      tx.run("MATCH (n:Counter {key:$id}) SET n.count = n.count + 1", id=node_id)
  with driver.session() as s:
      s.execute_write(bump, node_id)   # auto-retries the transient
  
  # WRONG for a multi-writer backend: a naked autocommit run treats the
  # conflict as a hard error. No data is lost or corrupted, but under
  # contention between replicas this call *fails* instead of retrying.
  # with driver.session() as s:
  #     s.run("MATCH (n:Counter {key:$id}) SET n.count = n.count + 1", id=node_id)
  

  The same holds for MERGE on a UNIQUE-constrained key — the classic “upsert this user from whichever replica saw the request first”. Two replicas running MERGE (u:User {id:$id}) … concurrently collide on the value, one gets the retryable transient, and a managed transaction converges it to a single node. Wrap it in execute_write and it is safe across any number of replicas; run it naked and a simultaneous upsert can fail.

• Write skew is the only case OCC does not catch (snapshot isolation is not serializable): two transactions each read a value and write disjoint rows based on it, so there is no write–write overlap to detect. This matters only for a multi-row invariant that genuinely requires serializability (e.g. “the sum of two balances must stay ≥ 0”, debited by two txns on different rows). The fix is not a lock — force the skew into a detectable write–write conflict by having both transactions write a shared sentinel/version node:

  // Both txns touch the same guard row → OCC now detects the conflict → retry.
  MATCH (g:AccountGroup {id:$gid}) SET g.version = g.version + 1
  

  If your backend has no such cross-row invariant (the common case), you need none of this — managed transactions are sufficient.

What makes this work: pipelined writes (PATINADB_PIPELINED_WRITES, default on). Concurrent autocommit writes to one database resolve in parallel and are admitted under the optimistic check above, rather than serializing behind each other’s whole Raft round-trip — which is why a conflict between two concurrent writers surfaces as the retryable transient instead of being avoided by making everyone queue. Both explicit transactions and plain autocommit statements go through it, so the retry advice above applies to both. PATINADB_PIPELINED_WRITES=0 is the rollback valve: writes then serialize per database — no lost updates either way, but ~4–5× less write throughput on a disjoint-write workload, and note this does not buy serializability (Bolt’s explicit transactions were already snapshot-isolated). See Configuration.

Measured operating limits (issue #1079, 2026-08-21, 62-GiB dev box, release build, real 3-node in-process cluster, loadavg ~7–9, two runs agreeing to within ~3%). The pipelined-write admission critical section (per-db write_lock: an openraft read-index quorum-RTT barrier, the constraint check, and the preimage build, all run BEFORE the cheap in-flight-conflict check that actually decides admit-or-reject) is a genuine capacity wall — the question was only where it sits relative to a real deployment’s rate.

arm	N	ok	goodput/s	p50 ms	p95 ms
disjoint	8	8/8	494	11.7	16.1
disjoint	32	32/32	443	39.7	61.5
disjoint	100	100/100	465	113.2	196.4
hot-key	8	8/8	258	19.1	30.9
hot-key	32	10/32	76	122.8	130.4
hot-key	100	10/100	23	418.3	441.4

Disjoint-write ceiling (different rows/keys per writer): ~465–494 writes/s per database. A planned rate of ~10 writes/s is ~2% of this — comfortably not acute. This ceiling is NOT 1/(single-write latency): a single uncontended write takes ~4.3 ms end-to-end, but resolve/propose/ replicate/apply pipeline across concurrent writers, so throughput exceeds the naive inverse-latency figure — only the write_lock-held admission section is genuinely serial (≤ ~2 ms of it).

Hot-key collapse (many writers contending the SAME row) is a different, real finding: throughput FALLS as writers are added — 258/s at N=8 down to 23/s at N=100 (11.5× collapse), and at N=100 only 10 of 100 writers succeed; the other 90 exhaust the retry budget (8 attempts × a fixed 2 ms backoff, no jitter) and are handed a retryable conflict. Every condemned admission still pays the full quorum-RTT + preimage cost under the shared lock before the cheap in-flight check would have rejected it for free — so added contention on one key buys negative throughput, not just diminishing returns. This is exactly the shape the write-skew guard/ sentinel-row pattern above creates on purpose (every conflicting transaction deliberately touches the SAME AccountGroup/guard node) — the pattern is still correct and still the right fix for a genuine multi-row invariant, but driving it at high concurrency on one guard row is the concrete case this ceiling applies to, not an abstract one.

At the ~10 writes/s rate this measurement was taken for, the hot-key collapse is unreachable (expected concurrency on any single key ≈ 0.02 at a ~2 ms serial section — i.e. two writers landing on the same key at the same instant is a ~2% event, and the collapse needs sustained concurrent contention, not an occasional overlap). It stays a documented limit, not a fix — the fix direction (cheap in-flight check BEFORE the quorum-RTT/preimage cost, then retry jitter) is already known and correctly ordered in issue #1079 for whenever a workload’s key distribution changes this calculus. This depends on the write workload NOT concentrating repeated writes on a small set of hot keys/rows (a shared counter, a single tenant-root node touched by every write, or the guard-row pattern above used at real concurrency) — if your workload does that by design, re-run the bench (cargo test --release -p patinadb-raft --test pipeline_admission_bench -- --ignored --nocapture --test-threads=1) against your actual key-contention shape before trusting “not acute” for it.

5. Verify (go-live smoke checklist)

Deployment rehearsal

Before the manual checklist below, run the automated dress rehearsal:

scripts/deployment-rehearsal.sh          # add --keep to retain logs

It boots a real 3-node cluster (real processes, real HTTP, real kill -9, real backup files) and drives the scenarios that actually bite in production: the cluster forms and elects a leader; unauthenticated access is refused; a write replicates to every node; errors carry a machine-readable retry decision; the cluster survives losing its leader to SIGKILL without losing an acknowledged write; a backup restores into a fresh node; and the killed node rejoins and catches up. Exit code 0 is the verdict.

This is a level above the in-process integration tests: those verify the engine, this verifies the deployment. Run it before launch and after any infrastructure change.

Two things it deliberately does not do. It cannot test your environment — network policy, TLS certificates, backup destination — so it prints a checklist of exactly those at the end. And its retry-contract scenario only proves the server’s half (that every error carries retryable); the client half is yours, see Client Retry Contract.

If you run it without a licence, the backup scenario falls back to the HEAD-state backup and says so: point-in-time recovery (?history=true) is a Pro/Enterprise feature and is refused with a 403 on Community.

Manual smoke checks

Run these against the cluster before cutting traffic over. Substitute your Bolt URL / credentials; the examples use Cypher you can run from the driver or via POST /cypher.

0. Validate the config first — before the node even boots. On each node, run the exact command line / env / config file you deploy with, plus --check-config. It resolves everything the way a real boot does, prints what it understood, and exits non-zero if a setting is invalid or a runtime env knob is set to a value the engine would silently ignore (e.g. a PATINADB_MAX_AGG_ROWS=5M that the runtime wants as 5000000). This catches the misconfiguration class that otherwise only shows up as surprising behaviour in production (Config dry run):

   patinadb-raft --config node.yaml --check-config || { echo "bad config"; exit 1; }
   

   Confirm the printed cache budget shows the caches ENABLED with the memory you intend — a typo’d --cache-limit silently disabling the cache is exactly what this step exists to catch.

1. Write + read a node (exercise the write path through Raft):

   CREATE (:HealthCheck {id: 'smoke-1', at: datetime()});
   MATCH (h:HealthCheck {id: 'smoke-1'}) RETURN h.at;
   

2. Time-travel read (AS OF). Capture the current engram, mutate, then read the past state (Time Travel):

   // note HEAD first
   CALL patinadb.engrams() YIELD id RETURN id ORDER BY id DESC LIMIT 1;
   // ... then, after a later write, read the earlier state:
   USE default AS OF '<engram-id>' MATCH (h:HealthCheck) RETURN count(h);
   

3. Full-text search (Full-Text Search):

   CREATE FULLTEXT INDEX docs FOR (n:Doc) ON EACH [n.body];
   CREATE (:Doc {body: 'patinadb production go-live'});
   CALL db.index.fulltext.queryNodes('docs', 'production') YIELD node, score RETURN node.body, score;
   

4. Vector search (Vector Search) — note the arg order queryNodes(indexName, k, queryVector):

   CREATE VECTOR INDEX emb FOR (n:Item) ON (n.vec)
     OPTIONS { indexConfig: { `vector.dimensions`: 3, `vector.similarity_function`: 'cosine' } };
   CREATE (:Item {vec: [0.1, 0.2, 0.3]});
   CALL db.index.vector.queryNodes('emb', 5, [0.1, 0.2, 0.3]) YIELD node, score RETURN node, score;
   

5. Cross-node consistency. After a write on the leader, confirm every node converges (each node serves its own applied state):

   for n in node1 node2 node3; do
     curl -u neo4j:"$PW" -s -X POST http://$n.internal:21001/cypher \
       -H 'content-type: application/json' \
       -d '{"query":"MATCH (h:HealthCheck {id:'\''smoke-1'\''}) RETURN count(h) AS c"}' | jq -c
   done   # every node must report the same count once replication settles
   

6. Failover drill. Kill the leader (docker stop patinadb-node1 or SIGTERM), confirm a survivor becomes leader (GET /mgmt/cluster), confirm writes resume, and restart the node — it rejoins and catches up. No acknowledged write is lost across a clean failover (Failover behaviour).

7. Multi-writer retry — only if your backend runs more than one replica. This does not test patinaDB; it tests your integration code, and it is the most likely place a multi-replica backend is subtly wrong. From your app (not curl), fire the same SET n.count = n.count + 1 on one node from several concurrent workers, each running the whole write through a managed transaction (execute_write / executeWrite), and assert the final count equals the number of writes. If it matches, your retry path works. If it comes up short — or you see Neo.TransientError.General.TransientError surfacing as an error to your caller — your writes are not wrapped in a managed transaction; fix that before go-live (see Concurrency & retries). The count is never wrong on the DB side; a shortfall means a conflict was dropped instead of retried.

Delete the :HealthCheck / :Doc / :Item smoke data afterward.

6. Backup & disaster recovery

The DR primitives are admin-only /mgmt/* REST endpoints, run against the leader (REST API). A backup is a portable, streamed dump of the whole registry (graph + constraints + compound / full-text / vector / point / edge-sorted indexes + WASM functions + tags + RBAC users + the anamnesis flag).

Take a backup (with time-travel history)

# Whole-cluster portable backup INCLUDING the engram timeline (PITR).
# ?history=true is a licensed (Pro/Enterprise) feature — 403 in Community.
curl -u neo4j:"$PW" 'http://<leader>:21001/mgmt/snapshot?history=true' -o patinadb-backup.json

Without ?history=true the backup is a HEAD-state dump (no AS OF over pre-backup history). Schedule this against the leader; a follower returns 503 naming the leader (the endpoint is leader-anchored/linearizable).

You do not need to quiesce writes. A backup is a consistent cut: every database’s HEAD label, its graph and its engram history are read from one pinned point in time, so a restored backup satisfies the same reconstruct(HEAD) == live graph invariant the source did, and its AS OF timeline is monotone — no matter how many writes land while the backup streams. Writes are briefly serialized only while that point is pinned (one O(1) pin per database), never for the duration of the stream: measured under issue #1105 on a 62 GiB/NVMe box, a 212 MB PITR backup taking 1.73 s moved a concurrent writer’s median latency from 3116 µs to 3333 µs (+7%); no stall resembling the backup’s own duration appeared. (That box was shared, so the measurement is honest about the median and about the absence of a stall — it is not precise enough to quote a tail-latency figure.) (Before #1105 the three parts were read at three different instants and a backup taken under load was silently torn — if you are on an older build, quiesce writes for the duration of the export.)

Watch the backup age. Every successfully completed backup stream (fully downloaded — an aborted download does not count) is stamped node-locally and exported as the Prometheus gauge patinadb_backup_last_success_timestamp_seconds (persisted across restarts; the series is absent until the first backup completes). The shipped alert rules PatinaDBBackupStale / PatinaDBBackupNeverRecorded page when the newest cluster-wide backup is older than your RPO or no backup ever finished — the “backup cron silently stopped” failure mode. See deploy/README.md → Backup & disaster recovery.

Restore

# Restore into a FRESHLY BOOTSTRAPPED, EMPTY cluster (leader). A non-empty
# target is refused (409) — and that refusal is the behaviour you want.
curl -u neo4j:"$PW" -X POST 'http://<leader>:21001/mgmt/restore' \
  -H 'content-type: application/json' --data-binary @patinadb-backup.json

Restore is additive, not a wipe — and ?force=true does not change that. /mgmt/restore replays the backup’s operations on top of whatever the target already holds; it never deletes existing data. The ?force=true flag only lifts the non-empty-target refusal so the restore is allowed to merge into a populated database — it is not “replace what’s there”. Restoring over a live, corrupted cluster therefore merges the backup into the corruption instead of replacing it. Always restore onto a freshly bootstrapped, empty cluster (see The whole cluster is lost under Recovery scenarios below and the Disaster Recovery chapter). This is stricter still for a ?history=true (PITR) backup: installing the history requires a pristine target — a database that has data or a timeline of its own is refused, with or without ?force=true.

Then verify with the shipped scripts/verify-restore.sh (readiness + node/edge counts + an optional AS OF probe), or re-run the §5 checklist.

PITR restore is replicated on peer protocol 21. Restore once through the leader after every member confirms support. Encoded history travels in bounded blocks and each applying replica verifies it before publication. Wait for followers to apply the restore and for newly joined learners to complete history backfill, then verify time-travel reads on each node. Older or unconfirmed clusters refuse the history restore before graph writes. Disaster Recovery describes disk/memory costs and the graph-only rollback boundary.

Portable CSV export. GET /mgmt/export?db=<name>&format=csv|parquet|arrow streams one database as a tar of neo4j-admin-style files for an external pipeline (REST API).

Recovery scenarios

• A node is lost. Bring up a replacement with the same --id and an empty --db, --join the cluster; it bootstraps from a streamed snapshot and catches up automatically — no manual restore. If the old voter is permanently gone, evict-voter it, add the replacement as a learner, and change-membership it in (licensed). See Removing a dead voter.

• The leader dies. Survivors auto-elect a new leader (election timeout 750–1500 ms); committed writes survive, in-flight uncommitted writes to the dead leader are retried by the driver. This is the automatic-failover guarantee the failover suite exercises.

• Logical corruption (a bad write). Prefer the live path: read the good state AS OF a pre-incident engram/tag and promote it back to HEAD with CALL patinadb.restore('<engram-id>') — no rebuild, no backup file, minutes of RTO (see Disaster Recovery → Bad write). If the bad write is older than the history-retention window (30 days on Community, unlimited on Pro/Enterprise), that path is gone — fall back to a ?history=true backup that predates the incident, restored onto a fresh, empty cluster as below. Do not restore over the live corrupted cluster: restore is additive, so you would merge the good backup into the bad state.

• The whole cluster is lost (all hosts/disks gone). Rebuild from your portable backup onto fresh, empty nodes:

  1. Provision new hosts; start node 1 with --bootstrap on an empty data directory (mandatory — restore is additive and a PITR restore refuses a non-pristine target outright).
  2. Copy the backup file onto the new node and restore it: POST /mgmt/restore --data-binary @patinadb-backup.json (no ?force — the target is empty).
  3. Add the remaining nodes as learners and promote them to voters; they pick up the restored HEAD graph via normal Raft replication.
  4. If you restored a ?history=true backup: the time-travel timeline is installed only on the node that ran the restore — re-run the restore on every node you need AS OF history on, or route time-travel reads to that node.
  5. Verify (counts, a known query, an AS OF probe for PITR backups) before repointing clients.

  The full step-by-step runbook, including the majority-lost case and the RPO consequences of your backup cadence, is in Disaster Recovery → Whole cluster lost.

The full disaster-recovery runbook — what each backup mechanism covers, the backup-cadence/RPO trade-off, per-failure RPO/RTO tables (including the majority-of-voters-lost case), the exact recovery procedures, and the post-restore verification checklist — is the Disaster Recovery chapter of this manual.

7. Monitoring & alerting

Every node exports Prometheus metrics at GET /metrics (auth-gated and global-Admin-only by default — the scrape needs an Admin credential). The repo ships a ready stack — deploy/prometheus.yml, deploy/prometheus/alerts.yml, and Grafana dashboards under deploy/grafana/ — documented in deploy/README.md.

Probes (Kubernetes / load balancer)

Both are auth-exempt and never shed by the concurrency limiter. /ready returns 503 with a reason (priority order): no_leader, no_quorum (this node leads but a quorum has not acknowledged it for over 5s, so it cannot commit — issue #1106), installing, degraded (community write-degrade), lagging (apply lag over --readiness-max-lag, default 50), shedding (concurrency limiter saturated). See Health vs. readiness probes and the Kubernetes example.

Key metrics to alert on

The shipped deploy/prometheus/alerts.yml defines these groups/rules:

The metric series behind them include: patinadb_raft_is_leader, patinadb_raft_replication_lag (this node’s own apply-backlog-of-known-log — does NOT catch a partitioned-but-reachable node, see below), patinadb_raft_has_leader (per-node “do I know a leader right now”, issue #532), patinadb_raft_follower_lag{follower_id} (leader-only, the genuine distance-behind-the-leader signal), patinadb_raft_apply_errors_total, patinadb_data_dir_available_bytes / _used_bytes / _total_bytes (the ENOSPC early-warning signal), patinadb_query_duration_seconds / patinadb_queries_total, patinadb_http_requests_total, the patinadb_cache_* fill/hit/eviction series, patinadb_mem_available_bytes, patinadb_entitlement_usage_ratio{axis} / _limit{axis}, and patinadb_resource_cap_rejections_total{cap} (issue #1153 — see below). Alertmanager is not wired by default — point the shipped rules at your own Alertmanager. The deploy/ Grafana dashboards are patinadb-cache.json and the cluster dashboard (patinadb-cluster.json, title “patinaDB Cluster”). For the full field reference see the “Metrics reference” table in deploy/README.md.

Watch the resource caps (patinadb_resource_cap_rejections_total)

patinaDB refuses a query that would exceed a configured resource bound rather than swapping or dying: the message names the knob to raise, and the error is a deterministic Neo.ClientError.Statement.ResourceExhausted (retryable: false — retrying changes nothing). Your client is told exactly what happened. You were not: until issue #1153 a cap hit reached /metrics only as a generic patinadb_queries_total{outcome="error"}, indistinguishable from a syntax error, so nothing showed that a workload keeps meeting a bound.

One series per cap, labelled with the environment variable that raises it, because the caps call for different actions:

The counters are monotonic since process start, so alert on the rate(). There is no shipped rule for them yet: what counts as “too often” depends on your workload, and a rule with a made-up threshold is worse than none.

Also useful operationally: GET /mgmt/cluster (live topology), GET /mgmt/queries (per-shape slow-query stats), GET /mgmt/dbsizes, GET /mgmt/cache, GET /mgmt/audit (node-local recent ring view backed by asynchronous durable storage; see audit caveats). Enable request tracing / an OTLP collector with --otel-endpoint (Request correlation & tracing).

8. Retention & disk

Your profile uses versioning, so the history retention window is an explicit operator decision:

• Unlimited retention (the licensed default). The engram history + snapshots keep the whole timeline — AS OF any past point works forever — but on-disk volume grows without bound over the deployment’s life. Budget disk accordingly and alert on patinadb_data_dir_available_bytes.

  Two different growth sources, budget both separately (issue #1098). “Volume grows without bound” above is about the delta history (one small record per write) — but every DEFAULT_SNAPSHOT_INTERVAL (50) commits, patinaDB ALSO takes a periodic full-graph snapshot (_engram_snapshots/*.snap, a complete copy of the graph at that point, used to bound AS OF/reconstruct replay depth). These snapshots are governed by a separate knob, PATINADB_MAX_SNAPSHOTS (unset/0 = unlimited — but the history-retention squash described below does not prune them: it only removes snapshots older than its cutoff, so within the retention window they still accumulate one file every ~50 commits). At a steady 10 writes/s that is a full-graph copy roughly every 5 seconds — measured (2026-08-20, Dataset on origin/main): 500 commits → 10 .snap files; 2000 commits → 40 files, 5.0 MiB. The corresponding TiB/day figure for a large graph is a calculation from that measured cadence, not itself measured — extrapolating (commits / 50) x graph-snapshot-size per day; budget disk against your own graph’s snapshot size, not just the delta-history rate quoted above. Pruning is strictly safety-preserving (AS OF/reconstruct of every engram stays byte-identical; only reconstruction whose nearest snapshot was pruned falls back to a longer delta replay — a performance cost, never a correctness or availability one), so bounding it costs nothing but occasional extra replay depth. The server binary defaults PATINADB_MAX_SNAPSHOTS=16 when you don’t set it (embedded/CLI keep the unlimited default — set it explicitly there if you want the same bound); raise it if you need a deeper local replay-speed cushion, or set 0 to opt back into unlimited .snap retention.

  The third line item: what a rejoining node has to transfer (issue #1099). RFC-0025 update (#1251, PR #1259): on a cluster eligible for physical v5, onboarding uses a state-only checkpoint including indexes plus the anchor history tail. Older history is backfilled separately. The historical v3 measurements below describe the earlier logical format, not current v5 wire size or onboarding throughput. A prepared checkpoint survives restart; its build and receiver installation still consume disk and I/O. Track current state catch-up separately from historical completeness.

  The two sources above are about disk on a running node. A Raft snapshot is a separate artifact — the file a rejoining or freshly bootstrapped node pulls — and openraft builds one every 5000 log entries, per node.

  It used to carry the periodic .snap graph-snapshot bodies as well, so its size tracked whatever PATINADB_MAX_SNAPSHOTS allowed to accumulate. Since #1099 it does not: those bodies are a reconstruction cache the receiving node rebuilds from the metadata and deltas it does get, so what a snapshot ships is one copy of the graph plus the change log. Measured on a 60 000-vertex fixture with .snap retention unlimited (the pre-#1098 default, which is still the embedded/CLI default): a further 400 commits over an unchanged graph used to add 4.7 MB to every snapshot and now add 69 kB. With the server’s PATINADB_MAX_SNAPSHOTS=16 the old cost was bounded rather than unbounded — at most ~16 full-graph copies per snapshot — so read that 4.7 MB as the unbounded-retention case and scale it to your own bound.

  The trade is that a first AS OF into a region the newly-bootstrapped node has not cached replays deltas instead of loading a cached snapshot — slower once, never wrong.

  The receive cap is not a sizing prerequisite (issue #1246). PATINADB_MAX_SNAPSHOT_BYTES (default 8 GiB) windows a single upload attempt, not the snapshot: a snapshot larger than the cap — measured in the #1145 rejoin drill: a 71 GiB graph yields a ~19.5 GiB compressed snapshot — installs over several probe/resume attempts. You do not have to raise the cap to fit your graph. What you SHOULD alert on is patinadb_raft_snapshot_receive_rejected_total{reason="over_cap"} climbing without the follower catching up: before #1246 that exact shape was a silent livelock (166 aborts, ~1.45 TB retransmitted, one node that never rejoined), and any regression of it is only visible in that metric.

  Do not extrapolate any of these figures: they depend on your graph size, your commit rate and your PATINADB_MAX_SNAPSHOTS together. Size each line item from your own measured growth, alert on patinadb_data_dir_available_bytes, and include snapshot transfer time in your expectations for how long a node takes to rejoin. See PATINADB_SNAPSHOT_LOGS_SINCE_LAST in Configuration for how often a build happens.

• A bounded window (via a license that sets history_retention_days, or Community’s forced 30-day cap). A background best-effort task periodically squashes engrams older than the cutoff, reclaiming disk — but time-travel / AS OF / diff beyond the window is then unavailable. See History retention in practice.

CALL patinadb.engrams() is unbounded BY DEFAULT (issue #1114). With no arguments it returns the ENTIRE engram history in one response — exactly as unbounded as the retention window above. At the recommended unlimited-retention profile (~10 writes/s), a month of history is already ~26M engrams: one operator/admin-tool CALL after weeks of otherwise-fine operation is a multi-GB response, not something triggered by load. Prefer CALL patinadb.engrams(limit) (newest first, at most limit rows, O(limit) walk + memory) for any script or tool that does not genuinely need the whole history in one call. Page further back with CALL patinadb.engrams(limit, before), where before is the id of the LAST (oldest) row a previous call returned — chained calls tile with no overlap and no gap. The bare no-args call stays available (and stays byte-identical to before this issue) for tooling that genuinely wants everything at once.

Caveat — retention under sustained time-travel read load. The background squash and time-travel reads are coordinated so a squash-delete can never tear an in-flight historical read (correctness is preserved — you never get a wrong historical result). The trade-off is that under a continuous stream of overlapping time-travel reads, the squash writer can be starved / delayed — it waits for a quiescent gap before deleting. Correctness is always preserved (you never get a wrong or torn historical read); the trade-off is squash lag, and a periodic squash runs on the same serial state-machine apply path, so a prolonged starvation could delay apply under this pathological load. Tightening that coordination (a bounded wait / fairness so squash can’t be starved indefinitely) is a tracked hardening item — issue #499. Recommendation for this profile: prefer unlimited retention (the licensed default) and manage disk with capacity + backups rather than an aggressive squash window — that sidesteps the interaction entirely. Only enable a bounded window if your disk budget requires it and your time-travel read load is light, or after #499 is closed.

Time-travel reads in this profile

Your backend uses AS OF, so two operational facts matter more here than the defaults suggest:

1. An unknown or squashed engram id now errors, it does not read as the empty graph (changed in issue #590). AS OF an id that is not in the database’s history (never existed, or squashed away by retention) fails with an actionable unknown engram … error, so a backend that stores engram ids and reads them back later can distinguish “that point in history is gone” from “nothing matched”. Even so, use tags for any point you intend to read again: a tag is pinned against squash and snapshotted, so it stays both reachable and cheap. See Time Travel.
2. The read-path knobs are safe to flip; the results never change. PATINADB_PARTIAL_TIMETRAVEL (default on) reconstructs only the subgraph a read touches; =0 is the rollback valve if you ever suspect it — every read then takes the full reconstruct, slower but byte-identical. PATINADB_INDEXED_SNAPSHOTS (default off) is worth enabling only if your backend re-reads a known set of historical points repeatedly; its sidecar is built lazily in the background and costs disk, so bound it with PATINADB_MAX_SNAPSHOT_INDEXES (default 16). Neither touches the write path. Full picture: Performance & tuning.

Loading the initial dataset

Seed the cluster before go-live rather than through a per-row Bolt write — a CREATE per row is the slowest way to fill a graph. For a large columnar file (or a coupled node+edge subgraph) already on the leader’s disk, use POST /mgmt/upsert/upsert-edges/upsert-graph (index-accelerated match-or-create, chunkable — see Bulk Upsert); for a streamed CSV, use LOAD CSV … CALL { … } IN TRANSACTIONS OF n ROWS (commits in bounded chunks instead of one giant transaction). Both, plus the throughput you should expect and the reason a single unbounded LOAD CSV … CREATE is a memory cliff, are in Bulk Loading & Import.

Every path above is versioned — the initial load is one changeset (or one per chunk) in the engram timeline, so it is a valid AS OF point right away. Take a tag right after the load (CREATE TAG initial-load) so there is a named point to read back to.

9. Known limitations for this profile

Read Limitations in full; the ones that bear on a 3-node Bolt cluster doing CRUD + versioning + search:

• Bolt transactions are snapshot isolation, not serializable — write skew is possible; use single autocommit statements or app-level guards for cross-row invariants, and retry the transient conflict error (The server).
• Bolt reads are always local (eventually consistent on followers). The linearizable read is REST-only; use the AS OF TAG clean-tag pattern for a lag-immune, follower-servable read.
• RBAC enforcement is reject, not row-filtering, and per-property / relationship-type RBAC is not implemented (moot for a single trusted tenant).
• Security audit persistence is asynchronous and node-local, not Raft-replicated. Committed entries survive restart, but a hard crash can lose the uncommitted tail; queue saturation or storage failures can prevent persistence. Ship patinadb::audit to a central sink and verify that pipeline’s delivery guarantees. See Security audit log.
• No encryption-at-rest — use OS-level disk encryption on the --db volume.
• Full-text: string properties only; read-modify-write postings (not for very high write throughput); prefix/fuzzy capped. Vector: IVF-Flat ANN (Full-Text Search, Vector Search).
• CREATE EDGE SORTED INDEX definitions are not carried in Raft snapshots — a node bootstrapped purely from a streamed snapshot falls back to traverse+sort for that shape until the DDL is re-issued (correctness unaffected).
• On-disk format upgrades may require a dump/reload: export via GET /mgmt/snapshot, upgrade the binaries, import — plan this into your rolling upgrade (Storage & scale, Rolling upgrades).

The parser / DoS input-hardening caveats in the limitations chapter concern untrusted input and are not relevant to this single-trusted-backend profile.