Production Deployment (3-Node Bolt Cluster)

This is an operator walkthrough for one specific, common deployment: a 3-node patinadb-raft Raft cluster with automatic failover, fronting a single trusted backend service that connects over Bolt with a Neo4j driver, and using graph CRUD + Cypher, versioning / time-travel (AS OF, engrams, diffs, tags) and search (full-text + vector indexes).

The threat model is a trusted, single-tenant backend — the only client is your own application, not arbitrary internet users. So this is a deploy + secure

• operate guide, not an untrusted-input hardening guide.

It ties together the reference chapters — High Availability, Authentication & TLS, Day-2 Operations, Editions & Limits, Configuration, Bolt, REST API and Time Travel — into one go-live sequence. Read those for depth; this chapter is the checklist.

1. Prerequisites & licensing

A production 3-node cluster requires a commercial license. Read this before anything else — it changes how you bring the cluster up.

The Community edition (the default with no valid license) caps max_voters at 1 (Editions & Limits). That has two concrete consequences for a 3-node cluster:

• POST /mgmt/change-membership refuses any voter set larger than 1 with 409 VoterCapExceeded. That endpoint is the documented way to grow a --bootstrap node into a 3-voter cluster and to replace a dead voter later (evict-voter → add-learner → change-membership) — a routine Day-2 op. In Community mode both are blocked.
• Community mode also enforces mandatory telemetry: if the node cannot phone home within the grace window (default 72 h), it degrades to read-only (writes return 503, reads keep working). --disable-telemetry is refused at startup without a valid license (fail-closed), so an isolated / air-gapped deployment cannot opt out. See Licensing & Telemetry.
• Community additionally caps ~5,000,000 nodes + edges (writes degrade to read-only past the cap) and 30-day rolling history retention (older time-travel history is squashed away — see §8).

A valid license lifts every one of these: max_voters becomes unlimited (or your tier’s cap), telemetry becomes best-effort and disableable, the scale cap is raised or removed, and retention defaults to unlimited. Install a license on every node before forming the cluster — see Installing a license and Getting a license. Precedence: --license / PATINADB_LICENSE (a file path or an inline token) → <db-root>/license.key.

Confirm the tier on each node after boot with the auth-exempt GET /version:

curl -s http://<node>:21001/version | jq '{tier, entitlements, usage}'
# tier must read "licensed" — "community" means the node found no valid license.

Honest nuance. You can technically stand up a fixed 3-voter set in one shot with a single POST /mgmt/init {"1":addr1,"2":addr2,"3":addr3} — that path calls Raft initialize directly and does not consult the voter cap. But you would then be unable to change-membership (add or replace a voter) without a license, and remain subject to the telemetry / scale / retention caps. It is not a viable production configuration — get a license.

Sizing

Disk sizing

Each node stores the whole graph (every voter has a full replica), its Raft log, periodic snapshots, and the engram history (the versioning timeline). With unlimited retention (the licensed default) the engram history and snapshots grow without bound over the deployment’s lifetime — budget for it, and read §8 to decide between unlimited history and a bounded retention window. Back up the entire --db directory as a unit (graph, Raft log, snapshots, engram log); each database is a subdirectory of --db (On-disk layout).

Monitor free space with the patinadb_data_dir_available_bytes gauge and the shipped PatinaDBDiskSpaceLow / PatinaDBDiskSpaceCritical alerts (§7).

2. Bring up the 3-node cluster

A cluster forms by bootstrapping node 1, joining nodes 2 and 3 as learners, then promoting all three to voters. The promotion (change-membership) is the step that requires a license (§1).

With Docker Compose

The repository ships a 3-node stack at deploy/docker-compose.yml (plus Prometheus + Grafana). Treat it as a frictionless local demo, not a production template: it runs --insecure-disable-auth with no license, and its cluster-init step promotes to a 3-voter set via change-membership [1,2,3] — which an unlicensed node refuses with 409, so a plain unlicensed docker compose up leaves node 1 leading with nodes 2 and 3 as learners, not a 3-voter HA set. For production, harden it: add a license, turn auth on, and mount secrets as files.

# production 3-node compose (adapted from deploy/docker-compose.yml).
name: patinadb-prod

x-node: &node
  image: your-registry/patinadb-raft:<version>   # build from the repo Dockerfile
  restart: unless-stopped
  networks: [cl]
  environment:
    # Secrets mounted as files (see §3 "Secrets"); never bake them into the image.
    PATINADB_AUTH_PASSWORD_FILE: /run/secrets/auth_password
    PATINADB_CLUSTER_SECRET_FILE: /run/secrets/cluster_secret
    PATINADB_LICENSE_FILE: /run/secrets/license   # REQUIRED for a 3-voter cluster
  healthcheck:
    # distroless has no shell/curl; the binary is its own probe.
    test: ["CMD", "/usr/local/bin/patinadb-raft", "--healthcheck", "http://127.0.0.1:21001/ready"]
    interval: 30s
    timeout: 5s
    start_period: 20s
    retries: 3

services:
  node1:
    <<: *node
    container_name: patinadb-node1
    command:
      - --id=1
      - --addr=patinadb-node1:21001
      - --bolt-addr=0.0.0.0:7687
      - --advertised-addr=patinadb-node1.internal:7687   # a reachable host:port
      - --db=/data
      - --bootstrap
    ports: ["21001:21001", "7687:7687"]
    volumes: ["node1-data:/data", "./secrets:/run/secrets:ro"]

  node2:
    <<: *node
    container_name: patinadb-node2
    command:
      - --id=2
      - --addr=patinadb-node2:21001
      - --bolt-addr=0.0.0.0:7687
      - --advertised-addr=patinadb-node2.internal:7687
      - --db=/data
      - --join=patinadb-node1:21001
    ports: ["21002:21001", "7688:7687"]
    volumes: ["node2-data:/data", "./secrets:/run/secrets:ro"]

  node3:
    <<: *node
    container_name: patinadb-node3
    command:
      - --id=3
      - --addr=patinadb-node3:21001
      - --bolt-addr=0.0.0.0:7687
      - --advertised-addr=patinadb-node3.internal:7687
      - --db=/data
      - --join=patinadb-node1:21001
    ports: ["21003:21001", "7689:7687"]
    volumes: ["node3-data:/data", "./secrets:/run/secrets:ro"]

networks: { cl: { driver: bridge } }
volumes: { node1-data: , node2-data: , node3-data: }

--join registers nodes 2 and 3 as learners automatically. Once all three report their peer versions, promote them to voters against the leader (substitute your password):

curl -u neo4j:"$PW" -X POST http://patinadb-node1:21001/mgmt/change-membership \
  -H 'content-type: application/json' -d '[1,2,3]'
# → {"ok": true}   (a 409 here means Community mode / no license — see §1)

Notes. The demo --advertised-addr values are 127.0.0.1:7687/7688/7689 because everything runs on one host. On a real multi-host cluster set --advertised-addr to each node’s reachable host:port so neo4j:// routing sends drivers somewhere they can connect (§4). The distroless image runs as uid 65532; a fresh bind-mounted host volume is root-owned — chown -R 65532:65532 it, use a Docker named volume, or set the Kubernetes securityContext.fsGroup: 65532.

Without Docker (bare processes / systemd)

Equivalent to the compose above, one process per host:

# Node 1 — bootstrap leader
patinadb-raft --id 1 --addr node1.internal:21001 --db /var/lib/patinadb --bootstrap \
  --bolt-addr 0.0.0.0:7687 --advertised-addr node1.internal:7687 \
  --license /etc/patinadb/license.key
  # PATINADB_AUTH_PASSWORD, PATINADB_CLUSTER_SECRET set in the unit's environment

# Nodes 2 and 3 — auto-join as learners
patinadb-raft --id 2 --addr node2.internal:21001 --db /var/lib/patinadb --join node1.internal:21001 \
  --bolt-addr 0.0.0.0:7687 --advertised-addr node2.internal:7687 --license /etc/patinadb/license.key
patinadb-raft --id 3 --addr node3.internal:21001 --db /var/lib/patinadb --join node1.internal:21001 \
  --bolt-addr 0.0.0.0:7687 --advertised-addr node3.internal:7687 --license /etc/patinadb/license.key

Then promote to a 3-voter set (as above). If you prefer to skip the bootstrap → learner → promote sequence, a single POST /mgmt/init forms the set in one shot — but re-read the §1 nuance about the voter cap:

curl -u neo4j:"$PW" -X POST http://node1.internal:21001/mgmt/init \
  -H 'content-type: application/json' \
  -d '{"1":"node1.internal:21001","2":"node2.internal:21001","3":"node3.internal:21001"}'

Verify the cluster formed

# One leader + three voters, from any node:
curl -u neo4j:"$PW" -s http://node1.internal:21001/mgmt/cluster | jq

# Every node is ready (auth-exempt):
for n in node1 node2 node3; do curl -s http://$n.internal:21001/ready | jq -c; done

For growing/shrinking membership, failover behaviour, and rolling upgrades see High Availability.

3. Secure it

The default posture is fail-closed: an empty --auth-password refuses to start unless you also pass --insecure-disable-auth (Authentication). Work through the security deployment checklist; the essentials for this profile:

Backend ↔ database link (Bolt auth + TLS)

• Auth. Set a strong --auth-password (the root-admin credential; REST Basic
  • Bolt LOGON). Your backend authenticates over Bolt at LOGON. For least privilege, create a dedicated user with only the roles it needs rather than logging in as neo4j (Users & roles).
• TLS. Native --tls-cert / --tls-key (PEM) secures the HTTP plane (REST + management + peer RPC) and the Bolt endpoint (bolt+s:// / wss://) with the same certificate — connect the driver with the +s scheme. It is opt-in; without it everything is plaintext, so a plaintext node logs a loud cleartext-credentials warning at startup. If you terminate TLS at a reverse proxy instead, set --advertised-addr to the public endpoint. See Native TLS for the HTTP plane and Reverse-proxy termination.

Internal cluster traffic (cluster secret + peer TLS)

• Cluster secret. Peer /raft/* RPCs authenticate with a dedicated --cluster-secret (PATINADB_CLUSTER_SECRET) carried in the X-Cluster-Secret header. All nodes must share the same value. If left empty it falls back to --auth-password, but a dedicated secret lets you rotate the two independently.
• Peer TLS. For a self-signed / private-CA cluster, give every node --tls-ca <ca.pem> so peers verify each other; the cert’s SANs must cover the peer --addr hosts.

Secrets (files & managers)

Do not pass secrets as plain env vars (they leak into docker inspect / /proc/<pid>/environ). Use the file or command conventions (Secrets management) for PATINADB_AUTH_PASSWORD, PATINADB_CLUSTER_SECRET, and PATINADB_LICENSE:

• <VAR>_FILE — mount the secret as a file: PATINADB_AUTH_PASSWORD_FILE=/run/secrets/auth_password.
• <VAR>_COMMAND — fetch from a secrets manager: PATINADB_AUTH_PASSWORD_COMMAND="vault kv get -field=pw secret/patinadb".

Precedence: direct env/flag > <VAR>_FILE > <VAR>_COMMAND > config file > default. All three are fail-closed — an unreadable file or a failing command is a loud startup error, never a silent “no secret”.

Online rotation

Rotate without a full restart (admin-only, per node): stage a new cluster secret on every node with POST /mgmt/rotate-cluster-secret {"secret":…} (both accepted during the grace window), then POST /mgmt/retire-cluster-secret on every node; rotate the root password with POST /mgmt/rotate-admin-password {"password":…}.

Bind & expose deliberately

Bind --addr and --bolt-addr to the private interfaces the backend and peers reach — not 0.0.0.0 on a public interface. /metrics is authenticated by default (its db=<name> labels leak database names); keep it that way, or open it only on a private monitoring network with --insecure-open-metrics. /health, /ready and /version are always open probes.

Multi-tenant note (usually N/A here). This profile is single-tenant, so the default open RBAC is fine. If you later host multiple isolated tenants, enable --rbac-closed (closed-mode tenant isolation).

4. Connect the backend

Point your Neo4j driver at the cluster. Prefer the routing scheme so reads spread across followers and writes go to the leader automatically (Cluster-aware routing):

neo4j://<any-node-host>:7687      # or neo4j+s:// with TLS

• neo4j:// vs bolt://. neo4j:// asks the cluster for a routing table and load-balances: writes → leader, reads → followers/learners, with automatic reconnection on failover. bolt:// pins to one node (no routing). Use neo4j:// (or neo4j+s:// with TLS). Routing hands drivers each node’s --advertised-addr, so those must be reachable from the backend.
• Read consistency. A default read serves the node’s local applied state (causally consistent within a database, eventually consistent across replicas — a follower may lag). The Bolt path always reads locally; the "consistency":"linearizable" opt-in is REST-only and leader-only. When you need a lag-immune read that any follower can serve, read AS OF TAG '<name>' (a replicated, deterministic point — the “clean-tag” pattern in Read consistency).
• Transactions are snapshot isolation, not serializable. BEGIN … COMMIT gets repeatable reads and first-committer-wins on write-write conflicts (a conflict raises the driver-retryable Neo.TransientError.Transaction.LockClientStopped — wrap writes in a retry loop). SI does not prevent write skew; for an invariant spanning rows one tx reads and another writes, use a single autocommit statement or an app-level guard. See Transactions.
• Pool & limits. Size the driver’s connection pool below the server’s --max-bolt-connections (default 1024) — a cap on authenticated connections; an unauthenticated socket (silent, stalled mid-handshake, or idling after HELLO) never spends this budget (issue #440), instead bounded in time by --bolt-handshake-timeout-secs (default 10) and --bolt-idle-timeout-secs (default 300). Tune the tx guards for your workload: --max-tx-ops (default 5,000,000 buffered ops per explicit tx) and --idle-tx-timeout-secs (default 300 — an abandoned BEGIN is rolled back and its connection closed). A client that RUNs a streaming read then never PULLs is reaped after --stream-pull-timeout-secs (default 300). See Connection resource limits and Transaction resource limits.

Concurrency & retries

If your backend runs multiple replicas, read this first. A horizontally scaled backend — say 3 app replicas behind a load balancer — means two replicas can issue a write to the same node or edge at the same time. That is a supported, everyday topology, and patinaDB does not lose an update or create a duplicate under it: the server detects the collision and lets exactly one writer through. But it handles the collision by rejecting the other writer with a retryable error, not by making it wait — so the one thing your backend MUST do is retry that error, on every replica, for every write. The mechanism below is how, and it is a requirement for a multi-writer deployment, not a recommendation. A backend that treats the retryable conflict as a hard failure will see write failures under contention (never lost or corrupted data — but avoidable failures). This is verified end-to-end: 16 concurrent clients hammering one counter over autocommit land every increment exactly once, and two replicas creating the same UNIQUE-constrained key collide so that exactly one succeeds (pipelined_bolt_autocommit_contended_counter_never_loses_an_update, bolt_occ_duplicate_unique_create_conflicts).

You do not lock the database. patinaDB has no pessimistic-lock primitive (no SELECT … FOR UPDATE) and does not need one. Concurrency control is optimistic and works out of the box:

• Ordinary write–write conflicts (two requests updating the same node/edge) are detected at COMMIT: the first committer wins, the second gets the driver-retryable Neo.TransientError.Transaction.LockClientStopped. Use your driver’s managed transactions — session.execute_write(tx_fn) (Python) / session.executeWrite(...) (JS/Java) — which retry a transient automatically. That is the whole mechanism: concurrent updates to the same data are serialized and retried with no locking and no lost updates.

  # REQUIRED for a multi-writer backend: the driver retries the transient for you.
  def bump(tx, node_id):
      tx.run("MATCH (n {id:$id}) SET n.count = n.count + 1", id=node_id)
  with driver.session() as s:
      s.execute_write(bump, node_id)   # auto-retries LockClientStopped
  
  # WRONG for a multi-writer backend: a naked autocommit run treats the
  # conflict as a hard error. No data is lost or corrupted, but under
  # contention between replicas this call *fails* instead of retrying.
  # with driver.session() as s:
  #     s.run("MATCH (n {id:$id}) SET n.count = n.count + 1", id=node_id)
  

  The same holds for MERGE on a UNIQUE-constrained key — the classic “upsert this user from whichever replica saw the request first”. Two replicas running MERGE (u:User {id:$id}) … concurrently collide on the value, one gets the retryable transient, and a managed transaction converges it to a single node. Wrap it in execute_write and it is safe across any number of replicas; run it naked and a simultaneous upsert can fail.

• Write skew is the only case OCC does not catch (snapshot isolation is not serializable): two transactions each read a value and write disjoint rows based on it, so there is no write–write overlap to detect. This matters only for a multi-row invariant that genuinely requires serializability (e.g. “the sum of two balances must stay ≥ 0”, debited by two txns on different rows). The fix is not a lock — force the skew into a detectable write–write conflict by having both transactions write a shared sentinel/version node:

  // Both txns touch the same guard row → OCC now detects the conflict → retry.
  MATCH (g:AccountGroup {id:$gid}) SET g.version = g.version + 1
  

  If your backend has no such cross-row invariant (the common case), you need none of this — managed transactions are sufficient.

What makes this work: pipelined writes (PATINADB_PIPELINED_WRITES, default on). Concurrent autocommit writes to one database resolve in parallel and are admitted under the optimistic check above, rather than serializing behind each other’s whole Raft round-trip — which is why a conflict between two concurrent writers surfaces as the retryable transient instead of being avoided by making everyone queue. Both explicit transactions and plain autocommit statements go through it, so the retry advice above applies to both. PATINADB_PIPELINED_WRITES=0 is the rollback valve: writes then serialize per database — no lost updates either way, but ~4–5× less write throughput on a disjoint-write workload, and note this does not buy serializability (Bolt’s explicit transactions were already snapshot-isolated). See Configuration.

5. Verify (go-live smoke checklist)

Run these against the cluster before cutting traffic over. Substitute your Bolt URL / credentials; the examples use Cypher you can run from the driver or via POST /cypher.

0. Validate the config first — before the node even boots. On each node, run the exact command line / env / config file you deploy with, plus --check-config. It resolves everything the way a real boot does, prints what it understood, and exits non-zero if a setting is invalid or a runtime env knob is set to a value the engine would silently ignore (e.g. a PATINADB_MAX_AGG_ROWS=5M that the runtime wants as 5000000). This catches the misconfiguration class that otherwise only shows up as surprising behaviour in production (Config dry run):

   patinadb-raft --config node.yaml --check-config || { echo "bad config"; exit 1; }
   

   Confirm the printed cache budget shows the caches ENABLED with the memory you intend — a typo’d --cache-limit silently disabling the cache is exactly what this step exists to catch.

1. Write + read a node (exercise the write path through Raft):

   CREATE (:HealthCheck {id: 'smoke-1', at: datetime()});
   MATCH (h:HealthCheck {id: 'smoke-1'}) RETURN h.at;
   

2. Time-travel read (AS OF). Capture the current engram, mutate, then read the past state (Time Travel):

   // note HEAD first
   CALL patinadb.engrams() YIELD id RETURN id ORDER BY id DESC LIMIT 1;
   // ... then, after a later write, read the earlier state:
   USE default AS OF '<engram-id>' MATCH (h:HealthCheck) RETURN count(h);
   

3. Full-text search (Full-Text Search):

   CREATE FULLTEXT INDEX docs FOR (n:Doc) ON EACH [n.body];
   CREATE (:Doc {body: 'patinadb production go-live'});
   CALL db.index.fulltext.queryNodes('docs', 'production') YIELD node, score RETURN node.body, score;
   

4. Vector search (Vector Search) — note the arg order queryNodes(indexName, k, queryVector):

   CREATE VECTOR INDEX emb FOR (n:Item) ON (n.vec)
     OPTIONS { indexConfig: { `vector.dimensions`: 3, `vector.similarity_function`: 'cosine' } };
   CREATE (:Item {vec: [0.1, 0.2, 0.3]});
   CALL db.index.vector.queryNodes('emb', 5, [0.1, 0.2, 0.3]) YIELD node, score RETURN node, score;
   

5. Cross-node consistency. After a write on the leader, confirm every node converges (each node serves its own applied state):

   for n in node1 node2 node3; do
     curl -u neo4j:"$PW" -s -X POST http://$n.internal:21001/cypher \
       -H 'content-type: application/json' \
       -d '{"query":"MATCH (h:HealthCheck {id:'\''smoke-1'\''}) RETURN count(h) AS c"}' | jq -c
   done   # every node must report the same count once replication settles
   

6. Failover drill. Kill the leader (docker stop patinadb-node1 or SIGTERM), confirm a survivor becomes leader (GET /mgmt/cluster), confirm writes resume, and restart the node — it rejoins and catches up. No acknowledged write is lost across a clean failover (Failover behaviour).

7. Multi-writer retry — only if your backend runs more than one replica. This does not test patinaDB; it tests your integration code, and it is the most likely place a multi-replica backend is subtly wrong. From your app (not curl), fire the same SET n.count = n.count + 1 on one node from several concurrent workers, each running the whole write through a managed transaction (execute_write / executeWrite), and assert the final count equals the number of writes. If it matches, your retry path works. If it comes up short — or you see Neo.TransientError.Transaction.LockClientStopped surfacing as an error to your caller — your writes are not wrapped in a managed transaction; fix that before go-live (see Concurrency & retries). The count is never wrong on the DB side; a shortfall means a conflict was dropped instead of retried.

Delete the :HealthCheck / :Doc / :Item smoke data afterward.

6. Backup & disaster recovery

The DR primitives are admin-only /mgmt/* REST endpoints, run against the leader (REST API). A backup is a portable, streamed dump of the whole registry (graph + full-text + vector + constraints + point indexes + tags + RBAC users).

Take a backup (with time-travel history)

# Whole-cluster portable backup INCLUDING the engram timeline (PITR).
# ?history=true is a licensed (Pro/Enterprise) feature — 403 in Community.
curl -u neo4j:"$PW" 'http://<leader>:21001/mgmt/snapshot?history=true' -o patinadb-backup.json

Without ?history=true the backup is a HEAD-state dump (no AS OF over pre-backup history). Schedule this against the leader; a follower returns 503 naming the leader (the endpoint is leader-anchored/linearizable).

Restore

# Restore into a cluster (leader). A non-empty target is refused (409) without ?force=true.
curl -u neo4j:"$PW" -X POST 'http://<leader>:21001/mgmt/restore?force=true' \
  -H 'content-type: application/json' --data-binary @patinadb-backup.json

Then verify with the shipped scripts/verify-restore.sh (readiness + node/edge counts + an optional AS OF probe), or re-run the §5 checklist.

Portable CSV export. GET /mgmt/export?db=<name>&format=csv|parquet|arrow streams one database as a tar of neo4j-admin-style files for an external pipeline (REST API).

Recovery scenarios

• A node is lost. Bring up a replacement with the same --id and an empty --db, --join the cluster; it bootstraps from a streamed snapshot and catches up automatically — no manual restore. If the old voter is permanently gone, evict-voter it, add the replacement as a learner, and change-membership it in (licensed). See Removing a dead voter.
• The leader dies. Survivors auto-elect a new leader (election timeout 750–1500 ms); committed writes survive, in-flight uncommitted writes to the dead leader are retried by the driver. This is the automatic-failover guarantee the failover suite exercises.
• Logical corruption (a bad write). Restore from a ?history=true backup, or read the good state AS OF a pre-incident engram/tag and promote it.

The repository’s docs/disaster-recovery.md carries the full RPO/RTO-per-failure runbook; ask your patinaDB contact for it if you don’t have the repo.

7. Monitoring & alerting

Every node exports Prometheus metrics at GET /metrics (auth-gated by default). The repo ships a ready stack — deploy/prometheus.yml, deploy/prometheus/alerts.yml, and Grafana dashboards under deploy/grafana/ — documented in deploy/README.md.

Probes (Kubernetes / load balancer)

Both are auth-exempt and never shed by the concurrency limiter. /ready returns 503 with a reason (priority order): no_leader, installing, degraded (community write-degrade), lagging (apply lag over --readiness-max-lag, default 50), shedding (concurrency limiter saturated). See Health vs. readiness probes and the Kubernetes example.

Key metrics to alert on

The shipped deploy/prometheus/alerts.yml defines these groups/rules:

The metric series behind them include: patinadb_raft_is_leader, patinadb_raft_replication_lag (this node’s own apply-backlog-of-known-log — does NOT catch a partitioned-but-reachable node, see below), patinadb_raft_has_leader (per-node “do I know a leader right now”, issue #532), patinadb_raft_follower_lag{follower_id} (leader-only, the genuine distance-behind-the-leader signal), patinadb_raft_apply_errors_total, patinadb_data_dir_available_bytes / _used_bytes / _total_bytes (the ENOSPC early-warning signal), patinadb_query_duration_seconds / patinadb_queries_total, patinadb_http_requests_total, the patinadb_cache_* fill/hit/eviction series, patinadb_mem_available_bytes, and patinadb_entitlement_usage_ratio{axis} / _limit{axis}. Alertmanager is not wired by default — point the shipped rules at your own Alertmanager. The deploy/ Grafana dashboards are patinadb-cache.json and the cluster dashboard (cypherlite-cluster.json, title “patinaDB Cluster”). For the full field reference see the “Metrics reference” table in deploy/README.md.

Also useful operationally: GET /mgmt/cluster (live topology), GET /mgmt/queries (per-shape slow-query stats), GET /mgmt/dbsizes, GET /mgmt/cache, GET /mgmt/audit (security audit — node-local, in-memory). Enable request tracing / an OTLP collector with --otel-endpoint (Request correlation & tracing).

8. Retention & disk

Your profile uses versioning, so the history retention window is an explicit operator decision:

• Unlimited retention (the licensed default). The engram history + snapshots keep the whole timeline — AS OF any past point works forever — but on-disk volume grows without bound over the deployment’s life. Budget disk accordingly and alert on patinadb_data_dir_available_bytes.
• A bounded window (via a license that sets history_retention_days, or Community’s forced 30-day cap). A background best-effort task periodically squashes engrams older than the cutoff, reclaiming disk — but time-travel / AS OF / diff beyond the window is then unavailable. See History retention in practice.

Caveat — retention under sustained time-travel read load. The background squash and time-travel reads are coordinated so a squash-delete can never tear an in-flight historical read (correctness is preserved — you never get a wrong historical result). The trade-off is that under a continuous stream of overlapping time-travel reads, the squash writer can be starved / delayed — it waits for a quiescent gap before deleting. Correctness is always preserved (you never get a wrong or torn historical read); the trade-off is squash lag, and a periodic squash runs on the same serial state-machine apply path, so a prolonged starvation could delay apply under this pathological load. Tightening that coordination (a bounded wait / fairness so squash can’t be starved indefinitely) is a tracked hardening item — issue #499. Recommendation for this profile: prefer unlimited retention (the licensed default) and manage disk with capacity + backups rather than an aggressive squash window — that sidesteps the interaction entirely. Only enable a bounded window if your disk budget requires it and your time-travel read load is light, or after #499 is closed.

Time-travel reads in this profile

Your backend uses AS OF, so two operational facts matter more here than the defaults suggest:

1. An unknown or squashed engram id reads as the empty graph, not an error. A backend that stores engram ids and reads them back later cannot distinguish “that point in history is gone” from “nothing matched”. Existence-check with CALL patinadb.engrams() before trusting an empty historical result, or — better for this profile — use tags for any point you intend to read again: a tag is pinned against squash and snapshotted, so it stays both reachable and cheap. See Time Travel. This is the single most likely time-travel surprise in a backend integration.
2. The read-path knobs are safe to flip; the results never change. PATINADB_PARTIAL_TIMETRAVEL (default on) reconstructs only the subgraph a read touches; =0 is the rollback valve if you ever suspect it — every read then takes the full reconstruct, slower but byte-identical. PATINADB_INDEXED_SNAPSHOTS (default off) is worth enabling only if your backend re-reads a known set of historical points repeatedly; its sidecar is built lazily in the background and costs disk, so bound it with PATINADB_MAX_SNAPSHOT_INDEXES (default 16). Neither touches the write path. Full picture: Performance & tuning.

Loading the initial dataset

Seed the cluster before go-live rather than through a per-row Bolt write — a CREATE per row is the slowest way to fill a graph. For a large columnar file (or a coupled node+edge subgraph) already on the leader’s disk, use POST /mgmt/upsert/upsert-edges/upsert-graph (index-accelerated match-or-create, chunkable — see Bulk Upsert); for a streamed CSV, use LOAD CSV … CALL { … } IN TRANSACTIONS OF n ROWS (commits in bounded chunks instead of one giant transaction). Both, plus the throughput you should expect and the reason a single unbounded LOAD CSV … CREATE is a memory cliff, are in Bulk Loading & Import.

Every path above is versioned — the initial load is one changeset (or one per chunk) in the engram timeline, so it is a valid AS OF point right away. Take a tag right after the load (CREATE TAG initial-load) so there is a named point to read back to.

9. Known limitations for this profile

Read Limitations in full; the ones that bear on a 3-node Bolt cluster doing CRUD + versioning + search:

• Bolt transactions are snapshot isolation, not serializable — write skew is possible; use single autocommit statements or app-level guards for cross-row invariants, and retry the transient conflict error (The server).
• Bolt reads are always local (eventually consistent on followers). The linearizable read is REST-only; use the AS OF TAG clean-tag pattern for a lag-immune, follower-servable read.
• RBAC enforcement is reject, not row-filtering, and per-property / relationship-type RBAC is not implemented (moot for a single trusted tenant).
• Security audit log is node-local + in-memory (bounded ring, not Raft-replicated, not persisted across restart) — ship the patinadb::audit tracing target to a central sink if you need durable audit.
• No encryption-at-rest — use OS-level disk encryption on the --db volume.
• Full-text: string properties only; read-modify-write postings (not for very high write throughput); prefix/fuzzy capped. Vector: IVF-Flat ANN (Full-Text Search, Vector Search).
• CREATE EDGE SORTED INDEX definitions are not carried in Raft snapshots — a node bootstrapped purely from a streamed snapshot falls back to traverse+sort for that shape until the DDL is re-issued (correctness unaffected).
• On-disk format upgrades may require a dump/reload: export via GET /mgmt/snapshot, upgrade the binaries, import — plan this into your rolling upgrade (Storage & scale, Rolling upgrades).

The parser / DoS input-hardening caveats in the limitations chapter concern untrusted input and are not relevant to this single-trusted-backend profile.