Day-2 Operations

Once a cluster is up, three things make it operable: a probe that tells a load balancer or Kubernetes when a node can actually serve, request correlation so a slow query is traceable end-to-end, and a way to remove a permanently-dead voter so it stops blocking membership changes and quorum math.

Backups, RPO/RTO planning, and step-by-step recovery for a lost node, a lost leader, or a whole-cluster restore build on the /mgmt/snapshot / /mgmt/restore mechanics in Production Deployment → Backup & disaster recovery and the removal procedure below — ask your patinaDB contact for the full disaster-recovery runbook if you don’t already have it.

Health vs. readiness probes

The server exposes two unauthenticated probe endpoints. Both are auth-exempt (like /version / /metrics) so a load balancer or orchestrator can reach them without credentials, and neither is ever shed by the concurrency limiter.

/ready returns 200 with {"ready": true, …} only when all of the following hold, and otherwise 503 with a short machine-readable reason (checked in priority order):

Example bodies:

// GET /ready  → 200
{ "ready": true, "current_leader": 1, "last_applied": 42, "apply_lag": 0 }

// GET /ready  → 503 (fresh node, no leader yet)
{ "ready": false, "reason": "no_leader", "current_leader": null, "last_applied": 0, "apply_lag": 0 }

Because /ready fails when a node is leaderless, lagging, mid-install, or degraded, pointing a load balancer at it keeps stale/failing reads out of the rotation. Keep /health as the liveness probe so a slow-but-catching-up node is not restarted while it recovers.

The distroless image ships without a shell, so the compose healthcheck can’t curl from inside the container — probe /health and /ready from the orchestrator / an external monitor instead.

Kubernetes example

livenessProbe:
  httpGet: { path: /health, port: 8080 }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5

Request correlation & tracing

Every HTTP request runs inside a tracing span carrying a request id:

• If the caller sends an X-Request-Id header it is honored (used verbatim); otherwise a fresh UUID is minted.
• The id is echoed on the response X-Request-Id header, and it appears on every log line emitted while handling the request — so a slow or failing query is traceable from the client through the server logs.
• An inbound W3C traceparent header’s trace_id is picked up into the span as trace_id, for OpenTelemetry-compatible correlation across services.

Setting --otel-endpoint <url> (or PATINADB_OTEL_ENDPOINT, e.g. http://otel-collector:4318) installs a real OTLP span exporter (OTLP/HTTP protobuf, /v1/traces is appended automatically) — every http_request span (carrying request_id and any inbound trace_id) is exported to the configured collector, and outbound /raft/* peer RPCs propagate a W3C traceparent derived from the current span’s context, so a trace that enters on one node continues across the Raft round-trip to whichever node actually applies it. This is independent of PATINADB_LOG=json (structured JSON logging) — set either, both, or neither; X-Request-Id correlation is always on regardless of this flag. A malformed or unreachable endpoint logs a warning at startup and the node runs normally with span export simply disabled (exporter init failure is never fatal, and once initialized, network failures during actual export are async/best-effort and never block a request). With no endpoint set, nothing OTLP-related is installed — zero cost.

Telemetry-degrade observability and break-glass

A community-mode node that can’t reach the telemetry endpoint for the whole grace window (default 72h) degrades: client writes are refused until a heartbeat succeeds again (see Licensing & Telemetry). Two things make that freeze operable instead of a surprise:

• Alert before it happens. /metrics exposes patinadb_telemetry_degraded (0/1, always present) and patinadb_telemetry_seconds_until_degrade (present only while community mode is armed and not yet degraded) — wire a Prometheus alert on the latter dropping below, say, one hour so you learn about a telemetry-endpoint outage well before writes actually stop.

• A time-boxed break-glass override, for the rare case where you need writes to keep flowing through a telemetry outage you can’t fix immediately:

  patinadb-raft --id 1 --addr 0.0.0.0:21001 --db ./data --bootstrap \
    --auth-password "$PW" \
    --telemetry-degrade-override-until 72h
  

  Flag --telemetry-degrade-override-until <value> / env PATINADB_TELEMETRY_DEGRADE_OVERRIDE. The value is always resolved to an absolute deadline, so the override must expire — accepted forms:

  Form	Example	Resolves to
  Duration from startup	72h, 3d, 30m, 90s	now + duration
  RFC3339 UTC timestamp	2026-07-12T09:00:00Z	that instant
  Bare unix-second integer	1799999999	that instant

  While the override is active, /metrics also exposes patinadb_telemetry_override_until_seconds (seconds remaining before the override itself expires) so you don’t lose track of it. The override is logged loudly at startup — it is a deliberate, visible escape hatch, not a silent bypass. Once it expires, the normal grace-window degrade behavior resumes exactly as if it had never been set.

Tenant isolation and read-proc guards

Two related operational knobs, covered in full elsewhere but worth knowing about when running a shared/multi-tenant cluster:

• --rbac-closed (database-level deny) stops a non-admin’s global role from reaching every database by default — see Authentication & TLS: database-level deny.
• Expensive read procedures are bounded, not unlimited. CALL patinadb.algo.betweenness/closeness (O(V·E), Reader-callable) are guarded by a cooperative deadline under --query-timeout-secs (the algorithm checks in periodically and bails cleanly instead of running past the timeout to completion) plus a static work budget, PATINADB_MAX_ALGO_WORK (env, default ~1e9 — see Configuration Reference), so a Reader can’t pin a blocking thread indefinitely even with no --query-timeout-secs set. Not yet implemented: a bounded blocking-thread pool and a per-user concurrent-read-procedure cap — today’s guards stop a single expensive call from running forever, but a burst of many concurrent expensive calls from different users is not yet rate-limited.

Removing a dead voter

A permanently-unreachable voter stays in the quorum set until you remove it, and it can block further membership changes. In a 3-voter cluster, one dead voter plus one more failure is a quorum loss — so evict a node you don’t intend to bring back.

curl -u admin:… -X POST http://<leader>/mgmt/evict-voter \
     -H 'content-type: application/json' -d '{"id": 3}'
# → 200 {"ok": true, "evicted": 3, "voters": [1, 2]}

• Admin-only (under /mgmt/), and must be issued on the leader — a follower returns a 503 leader hint (like a misrouted write).
• It performs a demote-then-remove in one joint-consensus membership change.
• Quorum guard: the request is refused (409) if the resulting voter set’s live members could no longer form a majority. “Live” requires both reachability over GET /health and genuine Raft replication progress (a voter that answers /health but has fallen behind — or dropped out of — the leader’s replication view is not counted live), so the endpoint won’t hand you a cluster that can’t actually commit. Evicting an unknown id, or the last remaining voter, is a 400.

Automatic dead-voter eviction

A leader-only background failure detector can auto-evict a voter that stays dead past a configurable window, reusing the exact same quorum guard as the manual endpoint above — it can never strand the cluster.

patinadb-raft --id 1 --addr 0.0.0.0:21001 --db ./data --bootstrap \
  --auth-password "$PW" \
  --auto-evict-after-secs 300

• Off by default (unset or 0) — no background task runs, behavior is byte-for-byte the manual-only path above.
• A voter must be continuously dead for the whole window before it’s evicted (a single bad probe never triggers it — the timer resets the instant the voter is seen alive again).
• If eviction would break quorum, the detector refuses and backs off — it never forces the removal. Every attempt (allowed or refused) is recorded in the audit log.

Learner→voter auto-promotion is a follow-on. Use GET /mgmt/cluster to see the live voter/learner topology before and after any eviction.

Reclaiming disk from a bloated .redb file

patinaDB’s B-tree storage engine pre-allocates its file and reuses freed pages internally — it never shrinks the file on its own. So after a bulk-load-then- delete, a mass DETACH DELETE, dropping a large label, or a retention squash, the per-db .redb MAIN file can stay far larger than the live graph, and the usual remedies do not touch it:

• PATINADB_MAX_SNAPSHOTS / a licensed node’s history_retention_days squash pass free only the separate _engram_snapshots/*.snap sidecar files.
• Deleting data frees pages inside the file (the store reuses them for the next write) but does not return them to the filesystem.

If GET /mgmt/dbsizes shows a database’s .redb file itself is the bloat (not its snapshot sidecars), reclaim it with one of the two paths below.

Offline: patinadb <db> compact

The patinadb maintenance tool (shipped alongside the server) provides a subcommand that runs the storage engine’s built-in compaction directly on a node’s .redb file:

patinadb ./data/mydb compact
# compacted ./data/mydb: 46141440 -> 6295552 allocated bytes (39845888 freed, 86.4%)

This is OFFLINE ONLY — the target database must not be open anywhere else (a running patinadb-raft node, or another patinadb process holding the same file open). Compaction needs exclusive access — no concurrent readers or writers at all (an inherent property of the single-file B-tree) — so it cannot safely run against a live server. There is deliberately no /mgmt/compact REST endpoint or live/online compaction path — attempting to open a database that’s still in use fails loud with a clear “database already open” error and touches nothing (never corruption, never a partial compaction). To use it against a clustered node: stop that one node, run compact, then start it back up (a follower rejoins and catches up normally; see the rolling rebuild below for a way to do this with zero downtime across the whole cluster).

Compaction relocates and frees pages, then shrinks the file — it changes only the physical file, never any logical content, so every graph, engram-history record, and index reads back byte-identically afterward; AS OF time-travel, constraints, and compound/fulltext/vector indexes are all unaffected.

Measured (this box; a 50,000-node graph, each node with 3 string properties, then 49,500 of them DETACH DELETEd — the allocated-block figures below are blocks() * 512, never metadata().len(), per this repo’s measurement discipline, since the storage engine’s sparse pre-allocated tail makes len() over-report):

A 7.3× reduction (86.4% of the allocated bytes reclaimed) for this shape; the exact ratio depends on how much of the file’s high-water mark is genuinely dead versus reused-in-place, so re-measure on your own data rather than assuming this number.

Zero-downtime: rolling rebuild onto a fresh directory

For a clustered node you don’t want to take fully offline, the same effect is achieved by rebuilding a follower from scratch via the existing streamed- snapshot bootstrap (docs/disaster-recovery.md covers the mechanics for a lost node; this is the identical procedure used deliberately, as a space-reclamation step, on a node that isn’t lost):

1. Stop the bloated follower and remove (or move aside) its --db data directory — a fresh, empty directory in its place.
2. Start it back up with the same --id/--addr; it rejoins as before and add-learner/normal catch-up streams a snapshot into it, which writes a compact file (no bloat — it’s built fresh from the current graph, not accumulated by however many past deletes/updates the old file absorbed).
3. Once it’s caught up, fail over onto it (or just leave it as a healthy follower) and repeat for the next node — one at a time, so the cluster never loses quorum.
4. Finally rebuild the former leader the same way once it’s no longer serving writes for that database.

This needs no new tooling — it’s the same rejoin path a genuinely lost node uses — and, unlike the offline compact command, it never takes the cluster down (only the one node being rebuilt, which the remaining voters cover for).

An allocated-vs-live-bytes gauge (so bloat is visible before it becomes a disk alert) is a documented follow-up, not implemented yet — for now, GET /mgmt/dbsizes (the on-disk size) alongside a rough live-graph estimate from MATCH (n) RETURN count(n) / GET /version’s usage block is the manual signal to watch for a growing gap between the two.