Day-2 Operations

Once a cluster is up, three things make it operable: a probe that tells a load balancer or Kubernetes when a node can actually serve, request correlation so a slow query is traceable end-to-end, and a way to remove a permanently-dead voter so it stops blocking membership changes and quorum math.

Health vs. readiness probes

The server exposes two unauthenticated probe endpoints. Both are auth-exempt (like /version / /metrics) so a load balancer or orchestrator can reach them without credentials, and neither is ever shed by the concurrency limiter.

/ready returns 200 with {"ready": true, …} only when all of the following hold, and otherwise 503 with a short machine-readable reason (checked in priority order):

Example bodies:

// GET /ready  → 200
{ "ready": true, "current_leader": 1, "last_applied": 42, "apply_lag": 0 }

// GET /ready  → 503 (fresh node, no leader yet)
{ "ready": false, "reason": "no_leader", "current_leader": null, "last_applied": 0, "apply_lag": 0 }

Because /ready fails when a node is leaderless, lagging, mid-install, or degraded, pointing a load balancer at it keeps stale/failing reads out of the rotation. Keep /health as the liveness probe so a slow-but-catching-up node is not restarted while it recovers.

The distroless image ships without a shell, so the compose healthcheck can’t curl from inside the container — probe /health and /ready from the orchestrator / an external monitor instead.

Kubernetes example

livenessProbe:
  httpGet: { path: /health, port: 8080 }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5

Request correlation & tracing

Every HTTP request runs inside a tracing span carrying a request id:

• If the caller sends an X-Request-Id header it is honored (used verbatim); otherwise a fresh UUID is minted.
• The id is echoed on the response X-Request-Id header, and it appears on every log line emitted while handling the request — so a slow or failing query is traceable from the client through the server logs.
• An inbound W3C traceparent header’s trace_id is picked up into the span as trace_id, for OpenTelemetry-compatible correlation across services.

Setting --otel-endpoint <url> (or PATINADB_OTEL_ENDPOINT) forces structured JSON logging on (regardless of PATINADB_LOG) so an OpenTelemetry Collector’s filelog receiver — or any log shipper — can ingest each event with its span context (request_id, trace_id). This is log-based correlation; a native OTLP span exporter is a documented follow-on. With no endpoint set, the X-Request-Id correlation still works — it’s always on.

Telemetry-degrade observability and break-glass

A community-mode node that can’t reach the telemetry endpoint for the whole grace window (default 72h) degrades: client writes are refused until a heartbeat succeeds again (see Licensing & Telemetry). Two things make that freeze operable instead of a surprise:

• Alert before it happens. /metrics exposes patinadb_telemetry_degraded (0/1, always present) and patinadb_telemetry_seconds_until_degrade (present only while community mode is armed and not yet degraded) — wire a Prometheus alert on the latter dropping below, say, one hour so you learn about a telemetry-endpoint outage well before writes actually stop.

• A time-boxed break-glass override, for the rare case where you need writes to keep flowing through a telemetry outage you can’t fix immediately:

  patinadb-raft --id 1 --addr 0.0.0.0:21001 --db ./data --bootstrap \
    --auth-password "$PW" \
    --telemetry-degrade-override-until 72h
  

  Flag --telemetry-degrade-override-until <value> / env PATINADB_TELEMETRY_DEGRADE_OVERRIDE. The value is always resolved to an absolute deadline, so the override must expire — accepted forms:

  Form	Example	Resolves to
  Duration from startup	72h, 3d, 30m, 90s	now + duration
  RFC3339 UTC timestamp	2026-07-12T09:00:00Z	that instant
  Bare unix-second integer	1799999999	that instant

  While the override is active, /metrics also exposes patinadb_telemetry_override_until_seconds (seconds remaining before the override itself expires) so you don’t lose track of it. The override is logged loudly at startup — it is a deliberate, visible escape hatch, not a silent bypass. Once it expires, the normal grace-window degrade behavior resumes exactly as if it had never been set.

Tenant isolation and read-proc guards

Two related operational knobs, covered in full elsewhere but worth knowing about when running a shared/multi-tenant cluster:

• --rbac-closed (database-level deny) stops a non-admin’s global role from reaching every database by default — see Authentication & TLS: database-level deny.
• Expensive read procedures are bounded, not unlimited. CALL patinadb.algo.betweenness/closeness (O(V·E), Reader-callable) are guarded by a cooperative deadline under --query-timeout-secs (the algorithm checks in periodically and bails cleanly instead of running past the timeout to completion) plus a static work budget, PATINADB_MAX_ALGO_WORK (env, default ~1e10 — see Configuration Reference), so a Reader can’t pin a blocking thread indefinitely even with no --query-timeout-secs set. Not yet implemented: a bounded blocking-thread pool and a per-user concurrent-read-procedure cap — today’s guards stop a single expensive call from running forever, but a burst of many concurrent expensive calls from different users is not yet rate-limited.

Removing a dead voter

A permanently-unreachable voter stays in the quorum set until you remove it, and it can block further membership changes. In a 3-voter cluster, one dead voter plus one more failure is a quorum loss — so evict a node you don’t intend to bring back.

curl -u admin:… -X POST http://<leader>/mgmt/evict-voter \
     -H 'content-type: application/json' -d '{"id": 3}'
# → 200 {"ok": true, "evicted": 3, "voters": [1, 2]}

• Admin-only (under /mgmt/), and must be issued on the leader — a follower returns a 503 leader hint (like a misrouted write).
• It performs a demote-then-remove in one joint-consensus membership change.
• Quorum guard: the request is refused (409) if the resulting voter set’s reachable members (probed over /health, self always counted live) could no longer form a majority — the endpoint won’t hand you a cluster that can’t commit. Evicting an unknown id, or the last remaining voter, is a 400.

Auto-eviction after a failure-detector timeout, and learner→voter auto-promotion, are follow-ons — this manual/admin endpoint is the current MVP. Use GET /mgmt/cluster to see the live voter/learner topology before and after.