Cache Observability & Tuning

This chapter is the operator’s field guide to the governed cache: every metric it exposes, the four surfaces that expose them, and a symptom → knob playbook for turning what you see into a configuration change.

It is the practical companion to Caching & Memory Tuning, which explains why the cache exists and how its memory budget is partitioned. Read that first for the concepts (the budget model, the governor, the four layers); read this to watch the cache in production and size it right. The raw knob table lives in Configuration → Cache memory budget.

Nothing to see while off. The cache is opt-in (PATINADB_CACHE_LIMIT unset or 0 — the default). While disabled every surface below is a truthful all-zeros “just the OS/redb page cache” report, never an error, at zero residual cost.

The four surfaces

The same per-scope accounting is exposed four ways, from quickest to richest:

All four read the same process-global governor, node-local by design — a node reports only its own cache. The cache is never replicated, so each node’s numbers stand alone (that is exactly why a per-node cache can never cause divergence — see Caching → The governor).

Metric reference

Every metric, grouped by what it tells you, with its name on each surface it appears on. A — means that surface does not expose it. The Prometheus column is the exact series name — a wrong name reads nothing, so these are verbatim.

Labels: {level} is the cache layer (l1.objects, l1.properties, l2.adjacency, l3.results); {db} is the numeric database id.

Fill — how much RAM the cache holds

Hit / miss — is the cache paying off

cache.stats reports hit_rate per (level, scope) — the finest grain, so you can see which collection is hot. The Prometheus hit_ratio is the per-level aggregate; /mgmt/cache carries both.

Eviction — is the cache under pressure

The free-floor-vs-cap split is the most operationally important pair here. Both are eviction, but they mean opposite things: cap eviction says the cache is bounded by your PATINADB_CACHE_LIMIT (raise it if you have RAM); free-floor eviction says the cache is bounded by the OS running low on free RAM (the cache is starving the page cache — shrink it).

Admission — is scan-resistance working

Invalidation — the write-churn tax

Both count the same thing from two directions — the cost of a write to cached data. gen_invalidations is the passive, next-read discovery; scope_invalidations is the active, up-front purge. A scope with high invalidation and a low hit ratio is telling you the cache cannot help that data (it changes faster than it is re-read) — expected, and not worth budget (see Caching → Honest limits).

Budget & free RAM — the governor’s operating envelope

The pair to watch together is patinadb_mem_available_bytes against patinadb_cache_min_free_bytes: the gap between them is the governor’s remaining slack before it starts shrinking the app cache to defend the page cache.

L4 disk victim tier

The L4 victim cache is the optional disk tier below the RAM L3 result cache: when an expensive L3 result is evicted (or admission-rejected), it is spilled to a redb file under the db root and served from disk on a future identical query instead of recomputed. It is off by default (PATINADB_L4_VICTIM_MAX_BYTES=0), so its whole metric block is absent until you enable it. Enabled, it reports as a distinct block — it is not a governed RAM level, so it never shows up under levels[] or the RAM cache_limit.

The headline pair is spills vs hits: L4 is earning its keep when hits are a healthy fraction of spills. If spills dominate, the workload is expensive-but-not- repeated (or too write-churny — see stale_drops), and the disk tier is pure overhead. The writer runs on a bounded background channel off the read/evict hot path, so a slow disk never stalls a query — under backpressure a spill is simply dropped (a future recompute, never a wrong answer).

Reading each surface

In-query: CALL patinadb.cache.stats()

The fastest look — no HTTP, no dashboard, runs in any session:

CALL patinadb.cache.stats()
YIELD scope, kind, bytes, entries, hit_rate, generation
RETURN scope, kind, bytes, entries, hit_rate
ORDER BY bytes DESC

• scope — the hot database / collection / query shape, rendered as a stable string: db:1, db:1/label:Ticket, or db:1/shape:1234 (a plan fingerprint).
• kind — the cache level (l1.objects, l1.properties, l2.adjacency, l3.results, and — when the disk victim tier is enabled — a single l4.victim pseudo-scope row).
• bytes / entries — resident size for that (level, scope).
• hit_rate — hits / (hits + misses), 0.0 when never accessed.
• generation — reserved; yielded as NULL today (the column is kept for schema stability).

With caching disabled it returns zero rows (no levels are registered), never an error. See Procedures → Cache observability.

Over HTTP: GET /mgmt/cache

The complete report as one JSON document (admin-only — it is under /mgmt/). This is the richest single call: it carries fields no other surface has (avg_entry_bytes, evicted_bytes, scope_invalidations, the full budget, and the Sankey).

curl -s -u neo4j:secret http://127.0.0.1:21001/mgmt/cache | jq

An enabled node returns roughly:

{
  "enabled": true,
  "budget": {
    "total": 15461882265,
    "cache_limit": 6184752906,
    "work_mem_limit": 6957846769,
    "headroom": 2319282256,
    "min_free": 1717986918
  },
  "total_resident_bytes": 41231360,
  "utilization": 0.0067,
  "governor": {
    "admissions": 128934,
    "rejections": 20514,
    "evicted_free_floor_bytes": 0,
    "evicted_cap_bytes": 0
  },
  "levels": [
    {
      "name": "l3.results",
      "bytes": 12058624, "entries": 214,
      "hits": 90233, "misses": 1201, "hit_rate": 0.9869,
      "utilization": 0.0019, "avg_entry_bytes": 56348,
      "evicted_entries": 0, "evicted_bytes": 0,
      "gen_invalidations": 88, "scope_invalidations": 3,
      "scopes": [
        { "scope": "db:1/shape:8123", "db": 1, "bytes": 8388608,
          "entries": 40, "hits": 61022, "misses": 210, "hit_rate": 0.9966 }
      ]
    }
  ],
  "sankey": {
    "total_lookups": 402118,
    "misses": 14002,
    "layers": [
      { "name": "l3.results",   "label": "L3 result",   "value": 90233 },
      { "name": "l2.adjacency", "label": "L2 adjacency", "value": 121444 },
      { "name": "l1.properties","label": "L1 property",  "value": 130221 },
      { "name": "l1.objects",   "label": "L1 object",    "value": 46218 },
      { "name": "miss",         "label": "miss → storage","value": 14002 }
    ]
  },
  "mem_available_bytes": 9663676416
}

The levels array is ordered deepest-cache-first (L3 → L2 → L1 property → L1 object) — the same order the Sankey reads. Each level’s scopes are sorted hottest-first (by hits, then bytes) so the top row is the hottest collection or shape. The sankey block is the read-flow: total_lookups is the source width (Σ hits + Σ misses), each layer’s value is the hits that layer absorbed, and the trailing miss layer is the summed fall-through to a storage decode. Because the levels are consulted independently (a query may touch several), it is an aggregate share of all cache lookups, not a strict per-query cascade.

While disabled, enabled is false, the budget is all zeros, and levels is empty. See REST API → GET /mgmt/cache.

Prometheus & Grafana

/metrics exports the full patinadb_cache_* set (refreshed from the governor at scrape time — always current, no background task). The Docker demo in deploy/ ships a ready-made patinaDB Cache dashboard (grafana/dashboards/patinadb-cache.json, 20 panels); it is auto-provisioned by the compose stack, so once a node is scraped it appears in Grafana with no import step. To load it into an existing Grafana, import that JSON and point it at your Prometheus datasource. Its three panel rows map onto the metric groups above:

• Cache Overview — five stat tiles: utilization % (sum(patinadb_cache_bytes) / max(patinadb_cache_limit_bytes)), resident bytes, cached entries, mem-available, and the overall hit ratio (rate(hits) / (rate(hits) + rate(misses))).
• Fill & Hit Rate — resident bytes / utilization / hit-ratio / entries per level, the hit-vs-miss rate, and a hot databases table keyed by patinadb_cache_bytes{level,db}.
• Eviction, Admission & Invalidation — the free-floor-vs-cap eviction-byte split (rate(patinadb_cache_evicted_free_floor_bytes_total) vs ..._cap_bytes_total), evicted entries and generation invalidations per level, admissions vs rejections, the miss → storage decode rate, and mem-available vs the free floor — the single most important page-cache-pressure panel.

The dashboard is all-zero until the cache is enabled. The full compose stack and its two provisioned dashboards are documented in the repo’s deploy/README.md.

Browser admin → Cache

The patinadb-browser admin dashboard has a Cache section (it proxies the node’s GET /mgmt/cache) with, at a glance:

• KPI tiles — resident / cache-limit and utilization, governor admissions & rejections, the floor-vs-cap eviction bytes, and live mem-available vs the free floor.

• Per-level bars — one row per level with a hit/miss track and the raw hits / misses / evictions / gen-invalidations / utilization.

• Hot scopes — the top scopes across all levels (which db / collection holds the hot bytes).

• The Layer Sankey — “Which layer catches how much”: one ribbon per cache layer sized by the hits it absorbed, plus a miss → storage fall-through:

  Cache lookups ─┬─▶ L3 result
                 ├─▶ L2 adjacency
                 ├─▶ L1 property
                 ├─▶ L1 object
                 └─▶ miss → storage
  

  A fat L1 property ribbon and a thin miss → storage tail is the healthy read-heavy picture — most reads are absorbed above the storage decode. A fat miss → storage ribbon means the cache is not catching your read pattern (too small, or the workload is write-churny / scan-heavy). In embedded (file) mode there is no server, so the section shows cache disabled.

Tuning from the metrics

This is the payoff: mapping what a surface shows to the knob that fixes it. Every knob below is documented in Configuration → Cache memory budget (each is an env var, a patinadb-raft flag, and a YAML key).

The two failure modes worth internalizing are the mirror image of each other:

• Too small shows as low hit ratio + cap eviction + a fat miss ribbon while RAM is free → raise PATINADB_CACHE_LIMIT.
• Too big shows as free-floor eviction + mem_available pinned at min_free → lower it. Oversizing the app cache and starving the OS page cache re-creates the exact swap cliff the cache was meant to avoid — the free-floor split and the mem-available panel exist to make that visible before it bites.

When in doubt, size conservatively and let a high hit ratio justify growing. If the working set fits RAM with the page cache alone and decode cost is already negligible, the honest answer is PATINADB_CACHE_LIMIT=0 (the default) — a fully supported, zero-residual-cost configuration; see Caching → When to disable.

Current limitations

• generation in cache.stats is reserved — the per-scope write-generation is not carried on the observability seam yet, so the column is always NULL. It is kept in the signature for schema stability.
• Prometheus is a subset of /mgmt/cache. avg_entry_bytes, evicted_bytes (per level), and scope_invalidations are only on /mgmt/cache (and the browser) — there is no Prometheus series for them. For alerting on those, scrape the endpoint directly.