Caching & Memory Tuning

Query normalization is an optimization, not an alternative syntax. Queries containing comments, backtick-quoted identifiers or non-ASCII text outside string literals use the original parser/cache key rather than literal-shape sharing. Unicode string values remain supported. Use explicit parameters for repeated application queries; neither cache hits nor misses may change results or result-column names (#1264).

For the structured overview of the on-disk layout, the hop read path, every cache level and the memory budget hierarchy, see How Storage Works.

patinaDB can keep a governed, RAM-budgeted cache of decoded graph objects above the OS page cache. The server defaults this cache ON, at PATINADB_CACHE_LIMIT=40% of the resolved memory limit, whenever you leave the flag/env var/config-file key unset — so a fresh node ships with the L1, L2 and L3 RAM caches engaged. The L4 disk tier stays off until you set PATINADB_L4_VICTIM_MAX_BYTES. Turn the RAM caches off with --cache-limit 0 (or PATINADB_CACHE_LIMIT=0, off, disabled, or cache_limit: "0" in the config file), any of which still win over the default. Cache-off is not a zero-cost mode: it disables the decoded caches and A2 entry-decision reuse. redb’s native page caches keep their automatic share (40% of PATINADB_MEMORY_LIMIT), the work-memory governor stays active, and the parse and plan caches keep working.

This chapter explains why the cache exists, the memory-budget model that keeps it from fighting the page cache, and how to size, observe, and tune it. For the raw knob table (env / flag / YAML) see Configuration → Cache memory budget; this chapter is the conceptual companion.

Status. All four RAM cache layers ship today: the memory-budget governor plus L1 decoded objects (vertices), L1 property values, L2 adjacency, and L3 query results — see the Layers section — all controlled by the one PATINADB_CACHE_LIMIT knob (on by default at 40%; 0 disables the decoded levels, while native page caches and work memory remain). A fifth, L4 disk-backed victim tier for expensive L3 results also ships, opt-in via PATINADB_L4_VICTIM_MAX_BYTES (off by default).

Eviction in decoded-object, vertex-property, adjacency, query-result and edge-property caches samples occupied entry positions (#1476). The cache budget, frequency admission, generation validation, database-share preference and L4 spill settings retain their existing meanings; no new tuning option is required. The combined change still needs whole-query and memory-pressure qualification; cache byte accounting remains an estimate rather than a bound on process RSS.

Why a cache above the page cache

The storage engine — and, beneath it, the OS — already keep hot B-tree pages resident. That is level 0, and it is good: it avoids disk I/O. But L0 caches bytes, and it stops there. Every read still re-decodes those bytes: postcard / encode_for_index → Vertex / Edge / AttributeValue, with each string value heap-allocated multiple times on the decode → hydrate → pack path. On a hot OLTP path (point lookups, MATCH (n) WHERE n.id = …, fan-out target reads) that decode CPU is paid again and again for the same object.

The patinaDB cache sits above the byte boundary and caches the decoded artifact, so a hit skips the decode and the string allocations entirely. It never mmaps or manages pages itself — L0 stays the I/O-avoidance layer, and the cache is strictly a read-side accelerator: writes always go straight to the B-tree store through the durable path, and the affected cache entries are invalidated, never written through. A cache hit can only make a read faster (or memory tighter), never return a stale or wrong result.

The memory budget model

The app cache is a second consumer of the same RAM the OS page cache needs. Grown naively it trades a decode-CPU win for extra page faults — a bad trade once the working set approaches RAM. So the budget is not a fixed number; it is an explicit, cgroup-aware partition that leaves the OS page cache a guaranteed floor.

How much do we have? (total discovery)

The number everything derives from is not the host’s MemTotal. In a container that is the host’s RAM, and trusting it is the classic Docker OOM-kill footgun (patinaDB ships in Docker). Discovery, in order:

1. cgroup limit — cgroup v2 memory.max, else v1 memory.limit_in_bytes (the real ceiling the kernel OOM-kills at);
2. host /proc/meminfo MemTotal (bare metal / unlimited cgroup);
3. TOTAL = min(cgroup_limit, MemTotal).

Discovery is all-or-nothing on the server. If any part of it fails (no readable cgroup membership or mount, an unreadable or malformed limit, no memory controller, a hierarchy that changes during discovery), TOTAL is unknown. The server then does not guess: it refuses to start unless PATINADB_MEMORY_LIMIT is an absolute size, PATINADB_CACHE_MIN_FREE defaults to 1 GiB, and an explicit ceiling must still fit any limit that was observed.

The partition and the free-floor

From TOTAL patinaDB resolves its own-heap ceiling MEMORY_LIMIT (this does not include the page cache — that is OS-managed L0, protected separately by the floor below), then carves four regions:

PATINADB_MEMORY_LIMIT itself defaults to auto = TOTAL − min_free. The page-cache floor is expressed against system free RAM, not the internal MEMORY_LIMIT — it is the governor’s promise to the OS, orthogonal to the internal cache-vs-action split.

Each knob accepts an absolute size (8GiB, 512MiB), a fraction of its parent region (40%), or auto/unset for the default. Precedence per knob is explicit-absolute > fraction-of-parent > default, and fractions compose: a PATINADB_CACHE_LIMIT=40% is 40% of the resolved MEMORY_LIMIT, which may itself be a fraction of the discovered TOTAL. Every knob is also a patinadb-raft flag (--cache-limit, --memory-limit, --work-mem-limit, --mem-headroom, --cache-min-free) and a YAML config key, with the usual explicit-flag/env > file > default precedence — see the Configuration reference.

Cache and work memory are separate allowances

The cache pool and work memory (“actions”) are fixed partitions of MEMORY_LIMIT, resolved once at startup. A query never evicts cache entries to obtain working memory: a work reservation that does not fit its query, transaction or pool allowance is refused (see Configuration). The only pressure between the two is indirect: when measured available memory falls below PATINADB_CACHE_MIN_FREE, the governor evicts cache entries (see The governor). Two nested guards are checked at startup: an internal one (cache + actions + headroom ≤ MEMORY_LIMIT) and an external one (MEMORY_LIMIT + CACHE_MIN_FREE ≤ TOTAL).

Fail-loud validation

At startup patinaDB resolves the whole partition and refuses to boot if it over-commits — the errors name the offending knobs and the resolved byte counts:

• CACHE_LIMIT + WORK_MEM_LIMIT + HEADROOM ≤ MEMORY_LIMIT (internal partition);
• MEMORY_LIMIT + CACHE_MIN_FREE ≤ TOTAL (leave the OS its floor).

The server logs two INFO events at startup: memory discovery, with what was found and any discovery issues, and resolved memory and native startup plan (bytes), with every resolved region and the per-file native cache plan. They are logged whether or not the entry caches are enabled. GET /mgmt/cache echoes the budget (total, cache_limit, work_mem_limit, headroom, min_free), and --check-config prints the same resolution without starting the node.

A worked sizing example

Take a container with memory.max=16GiB and the defaults. TOTAL discovers as 16GiB; MEMORY_LIMIT=auto resolves to TOTAL − min_free (the floor keeps ≥1.6GiB of system RAM free so the storage engine’s L0 breathes); the three internal regions then split it roughly Cache ≈ 5.8GiB · Actions ≈ 6.5GiB · Headroom ≈ 2.2GiB. Tune from there:

• Read-heavy / point-lookup workload — leave more RAM to the page cache: PATINADB_MEMORY_LIMIT=10GiB (≈6GiB stays with L0), and the decoded-object cache still captures the hot decode CPU.
• Feed / dashboard workload with heavy repeated reads — bias toward the cache: PATINADB_CACHE_LIMIT=70%, where repeat-read cache value dominates.

The governor

The budget is the contract; the governor is the runtime feedback loop that enforces it, “always keeping free RAM in view”:

• Observe. While the decoded caches are enabled, a background sampler estimates available memory once a second: host MemAvailable (/proc/meminfo), capped at every readable cgroup-v2 ancestor by that ancestor’s remaining allowance, crediting only clean inactive file pages (dirty, writeback and shared-memory pages count as used). So pressure inside a container is visible even when the host has RAM to spare.
• Protect the page cache first. When free RAM drops below PATINADB_CACHE_MIN_FREE, the governor shrinks the app cache by the deficit before the OS starts reclaiming the page-cache pages the storage engine depends on. A hysteresis window keeps a steady stream of below-floor samples from re-evicting every tick (no thrash). Its standing bias is “shrink app caches first under pressure.”
• Admission = scan resistance. Admission uses W-TinyLFU (a small frequency sketch): a candidate is admitted only when it is estimated hotter than the entry it would evict. So a one-shot full analytics scan streams through without evicting the hot OLTP working set — a full table scan won’t flush the cache. A candidate must also have been seen at least twice before it is admitted at all. Eviction is sampled recency: a level samples eight resident entries, evicts the least recently used of them, and repeats until enough bytes are free. The sample is biased towards databases that hold more than their equal share of the level. When the governor must free bytes across levels (floor or cap), it takes them from each level in proportion to its resident bytes. There is no cost or value-density weighting. On top of the frequency filter, the L2 adjacency level applies a size ceiling (issue #808): a fully-drained incident-edge list longer than PATINADB_CACHE_ADJACENCY_MAX_EDGES (default 8192 edges, 0 disables) is never admitted. This bounds the recording buffer and prevents one large incident list from consuming the cache share intended for reused anchors. Cached lists retain the optional target-label IDs decoded by the storage scan. Eligible property-free target checks reuse these IDs; secondary-label membership is still checked, and legacy admissions without IDs retain the normal target lookup. Cache accounting charges the capacity of both backing arrays and their Arc/Vec allocations; it remains an estimate of allocator and map overhead, rather than an exact process-memory measurement. No performance improvement is claimed for this representation until its correctness follow-up and native comparison are complete.
• Node-local, never stale. Caches are strictly node-local — nothing travels over Raft, and there is no cross-node coherence protocol. Correctness comes from snapshot-relative generation tags. L1 objects depend on a structural label generation; L1 properties depend on that generation and the generation of their specific (label, property). Overwriting an existing summary in an ordinary deferred atomic write therefore need not discard cached labels or unchanged issue_key values. Direct non-deferred physical commits additionally change a per-open certificate on L1, L2 and edge-property entries. This conservatively rejects reuse across such a commit even for unchanged dependencies. The certificate belongs to the actual native read snapshot, so an older pinned reader cannot republish a value for a newer pin in the interval before logical sync(). A pin racing publication bypasses these caches. L3/L4 additionally stamp a persisted physical-mutation revision from the same read snapshot. This survives reopen and prevents a spilled old result from validating before logical sync(). Old L4 entries without that certificate miss. Failed reads and counter exhaustion bypass result caching. This is not a new atomicity guarantee for direct writes. Create/delete, label churn, property addition/removal and conservative bulk paths still invalidate the broader structural scope. This is not per-entity invalidation: overwriting a property invalidates that property’s entries for the label. L2 retains its existing dependencies. L3/L4 use structural and property dependencies for supported scalar query plans: properties read only by a filter, ordering or aggregate still count. An overwrite of an existing, unread property can preserve a result hit. Whole entities/property maps, ambiguous bindings and unsupported plan shapes retain broad label invalidation. Traversed edge types keep broad type epochs. Property writes cover primary and secondary labels; changing the secondary label set invalidates its old/new populations and the primary entity scope. Generations are updated through the transaction’s metadata and read from the querying transaction’s snapshot, not from a global latest-version counter. A mismatched entry is a miss, not permission to serve an old value. Time-travel (AS OF) reads and the copy-on-read write-resolve mirror both bypass the live cache.

What gets cached (the layers)

The design is a hierarchy above the page layer, extended incrementally:

L3’s query-shape admission history is bounded separately from result payloads: 16 fixed shards hold at most 256 counters each (4096 counters across all node-local databases). An unseen shape arriving at a full shard resets that shard’s history. A forgotten shape must satisfy the configured repetition threshold again; valid cached results and their generation checks are unaffected. Database invalidation and cache reset also discard the corresponding history.

The plan cache (512 plans), parse cache (1,024 entries) and planner statistics cache (4,096 entries or 4 MiB) have fixed caps and are not governor-accounted. The A2 entry-decision cache is governor-visible as planner.entries (#1467). It retains at most 64 entries and min(Q/16, 4 MiB) estimated bytes, where Q is the governor’s auxiliary cache quantum after native/CSR adjustments. Keys are estimated and reserved before copying; oversize or unsupported keys use the ordinary cost model. No new environment switch is needed: PATINADB_CACHE_LIMIT controls this region too, and zero disables it. Its lookup and publication skip a contended mutex; metadata validation runs outside that mutex (#1466), retaining the normal dependency checks.

The level’s byte count includes construction and evicted entries still held by a reader. Per-database statistics list only entries still in the directory, so their sum can be smaller. Hit/miss counters for this region are not collected (the reported zeros are not a measured hit rate). Accounting uses the shared value model, including map key text, named time zones and polygon ring headers; it estimates ownership, not exact allocator consumption or RSS. Besides the four layers, the governor accounts the membership, edge-property, CSR (derived.csr) and A1 degree-catalog (stats.degree) regions; see How Storage Works → Caches.

The RAM layers share one budget + the generation-tag invalidation discipline; GET /mgmt/cache and CALL patinadb.cache.stats() report each level’s live bytes + hit rate. L4 is a disk tier with its own byte cap (not part of the RAM budget) — see below.

L4 victim cache (disk)

The L3 result cache is pure-RAM under the governor’s byte budget, and it discards a result in two cases regardless of how expensive it was: on eviction (the coldest entry is dropped when over cache_limit) and on admission rejection (a fresh result that loses the scan-resistance comparison). In both cases the expensive CPU/IO of computing the result is thrown away and the next identical query pays full price. Meanwhile the box usually has spare disk.

The L4 victim cache catches exactly those victims: when an L3 result whose measured compute cost exceeds a threshold is evicted, it is spilled to a separate file under the database directory (<db_root>/_l4_victim/), and a future identical read is served from disk (deserialize an already-materialized result) instead of recomputed. The read order becomes L3-RAM → L4 victim → recompute, and an L4 hit is promoted back into RAM.

• Provably as safe as L3-RAM. An L4 hit is validated against the same dependency stamp L3-RAM uses — persisted beside the payload. A write that advances a recorded structural, property, broad-label or edge-type generation makes the entry miss and be recomputed. Unknown dependency encodings miss too. This increment uses disposable L4 format version 3: older victim files are discarded on open and warmed again, without changing the database format.
• Survives a restart (unlike the pure-RAM tiers): the persisted key and stamp are db-id-free — db identity is the file location and validity rests on the dependency generations, which are durable in the main database. So an expensive dashboard query stays warm on disk across a node restart.
• Cost-gated. Only a result costing more than PATINADB_L4_VICTIM_MIN_COST_MS (default 50 ms) is spilled. Tune this policy using observed reuse and latency; it is not a measured break-even guarantee for every storage device.
• Bounded by its own cap with LRU eviction: PATINADB_L4_VICTIM_MAX_BYTES (0 = off, opt-in, the shipped default). L4 is not part of the RAM budget. Its database handles and the OS page cache still consume memory. The cap applies to accounted entries per database, not physical file allocation; allow for storage-engine overhead and sum the budget across nodes/databases.
• Off the hot path. L4 is read only on an L3-RAM miss and written only on the eviction or RAM-admission rejection path. RAM-L3 hits do not consult L4. It inherits L3’s admission analysis verbatim (untyped-edge / unlabeled-scan / path-binding / procedure / correlated-subquery / non-deterministic shapes are never cached), is bypassed for time-travel (AS OF) and the write resolve mirror, and is truncated on clear_graph / snapshot install.

Enable it alongside the RAM cache (it catches RAM’s victims): e.g. PATINADB_CACHE_LIMIT=4GiB PATINADB_L4_VICTIM_MAX_BYTES=8GiB. Observability (a dedicated l4_victim metrics block + CALL patinadb.cache.stats row) ships alongside it — see Cache Observability & Tuning.

An enabled L4 is not proof that the workload uses it: require actual spills and valid hits, and test reuse after restart. Under heavy writes, property-wide dependencies can invalidate results even when the changed node would not match the query predicate. L4 is not a replacement for sufficient RAM or a query deadline policy. The local ten-hour qualification (#1261, repository docs/cache-result-dependencies-1261.md) exercised L4 under mixed traffic; separate undersized-RAM trials still exceeded the backend deadline. Full-capacity L4 eviction and OS-cold L4 latency were not qualified by that run.

Observability & tuning

The cache exposes the same per-scope accounting three ways — CALL patinadb.cache.stats(), GET /mgmt/cache, and the patinadb_cache_* Prometheus series (with a ready-made Grafana dashboard) — so an operator can see which database, collection (label), or query shape is hot and how much RAM it holds.

The full treatment — every metric with its name on each surface, how to read each one, the Grafana panel walk, and a symptom → knob tuning playbook — lives in its own chapter: Cache Observability & Tuning. The short version: a high hit ratio on growing patinadb_cache_bytes means the cache is earning its keep (grow it); free-floor eviction with mem_available pinned at min_free means it is starving the OS page cache (shrink it).

When to disable. Set PATINADB_CACHE_LIMIT=0 explicitly to opt out of the 40%-on-by-default behavior. If the working set comfortably fits RAM with the page cache alone and decode cost is already negligible, the app cache adds accounting overhead for little gain. Disabling is a fully supported, tested configuration. It disables the decoded caches, A2 entry-decision reuse and the sampler (and with it the periodic allocator reclamation); native page caches, work memory and the count-bounded plan and parse caches remain.

Honest limits

• Write-heavy scopes self-limit. A label under constant write churn bumps its generation constantly, so its cached entries rarely survive to a second hit — entries for churny data are invalidated before they are reused, and they are evicted like any other cold entry. The win therefore depends on the workload: repeated reads of rarely written data benefit, write-saturated labels gain little.
• The biggest failure mode is oversizing the app cache and starving the OS page cache — inducing the exact page-cache eviction and swap cliff the cache was meant to avoid. The page-cache floor and the governor’s “shrink app caches first” bias exist precisely to prevent this, and the gauges above make a misconfiguration visible rather than silent. When in doubt, size the cache conservatively and let a high hit rate justify growing it.

L1 property entries can also record that a vertex property is absent. Repeated reads can reuse that result under the same admission and memory budget as present values. SET, REMOVE and graph replacement invalidate it using the existing property generation and cache invalidation rules (RFC 0040, #1442).