How Storage Works
This chapter is the structured reference for how patinaDB stores a graph, reads it back, caches it and budgets memory for it. It is written for operators and developers who size, tune and troubleshoot a deployment. After reading it you should know where your data lives, what a hop and a property read cost, what the caches hold, how memory is divided, and which settings matter.
Other chapters keep the details and are linked where they go further: Caching & Memory Tuning and Cache Observability & Tuning for the caches, Configuration → Cache memory budget for every memory setting, Data Model for the logical model, Day-2 Operations for disk reclamation and upgrades, and Limitations for the open edges.
Three rules hold throughout:
- The redb file is the only source of truth. Every cache, statistic and derived structure described here is a copy that can be dropped at any time. A cache hit can make a read faster; it never changes a result. A cached entry whose generation stamp does not match the reader’s snapshot is a miss, not a stale answer.
- Everything in RAM is node-local. Caches, statistics, the degree catalog and CSR partitions are never written to the Raft log, never replicated and never sent to another node. Each node builds its own from its own copy of the data. Only L4, the optional result spill file, lives on local disk.
- Numbers in this chapter carry their source. Defaults and limits are quoted from the source code. Every measured number names the measurement, its conditions and its issue (#1442 unless stated otherwise). Measurements come from one shared development host and are indicative, not a promise for your workload. Values marked calculated are arithmetic from the source formulas, not observations.
The picture at a glance
| Layer | What it is | Where | Survives a restart |
|---|---|---|---|
| Graph file | One redb B-tree file per database: vertices, labels, adjacency, properties, indexes, search, history, metadata | <data>/<db>/patinadb.redb | yes |
| History bodies | Engram snapshot bodies | <data>/<db>/_engram_snapshots/ | yes |
| Raft state | Replicated log, vote, state-machine metadata, users, IAM | <data>/_raft/patinadb.redb | yes |
| Audit | Node audit log | <data>/_audit/patinadb.redb | yes |
| Native page caches | redb’s own page cache per open file | RAM, inside the cache pool | no |
| Decoded caches | L1 objects and properties, L2 adjacency, L3 results, auxiliary caches | RAM, inside the cache pool | no |
| Derived structures | Membership sets, CSR partitions, planner statistics, degree catalog | RAM | no |
| L4 victim tier | Spilled expensive L3 results (off by default) | <data>/<db>/_l4_victim/ | yes (revalidated on use) |
| OS page cache | Kernel cache of file pages | RAM outside patinaDB’s heap | not reliably |
Figure 1. What one node stores on disk. Table names are those in the source. The red strip lists what lives only in RAM.
Where your data lives
Files and directories
The server’s --db directory is the registry root. Each database is a
subdirectory named after the database, holding one redb file named
patinadb.redb. default and system are reserved database names. Directory
names that start with _ are reserved for internal state and are never opened
as databases:
| Path | Contents |
|---|---|
<db>/patinadb.redb | the whole graph of one database, with its indexes and history tables |
<db>/_engram_snapshots/ | Engram snapshot bodies ({id}.snap, {id}.snapidx sidecars); staging files left by a crash are removed at open |
<db>/_l4_victim/ | the optional L4 result spill file (only when L4 is enabled) |
_raft/patinadb.redb | Raft log (raft_logs, keyed by log index), vote and state (raft_meta, raft_sm), users and IAM |
_audit/patinadb.redb | the audit log (raft_audit, raft_audit_meta), in its own file so audit I/O does not wait on the Raft writer |
_raft_snapshots/ | Raft snapshot files |
_physical_install/, _snapshot_install_intent/, _maintenance/, _diff_jobs/ | in-progress install, maintenance-job and diff-job state |
Full-text and vector indexes have no files of their own: they are tables
inside the database file. Back up the whole --db directory as a unit, or use
the portable backup.
Tables inside a database file
Every table is an ordered byte-key B-tree. Keys sort bytewise, so the layout of
a key decides which scans are cheap: anything that shares a key prefix is
stored together and read with one seek and a range scan. A UUID inside a key is
17 bytes (a 0x10 tag plus the 16 raw bytes), so UUID order is key order.
| Group | Table | Key → value |
|---|---|---|
| Vertices | __patinadb_default__ | vertex UUID → primary label id |
vertex_labels | primary label id + UUID → empty (the label index) | |
| Secondary labels | vertex_extra_labels | UUID → secondary label ids |
vertex_extra_labels_by_label | label id + UUID → primary label id (the inverse index; trusted only while its READY marker is present) | |
vertex_inline_exempt | UUID → two bits: in-degree or out-degree above 4,096 | |
| Adjacency | edge_ranges, reversed_edge_ranges | 42-byte edge key → inline secondary labels (next section) |
| Properties | vertex_properties | UUID + property name id → value frame (≤ 256 bytes inline) |
vertex_property_payloads | property name id + UUID → values larger than 256 bytes | |
edge_properties | edge + property name → value | |
| Indexes | vertex_property_values | name id + label id + encoded value + UUID → empty |
compound_index_defs, compound_index_entries | definitions; label and fields + encoded values + UUID → empty | |
edge_property_values | name + value + edge → empty | |
edge_sorted_index_defs, edge_sorted_index_entries | definitions; index + anchor + encoded target property + target → empty | |
| Search | fulltext_catalog, fulltext_index | index definitions; postings, document lengths and corpus statistics |
vector_catalog, vector_postings, vector_assign | index definitions; vectors per IVF centroid; centroid per vertex | |
| History | engrams, engram_deltas, head, snapshots | Engram metadata, per-commit delta lists (optionally zstd-framed), current HEAD, snapshot markers |
| Metadata | metadata | schema version, the label/type/property-name registry, durable generation counters, READY markers, index and constraint definitions |
idempotency | client idempotency tokens | |
| Cache validation | __patina_direct_result_revision | revision → little-endian u64, advanced atomically by direct non-deferred mutations; read by L3/L4 against the query snapshot |
Index values are empty: the key is the entry, and an index lookup is a seek
plus a range scan over keys. Labels, relationship types and property names are
interned once in an append-only registry and stored as 4-byte ids; the registry
holds at most PATINADB_MAX_LABELS names (default 1,000,000, see
Day-2 Operations).
Adjacency keys and inline labels
Each relationship is stored twice: once in edge_ranges under its source and
once in reversed_edge_ranges under its target. The 42-byte key is
source UUID (17) | relationship type id (4) | target primary label id (4) | target UUID (17)
and the reverse tree swaps the endpoints, so in both trees the label slot holds the primary label of the vertex the hop lands on. Consequences:
- Prefix scans. All relationships of one vertex share a 17-byte prefix, all of one type a 21-byte prefix, and all of one type whose target has a given primary label a 25-byte prefix. A typed hop is one seek plus one range scan. Within a 21-byte prefix, entries sort by target primary label, then target UUID.
- No relationship id. A relationship is identified by
(source, type, target). There is no separate id and no second relationship of the same type between the same two vertices (see Data Model → Parallel edges). - Primary-label checks are free. The target’s primary label is in the key,
so
(a)-[:R]->(b:Person)withPersonasb’s primary label needs no read ofb.
The value of an adjacency entry carries the landing vertex’s secondary labels (schema 8, RFC 0040 increment 1):
- an empty value means the vertex has no secondary labels;
- otherwise the value is the vertex’s secondary label ids, sorted;
- a single byte
0x00(NO_INLINE) means “not stored here, look it up”.
A vertex whose in-degree or out-degree, over all relationship types together,
exceeds 4,096 is marked in vertex_inline_exempt. From then on, entries that
point at it in that direction carry NO_INLINE instead of its labels. The mark
is sticky: it stays when the degree drops again, and is removed only when the
vertex is deleted. It exists because a label change on a vertex rewrites every
non-exempt adjacency entry that describes it, in the same write transaction.
Without the bound, relabelling a hub would rewrite all of its entries.
Readers trust inline values only while the database’s inline-labels READY
marker is present. A store without READY (an upgraded schema-7 store before
its rebuild, or an interrupted rebuild) falls back to the lookup path for every
secondary-label check, so results never depend on the marker. See
Day-2 Operations → Inline secondary labels
for the open-time rebuild and PATINADB_INLINE_LABELS.
Vertex properties: inline and external
vertex_properties stores one entry per (vertex, property). A serialized value
of at most 256 bytes is stored inline in the entry. A larger value is written
once to vertex_property_payloads, and the primary entry keeps a one-byte
marker. The payload table is keyed property-first, then owner. A missing
payload is reported as corruption, never as NULL. Edge properties are always
inline. The 256-byte threshold is a fixed heuristic (#1399), not a measured
optimum; see
Query Planning → Inline and external vertex properties.
What is not on disk
Planner statistics (label counts, column statistics, fan-out) and the RFC 0042 A1 degree catalog are computed on demand and kept only in RAM. They are capped at 4,096 entries or 4 MiB, never written to the metadata tree, never replicated and lost on restart (see How the Query Planner Works → Statistics). The same holds for every cache level except L4 and for CSR partitions. After a restart a node is correct immediately and fast again once its caches and statistics have been rebuilt by use.
Format versions
| Version | Value | Meaning |
|---|---|---|
SCHEMA_VERSION (database file) | 8 | 5: postcard codec with zstd frames; 6: 42-byte edge keys; 7: inline/external vertex properties; 8: inline secondary labels and vertex_inline_exempt |
SNAPSHOT_FORMAT_VERSION (Raft snapshot) | 8 | format 8 carries schema-8 physical bodies |
PROTOCOL_VERSION (peer protocol) | 22 | every member must report 22 before physical snapshots are used |
Opening a schema-7 store migrates it in place to 8 and then rebuilds the inline values. Schema 5 is refused with a re-import error; other older versions fail with an “export and import” error; a store written by a newer binary is refused. See Limitations → Storage & scale.
What one hop costs
A hop such as (a)-[:R]->(b:Label {p: …}) runs in three stages for every
anchor a. Each stage tries its sources in order and stops at the first that
can answer.
Figure 2. The read path of one hop. The times are the A2-cal v1 primitives in the table below (#1442).
- Neighbours of
a.- A CSR partition, when one is built for this relationship type and
direction, its snapshot matches the reader’s and
ahas not been written since the build. Under the shippedPATINADB_CSR=autoonly deduplicating variable-length traversals use it; see CSR adjacency. - Else the L2 adjacency cache: live reads of a single relationship type, for lists of at most 8,192 entries, validated by the anchor’s label generation and the relationship type’s generation.
- Else a redb range scan: seek to the 21-byte
(a, type)prefix (17 bytes for an untyped or multi-type hop, then a type filter) and read the entries. A fully read list that fits the L2 limit is offered to L2.
- A CSR partition, when one is built for this relationship type and
direction, its snapshot matches the reader’s and
- Labels of each
b.- A primary label is compared against the key’s label id.
- A secondary label is answered from the inline value when the store is
READY and
bhas no secondary label or exactly one. - Otherwise (two or more secondary labels, NO_INLINE, or no READY marker)
the membership cache answers: a per-label set of holders, built after
65,536 checks of one label within one generation, else a point read of
vertex_extra_labels.
- Properties of
b, only those the query reads: the L1 property cache (which also caches confirmed absence), else a point read ofvertex_properties, plus a second read ofvertex_property_payloadsfor a value larger than 256 bytes. Relationship properties use the separatel1.edge_propertiescache in the same way.
Historical (AS OF) reads and the write-resolve mirror bypass the live caches
and read the tables directly. Parallel workers (PATINADB_QUERY_THREADS above
- use a per-query shared cache instead of the process-wide L2.
Primitive costs (RFC 0042 A.5)
The cost model is calibrated against these primitives. They come from the A2-cal
v1 calibration (#1442) and are the V1 constants in
crates/patinadb/src/plan/cost/constants.rs:
| Primitive | ns per operation | Scope of the measurement |
|---|---|---|
| next entry, typed L2 list | 4.3 | payload iteration only; excludes the cache lookup |
| inline secondary-label accept | 8.9 | one InlineTargetCheck accepting a secondary label |
| next entry, warm redb range | 24.8 | raw cursor step on the SF1 store; excludes decoding |
| L1 property hit | 43.1 | resident integer property; excludes any redb read |
| secondary-label membership hit | 154.3 | resolved membership check |
| seek to a key prefix, warm redb | 294.5 | cursor positioning only |
| next entry, advisory-cold redb | 879.7 | first 65,536 entries after POSIX_FADV_DONTNEED; a sequential read-ahead regime |
Figure 3. Hop primitives, A2-cal v1: chris-aibox (Ryzen 9 9950X3D2), cores 0–3, 2026-10-16, median of seven batch minima; seek and redb steps on a reflink copy of the LDBC SF1 store, the rest on generated fixtures (RFC 0042 A.5, #1442).
How to read them:
- Each value is one primitive’s own scope, not a hop latency. A real hop adds decoding, row construction, filters and cache bookkeeping. Do not add them up and quote the sum as a measured hop cost; such a sum is unverified.
- The ratios are what matters for design. An L2 entry is about a sixth of a warm redb entry, a seek costs as much as about twelve warm redb entries, an inline label check is about a seventeenth of a membership hit, and a cold entry is about 35 times a warm one (ratios of the table values, #1442).
- Cold is advisory. The cold figure is a page-cache-evicted sequential read on one machine. The planner never assumes it by itself; it prices warm L2 or warm redb.
What RFC 0040 changed
RFC 0040 (“hop-local storage layout”, #1442) attacked the per-hop cost in two steps:
- Increment 0, no format change. Hops take the target’s primary label from
the key instead of reading the vertex; membership checks use a set held per
read view instead of a global lock; cache bookkeeping got cheaper; absent
properties are cached. On the synthetic
ldbc_pinsgraph, vertex label reads fell from 5,383 to 1,427 for IC14 and from 320 to 21 for IC9. Paired SF1 runs measured IC3 at 0.62 and IC9 at 0.69–0.72 of the previous time (RFC 0040 “Increment 0 results”, #1442). - Increment 1, schema 8. Secondary labels moved inline into the adjacency value, with the 4,096-degree exemption. On a compacted SF1 copy, allocated file size grew by 0.68% (10,114,277,376 to 10,183,208,960 bytes) and the first open with the rebuild took about 95 s. Paired SF1 reads were 10–17% faster on IC4, IC6, IC8, IC11 and IC12, with no significant regression. Costs: about 14–15% on a deferred in-memory edge-insert microbenchmark (no regression in committed workloads) and about 8× for a label change on a vertex with 16 incident entries (RFC 0040 “Increment 1 measurements”, #1442).
Rejected in the same RFC: per-vertex adjacency blocks (a native redb comparison measured a 1.4–2.2× insert penalty, ADR 0017, #1442) and fixed-width key tables (not adopted; their savings were never measured).
Caches
Figure 4. How a server node divides its RAM, with the automatic defaults. The values are calculated from the source formulas for a 2 GiB and a 16 GiB ceiling, not observed.
What each cache holds
| Cache | Holds | Key | Size bound | Governed |
|---|---|---|---|---|
| Native redb page cache | file pages of each open redb file | page | per-file request planned at startup from C − E | yes, startup plan |
l1.objects | a vertex’s decoded primary label | (db, UUID) | E / 4 | yes |
l1.properties | decoded vertex property values, including confirmed absence | (db, UUID, property) | E / 4 | yes |
l2.adjacency | the raw entries of one anchor’s adjacency list, per side and type | (db, anchor) | E / 4; lists over 8,192 entries never admitted | yes |
l3.results | materialized results of repeated pure-read queries (seen at least twice, at most 1 MiB each) | kind, db, schema epoch, exact text, parameters | E / 4 | yes |
l1.edge_properties | decoded relationship property values | (db, edge, property) | C / 8 quantum | yes |
l1.label_membership | holder set of one secondary label | (db, label) | C / 8 quantum | yes |
derived.csr | CSR partitions, see CSR adjacency | (db, type, direction) | E / 9 once reserved | yes |
stats.degree | the A1 degree catalog | per database | min(C/64, 4 MiB) blob within a C/32 region | yes |
| shared worker cache | adjacency and properties for one parallel query | per query | C / 8 quantum | per query |
| L4 victim tier | expensive evicted L3 results, on disk | db-free key and stamp | PATINADB_L4_VICTIM_MAX_BYTES per database (default 0 = off) | own disk cap |
| Plan cache | rewritten operator trees | db + normalized text | 512 plans, LRU | no |
| Parse cache | parsed queries | text | 1,024 entries | no |
planner.entries (A2 entry decisions) | completed cost-model entry choices | pattern + parameter values | 64 entries and min(Q/16, 4 MiB) estimated bytes | yes |
C is the cache pool (PATINADB_CACHE_LIMIT) and E = 4 × ⌊C/8⌋ the decoded
entry allowance (next section). Q is the auxiliary cache quantum after
native/CSR adjustments; absent those adjustments, Q = floor(C/8). A2 reserves
its estimated key bytes before copying and keeps evicted-but-pinned entries
charged until their final reader drops them. The plan and parse caches remain
count-bounded and outside the governor; see
How the Query Planner Works.
Not every level is in use for every query: L1/L2 only serve live reads, L3 only caches result shapes it can prove deterministic (see Caching → What gets cached), and the membership set is built only after 65,536 checks of one label.
Budget shares
With the shipped defaults the cache pool C is 40% of the own-heap ceiling
PATINADB_MEMORY_LIMIT (Figure 4). It splits as follows (source:
cache/level_budget.rs, cache/native/startup_plan.rs, cache/mod.rs):
- Native and decoded halves.
E = 4 × ⌊C/8⌋goes to the four core decoded levels; the remainderC − E, about half, goes to redb’s native page caches. Startup divides the native half into fixed per-file requests for every graph file, the Raft file, the audit file and a set of scratch slots of at most 64 MiB each. - Four core levels. L1 objects, L1 properties, L2 adjacency and L3 results
get
E/4each. - L3 unusable. If
PATINADB_L3_MAX_RESULT_BYTESis below L3’s fixed 256-byte entry overhead (for example1), L3 can admit nothing and the other three levels getE/3each (RFC 0042 B.4). - CSR share. Once the CSR share is reserved, each core level keeps 8/9 of
its cap (2/9 of
E) and the CSR gets 1/9 ofE. WithPATINADB_CSR=onthe share is reserved at start-up; with the shippedautoonly after an admitted query has used a CSR. Theautorule exists because an unused reserved share made SF1 IC6 1.9× slower (#1442,cache/csr.rs). - Auxiliary caches (membership, edge properties, the shared worker cache)
each self-bound at the older
C/8quantum (× 8/9 with the CSR share). These caps are not carved out ofE, so the sum of all caps can exceedC.
The decoded caps count accounted bytes (entry payload plus a fixed per-entry overhead of 160–256 bytes), not allocator bytes. They are not an exact RSS bound.
PATINADB_CACHE_LIMIT=0 (also off, disabled or an empty value) turns off
the decoded caches and A2 entry-decision reuse. The native page caches keep the
automatic 40% share, the work-memory governor stays active, and the parse and
plan caches keep working. Cache-off is therefore not a zero-memory mode.
Admission and eviction
- Admission is W-TinyLFU with a 4 MiB frequency sketch: an entry is admitted on its second touch, and when a level is full a candidate must be estimated at least as hot as the entry it would displace. A one-off scan therefore streams through without evicting the hot set. A scan repeated in a cycle does pass the doorkeeper; that is the thrash case below.
- Eviction samples eight resident entries and evicts the least recently used of them, repeated until enough bytes are free. A database holding more than its equal share is preferred as the victim source.
- Governor tick. Once a second the server’s governor samples available
memory. Below the page-cache floor it evicts the deficit, with hysteresis;
above
Cit evicts the excess.
Invalidation by generations
No write updates a cache entry. Writes advance durable generation counters in
the metadata table inside the same transaction, and a reader validates every
entry against the generations it reads from its own snapshot:
| Cache | Validated against |
|---|---|
| L1 objects | the label’s structural generation |
| L1 properties | max(structural generation, the (label, property) generation) |
| L2 adjacency | anchor label generation, relationship-type generation and the inline READY epoch; written anchors are also dropped eagerly |
| L3, L4 | every label, structural, property and relationship-type scope the plan reads, plus the schema epoch in the key |
| Membership | the label’s structural generation at build time |
| CSR | build snapshot, type generation, READY epoch, per-anchor dirty marks |
| Plan cache | DDL and graph clears; join-model plans also check accepted degree-catalog publications for their database (plain data writes retain plans) |
A generation advances once per dirty label or relationship type when a
transaction synchronizes, not once per row. One consequence: overwriting a
property invalidates that property’s L1 entries for the whole label, not just
the written vertex. Snapshot install, restore and clear_graph reset the
generations, drop every RAM level for the database, truncate L4 and clear the
plan cache. Older pinned readers keep seeing entries that match their
snapshot.
Direct non-deferred physical writes can precede the logical generation bump.
L1, L2 and edge-property entries additionally match a per-open certificate from
the actual native read snapshot. A direct commit changes that certificate for
the whole physical database; raced pins bypass those caches. Deferred atomic
writes retain the dependency policy above. L3/L4 use a persisted revision in
__patina_direct_result_revision, copied with native checkpoints and retained
on reopen. Empty repair commits do not advance it. Missing legacy certificates,
failed reads and an exhausted revision bypass result reuse. The revision read is
memoized per native query snapshot; its runtime impact is not yet qualified.
See limitations.
CSR adjacency
The CSR (compressed sparse row) cache is a derived, read-only, in-memory copy of one relationship type’s adjacency in one direction (RFC 0042 part D). It is never written to disk or replicated.
- Layout. UUID-sorted anchor ids, per-anchor offsets and dense target numbers, with side arrays for the target’s primary label and inline secondary labels (NO_INLINE is kept).
- When it builds. On demand, in the background, from one ordered pass over the direction’s adjacency tree on a pinned snapshot. The requesting query does not wait; it uses the next source until the partition is published. A failed build backs off for one second before the same partition is retried.
- When it is used. Only if the reader’s snapshot is at least the build snapshot, the relationship-type generation and READY epoch match, and the anchor is not marked dirty. A partition more than a quarter dirty is not used and gets rebuilt. The quarter and the one-second backoff are unverified heuristics.
- Write fences. Every adjacency write marks the touched anchors dirty inside the same scope that invalidates L2, and deferred writes keep that fence until the actual backend flush. A reader that cannot prove the order of its pin relative to a write declines the CSR.
auto(shipped default) vson.autoadmits only deduplicating variable-length traversals (BFS wherecount(DISTINCT …)makes path multiplicity irrelevant), by cost, and only with a cost model enabled. Shortest paths and fixed hops stay on L2/redb. The share is reserved only after the first admitted use.onalso serves shortest paths and single-type fixed hops (D2) and reserves the share at start-up.offdisables it.
- Memory. Partitions are charged to the CSR share (
E/9) before they grow; a partition that does not fit is refused and the query keeps the reference reader. Dense visited sets are charged to the query’s memory scope; when that reservation is refused, the traversal falls back to its hash-based state.
Measured sizes and effects (owned SF1 copy, in-process, #1442):
- One
KNOWSpartition was charged 3,074,896 bytes per direction and took 5.4 s (forward) and 4.5 s (reverse) from request to ready (RFC 0042 D1). - All 30 SF1 partitions together were charged 989,749,944 bytes and took 104 s of sequential warm construction (RFC 0042 D2). This is not cold startup or RSS.
- Under
auto, a*1..3count(DISTINCT …)BFS ran at 0.42 of its off time (34.66 ms to 14.49 ms); IC1, IC13 and IC14 did not engage and stayed within 0.95–1.04 (RFC 0042 “D auto”). - With
on, a generated 1,024-vertex graph written at 1,000 writes/s became fully dirty and read throughput fell from 341,527/s to 279,626/s (RFC 0042 D2). This is why fixed hops are not inauto.
EXPLAIN neither builds nor reports a CSR; PROFILE shows csr.admission and
traversal.source decisions. See
How the Query Planner Works → CSR adjacency cache.
Memory model
Regions
At start-up the server resolves one memory budget (Figure 4; source
crates/patinadb/src/resources/config.rs):
- Effective RAM (
TOTAL) is the smallest of host RAM and every visible cgroup limit (v2: every ancestor’smemory.max; v1: the hierarchical limit). If discovery is incomplete the server does not guess: it requires an absolutePATINADB_MEMORY_LIMIT. - Page-cache floor
PATINADB_CACHE_MIN_FREE, defaultmax(1 GiB, TOTAL/10): memory left to the kernel’s page cache and other processes. - Own heap
PATINADB_MEMORY_LIMIT, defaultTOTAL − floor, divided into the cache pool (40%), work memory (45%) and headroom (15%). - Work memory is divided into apply (25%, committed Raft entries,
replay and recovery), background (10%, optional maintenance scratch)
and foreground (the rest, client statements). A client transaction may
hold at most
min(256 MiB, foreground/4); one statement at mostmin(128 MiB, transaction/2).
Each region accepts an absolute size, a percentage of its parent, or auto;
the four work budgets (PATINADB_APPLY_MEM_LIMIT,
PATINADB_BACKGROUND_MEM_LIMIT, PATINADB_MAX_TX_MEMORY_BYTES,
PATINADB_MAX_QUERY_MEMORY_BYTES) accept sizes or auto only. Startup refuses
a partition that over-commits. A ceiling of 1 GiB or less leaves no own heap
with the default floor and is refused. For a 2 GiB ceiling the defaults give a
1 GiB own heap, a 409.6 MiB cache pool, 51.2 MiB per core cache level, a
78,517,370-byte transaction cap and a 39,258,685-byte statement cap
(calculated; the same arithmetic as
Configuration).
From an 8 GiB ceiling on, the transaction and statement caps stop growing at
256 MiB and 128 MiB.
Embedded use (the patinadb library without the server) installs no
work-memory governor and starts with caches off. Setting
PATINADB_CACHE_LIMIT enables the decoded caches with that size as their
whole budget, without a page-cache floor and without the sampler.
What is accounted, and what happens on refusal
The work budgets count reservations on covered paths, not allocations.
The server gives every Bolt statement a query scope inside its transaction
scope (autocommit statements get a one-statement transaction); HTTP /cypher
reads and writes are admitted too.
| When a reservation is refused | Paths |
|---|---|
the statement fails with Neo.ClientError.Statement.ResourceExhausted | full sorts, materialized results, row building, UNION/DISTINCT accumulators |
| the query continues on the reference path | factorized aggregation, CSR visited sets, cost-model catalogs (16 MiB each), the covering-input spool |
| not byte-accounted; bounded by row counts instead | hash-join build sides and plain GROUP BY (PATINADB_MAX_AGG_ROWS, default 5,000,000), cartesian products (PATINADB_CARTESIAN_CAP, default 10,000) |
The refusal message names the setting and the numbers:
memory reservation refused: scope exhausted; PATINADB_MAX_QUERY_MEMORY_BYTES=…, accounted=…, requested=…, phase=….
A refused statement has no effect (Bolt reports patinadb_effect: "none").
A write is also checked before it is proposed to Raft: its counted data, times
six plus 64 bytes, must fit the apply allowance. A replica never refuses an
entry that is already committed. Refusals are counted in
patinadb_resource_cap_rejections_total (label cap names the setting), and the pools in the
patinadb_memory_* gauges (see
Cache Observability → Accounted work memory).
On Linux the server rechecks the visible memory limits every two seconds. If a
limit drops below the startup bound, new foreground and background
reservations are refused (patinadb_memory_admission_stopped = 1) until the
node is restarted with a valid configuration. Committed apply is never
stopped.
The governor sampler and allocator reclamation
While the decoded caches are enabled, the server samples available memory
every second. The sample is host MemAvailable capped by every readable
cgroup-v2 ancestor’s remaining allowance, crediting only clean inactive file
pages; dirty, writeback and shared-memory pages count as used. Below the
page-cache floor the governor evicts cache entries.
On Linux with glibc the same sampler asks the allocator to return freed pages
to the OS (malloc_trim) every 30 seconds, and at most once a second while
below the floor (#1306). This was added after a soak in which retained free
allocator arenas, not live data, exhausted the memory reserve (#1306). It
frees nothing that is in use and cannot release fragmented pages. With the
decoded caches off there is no sampler and therefore no trimming.
None of these mechanisms is a hard RSS limit. Native pages, allocator slack, encoded network buffers and paths not listed above are outside the counters; keep cgroup and process memory under observation.
Durability and recovery
- Write path. The leader resolves a writing statement against a throwaway
mirror of the graph, then proposes the resolved operations as one Raft entry.
The entry is appended to
_raft/patinadb.redbin one durable redb transaction before it counts toward the commit. Each node applies committed entries to its database file as one crash-atomic unit per entry, together with its Engram history, so an entry lands completely or not at all. - Group commit. Several committed entries handed to apply together share
one deferred-durability scope and then one durability barrier (one fsync) per
touched database, before the applied index is persisted. A crash before the
barrier rolls the whole batch back to the previous durable root, and the
entries are applied again from the log. Entries that change schema or
databases apply one by one.
PATINADB_RAFT_GROUP_APPLY=0disables grouping. - Quick repair. Every durable commit uses redb’s quick-repair mode, so reopening after a crash does not need a full-file repair scan (#1307). At start-up, after any pending snapshot install is recovered and before Raft starts, each database gets one empty durable commit that moves pending native cleanup in front of client admission.
- History snapshots. Each database writes an Engram snapshot body every 50
commits or once 1 MiB of deltas has accumulated
(
PATINADB_SNAPSHOT_DELTA_BYTES), so time-travel reads replay a bounded delta chain. Periodic snapshot scans are paced at 16 MiB/s by default (Configuration). - Raft snapshots. Every 5,000 log entries
(
PATINADB_SNAPSHOT_LOGS_SINCE_LAST) the node takes a Raft snapshot. When every member reports protocol 22 it is physical: a fresh redb checkpoint of each database’s state tables, built from a snapshot pinned at the cut, plus history streams. Otherwise it is logical: record streams that the receiver replays. Afterwards the log prefix is purged: the purge floor is written durably first and the physical deletion follows in small batches. - What a restart keeps: database files, history tables and snapshot bodies, the Raft log, vote and applied index, Raft snapshots, the audit file and an enabled L4 file.
- What a restart loses: every RAM cache, membership sets, CSR partitions, planner statistics, the degree catalog and the plan, parse and A2 entry-decision caches. The first queries after a restart are slower until these are rebuilt; results are the same.
Operational detail: Day-2 Operations (startup recovery, disk reclamation), Disaster Recovery and Limitations → Mixed workloads under severe RAM pressure.
Practical guidance
Sizing RAM against data
There is no measured general formula; size from measurements of your own data. These measurements give orientation:
- File size per relationship. On a 200,000-edge generated fixture, the whole compacted file used 66.6 allocated bytes per adjacency entry with empty values and 70.8 with inline labels; every relationship has two entries (RFC 0040 1.B2, #1442). Real files also hold properties, indexes and history.
- A real store. The compacted LDBC SF1 store (3.18 M vertices, 17.3 M relationships) used 10,114,277,376 allocated bytes before and 10,183,208,960 after the schema-8 inline rebuild (RFC 0040, RFC 0042 D0, #1442).
- Working set, not file size, decides speed. In the Velrbench A1 runs below,
the query’s L2 adjacency working set was 79.4 MiB of accounted L2 bytes
(295,439 lists) at 8 GiB (#1442,
docs/benchmarks/storage-manual). With the 2 GiB defaults, L2 is capped at 51.2 MiB (calculated), and the query took more than twice as long. - A small budget can still be stable. A five-node, three-hour mixed soak with at least 10 GiB of allocated graph files per node and a hard 2 GiB memory limit per node passed its original deadline, fault and OOM checks (#1409; see Limitations). It used configured background pacing and synthetic data; it is not a latency guarantee.
- Measure allocated bytes. redb grows files copy-on-write and does not
shrink them; use
du(notdu --apparent-sizeor the file length) and reclaim space offline withpatinadb <db> compact(Day-2 Operations).
Read Figure 4 the other way round when sizing: decide how large the L2 and L1 shares must be for the hot part of your graph, multiply by 8 to get the cache pool, and divide by 0.4 for the own heap. Leave the page-cache floor for the kernel, which caches the file pages that the native caches miss.
Signs of cache thrash
Figure 5. The same query and plan under different memory budgets: Velrbench A1,
one query thread, L3 and L4 off, cores 0–3, 2026-10-16, main 81e0ab06 (RFC 0042
B.4, #1442; data in docs/benchmarks/storage-manual/).
What the runs show:
- Same plan, same rows, twice the time. At 2 GiB the L2 share (51.2 MiB, calculated) is smaller than the 79.4 MiB working set. The query cycles through its adjacency lists, so they pass admission, evict each other and miss again: hit rate 0.58 and about 639,000 evicted entries per query. In the original diagnosis, PROFILE counted 481,387 backend read requests per run at 2 GiB against none at 8 GiB (RFC 0042 B.4, #1442).
- The budget, not the cgroup, is the cause. An 8 GiB cgroup with the
application budget of a 2 GiB host (
PATINADB_MEMORY_LIMIT=1GiB,PATINADB_CACHE_MIN_FREE=1GiB) is as slow as the 2 GiB cgroup, and forcing redb’s page cache down to 13.65 MiB at 8 GiB changes nothing. No refaults or major faults were observed (RFC 0042 B.4, #1442). - More memory past the working set does not help. 3, 4 and 8 GiB all run in about 2 s with hit rate 1.0 after the first call.
Look for these signs on your own workload:
- a hot level near its cap in
GET /mgmt/cacheorpatinadb_cache_bytes, withpatinadb_cache_evictions_totalandpatinadb_cache_misses_totalrising while the query mix is stable; - a repeated query whose
PROFILEStorage I/Ofooter reports nonzeronormal readson a warm node (these are requests that missed redb’s page cache; cache hits never reach this counter); evicted_free_floor_bytesrising in/mgmt/cachegovernor, which means the page-cache floor, not the cache cap, is forcing evictions.
What to read
| Question | Where |
|---|---|
| How much does each cache hold, and does it hit? | CALL patinadb.cache.stats(), GET /mgmt/cache, patinadb_cache_{bytes,entries,hit_ratio,hits_total,misses_total,evictions_total} per level (reference) |
| Is the governor evicting for the floor or for the cap? | patinadb_cache_evicted_free_floor_bytes_total, patinadb_cache_evicted_cap_bytes_total |
| Are writes invalidating cached data? | patinadb_cache_gen_invalidations_total |
| Is work memory refusing queries? | patinadb_resource_cap_rejections_total, patinadb_memory_pool_refusals{pool}, patinadb_memory_admission_stopped |
| Did this query reach the file? | PROFILE Storage I/O footer (backend requests on the query thread, not physical disk bytes) |
| Where did a traversal read its neighbours? | PROFILE decisions traversal.source (csr, l2, redb) and csr.admission |
| Which read-ahead policy does a store use? | EXPLAIN Storage: line (Query Planning) |
| Is L4 used? | patinadb_l4_victim_*, the l4_victim block of /mgmt/cache |
patinadb_mem_available_bytes reports host MemAvailable; the sampler’s own
cgroup-capped estimate can be lower.
Knobs to try
Try one at a time and compare with a paired measurement.
| Symptom | Try | Notes |
|---|---|---|
| A hot level at its cap, high evictions | raise PATINADB_CACHE_LIMIT (or the node’s memory) | grows native and decoded halves together; cache + work + headroom must still fit the own heap |
| L3 never hits (few repeated queries) while L2 is at its cap | PATINADB_L3_MAX_RESULT_BYTES=1 | L3 off; the other three core levels get E/3 each; restart needed; no measured speedup yet (RFC 0042 B.4, #1442) |
| Expensive repeated reads lost on eviction or restart | PATINADB_L4_VICTIM_MAX_BYTES | disk tier; see Caching → L4 |
| Huge adjacency lists flooding L2 | keep PATINADB_CACHE_ADJACENCY_MAX_EDGES at its default 8,192 | 0 removes the ceiling (not recommended) |
| Memory floor evictions on a read-heavy node | lower PATINADB_MEMORY_LIMIT | leaves more RAM to the kernel page cache |
| Statements refused for memory | raise PATINADB_MAX_QUERY_MEMORY_BYTES / PATINADB_MAX_TX_MEMORY_BYTES within foreground | or add LIMIT, or reduce materialization |
| Deep BFS traversals | PATINADB_CSR=auto (shipped) or on for shortest paths | on reserves the share at start-up; check write rate first |
| Diagnosing a cache-related difference | PATINADB_CACHE_LIMIT=0, PATINADB_CSR=off, PATINADB_PRIMARY_POINT_IO=off | for comparison only; restart required |
Known limitations
- Memory budgets are reservation counts on covered paths; native pages, allocator retention and some buffers are outside them. There is no hard RSS limit (Limitations → Transaction memory admission).
- Cache shares are fixed at start-up: an idle level does not lend its share
to a busy one, and auxiliary caps are not reserved within
E(docs/arch/caching.mdrecords the deferred design). - A cyclic working set larger than its level thrashes despite W-TinyLFU (Figure 5).
- Invalidation is per label and property, not per vertex: write-heavy labels rarely keep cache entries.
- One redb file per database on one machine; no sharding (Limitations → Storage & scale).
- redb files do not shrink online; compaction is offline.
- Statistics and the degree catalog are rebuilt after every restart.
Sources for the figures
The figures are generated by scripts/docs_charts/storage_figures.py (standard
library only, deterministic; --check verifies the committed files). Figures 1
and 2 are diagrams checked against the source. Figure 4 evaluates the budget
formulas of resources/config.rs and cache/level_budget.rs. Figure 3 reads
the A2-cal v1 constants from plan/cost/constants.rs. Figure 5 reads
docs/benchmarks/storage-manual/mem-2g-a1.json, an extract of the RFC 0042 B.4
coordinator runs described in that directory’s README (#1442).