Query Planning & Performance
patinaDB compiles each query into a streaming operator tree (the “Op tree”), picks a physical access path for every entry point, and — on skewed data — uses a statistics catalog and a cost model to choose between competing plans. This chapter shows how to read what the planner chose and which fast paths make a query cheap.
Start with How the Query Planner Works. That chapter is the structured reference: the planning pipeline, entry and join choice, statistics, the experimental RFC 0042 switches and how to read the
Decisions:section. This chapter keeps the detailed per-feature notes.
Immediate IN operands
When an IN predicate has a single parameter or literal as its right-hand operand, execution borrows that value for the membership check, including inside compound Boolean predicates. The source value remains owned by the statement or plan. This avoids making a new owned RHS copy for every candidate row. Row-dependent RHS expressions continue through the ordinary evaluator. Query memory limits still apply to owned inputs, intermediate work and results; this behavior introduces no configuration switch or higher limit.
WHERE placement
A WHERE comparison of a property with a literal or parameter is checked
inside the pattern itself. This applies in either operand order ($d > m.date
is the same as m.date < $d), in chained comparisons, and in every MATCH
after a WITH. It applies only to the MATCH clauses written after the last
UNWIND, CALL or LOAD CSV before the WHERE, so a filter never skips an
error raised by such a clause.
WHERE n:Label on a node of a required pattern becomes a pattern label, so
the label can choose the scan. A comparison of a single relationship’s
property with a literal or parameter (WHERE r.since > $d) is checked while
the relationship is expanded; a failure to read that property fails the query
rather than dropping the row.
In a MATCH with several comma-separated patterns, a condition that reads only
one of the joined patterns runs on that pattern before the next one is
expanded. This covers =, <> and IS [NOT] NULL on its variables,
properties, literals and parameters, combined with NOT, AND and OR.
Conditions that can raise an error stay where they were written.
Reading EXPLAIN and PROFILE
For a direct property projection, RETURN n.key AS key ORDER BY key can use
the same ordered property scan as ORDER BY n.key for an eligible single-node
shape, subject to index/population safety checks. Traversals, DISTINCT and
multi-key ordering keep their previous scan hints and post-sort behavior.
Computed aliases such as toLower(n.key) do
not imply that the original property’s index order is usable. An ascending
STARTS WITH scan can seek to the prefix lower bound while retaining residual
filters before applying the output window. Deep SKIP still requires advancing
past qualifying rows; use keyset pagination when the application can express a
stable cursor. These are generic planner rules, not special cases for ticket IDs.
EXPLAIN renders the chosen plan without running the query; PROFILE runs
it and prefixes the elapsed time.
For buffered Bolt EXPLAIN/PROFILE requests outside an explicit transaction,
the effective client/server query deadline covers the complete execution,
including time waiting for a blocking worker. Native execution runs off the
async connection thread. Deadline enforcement remains cooperative; it cannot
interrupt every individual storage call. No separate profiling timeout is used.
EXPLAIN MATCH (p:Person {name: 'Ada'}) RETURN p
plan: 'Project items=1 distinct=false return_star=false
MatchScan required=[[entry](p:Person) → PropertyIndex[name] = est.1]
'
Read the tree bottom-up: the MatchScan is the leaf that produces rows, and each
line above consumes them. The arrow annotation is the access path and its
estimated cardinality — here a PropertyIndex[name] = est.1 point lookup on the
value index (one estimated row), not a full label scan.
A range predicate with an ORDER BY that a value index can serve turns into a
sorted seek and emits a Physical: footer naming the executed access path:
EXPLAIN MATCH (r:Reader) WHERE r.age > 40 RETURN r.name ORDER BY r.age
plan: 'Project items=1 distinct=false return_star=false
MatchScan required=[[entry](r:Reader) → PropertyIndex[age] SORTED SCAN ASC est.2]
Physical:
entry.range_seek
'
The Physical: footer describes the planned physical rules from the shared
registry. Runtime admission and preparation can still retain another path; use
PROFILE’s executed-access observations to distinguish the actual choice from
the plan. Common footer names:
| Footer | Meaning |
|---|---|
entry.range_seek | Value-index range seek (aligned >/>=/</<= bound). |
entry.cost_override: <rule> (≈N rows) beats <priority-rule> on cost | The cost model overrode the default priority pick (see below). |
limit.bounded_topk | ORDER BY … LIMIT k via an O(k)-memory top-K heap (no full sort). |
limit.edge_sorted_topk | Traversal + ORDER BY target.prop LIMIT k served by an edge-sorted index. |
limit.reverse_probe (fanout est.N → probe est.M) | Same shape as above with no edge-sorted index: instead of expanding the whole adjacency and sorting it, the planner walks the target label’s value index in sort order and probes each candidate for the edge, stopping once LIMIT is filled. Cost-chosen against limit.bounded_topk (the two estimates are shown), and never preferred over a real edge-sorted index. |
Choosing an equality property
For an unsorted pattern with several eligible equality properties, the property
rule compares bounded index probes instead of using the first property in the
query. For example, a shared run_id need not drive every lookup when id
identifies a much smaller set. The selected predicate and all other conditions
are still enforced, including repeated conditions on the same property.
Each candidate probe shares a raw-key budget across its numeric representations
and any overlay source and local entries. Probes that reach the limit tie by
field name and encoded value; they do not compute a full distribution. The
estimates use committed data and can omit pending buffered writes. This can
change which plan is preferred, but execution still reads its own writes.
These estimates never prove that a result is absent. Existing UUID, IN-list and
sorted access rules retain their precedence. EXPLAIN can separately collect
its existing cardinality estimates from the statistics catalog.
For label-only entry selection, a zero count in the primary-label index does not prove that a registered label is empty: vertices may carry it as an additional label. In that case the planner uses a coarse source-total estimate, or an unknown estimate when the source total is zero. A label absent from the registry can still estimate zero. This changes plan choice, not matching or result completeness. Positive primary-label estimates are retained and may still underestimate labels that also have secondary holders. There is no new setting for this fallback.
Secondary-label decoding keeps up to 16 successful registry ID-to-label mappings per storage manager in fixed inline slots. A slot collision uses the ordinary registry lookup; missing IDs and errors are not cached. The memo is local to its storage view and remains valid across sync and graph clearing because registry IDs are never recycled. It does not cache a vertex’s current label membership. There is no new configuration setting; query memory limits and error handling are unchanged.
For multi-pattern inner joins, the planner can check an unestimated fixed-length path earlier when at least two distinct nodes in that path are already bound, or when it is a closed path whose starting/closing node is already bound. It uses the existing seeded nested-loop execution instead of building the entire unbound path. The initial driving pattern stays unchanged. Variable-length and open single-anchor paths do not qualify for this rule. This is a heuristic: unknown fan-out can still make a plan expensive, and no latency bound is implied. It adds no setting and does not change matching, edge uniqueness, or query deadlines.
The statistics catalog
The planner keeps a lazy, cached statistics catalog. Per label it holds a
vertex count, taken from the primary-label index. Per property it holds column
statistics from the value index: the present count, distinct-value count (NDV),
min/max and a 16-bucket equi-depth histogram. CALL patinadb.stats computes
the same quantities exactly and fresh, including secondary-label holders, so its
output can differ from the planner’s cached, primary-label-only estimates:
CALL patinadb.stats('Person')
For a small Person dataset (3 nodes) this yields, per property:
| property | count | ndv | min | max |
|---|---|---|---|---|
| (label) | 3 | — | — | — |
active | 3 | 2 | false | true |
age | 3 | 3 | 32 | 40 |
name | 3 | 3 | Ada | Grace |
In the procedure output the label summary row has a NULL property. The
catalog is computed on demand and cached in RAM only. Each entry is tagged with
its label’s write generation and served until it is more than
PATINADB_STATS_STALENESS_BUDGET generations (default 64) old. A generation
advances once per dirty label per storage synchronisation. Because statistics
only change which plan runs, never the result, they are never replicated
across a cluster; each node computes its own. The degree catalog and freshness
rules are described in
How the Query Planner Works.
Cost-based selection
Two things use the catalog:
-
Entry-point selection. For a skewed EQ predicate (a value with many rows) or a range predicate, the estimator tightens its guess using the catalog (
present_count / NDV, or histogram interpolation) instead of a pessimistic whole-label count. That can move the entry point of a multi-node pattern to the genuinely cheapest node. -
Physical-rule ranking. When both a covering compound index and a selective single-property seek apply to the same node, the planner ranks them by an estimated
Cost { rows, io }rather than static priority — so a non-selective compound index correctly loses to a selective single-property seek. When the cost model overrides the default pick,EXPLAIN’s footer says so (entry.cost_override: …). -
Join ordering. A multi-pattern comma-
MATCHis reordered so the most selective pattern drives the join and the intermediate result stays small (a bounded Selinger DP for ≤ 8 patterns, greedy above that). Connectivity is preserved, so the reorder never introduces a cartesian product the written order avoided.
On valid data, join order and access-path changes preserve the result multiset.
Different access paths can visit different records and change storage-error order,
or whether corruption in an unrelated record is observed. Errors from accessed
records still propagate. A query without ORDER BY returns rows unordered;
one with ORDER BY post-sorts.
Fast paths that make queries cheap
The executor recognizes a number of shapes and serves them without a full scan or
a blocking sort. You don’t opt into these — they fire automatically when the
shape and the available indexes match. The Physical: footer tells you which
fired.
| Shape | Fast path |
|---|---|
MATCH (n:L {p: v}) | Value-index point lookup (PropertyIndex[p] =). |
WHERE n.p IN [...] | IN-list seek — a union of point seeks. Eligible secondary-label matches use primary-label value seeks and membership checks; larger label/probe counts retain the secondary scan. |
WHERE n.p STARTS WITH 's' | Prefix-range scan over the order-preserving string index. |
WHERE k1=X AND k2 > Y ORDER BY k2 (compound (k1,k2)) | Compound range-seek to the range boundary. |
ORDER BY p [DESC] LIMIT k (index-served) | O(log N + k) sorted scan, LIMIT pushed in. |
ORDER BY p [DESC] SKIP n LIMIT k | Deep-SKIP key-only cursor advance (no per-skipped-row fetch). |
WHERE p > $cursor ORDER BY p LIMIT k | Keyset pagination — O(log N + k) value-cursor seek. |
ORDER BY a, b [… ASC/DESC mixed] [LIMIT k] (single label, stats-gated) | Covering-compound (same direction) or leading-key partial-prefix scan (any direction mix) — no post-sort. |
ORDER BY … LIMIT k (not index-served) | Bounded top-K heap — O(k) memory, not a full sort. |
traversal + ORDER BY target.prop LIMIT k | Edge-sorted top-K when a covering index exists. |
[NOT] EXISTS { (n)-[:R]->() } | Bare pattern-existence probe — one edge-index seek, no sub-plan. |
WHERE NOT EXISTS { (n)-[:R]->(m) } (single hop, no inner WHERE) | Anti-join memo (exists.anti_join) — the existence verdict is answered once per distinct anchor node, not once per outer row. |
RETURN n.p, count(*) grouped by n.p | Group-by-count run-length value-index scan. |
MATCH (n:L) RETURN count(n) / count(*) (no WHERE) | Label-count key-only label-index count (agg.label_count_scan) — no vertex reads at all. Declines when L is also used as a secondary label. |
MATCH ()-[r]->() RETURN count(r) / count(*) | Exact global relationship count (agg.global_directed_edge_count): reads current forward-table cardinality for a bare directed one-hop pattern. Filters, labels, grouping, DISTINCT, repeated aliases, overlays and explicit liveness probing retain ordinary execution. |
MATCH ()-[r:T]->() RETURN count(r) (also T1|T2) | Typed relationship count (agg.typed_directed_edge_count): one key-only pass over the forward edge table comparing the stored type id, without the all-node scan or per-node seeks. Still proportional to the total number of relationships (unlike a maintained count store); the same pattern restrictions as the global count apply. |
Fixed linear MATCH with unequal terminal nodes, RETURN count(*) | Terminal inequality count (agg.endpoint_inequality_count) for supported paths: streams interior matches and subtracts equal endpoint identities from terminal counts. Requires query memory admission and a proof of relationship uniqueness; unsupported predicates/shapes retain ordinary execution. It may reduce row work without reducing every kind of read; no general latency improvement is guaranteed. |
WHERE distance(n.p, c) < D / withinBBox(...) | Spatial curve-range seek (entry.spatial_seek) when a point index exists. |
ORDER BY distance(n.p, c) LIMIT k | Spatial kNN expanding-ring search over a point index. |
The advisor
When a traversal + ORDER BY target.prop LIMIT k shape could be served by an
edge-sorted index but none exists, both EXPLAIN and the advisor procedure tell
you exactly which index to create:
EXPLAIN MATCH (c:Company {name:'Acme'})<-[:WORKS_AT]-(p:Person)
RETURN p ORDER BY p.age DESC LIMIT 3
plan: 'Limit skip=0 count=3
Project items=1 distinct=false return_star=false
Sort keys=1
MatchScan required=[(c:Company) -[entry](p:Person) → PropertyIndex[age] SORTED SCAN DESC est.3]
Physical:
limit.bounded_topk
Advice: CREATE EDGE SORTED INDEX FOR (m:Person)<-[:WORKS_AT]-() ON m.age (would serve this ORDER BY … LIMIT via edge_sorted_topk instead of a fan-out top-K)
'
The same suggestion is available programmatically:
CALL patinadb.advisor(
'MATCH (c:Company {name:"Acme"})<-[:WORKS_AT]-(p:Person) RETURN p ORDER BY p.age DESC LIMIT 3'
) YIELD suggestion
suggestion: 'CREATE EDGE SORTED INDEX FOR (m:Person)<-[:WORKS_AT]-() ON m.age'
Creating that index flips the plan to limit.edge_sorted_topk and the advice
disappears. See Edge-Sorted Indexes.
Bounded-memory guards
Reducible aggregates consume input incrementally; their group state and
DISTINCT/collect buffers remain bounded. Blocking operators (non-top-K ORDER BY,
UNION-distinct, and hash-join build sides), plus the row expansion of UNWIND inside a correlated
subquery, pattern-comprehension result lists, and each partial-prefix sorting group (including the NULL group),
are capped at PATINADB_MAX_AGG_ROWS (default 5,000,000).
Past the cap they raise a clear, actionable error instead of an OOM — the default
is far above any normal query, so below it the result is byte-identical. See
Configuration and Caching & Memory Tuning
for the RAM budget model, and Cache Observability & Tuning
for the read-cache hit rates that make repeated hot queries cheap.
Subquery planning and warnings
Scalar expressions share recursive planning in RETURN, WITH, WHERE and function
arguments. This includes COUNT/EXISTS nested in lists, maps, CASE, list
comprehensions, REDUCE and ALL/ANY/NONE/SINGLE. Iterator/accumulator variables
are local: [x IN [1,2] | x] uses its own x even if an outer x exists.
NULL/empty-list quantifier semantics and short-circuiting are preserved.
For eligible single-pattern bodies, COUNT counts matches incrementally and EXISTS stops on a successful match. Complex WITH/RETURN tails can still use bounded materialization; pattern comprehensions collect bounded result lists. Subquery storage errors, query timeouts and memory-limit errors fail the query; they are not zero counts, false existence results or partial lists. Missing properties still yield NULL; failed property reads fail the query, including dynamic/entity-valued access and traversal target-property filters. Unused CASE/coalesce branches do not perform those reads.
EXPLAIN and PROFILE append deduplicated Warning [code] lines:
| Code | Execution boundary | Possible change to evaluate |
|---|---|---|
PDB1230-COUNT | COUNT retains per-input execution outside eligible projection reuse and bounded equality grouping; no general decorrelation | Use EXISTS/NOT EXISTS if only existence is needed. For repeated exact counts, consider a grouped CALL/join. |
PDB1230-EXISTS | An EXISTS body with operators beyond a scan and optional filter materializes its bounded subplan result | Filter outer rows first. Simplify to a pattern only when aggregation, pagination, errors and empty-input behavior are preserved. |
PDB1230-COLLECT | Pattern comprehension materializes a list per input row | Consider COUNT or EXISTS with the same pattern and filter only if discarding projection evaluation is acceptable: projection errors and volatile expressions can change behavior. Use MATCH to return streamed rows if that result shape is suitable. |
PDB1230-APPLY | Identifies either bounded reuse eligibility for identical proven CALL dependencies, or complete declared imports when unproved (complete inputs for internal APPLY) or a plan not proven repeatable; distinct inputs still execute per row | Filter the outer input first; deduplicate keys only when duplicate outer rows are not part of the required result. |
Eligible repeatable serial projections reuse complete projected rows using the union of their proven input dependencies. Changing an unused outer column does not force another COUNT evaluation. Changing a returned column remains relevant to the key even if the COUNT itself depends on fewer columns. This is whole-row projection reuse, not independent scalar-subquery caching or grouped decorrelation. Unknown dependency forms (including wildcard subplans and opaque expressions) retain the complete typed input key. The existing row and 8 MiB serialized-data budgets apply; key serialization is bounded and does not clone input values.
Within an eligible projection, filter, aggregation, sort or UNWIND source, each COUNT expression may also cache its own integer result using its proven typed input dependencies. This includes graph patterns and trailing subplans, even when unrelated output columns change. Unknown dependency forms keep full input keys. Reuse is scoped to one operator and snapshot; faults are not cached. Volatile or unproven effects disable it. Each retained scalar entry charges its encoded key plus 264 bytes; all scalar entries and equality tables below share one 8 MiB auxiliary accounting budget and entry cap. Oversized keys and exhausted budgets keep ordinary execution; already retained entries remain usable. Different typed keys still execute the inner plan unless the independent equality-scan path below applies; otherwise mostly-distinct inputs can pay overhead without a benefit.
Boolean wrappers also retain native planned contents: XOR and scalar expressions used as predicates do not hide nested COUNT from these contexts. Predicate values must be Boolean or NULL; other types raise an argument-type error. XOR evaluates both operands once, left to right, while enclosing AND/OR and CASE retain their short-circuit behavior. This does not make volatile expressions reusable.
A snapshot-proven projection, filter, aggregation, sort or UNWIND source may also retain equality counts for
COUNT { (n) WHERE n.property = key } expressions (either equality order),
including nested arithmetic, functions, CASE, lists/maps, indexing/slicing,
list predicates/comprehensions, REDUCE and pattern-comprehension expressions.
Here key may be a parameter or a snapshot-stable expression over outer
bindings, including node/relationship or map properties, arithmetic and
containers. The dependency proof must exclude every inner binding; unknown or
volatile expressions retain ordinary execution. Direct id()/elementId()
comparison operands retain their special validation/coercion path. On an empty
inner relation the key is not evaluated, including later table lookups. On the
first nonempty scan its evaluation preserves operand error order; later key
errors still propagate through the evaluator.
The body may be a single-node scan or a graph path such as
COUNT { (n)-[:R]->(m) WHERE n.property = key }. Any named node or relationship in that path may supply
the compared property; a proven snapshot-stable inner-only expression such as
n.v + 1 or n.v + m.v is also eligible. Its dependencies must be known and
contained entirely in the inner path. Arithmetic, functions and containers use
the existing expression evaluator in original operand order. Expressions mixing
inner and outer bindings, volatile expressions and direct id()/elementId()
operands retain ordinary execution. A failed evaluation never populates the
retained table. Paths may span multiple fixed or variable-length relationships,
including zero-length and unbounded ranges. They use the original traversal
and preserve path multiplicity; this does not bound the cost of the first traversal.
All named nodes, relationships and internal positional path bindings must be independent
of outer bindings, with no repeated names, path binding, optional match,
dynamic property constraints, explicit seek hints or trailing clauses. Literal
and parameter property filters on nodes and relationships are eligible; the
original match iterator still applies them.
A leading AND prefix such as n.active AND n.v=key is eligible only when its
dependencies are proven inner-only and snapshot-stable. It runs before the
equality in the original order: false skips both comparison operands, whereas
NULL still evaluates them for errors but contributes no count. A prefix error
prevents retaining the table. All-false prefixes also skip lookup-key evaluation
on reuse. Trailing, volatile or outer-dependent predicates keep ordinary
execution; the planner does not reorder them to force admission. Directions and match multiplicity use the same match
iterator as ordinary execution. The first
required scan computes its normal answer while building auxiliary counts;
subsequent supported scalar keys may reuse them even when other output columns
change. Empty outer input does no inner scan. Indexed and seeded patterns retain
their existing access paths. Complex inner plans retain their normal execution.
Each expression uses its current lexical bindings; no COUNT is evaluated before
its branch or loop reaches it. All auxiliary equality tables in one operator
share the entry and byte budgets below; this does not grant a fresh budget per
nested COUNT. Reused lookups retain cancellation checks within a single row.
Filter contexts cover ordinary WHERE, seeded CALL execution, streaming COUNT predicates and both simple EXISTS predicate loops. Hoisted IN membership retains its prepared candidate set and uses the same context for its COUNT needle. An inner loop owns its context for that invocation; it never shares results with another CALL import or snapshot. EXISTS still stops at its first matching candidate, and short-circuited branches are not evaluated early. Volatile predicates disable reuse. Serial aggregation shares its context across group-key computation, aggregate arguments (including collect with entity preservation), and group-key output. All expressions share one auxiliary budget for that operator, rather than one budget per group. Its existing group and collected-value limits still apply. Parallel aggregation gives each worker its own context on the pinned snapshot; group-key finalization uses a separate context on that same snapshot. Budgets apply per worker, with separate later finalization accounting; 8 MiB is not a global query heap limit. No cache crosses workers or query invocations. Full sort and bounded top-K retain one context across key-computation windows, including seeded CALL sorting. Input and all sort expressions must pass the effect proof; unknown callers retain ordinary evaluation. Stable tie order, NULL ordering and the existing heap/input limits are unchanged. Mostly-distinct keys can cost more; use PROFILE and representative measurements. This is bounded reuse, not general decorrelation.
UNWIND prepares function, list and general expression sources once in the plan. Each source keeps a bounded context across seeds within one repeatable APPLY execution; nested invocations and snapshots do not share it. Literal and plain variable sources retain their existing resolution rules. List entity values, NULL expansion, lazy errors and the existing APPLY/UNWIND row caps are preserved. Procedure arguments and unproven effects retain ordinary evaluation.
The table keeps exact integer keys separate from floating-point comparison buckets, preserving existing mixed numeric equality, including large integers. NULL and NaN do not match. Unsupported inner values or exhausted auxiliary budgets discard the table and retain normal COUNT execution; unsupported outer keys use normal execution. The auxiliary budget charges 256 bytes per retained map entry plus string payload, up to 8 MiB and the aggregation row cap (integer keys can require two entries). This is accounting for retained state, not a measured total query heap bound, and is separate from the projection memo. The first scan incurs grouping work even if no later row reuses it. This is a bounded optimization for an admitted shape, not general decorrelation or a cost-based promise. See the measurement ledger.
Independent COUNT scans with additional repeatable predicates can retain their scan rows when an AND branch contains a proven inner/outer equality. The original complete predicate is evaluated for each outer row in source order; trailing errors, NULL reachability and outer-dependent conditions are not reordered. The first evaluation uses ordinary execution. The second may build a bounded scan cache; later evaluations can avoid graph traversal, but still decode rows and evaluate the filter. This is not a hash join or constant-time count lookup. The cache shares the context’s row/8 MiB auxiliary budget, charges serialized payload plus overhead, and is discarded on overflow or evaluation errors. Volatile or unproven effects, correlated scan bindings and unsupported scan shapes retain ordinary execution. Small input counts may not amortize cache construction; measure representative queries.
Individual repeatable EXISTS predicates also retain bounded Boolean results by proven typed dependencies, so changing unrelated outputs need not repeat them. Unknown dependencies use the full input row. The memo is owned by the same projection, filter, aggregation, sort or UNWIND context as COUNT and shares its entry/byte budget. Cache misses preserve the original edge probe, lazy pattern execution or bounded complex subplan; errors are not stored. Hits still check cancellation, and separate CALL invocations, workers, writes and snapshots retain their existing context boundaries. Distinct keys can add overhead without reducing execution. This is bounded reuse, not a streaming rewrite of complex EXISTS tails.
These are static plan diagnostics, including conditional expression branches;
they are not measured cost estimates or automatic rewrites. They appear in plan
text, not as a new Bolt notification protocol. Check PROFILE on representative
data and preserve zero counts, NULLs, grouping, duplicates and ordering.
For example, EXISTS { (n:Absent) RETURN count(*) AS c } is true even when
there are no matching nodes: the inner aggregate still returns one row. Removing
that aggregate changes the answer. Grouping by an inner property instead returns
no row on empty input. A projected NULL still counts as a row for EXISTS;
DISTINCT and WITH-level SKIP/LIMIT can remove that row. Complex tails retain
reached projection errors even after an earlier row could establish existence;
an enclosing unchosen CASE branch still skips the entire subquery. Do not drop
those projections or stop early without proving error/effect preservation.
A missing warning does not prove that every runtime guard admits an optimized path.
For an independent count reused across rows, a unit CALL expresses computing it once explicitly:
CALL { MATCH (m:Metric) RETURN count(*) AS total }
MATCH (n:Customer)
RETURN n, total
A correlated count is not interchangeable with that query. COUNT/EXISTS bodies accept one pattern and optional WHERE followed by WITH/RETURN stages. DISTINCT, filters, aggregation and LIMIT affect the final row count or existence. These stages retain the normal intermediate-result bounds. Eligible repeatable expressions may reuse results for repeated bindings within one operator. Use CALL for other supported inner-query forms, including UNION. General automatic decorrelation remains tracked in #1230.
CALL subqueries can combine RETURN branches with UNION or UNION ALL, including
imported outer bindings. UNION removes duplicates; UNION ALL preserves them.
When ORDER BY repeats a projected aggregate, its DISTINCT modifier is part of
the aggregate identity: count(x) and count(DISTINCT x) may sort differently.
The same identity applies inside scalar sort expressions such as
abs(count(DISTINCT x)) or count(x)+1. Sorting by the explicit aggregate alias
also selects the intended result. Unaliased aggregate output names such as
count(n.value) are treated as exact column names when reused in these expressions.
WITH DISTINCT inside CALL applies before later aggregates, and WITH * can carry bindings alongside explicitly calculated aliases. These paths retain bounded intermediate results; using CALL does not imply automatic decorrelation.
Equality filters inside COUNT, EXISTS and pattern comprehensions can use property
indexes, including $parameters and bare outer values (n.key = wanted or
wanted = n.key). The supported combination is a conjunction of simple
equalities. Complex filters and outer property expressions retain their general
path; this avoids changing evaluation errors. The filter is still checked after
the index lookup. This improves access per outer row and does not imply that the
subquery has been decorrelated or cached across rows.
coalesce() evaluates arguments from left to right and stops at the first
non-NULL value. A later subquery fallback is therefore not executed when an
earlier value is present. False and empty lists count as present values.
Errors in an argument that is actually needed still fail the query.
Correlated relationship bindings
COUNT, EXISTS, pattern comprehensions and CALL retain the identity of an outer relationship reused by name in an inner pattern. OPTIONAL MATCH candidate caching includes bound relationships and resolved scalar seek values. Rejected candidates do not consume the materialized-row limit, but still count toward query deadline checks. A relationship identity filter can still require scanning candidate edges; it does not imply general decorrelation or a direct edge seek.
A sorted scan inside correlated CALL also honors SKIP, including parameterized offsets. It applies the offset after binding validation and before bounding the collected rows. This path may visit the skipped candidates, unlike standalone key-only index pagination; an index does not guarantee identical pagination cost inside and outside CALL.
Bounded reuse of repeated CALL inputs
Within one APPLY execution and storage snapshot, deterministic read plans can reuse the result for identical declared CALL imports. Other outer columns do not enter a CALL scope or its memo key. CALL without imports has an empty seed and can reuse one result when outer rows exist. Internal planner APPLY retains the complete outer row as its dependency key. Outer duplicates and ordering remain present in the output; empty inner results remain empty. Keys preserve runtime types and representations, so Integer and Float inputs do not share a memo entry merely because Cypher considers their values equal.
The memo holds at most the configured row limit in entries and result rows, and at most 8 MiB of encoded keys/results. Oversized inputs or a full memo keep ordinary execution. Query deadlines, output limits and evaluation errors still apply. Each APPLY invocation owns its memo; it is not reused across queries, parameter changes or transactions.
Admission uses a snapshot-local effect analysis, separate from cross-transaction cache invalidation rules. It recursively covers read-only COUNT/EXISTS, list predicates/comprehensions, CASE, REDUCE, OPTIONAL MATCH, UNION and nested CALL, including unlabeled nodes and untyped relationships. Executable pattern and seek expressions are checked too. Procedures, writes, external input, volatile functions and unproved expression shapes remain excluded. Shortest-path escape hatches currently retain ordinary execution. An explicit non-wildcard output projection allows proven unused imports to be omitted from the memo key. Unknown dependencies, wildcard output and unsealed output retain all imports. Different required imported values still execute separately; this is not general grouped decorrelation. EXPLAIN/PROFILE identifies that remaining work and suggests filtering the outer input or a semantic-preserving grouped rewrite where appropriate.
When APPLY reuse is excluded, additional plan diagnostics identify the cause:
| Diagnostic | Cause and suggested action |
|---|---|
PDB1230-VOLATILE | Random/clock-dependent evaluation; bind once outside CALL only if one value per query is intended. |
PDB1230-PROCEDURE | Procedure effects are unproved; batching or deduplication must preserve invocation effects. |
PDB1230-EXTERNAL | External input may change; read once only if a single input snapshot is intended. |
PDB1230-MUTATION | Writes cannot reuse prior results; reducing duplicate writes changes semantics unless intended. |
PDB1230-PROOF | Unsupported proof shape; retain per-row semantics and use PROFILE before a grouped rewrite. |
These messages do not recommend silently replacing per-row randomness, external reads or writes with a once-only operation.
Aggregate expressions in lists, maps, index/slice expressions and CASE
conditions are evaluated at their enclosing WITH or RETURN grouping stage.
For example, UNWIND [1,2,3] AS x RETURN [count(*)][0] AS c returns one row
with c = 3. Aggregates within an inner COUNT/EXISTS query keep the inner scope.
Aggregates in iterator sources belong to that enclosing grouping stage too:
UNWIND [1,2,3] AS x RETURN size([y IN collect(x) | y]) AS n returns n = 3.
REDUCE may consume an aggregate list and use an aggregated initial value.
Local iterator variables are available in the body; other outer references
must be valid grouping keys when the expression aggregates.
Comprehensions, REDUCE and ANY/ALL/NONE/SINGLE accept the same scalar source
expressions, including direct list parameters, slices, CASE, nested
comprehensions and COALESCE. For example, [x IN $items[0..10] | x] does not
require an intermediate WITH solely to name its source. NULL and empty-list
behavior is unchanged.
Introspection functions evaluate their argument first: keys(box.value) and
properties(box.value) inspect the selected value, not box. Entity-valued
fields retain their node/relationship identity for id, elementId, labels,
type, startNode and endNode.
Runtime argument errors inside iterator bodies fail the query. type() accepts
a relationship or NULL; a node or other non-relationship is an argument error.
Strict toBoolean/toInteger/toFloat/toString reject list, map and graph-entity/path
arguments. Use the corresponding ...OrNull function when NULL on unsupported
types is intended. This is distinct from an unparseable conversion string,
which can still return NULL.
Inline aggregates in iterator bodies (for example [x IN [1] | count(*)])
are a planning error, including when the input is empty. Compute the aggregate
in a preceding WITH and reference its value in the iterator, or use an explicit
COUNT/EXISTS subquery with its own grouping scope. Aggregate sources such as
[x IN collect(n) | x] remain supported.
Within an aggregating CASE expression, every reference outside aggregate
arguments must be a projected grouping key. This applies to WHEN conditions,
THEN/ELSE values and simple CASE scrutinees. For example, project x alongside
CASE WHEN count(*) > 0 THEN x ELSE 0 END to group by x; without that grouping
key the query raises AmbiguousAggregationExpression.
Grouping aliases preserve the input variable’s value inside correlated COUNT/ EXISTS queries, pattern comprehensions, paths and boolean predicates. Local iterator variables retain lexical shadowing. Internal group/aggregate names are chosen to avoid user identifiers; using an alias that matches another input or local variable does not change that variable’s value.
A property-only grouping key such as n.v also retains that property inside
correlated scalar subqueries. It does not retain the whole n or unrelated
properties: using those in the same aggregating expression raises
AmbiguousAggregationExpression. Project the additional grouping key only if
its finer grouping is intended; otherwise compute the needed value before the
aggregation. Grouping by the whole entity retains its identity and properties.
Dynamic properties inside scalar patterns
COUNT, EXISTS and pattern comprehensions evaluate dynamic pattern properties against the current row, including bindings, property accesses, arithmetic and functions. This also applies when COUNT or EXISTS has a WITH/RETURN tail. Anonymous elements remain anonymous in wildcard output and DISTINCT semantics. For a variable-length relationship, dynamic property conditions apply to every relationship in the path; a zero-length path satisfies that condition vacuously. Unreached CASE branches and empty candidate scans do not evaluate the property expression. Reached evaluation errors propagate. These correctness rules do not imply that a dynamic property pattern is eligible for grouped COUNT reuse.
Dynamic pattern properties
Dynamic node and relationship properties in MATCH are evaluated against the
current row, including anonymous patterns such as MATCH (:Item {value: k}).
Their internal bindings do not appear in wildcard results or change DISTINCT
row identity. In OPTIONAL MATCH these conditions belong to the optional clause:
when no candidate satisfies them, the incoming row remains. Variable-length
relationship conditions apply to every edge; a zero-edge path has no edge
condition to evaluate. As with an explicit WHERE expression, a reached invalid
expression reports an error; absent candidates do not evaluate that expression.
RETURN DISTINCT * exposes the bindings still visible after the preceding
clauses. ORDER BY can refer to those bindings and their properties, including
within scalar subquery expressions. A name removed by WITH or never introduced
is still unavailable; an explicit DISTINCT projection retains only its projected
columns for ordering.
Filters before later reading clauses
A required MATCH ... WHERE filter is applied before a following MATCH,
OPTIONAL MATCH, UNWIND or CALL. Dynamic pattern properties obey the same
boundary. A row discarded by the required clause does not evaluate later
expressions; an error reached by a surviving row remains an error. Patterns
separated by commas within one MATCH are evaluated together before its filter.
A literal equality selecting a single anchor before a separate traversal MATCH can still use a covering edge-sorted index for top-K and receive its index recommendation. Extra anchor predicates and unproven shapes retain ordinary execution; multiple matching anchors retain the existing fallback.
OPTIONAL followed by another reading clause
A following MATCH, UNWIND or CALL consumes the rows produced by the preceding OPTIONAL MATCH, including its NULL bindings. Required matches cannot bind those variables before the optional clause applies its own predicate. A missing optional node followed by a required relationship from that node produces no rows; an independent required pattern can still combine with the NULL row.
Reading order before MATCH
A preceding UNWIND or CALL executes before the subsequent MATCH.
For example, UNWIND [1/0] AS k MATCH (n:Missing) RETURN k reports the
arithmetic error even when the later label has no matches. Empty prefixes
still produce no downstream rows, and duplicate prefix rows retain their
multiplicity. Explicit WITH is not required to preserve this boundary.
OPTIONAL clauses with multiple patterns
Comma-separated patterns inside one OPTIONAL MATCH must all match and satisfy
the clause’s WHERE. Otherwise the input row survives once, with all newly
introduced bindings set to NULL; existing outer bindings retain their values.
Two separate OPTIONAL MATCH clauses are evaluated independently in order.
Relationship identity across patterns
Within one MATCH or OPTIONAL MATCH clause, different relationship occurrences must use distinct physical relationships, including across comma-separated patterns, anonymous relationships, named paths and variable-length paths. Separate MATCH clauses may reuse a relationship. A zero-length path contributes no relationships to this check. An OPTIONAL clause without a complete valid match retains its incoming row with new bindings set to NULL.
Internal bindings used for these checks are not exposed by wildcard projections.
Repeated occurrences of the same node variable, such as a self-loop’s endpoints,
appear once in RETURN * and WITH *, including DISTINCT projections.
The existing intermediate row and cartesian-product limits still apply.
The relationship uniqueness check completes at the MATCH clause boundary, before subsequent UNWIND, CALL or OPTIONAL MATCH clauses run. Rows rejected by that check do not evaluate expressions in those later clauses. Errors on valid matching rows remain observable.
Automatic integer covering input (#1399)
A broad plain single-label aggregate reading one integer property automatically uses a covering value-index scan when the shape, row estimate and scratch budget pass admission. Stores/inputs estimated below 4096 rows retain existing access. Configured parallel workers also retain existing aggregate admission; this initial policy does not compare covering against parallel throughput. A bounded external merge restores the original UUID order before the existing evaluator consumes values. This avoids changing SUM intermediate overflow, floating-point expression accumulation, group encounter order or DISTINCT behavior merely because an index sorts by value. Missing properties retain NULL semantics. Original group/buffer limits still apply.
The candidate declines floats (index encoding canonicalizes signed zero), wide
values, secondary labels, pending derived indexes, copy-on-read mirrors, selective
patterns and unproved expressions. Existing grouped-count index execution keeps
precedence. EXPLAIN exposes the selected candidate or its fallback reason,
including estimated reorder work and insufficient scratch. Encoded key bytes
exclude page overhead; current value-index payloads are empty, and primary I/O costs
remain unknown. A logical byte estimate is not a prediction of physical reads.
Limited synthetic memory-pressure qualification covers this narrow increment;
cost selection remains heuristic and broader predicate/type coverage is follow-up
work. PATINADB_READ_COVERING=off disables covering for diagnosis. See the
configuration controls.
EXPLAIN describes the proposed access without running the query. PROFILE
keeps that pre-execution plan and appends Executed aggregate access (query
thread), collected at runtime. For example, eligible execution reports
agg.covering_integer_uuid with the actual index-record count and reorder=memory
or reorder=external merge. Retained execution reports a covering-decline reason
and agg.existing_input (serial aggregate) or (parallel aggregate).
Existing grouped/count fast paths report their own agg.* name.
The planned choice can differ from execution: an underestimated scratch budget
or unavailable scratch during preparation falls back before emitting rows.
PROFILE reports that fallback rather than claiming the proposed covering path
ran. Index/storage errors, cancellation and duplicate/orphan identities are not
converted into successful results. Diagnostics record at most 64 aggregate events, followed by
an omitted-event count; they are not per-worker or per-row timings. For a query
without observed aggregation the output says so explicitly. elapsed_ms still
measures execution, including abandoned preparation, not EXPLAIN planning.
Bounded preparation for filtered aggregates (#1399)
Eligible residual string predicates can obtain candidate node IDs from the value index before aggregation. This preparation is chosen automatically; existing selective seeks keep precedence. It is separate from the integer covering input and its diagnostic switch.
The filter first reuses a prefix of at most 64 decoded records or 64 KiB of encoded keys, with one final key allowed to cross the byte threshold. Already cached column population and observed key width can predict preparation that would exceed its 64 MiB logical key-work budget. The policy never rebuilds a full statistics catalog just to decide admission. A value-sorted prefix is biased, so its match rate is extrapolated only when the first and last encoded values prove that the column is uniform. Missing column statistics retain runtime bounds; label cardinality is not substituted for property population. Stale estimates can change the selected path but do not supply query results.
For supported COUNT grouping, a small exact set of selected roots can keep ordinary property reads instead of preparing the full grouping column. The current heuristic applies to at most 16 selected roots, an estimated label with at least 1,024 nodes and a ratio greater than 64:1. A bounded width prefix can also reject estimated group preparation exceeding the existing 16 MiB retained-value budget. These thresholds are admission heuristics, not a calibrated I/O price.
EXPLAIN reports a planned bounded prefix probe, available advisory counts and
budgets. It does not evaluate the predicate to measure selectivity. PROFILE’s
executed-access section reports filter.covered_candidates and, when attempted,
filter.group_values, with the selected path or fallback reason. Record/key-byte
counters cover consumed preparation entries; they exclude metadata/domain probes,
page metadata and physical I/O. Worker-local preparation is not included in the
query-thread capture. Copy-on-read overlays and pending indexes retain existing
access before creating the covering iterator.
Runtime candidate, retained-value, byte and deadline limits remain active even when estimates are too low. One large encoded value must still be read to inspect it; these bounds do not promise total process RSS. The policy can discard a bounded amount of preparation before fallback and does not guarantee that every filtered query becomes faster.
Inline and external vertex properties (#1399)
The logical primary key still begins with the owner UUID, preserving per-owner prefix reads. Schema 7 keeps canonical values up to 256 serialized bytes inline and places wider canonical values in a property/owner payload table. The owner entry carries a tag. Ordinary point reads and hydration resolve that tag using the same storage snapshot; writes update both tables in one redb transaction. This routing is automatic and has no activation flag. The threshold is a fixed layout heuristic, not a measured optimal boundary for every workload.
Keeping small values apart from wide neighboring values helps workloads that need a small property through primary access. Existing costed index/covering, selective and aggregate choices remain in place. The evaluator still decides which values to read and in which order. A property-ordered scan is not selected: its prototype did not demonstrate enough benefit to justify another access path. Broad reads of the wide values themselves are not guaranteed to improve; warm point/hydration overhead and write/index costs still matter.
EXPLAIN’s Vertex property storage line describes the physical representation,
separately from the query operators and storage I/O policy. PROFILE adds
Resolved vertex property storage: actual inline/external resolutions and their
canonical byte counts on the query thread. These include point and range reads;
worker threads, caches and buffered values may bypass them. They are not disk
traffic, evaluator-call counts or a total-memory measurement.
The payload table retains canonical bytes, including nested values and exact floating-point representations. Missing external data fails as a storage error; it is not interpreted as a missing property. Value-index keys keep their full ordered value, and repeated wide values are not interned across owners. Schema 6 stores require recreation/reimport; no on-disk compatibility migration or dual decoder is included in this format change.
Wide value-index keys (#1399)
Value indexes keep the encoded property value in the key so equality, prefix, range and ordered scans retain their existing semantics. The authoritative property tree keeps the original value. The secondary payload is empty for both direct setters and batch/rebuild paths.
redb 4.3.0 abbreviates byte-key separators in internal B-tree pages where the neighboring keys permit it. Full keys remain in leaf pages: this is neither string interning nor removal of repeated leaf-key bytes. Long common prefixes can prevent useful shortening, and wide full-index scans still visit wide leaf data. This backend representation is automatic; EXPLAIN and PROFILE keep reporting the same logical access paths. No planner flag enables it.
Smaller indexes do not guarantee fewer physical reads for every access path. Constrained-memory qualification found primary scans that read more from disk with the new file layout, despite similar requested read volume. Account for that tradeoff when measuring a workload dominated by wide primary properties.
The final wide-key investigation retained this representation. A fixed-prefix multimap can save space but may scan a whole suffix group for a selective lookup or late page. A separate prefix dictionary restores suffix seeks, yet adds indirection that regressed distinct-wide reads in the native controls. Grouping complete identical values benefits duplicate-heavy data but does not shorten unique values. These experimental representations are not product options.
The source repository records the method and counterexamples in
docs/wide-index-layout-decision-1399.md; earlier access-path work is in
docs/read-path-1399-plan.md. This concludes the bounded optimization round,
without claiming customer/TB qualification or universal optimality.
Primary-property I/O under memory pressure (#1399)
Logical access and aggregate ordering are unchanged. For an existing Linux
store larger in allocated bytes than detected RAM, pure primary-property point
reads can use a separate no-read-ahead file handle. This avoids applying the
same hint to index/range scans or writes. The policy is selected at open and is
reported by EXPLAIN’s Storage: line. Smaller stores and unsupported/unavailable
platform facilities retain native I/O; PATINADB_PRIMARY_POINT_IO=off is a
diagnostic override requiring reopen.
PROFILE’s Storage I/O footer distinguishes backend requests through the
primary handle from native requests on the query thread. It excludes worker
threads and redb cache hits and does not measure physical disk bytes. Cold
latency can still trade off against speculative reads; this is not a universal
scan speedup or a replacement for workload-specific measurements.
The counters instrument the Linux file backend only; memory, simulation and
other platform backends are uninstrumented, rather than evidence of zero I/O.
Fixed traversal to an already bound node
A fixed, single-type hop whose destination has UUID equality can test the typed edge key directly. Directed and undirected relationship semantics, edge properties, destination labels/properties and relationship uniqueness still apply. Other traversal shapes retain the general iterator. This rule has no configuration option; a speedup for a complete query is not yet established by the current LSQB work (PR #1410).
UUID point entry access also completes at the selected vertex: required labels are checked against that vertex’s primary and secondary labels, followed by all remaining property predicates. It does not append the general secondary label scan. Missing vertices yield no row; errors reading the candidate’s own labels or properties remain errors. Other entry access methods retain their existing candidate enumeration (PR #1410).
For property-free targets, traversal can reuse the primary-label ID already stored in the edge key, including targets without a label constraint and secondary-label checks. Secondary membership still has to be read and validated; this only removes a redundant primary-label lookup. Cached edges without an inline ID and property/multi-label constraints retain the full validation path. The existing optional liveness check remains in effect. No new setting or whole-query speedup guarantee follows from this access refinement.
Counting independent path branches
For eligible count(*) aggregates, agg.disjoint_path_count counts each side
of a fixed path around a pivot and multiplies the counts. It avoids constructing
every combination of the two sides. Each branch streams; count arithmetic checks
for overflow and execution continues to enforce the query deadline.
Explicitly configured parallel query workers retain existing parallel admission.
Eligibility requires one required path, no grouping or DISTINCT, and disjoint
relationship types across the split. Binding names may not repeat. Optional or
variable-length paths, property/seek constraints and path bindings retain the
existing execution. A proven filter may move to one factor when every referenced
binding is a vertex of that factor. Supported expressions are vertex = / <>,
AND, NOT, and bare EXISTS for one fixed typed edge between bound vertices.
The EXISTS form permits at most one label constraint per endpoint, but no local
bindings, properties, seeks, named edges or additional scope. Cross-factor
predicates, functions and other subquery forms retain the existing execution.
For the newly admitted compound or EXISTS predicates, the opposite factor is counted first. If it is empty, the predicate is not evaluated: there are no joined rows on which to evaluate it. Otherwise predicate evaluation retains ordinary bindings and fallible errors propagate immediately, without replay or fallback. Count products and sums remain checked; deadlines and existing limits remain. No isolated speedup is claimed for this extension.
For an eligible pure count factor with a fixed endpoint, typed single-edge EXISTS checks may reuse a neighbor set scoped to that factor. A complete set can prove absence; possible matches still use the ordinary point check. Duplicate paths retain their multiplicities. Set memory is charged to the query budget in addition to the existing pair memo, with a separate bounded scan cap. An incomplete capped scan is discarded and ordinary probes continue. Memory refusal, cancellation and read errors propagate without retry. Only live index views admit this optimization; overlay views retain ordinary probes. No new configuration option is introduced (PR #1410).
EXPLAIN reports the selected pivot as a path-element offset; PROFILE confirms actual use. Selection balances branch lengths without a graph-wide statistics scan. This is a heuristic, not a universal cheapest-plan guarantee. The original LSQB comparison at 928cd9ad completes Q1/Q6 with exact counts; other query regressions remain. This is not general performance qualification (PR #1410).
The development extension agg.grouped_disjoint_path_count applies the same
factorization to one vertex grouping key at the pivot and one count(*).
It retains the ordinary full-path entry choice, traverses that entire half,
and counts its rows by the grouped vertex using the existing bounded aggregate.
The opposite half is counted once per source group and its count is multiplied
with the source multiplicity using checked arithmetic. Only the pivot binding
is retained from each source row; duplicate paths still contribute to the count.
Groups without an opposite match are removed before the eager result is exposed.
Existing group/output row bounds remain in force. Intermediate source groups
can include vertices with no opposite match, so this strategy can refuse at the
group cap even when the final group set would fit. It does not raise the cap or
retry through another strategy after a resource error. Filters, scalar or
multiple grouping keys, DISTINCT, optional paths and unproved sources retain
the ordinary executor. Result order without ORDER BY may change; explicit
sorting and limits retain their operators. PROFILE records the chosen entry
and source-side grouping. The source-entry correction has focused correctness
and work-count controls; no isolated speedup is claimed.
Eligible predicate-free COUNT branches reuse the ordinary traversal state machine while omitting terminal row payloads. Candidate checks, relationship uniqueness, label validation and deadlines still run for each match. Branches that evaluate a row predicate retain their bindings. Repeated bindings, middle-entry products, variable-length paths, optional/path bindings, property/seek constraints and active join guards retain ordinary row emission. This changes materialization, not cardinality or traversal semantics; no configuration toggle is added.
A development COUNT adapter can also compare two uniquely bound vertices for
= or <> directly from their UUIDs during successful path emission. It uses
the same traversal, target validation and relationship-uniqueness checks, and
counts complete matches with the existing overflow checks. Unsupported bindings
or expressions retain row evaluation before consuming the iterator. This avoids
rows for that proven predicate; it does not skip graph validation or prune paths.
Focused and restored Q6 controls plus three genuine mutations passed. The later
PROFILE validation also passed the retained path-count tests after adaptive
removal. Combined broad source validation passed (PR #1410). Ordinary-binary
comparisons of the combined changes are recorded in
the LSQB measurement ledger;
those bundle measurements do not isolate this adapter’s contribution.
A development scalar COUNT alternative can choose an entry within the shorter
independent factor when its primary-label cost estimate is smaller than the
original pivot’s. It reuses the grouped admission proof, source_half split and
shared traversal, then combines each pivot’s count with its opposite branch using
checked arithmetic. Exactly one existing MemoryHashMap<Uuid, i64> retains the
counts; its actual table allocation and growth consume statement memory credits.
The existing aggregate group cap remains. No representative-row collection or
second key set is retained.
For a uniquely bound stable vertex, the source factor can emit its UUID directly
through the same completed traversal rather than build a binding row only to
extract that UUID. Admission requires the existing endpoint/fixed-path proof;
unsupported shapes retain row evaluation before consuming input. Match order,
duplicates and visited errors are preserved. This later source-emission change
passed focused/restored and mutation controls. The later b338 combined-binary
comparison includes this change; it does not isolate the gain from UUID emission.
Legacy calls without a statement memory scope retain the streaming path. Group
bindings _, _anon and internal property bindings also decline before
execution. Once execution starts, visited errors, cancellation and resource
refusals propagate without replay. The additional retained groups can cause a
memory refusal even where the streaming alternative would have completed.
Primary-label counts do not describe the full secondary-label population and
are used only as cost heuristics. Cold planning uses bounded index probes,
adding planning work; this does not prove a better traversal order. EXPLAIN
shows the enclosing candidate, while successful PROFILE execution records
agg.local_anchor_path_count with the chosen pivot and entry. Focused tests, the actual hook-bypass counterexample and restored-source lint
checks passed. Combined broad source validation and the b338 ordinary-binary
comparison are recorded in the measurement ledger. The comparison is descriptive,
uses shared OS cache and includes other changes; it does not establish a universal
benefit or performance under RAM pressure (PR #1410).
The experimental bounded suffix-count memo was removed after its ordinary-binary
comparison established no benefit. Scalar counts retain direct branch traversal
and the general row-elision paths described above. Historical measurements are
recorded in docs/lsqb-sf01-coverage-2026-10-11.md; the experiment is not a current
execution strategy or configuration option.
Grouping consumes owned evaluated keys without copying their heap-backed payload a second time. Borrowed callers retain copying behavior. Numeric equivalence, NULL grouping, entity identity and first-occurrence output representatives remain unchanged. This removes one copy; it does not make grouping allocation-free or remove its existing memory and group limits. No isolated speedup is claimed for these materialization changes.
Closed paths and endpoint joins
The development planner preserves a join alternative for two fixed paths sharing both distinct endpoint bindings when each relationship has one explicit type and the type sets do not overlap. The normal join optimizer still chooses execution order and hash versus seeded execution. Clause-wide relationship identity checks remain. Open chains and shared, untyped or variable relationship domains keep the existing fold policy. The original LSQB Q2 comparison at 83002192 still selects seeded execution and times out. Preserving a join alternative alone does not establish improved performance (PR #1410).
The single-probe cost comparison no longer selects repeated seeded execution for multi-hop paths with unbound interior vertices: it does not estimate the intermediate traversal work. Such cases retain hash execution unless the separate policy for unknown anchored paths applies. One-edge and fully bound interior paths retain their existing cost comparison. Performance validation of this development refinement is pending (PR #1410).
Streaming APPLY and partial join counts
APPLY streams its results one outer seed at a time when the snapshot-read analysis proves the entire consumer plan repeatable. Recursive iterator construction inherits that decision; unproven procedures inside scalar COUNT/EXISTS keep the eager read boundary. The outer input and each seed’s result are still collected, and nested consumers that require rows still collect them. Cumulative output limits and deadlines remain. Unproven procedures, mutations, external input and volatile expressions keep eager evaluation, including before a downstream LIMIT or early iterator drop. A downstream LIMIT or early drop can avoid evaluating later pure-read seeds; errors or cap breaches confined to those unconsumed seeds are therefore not reported. Errors encountered in the consumed prefix still propagate. This is lazy read evaluation, not preservation of the former eager error timing.
After ordinary storage-aware join ordering, eligible ungrouped count(*) over
an inner join can group both inputs by their shared vertex keys, join those
counts, and sum their products. This preserves multiplicities while avoiding
materializing the corresponding cross-product rows. Scalar keys, DISTINCT,
optional matches, user grouping, intervening filters and unproved source plans
are excluded. A selected APPLY plan is retained. Existing group and memory caps
apply; changed physical retention can change when resource refusal occurs.
Relationship identity guards between comma-separated patterns are omitted when their explicit relationship type domains are disjoint. Overlapping alternatives and wildcard types keep their guards. No graph label names or benchmark query texts select these rules.
Exact join membership during fixed-path traversal
For structurally pure keyed inner joins over a correlated fixed path, the executor can reuse the existing join hash table to reject vertices that cannot contribute a result before expanding their remaining path. The original join still emits every matching row, preserving duplicate witnesses. This requires proven vertex bindings and ordinary variable identity; anonymous internal names, scalar keys, optional paths, variable-length paths and predicate/effectful boundaries retain ordinary execution. No graph label or benchmark name selects this optimization.
The filter shares the join table instead of building another set of keys. Additional fixed control structures are charged to the query memory scope; the existing join-table accounting remains unchanged. At most eight keys and eight filters are admitted; larger shapes use the ordinary executor. There is no configuration toggle. Deadline and memory limits are unchanged. Work pruned before expansion performs no downstream reads, so errors in those unvisited branches are not evaluated. Errors from consumed storage reads still propagate.
The extension join.grouped_identity_membership also checks an identity vertex
grouping key between two eligible grouped COUNT(*) inputs. Both inputs must
satisfy the fixed-path grouped-count proof, and the inner join must use their
single output key. Output aliases are supported; anonymous/internal bindings,
inherited membership filters and effectful consumers retain ordinary execution.
It borrows the already built right join table while evaluating the left
aggregate, after source-side grouping and before counting each group’s opposite
branch. Source rows and group buffers are still constructed; only the unmatched
opposite-branch work is saved. Counts for retained groups and the original
join’s multiplicity are unchanged. A deadline check precedes every membership
decision, including empty right inputs. Encountered source, allocation and
traversal errors propagate; skipped opposite branches do not evaluate potential
storage errors or count overflow.
Fixed-hop storage errors
Fixed-hop traversal preserves upstream adjacency storage and decoding errors through relationship-label and property filters. Such an error fails the query rather than silently removing the affected row. This also applies when grouped COUNT uses the same traversal. Existing property-lookup and missing-target handling are unchanged.
Secondary-label membership scans
Single-label scans of secondary labels avoid temporary decoded ID/label vectors when only membership is needed. Each visited record still validates all label IDs, even after a match, and preserves primary/secondary deduplication. Multi-label scans keep the existing path.
An experiment that propagated required labels between MATCH patterns before plan construction was withdrawn after a measured planning regression (PR #1410); the anchored-cycle policy before it is in effect.
Direct EXISTS for a bound typed edge
An EXISTS over a single fixed typed edge between two uniquely bound vertices uses a direct index seek. It preserves endpoint/label checks, direction and errors from performed reads; unsupported shapes use ordinary execution. The seek may change which unrelated records are visited. Key construction reuses the target primary label just read and validated in the same transaction; it adds no cache and assumes nothing from a label carried in an input row. No setting is added. In the LSQB comparison of this change together with the planner rollback, Q3 recovered while Q8 and Q9 kept their 8 s timeouts; no isolated speedup is claimed. See the LSQB measurement ledger.
Completed-prefix filter pruning
For a pure fixed path with unique bindings, a leading AND condition may depend only on a prefix of the chosen endpoint traversal. After the first complete row has passed the normal path consistency checks, a false leading condition can skip the remaining suffixes of that same prefix. Duplicate prefix visits and physical-edge uniqueness retain the ordinary iterator semantics. UNKNOWN still evaluates the right condition; performed-read and evaluation errors propagate. An empty suffix never causes an earlier predicate evaluation. Pruned unread suffixes need not expose errors that ordinary expansion would have encountered.
Optional or variable paths, repeated bindings, seek hints, inherited join guards, and unproven effectful consumers retain ordinary execution. A middle entry or a condition depending on the final leaf retains the ordinary iteration on the same constructed source. No query name, graph schema, new cache or forced entry order selects this optimization.
Shared mandatory labels and branch access
After selecting ordinary join order and Hash/Apply strategy, the planner can propagate an unambiguous label constraint between occurrences of the same joined vertex within one mandatory MATCH scope. It then refreshes leaf scan strategies before eligible partial COUNT grouping. The rule preserves the selected join tree and does not cross optional matches, dependent predicates, scope rebinding or writes. It has no separate configuration switch.
Label population is only an entry estimate; fewer starting vertices can still produce more traversal work, so this rule does not guarantee a faster plan. Separate MATCH clauses and unsupported scopes retain their existing access planning.
Lazy compound IN membership
A sole immutable parameter or literal IN operand may use a lazily constructed, operator-local membership set when execution first reaches that predicate. AND, OR, XOR and NOT retain the ordinary recursive evaluator: skipped branches build nothing, and lhs/null and sticky-error handling precede membership lookup. Only stable plan predicates participate. Safe scalar candidates reuse the existing numeric-equivalence guard; collection needles or unsupported candidate values retain linear three-valued membership. Row-dependent RHS evaluation is unchanged.
Hash tables and retained keys use the existing query memory admission APIs. Total admitted candidate entries are bounded per operator; admission failure is propagated without replay, and context destruction releases retained owners. This introduces no shared result cache or new configuration flag. No isolated speedup is claimed.
Factor consumers
With PATINADB_FACTORIZE on (the default since 2026-10-17, #1442; off restores
the ordinary operators), three more operators can work on a pattern that
splits at an interior pivot into two independent branches (RFC 0042 B3–B5,
#1442). They read each branch once per pivot instead of building every
combination:
- Top-k within a factor.
ORDER BY … [SKIP s] LIMIT k, when each sort key belongs to the pivot or to one branch.- Each branch is sorted once per pivot, and only combinations that can still enter the page are expanded.
- The edge-sorted and ordered-frontier paths keep priority.
- If the page contains tied keys on different rows, the ordinary bounded top-k runs instead, so tie order is unchanged.
- DISTINCT over one factor.
RETURN DISTINCTunderORDER BY(with or withoutLIMIT), andWITH DISTINCT …feeding an ungroupedcount, when the projection reads the pivot and at most one branch. The other branch only has to exist, so it stops at its first matching row. - Counts with a cross-branch comparison. Ungrouped
count(*)with onea <> b,NOT a = bora.p <> b.q(or=) between the branches is counted per pivot by inclusion–exclusion.
A branch with a single element at a pivot is folded into the pivot (flatten). When the factors stay degenerate, for example a single hop from an anchored node, the whole operator runs the ordinary plan instead. That choice is made from the operator’s own input and never changes results (ADR 0019).
PROFILE lists the executed choices under factor.topk, agg.factorized,
factor.exists and factor.flatten. EXPLAIN is unchanged.
They have no cost admission: every proven shape engages, with or without the cost model. See limitations.
Factor-local grouping
PATINADB_FACTORIZE (on by default since 2026-10-17, #1442) enables a grouped aggregate for proven
independent fixed branches. For two branches around a pivot vertex it reads
each branch’s group keys once per pivot and combines the key multisets directly
into the group table; the existing ORDER BY and LIMIT still apply afterward.
PROFILE records agg.factorized when this path executes.
It requires an admitted read memory scope. Unsupported patterns or value domains
use the reference aggregate, and factor memory refusal falls back before output.
Branches may share a relationship type when they reach the pivot through it in the same way, for example two comments of one ticket:
MATCH (u1:User)-[:AUTHORED]->(:Comment)<-[:HAS_COMMENT]-(t:Ticket)
-[:HAS_COMMENT]->(:Comment)<-[:AUTHORED]-(u2:User)
WHERE u1.name < u2.name
RETURN u1.name, u2.name, count(DISTINCT t)
Relationship uniqueness then requires two different comments. The engine checks this per ticket at run time, also when a comment has several authors. Floating-point aggregation and SUM with overflowing positive/negative subtotals are not currently admitted.
With PATINADB_COST_MODEL set to entry or join (the default join), the grouped path engages
only when the degree catalog predicts that it is cheaper than the flat aggregate; without a catalog
the flat aggregate runs. The catalog comes from the background sample the cost model requests on a
miss, or from CALL patinadb.stats.degrees(). With PATINADB_COST_MODEL=off every proven shape
engages. The weights are fitted on one benchmark host, so treat them as indicative; see RFC 0042
B1/B2 (#1442).