First Customer: EU Writers, Global Readers, One Bolt Backend

For revision 0.15.0, the earlier two-learner 1M run failed its original relative latency gate (#1261). The user subsequently selected p90 <= 300 ms; relative 3x increases now warn in explicit backup/fan-out profiles. Old reports stay.

This is the scoped acceptance checklist for the deployment reviewed under issue #1261: three voting nodes in Europe, nonvoting learners near read traffic, and one trusted backend type that owns application RBAC. Application traffic uses Bolt; private management HTTP/CLI remains available to operators. This is not a production certification.

Repository scripts/search-blackbox/ adds an independently authored Rust/neo4rs search backend and source-derived HTTP oracle (#1261). The local two-hour test on 334,770 Jira summaries passes search/index correctness, bounded write-token recovery, ANN recall and offered-load admission across backend/DB restarts. Fault-time unavailability remains explicit:40 of14,400 read operations did not complete successfully, but no wrong answers or unresolved writes remained. This one-DB test does not qualify regional routing, concurrent multiwriter load or larger corpora. Method and full evidence: repository docs/search-blackbox-1261-2026-09-17.md. The Rust sample uses optional X-Write-Token with payload/principal-scoped native deduplication and a bounded replay window, not an indefinite request ledger. Its pinned neo4rs driver ignores URI routing-context parameters; the sample rejects those parameters rather than silently selecting another region. Configure an explicit regional bolt+s:// reader address separately from the routed neo4j+s:// writer connection. This is static reader selection, not automatic regional discovery or proven failover. Private CAs use NEO4J_CA_FILE. Read its API-SPEC and the token contract.

Driver qualification boundary: an earlier #1261 run exposed invalid write acknowledgements after a timeout. A separate scripted control reproduces pinned neo4rs’s own receive timeout returning a connection with outstanding protocol replies to its pool. The owned-task protection below covers caller cancellation, not this internal timeout. The sample now pins a local transport-safety patch that discards connections with unfinished replies or failed I/O instead of reusing them. Negative controls retain the unmodified published driver. The subsequent six-hour five-node/three-backend run passes216000reads and2160writes, both disjoint/shared phases, all eight controlled faults and final checks (#1261, docs/search-cluster-soak-1261-2026-09-17.md). This is local qualification, not independent-host production approval; results do not establish compatibility with unpatched neo4rs. The HTTP boundary now rejects empty/malformed write bookmarks as unknown outcomes rather than returning a successful ACK; no causal bookmark is invented.

The sample also renews its driver Graph after routing-discovery failures, for the next operation only. Pinned neo4rs can otherwise retain a writerless routing table after leader loss (#1261, cluster ledger and tests/routing_recovery.rs). This is an application-side mitigation, not an upstream driver fix: renewal requires the configured seed to be reachable. It does not retry the failed mutation, change its token or bookmarks, or turn an unknown write into an abort. Concurrent operations already using the old Graph finish independently; their late failures cannot discard a newer Graph. Live failover qualification remains required before relying on this sample for production.

The vendored driver additionally invalidates a cached database routing table after an explicit NotALeader or ForbiddenOnReadOnlyDatabase rejection in auto-commit run/execute, before its existing retry. This fixes retries targeting the former writer until the routing TTL expires. Discovered router entries are retained. It does not increase deadlines, add retries for ambiguous transport failures or replay explicit transactions; the writerless-Graph mitigation above remains separate. This local patch is not an upstream release.

The corresponding six-hour SSD fault run passes 216,000 reads / 2,160 writes, all eight faults and final oracles, with no failures outside injected-fault intervals plus the 60-second recovery window (#1261/#1304, docs/ssd-fault-soak-next-1261.md). The baseline had 21 late failed attempts; successful eventual token reconciliation alone had hidden that recovery defect. Raw failures during faults remain in the report. This qualifies the pinned sample/driver combination on one host, not unpatched neo4rs or a real WAN.

The follow-up cluster test reproduces a cancellation hazard in pinned neo4rs0.9.0-rc.10: cancelling an in-flight RUN can leave protocol replies that corrupt the next pooled request (#1261, repository cluster-soak report and tests/cancelled_driver.rs). The sample now bounds response waits without cancelling driver operations: at most16 owned tasks drain them before returning connections. Timed-out writes remain unknown outcomes, not rollbacks. Stalled tasks retain admission slots, so this trades saturation for safe bounded behavior; it is an application mitigation, not an upstream driver fix or production gate.

A documentation-driven sample backend and independent HTTP-only consistency simulator now live in repository scripts/customer-blackbox/ (#1261). They test backend-owned project/role authorization, opaque causal bookmarks across two backend processes, pagination/counts and recovery with three voters/two learners. Two local seeds pass; a deliberate value divergence is correctly rejected by the oracle. This is a small generated application fixture, not the customer’s actual backend or a1M-scale qualification. See repository docs/customer-blackbox-1261-2026-09-16.md for failed attempts, source provenance, limits and reproducible commands. The sample HTTP service is not a hardened public web server; keep it on loopback unless separately hardened.

The separate one-million-ticket extension passes its local steady-state gate with three voters, one shaped learner and two backends: 2,400 paced reads, 90 updates and 90 cross-backend causal reads, no dropped samples or wrong answers (#1261, docs/customer-blackbox-million-1261-2026-09-16.md, final run9). Mixed query p90 ranges from 3.757 to 116.561 ms; updates 8.784 ms and causal reads 18.918 ms. The sample uses separate global-write/regional-read drivers with redundant seed resolution. Restart/partition recovery retains acknowledged state, but leader loss still exposes unknown write outcomes and minutes-long fresh-reader catch-up. Unbounded first-page checks remain above the 300 ms paced gate. This is not full production approval or a substitute for idempotent write reconciliation and real WAN tests.

The RFC0032 archive/admin path passes the local 1M-ticket post-squash mixed-load backup and fresh-cluster HEAD/PITR restore gate (#1261, 2026-09-14; engineering ledger docs/archive-qualification-1261-2026-09-14.md). Tagged history, ordinary Bolt reads and non-publication controls pass. This is not off-host DR, actual disk exhaustion or physical learner throughput qualification. The earlier JSON PITR run exceeded its configured byte cap and did not reach restore; its failed report remains unchanged. See offline archive administration for its operational contract.

Latest-first learner qualification uses a separate end-to-end budget: checkpoint construction through the first successful causal Bolt read, not transfer alone. Historical completeness and historical content are checked separately; a node answering current reads does not imply all earlier Engrams are locally available. The renewed local gate passes causal/current reads before history completion and subsequent pre-cut historical-content checks. Its measured full-interval rate is 50.056 MB/s, so the strict >100 MB/s speed gate remains failed (#1261, 2026-09-15; engineering ledger docs/latest-onboarding-1261-2026-09-15.md). Follow-up transport batching reduces per-fragment file I/O scheduling, without removing verification or durable installation. The post-merge repeat does not demonstrate an end-to-end speedup; component benchmarks do not override this failed onboarding gate. Causal reads succeed after 84.693 s and history completion is observed after 114.257 s in that local 1M run, not an actual WAN deployment.

Recommended deployment contract

• Put the three voters in independent European failure domains. Measure actual inter-voter latency; “Europe” and a timezone are not latency budgets. Losing one voter preserves a majority; losing two stops writes.
• Keep remote replicas as learners. They are full-data replicas, not shards, and cannot automatically take over if the European majority is lost.
• Use an EU routing driver for writes and explicitly selected regional direct drivers for reads. Use verified TLS (neo4j+s:// / bolt+s://). A direct driver URI selects one endpoint, not a regional failover list; the backend must manage that policy. Alternatively use the opt-in regional routing from RFC-0027 with scoped bookmarks; see Multi-Region Topology.
• Pass causal bookmarks between write/read pools and backend instances whenever a request must observe an earlier write. A bookmark timeout is a real failure, not permission to retry without the bookmark. Unbookmarked replica reads may be stale; /ready is not a bound on WAN delivery lag.
• Configure max_leader_contact_age_secs on regional learners using the application’s chosen admission budget. This opt-in guard also checks the last advertised leader commit against local apply progress and readiness_max_lag, and rejects stale admission through routing, readiness, local HTTP and Bolt. It is disabled by default; it is not a quorum lease or a guarantee about the age of every result in a long-running transaction.
• Keep database ports private. End users never receive DB credentials or send arbitrary Cypher. Give the backend a non-admin, database-scoped service account; keep migrations, backups, peer secrets and root credentials separate.
• Install entitlements allowing three voters and the intended data/history size on every node. Community defaults do not authorize this voter topology.

RBAC is an application responsibility, not disabled authentication

Backend-owned authorization fits this first customer if the backend is the only trusted data client. It does not require implementing every pending database property-level privilege in #1231 first. Database authentication, least-privilege service accounts, parameterized queries and network isolation still matter.

Define where authorization state lives. If permission checks read ACLs or revocations from this graph, an arbitrary regional read is not automatically fresh enough. Causal bookmarks order reads relative to known writes; they are not an unconditional linearizable read API. Bolt does not expose the REST linearizable-read option. Either keep authoritative authorization in the backend’s own consistent system or explicitly resolve this contract before launch. Do not silently treat a stale allow decision as current.

What needs proving before launch

The initial customer proxy workload is the repository’s Jira benchmark, starting at one million tickets before six million and the eventual sixty-million target. Local load targets are two reads/second and one ticket update every ten seconds, with a separate configurable nightly-upsert workload. The backend owns ACLs internally and connects using a scoped technical user; its source and actual driver are not available for this test. Explicit backup/fan-out profiles require per-query p90 <= 300 ms and warn above three times quiet-system p90 (user decision, 2026-09-08, #1261). Errors and missing coverage still fail; older measurements and Q6/Q8 exceptions are historical, not silently regraded. One-host measurements are not independent-region or off-host evidence.

The large onboarding harness uses real HTTP snapshot transport and Raft write probes. It does not replace the customer’s Bolt workload or a multi-continent network test. The 1M local measurements and their acceptance boundaries are in docs/jira-rollout-execution-2026-09-06.md in the source tree.

Priorities, not a larger prerequisite backlog

0.17.0 candidate: remaining launch gates (2026-09-10)

The fulltext optimization is measured on 334,770 real Jira summaries, not a combined graph/vector/fulltext workload at the eventual customer scale. The 0.17.0 results do not replace the release gates in #1261. Prioritize the following evidence rather than expanding language features:

1. Recoverability first. The post-0.17.0 local follow-up now passes retained 1M-ticket HEAD and PITR restores, historical probes and Q1–Q10 reads (#1261). It fixes per-entry durable writes during compound-index construction; the earlier timeout remains a failed baseline. This follow-up is not yet a merged release. Next prove restoration into a fresh deployment from an independent off-host copy, with history, indexes and service-account checks, and agree on measured RPO/RTO. See source-tree docs/restore-1261-2026-09-10.md (#1261).
2. Combined workload and recovery. Run the final candidate’s automated release gate, then the representative 1M Jira workload with the indexes that the backend actually uses, nightly upserts and concurrent learner onboarding. Progress to 6M only after passing, with RAM/disk measurements. Repeat voter loss, learner restart and partition recovery while preserving causal reads. Separate graph readiness, index readiness and complete historical readiness.
3. Cold onboarding. The measured large prepared-checkpoint transfer is not proof of the requested >=100 MB/s cold path. Checkpoint creation included, the recorded result remains below that target (#1251). Re-measure the whole path under workload; initial indexing improvements alone do not establish a faster physical checkpoint copy. See Learner onboarding. A fresh logged 1M-ticket run on 2026-09-10 passes graph/catch-up checks: build 31.268 s, join/current HTTP read 11.697 s, then 25.678 s for history completeness (#1261). Build-plus-join is 161.783 MB/s allocated-source but 98.659 MB/s snapshot bytes; this does not pass a strict 100 MB/s snapshot-byte gate. OS caches were uncontrolled; causal Bolt and query/index readiness remain unqualified by this instrument. See the repository ledger docs/cold-onboarding-1261-2026-09-10.md. The subsequent local 1M combined run, with a not-yet-released snapshot staging fix (#1287), passes 1,200 mixed reads, 60 updates and 30 batches without errors or drops; highest source/learner query p90 is 122.618/110.617 ms (#1261). Its continuous trigger-to-causal-Bolt-read timer is 79.343 s (53.431 MB/s snapshot bytes), so the strict onboarding speed target remains unmet. Learner SIGKILL recovery passes via retained logs and via snapshot plus tail; release/partition controls pass separately at small fixture sizes. See docs/combined-rollout-1261-2026-09-10.md. 6M is explicitly deferred. Administrative CREATE TAG caused foreground write stalls in an earlier run; the passing mixed workload does not qualify tag creation under load.
4. Actual backend and network. Qualify the real driver and query templates, cross-pool bookmark propagation, regional selection/fallback, ambiguous commit retries and token retention/capacity. Run verified TLS and actual WAN tests; a one-host TCP impairment test is narrower. Backend-owned RBAC avoids requiring per-property database privileges, but still needs private endpoints, scoped credentials and an explicit freshness contract for authorization.
5. Operational acceptance. Verify alerts and runbooks for failed backups, free disk, replication/apply lag, incomplete history, clock skew and expiring certificates/entitlements. Three European voters need independent failure domains; worldwide learners do not provide an automatic write quorum after loss of the European majority.

Before releasing a candidate, run python3 scripts/release-gate.py from its source checkout with an explicit new output directory and the driver Python environment. The source-tree docs/release-gate.md gives the full command and prerequisites. It bundles token/retention/capacity, ambiguous-retry soak, crash, causal recovery, local restore and official-driver checks. Missing tests or missing actual dedup hits fail rather than producing a misleading green verdict. The user selected idempotency tokens for the first customer on 2026-09-08; capacity/retention and clock-monitoring requirements remain mandatory. The JSON result is local engineering evidence, not customer-backend, real-WAN or off-host qualification, and never publishes a release automatically.

Fix correctness/security regressions before throughput tuning. Track durable whole-install crash recovery in #1263, current cache availability in #1258, conservative startup-format selection in #1257, public version metadata in #1260, native Bolt ROUTE semantics in #1262, and delivery-lag freshness in #1133. The integration sweep and its current validation are recorded in docs/p1-p2-sweep-2026-09-06.md; merged fixes do not close those release obligations.

The follow-through audit also fixes query normalization (#1264) and Bolt authentication transitions (#1265). Checkpoint publication now separates immutable .json artifacts from mutable .recv transfer files; an old-version partial transfer can restart from zero during upgrade. See docs/bug-audit-2026-09-06.md in the source tree for validation boundaries.

PRs #1266 and #1267 are merged. The current local rollout wave adds six bounded acceptance cases: the three-voter/five-learner entitlement boundary, learner outages plus leader loss, replicated RBAC role downgrade through snapshot and failover, and a portable PITR restore after source-data loss with graph, index, user-role and historical-read checks, unconfirmed-learner removal recovery, and eviction refusal with a health-up but Raft-partitioned voter. These are local regression cases, not an aggregate gate result or customer qualification. The external workload, WAN, off-host restore, operations and retry-release evidence in the table above remain required before launch.

The first local 1M-ticket measurement is not a release approval. Physical current-state onboarding and current-state recovery pass their local checks, but two deep traversal-page query families exceed the requested relative latency budget under updates; the user accepted those measured Q6/Q8 exceptions for this workload. A separate pre-cut tag check exposed an improperly finalized raw benchmark import, not qualified production history. Unlogged imports need an explicit history base; delta-only asynchronous backfill does not transport snapshot-derived bases. Do not treat current readiness or the history status flag alone as proof that historical answers are complete. See docs/jira-rollout-execution-2026-09-06.md in the source tree.

The fully logged 1M-ticket local worker drill additionally passes real SIGKILL and same-directory restart while voters acknowledge creates, updates and deletes: both retained-log catch-up and a purged physical snapshot followed by newer log entries. See docs/jira-worker-restart-2026-09-07.md for exact evidence and scope. A restarting worker may refuse a causal read with Neo.TransientError.Transaction.BookmarkTimeout while behind. Snapshot directory replacement may invalidate an existing session with Neo.ClientError.Security.Unauthorized and authorization changed; authenticate again. The backend needs bounded read retries preserving the bookmark and, for that specific authorization-generation change, a fresh authenticated connection. Do not retry arbitrary authorization failures or replay acknowledged writes.

Region-aware server routing, native Bolt client-certificate authentication, property-level DB RBAC and epoch/cold-history storage (#1158) are optional for this private-backend contract unless explicitly required. They should not obscure the missing workload, failure and restore evidence.

See Production Deployment, Multi-Region Topology, and Disaster Recovery for operating procedures.