Bulk Loading & Import

patinaDB offers three ways to get a lot of data into a running server, all of them versioned (recorded in the engram log) and, on a cluster, replicated:

Bulk upsert (large files, match-or-create)

POST /mgmt/upsert, /mgmt/upsert-edges, and /mgmt/upsert-graph resolve and apply a whole columnar file (CSV / Parquet / Arrow) — or a coupled JSON node+edge subgraph — as one changeset, per-row match-or-create against a match key, index-accelerated. A chunk_rows field splits a very large load into many bounded, independently-committed engrams so peak memory (and Raft entry size) stays O(chunk) instead of O(rows).

curl -s -u neo4j:secret -X POST localhost:21001/mgmt/upsert \
  -H 'content-type: application/json' \
  -d '{"db":"default","file":"/srv/import/people.csv","label":"Person","match":["email"]}'

The file path is confined to the server’s filesystem sandbox (see below); this is the fastest, largest-scale online path — see Bulk Upsert for the full request shape, merge/conflict policies, and chunking guidance.

Online: LOAD CSV

LOAD CSV streams rows from a CSV file into a running query as a row source — like UNWIND, but from a file. It goes through the normal write path, so each load is versioned (an engram) and, on a cluster, replicated.

LOAD CSV WITH HEADERS FROM 'file:///data/people.csv' AS row
CREATE (:Person {id: toInteger(row.id), name: row.name})

• WITH HEADERS makes each row a map keyed by header (row.name); without it each row is a list of string cells (row[0]).
• Cells are strings — coerce with toInteger / toFloat / toBoolean. A coercion may sit directly in a CREATE/MERGE pattern property (as above); a row[i] list-index expression still needs a WITH stage first.
• FIELDTERMINATOR '<c>' overrides the , delimiter. http(s):// URLs are rejected; only file:// and bare/relative paths are read.

The full clause reference is in Cypher Support → Loading CSV.

The un-chunked memory cliff

A bare LOAD CSV … CREATE streams the source row by row, but buffers all resolved write operations into one transaction (on the server, one Raft entry) before committing — an O(rows) memory cliff for a large load. A LOAD CSV … RETURN row read query streams end to end and has no such cliff.

Online + chunked: CALL { … } IN TRANSACTIONS

Wrapping the write subquery in CALL { … } IN TRANSACTIONS chunks the load into many small, independently-committed transactions — the online answer to the RAM cliff. Peak memory drops to one chunk, and on a cluster each chunk replicates as one small Raft entry instead of one unbounded one.

LOAD CSV WITH HEADERS FROM 'file:///data/more_people.csv' AS row
CALL {
  WITH row
  CREATE (:Reader {id: row.id, name: row.name, age: toInteger(row.age)})
} IN TRANSACTIONS OF 2 ROWS

Loading a 5-row file OF 2 ROWS produces ⌈5/2⌉ = 3 chunks — and three separate engrams, one per committed chunk:

MATCH (r:Reader) RETURN count(r) AS n
-- n: 5
CALL patinadb.engrams() YIELD id RETURN count(id) AS n
-- n: 3 (for this one load, on a fresh database)

• Each chunk is its own commit ⇒ its own engram, so batched ingest is versioned and time-travellable. A later chunk sees the writes committed by earlier chunks (read-your-writes).
• OF <n> ROW[S] sets the chunk size (default 1000).
• ON ERROR — FAIL (default) aborts the whole statement on a failing chunk; CONTINUE skips it and commits the rest; BREAK stops after it, keeping earlier chunks. A skipped/failed chunk changes nothing — it is resolved against a throwaway copy of the graph, so a partial chunk is never left behind.
• Crash-atomic per chunk: a crash between chunks recovers to a chunk boundary. The graph and the engram log always agree.
• The CALL { … } IN TRANSACTIONS must be the final clause of the query, and it cannot run inside an explicit Bolt BEGIN … COMMIT transaction.

On the server the driver proposes one client_write per chunk, so a 250-row load OF 100 ROWS advances the applied index by 3, not 1 — bounded Raft entries, read-your-writes preserved across chunks on the leader.

Server file-I/O sandbox

Reading a file:// URL, or writing with the export procedures, touches the server’s filesystem. On the server both are deny-by-default and gated by two layers:

1. Directory sandbox — --allow-csv-dir <dir> (reads) and --allow-export-dir <dir> (writes) whitelist directories; unset means every file access is refused. Paths are canonicalized, so .. traversal and symlink escapes are rejected.
2. Authorize by effect — any query that does file I/O is raised to require the global Admin role, because file access is a host-level capability, not a graph-data one. A per-database Writer cannot read or write host files.

/mgmt/upsert and its siblings confine their file path through the same --allow-csv-dir sandbox and are admin-only by virtue of living under /mgmt/. See Authentication & TLS → Cypher-driven file I/O.

Which path should I use?

• A large columnar file (or subgraph document) already on the server → POST /mgmt/upsert/upsert-edges/upsert-graph (fastest, index-accelerated match-or-create, chunkable).
• Live ingest into a running / clustered database from a streamed CSV → LOAD CSV … CALL { … } IN TRANSACTIONS OF n ROWS (versioned, replicated, bounded memory).
• A small load, or you need the CSV as a read source → plain LOAD CSV … CREATE / LOAD CSV … RETURN row.