History epochs (RFC-0038)

A history epoch is a contiguous section of one database history. It is written once, as a verified, immutable archive object, to a volume the operator provides. After an explicit, cluster-wide hot release, the members remove the epoch’s delta and snapshot bodies from local (“hot”) storage. AS OF reads, diff and tag exports of released history are then served from the archive.

Everything is opt-in. Without a policy or an operator request nothing is cut, nothing is released and nothing is deleted.

Requirements

• Every cluster member runs a release with peer protocol 18. Sealing is refused until all members are confirmed at 18. Once the cluster holds any epoch record, a new member must also speak 18.
• The database has an incarnation. A database gets one automatically once the cluster is confirmed at protocol 17.
• PATINADB_EPOCH_ARCHIVE_DIR is set to the same absolute directory path on every member, and every member sees the same shared volume. patinadb-raft --check-config rejects a relative path. The value is node-local; it never enters Raft.

A shared volume is not a backup. Losing it loses the archived copy, and after a release that copy is the only one. Back the volume up yourself, or take self-contained portable backups (see Backups and restore).

Sealing an epoch

An operator cuts an epoch:

curl -X POST http://leader:7474/mgmt/epochs \
  -H 'content-type: application/json' \
  -d '{"database": "sales", "upper": "<engram id>", "request_id": "<optional uuid>"}'

• upper must be an engram on the database’s current history chain.
• The first epoch starts at the chain root. Every later epoch starts exactly where the previous epoch ended. The server chooses that lower boundary.
• The leader validates the whole section against its own history and refuses if it cannot vouch for it, for example because of a history floor or fence.
• The call requires the admin role and access to the database, and is recorded in the audit log.

While an epoch is sealed, squashes of that database are refused, because a squash would rewrite the history being archived:

• CALL patinadb.squash fails with a non-retryable conflict.
• Prepared squash jobs are refused.
• Licence- or retention-driven squashes are deferred and logged.

Ordinary writes continue.

Automatic cuts (policy)

A per-database policy can cut epochs automatically. It is off by default and has no shipped thresholds:

curl -X PUT http://leader:7474/mgmt/epochs/policy \
  -H 'content-type: application/json' \
  -d '{"database": "sales", "max_engrams": 100000, "max_age_days": 90, "release_after_days": 7}'

• max_engrams cuts once that many engrams follow the previous epoch.
• max_age_days cuts at the newest engram at least that old.
• With both set, the newer boundary wins.
• release_after_days releases a committed epoch automatically after that grace period. Without it, release stays a manual step.

Every field is optional; a request without any field clears the policy. GET /mgmt/epochs/policy?database=sales shows it. The leader evaluates policies periodically and only while no seal is open. Engram timestamps only decide when to cut; the boundaries come from the ancestry. If HEAD’s timestamp lies more than five minutes in the future, or timestamps inside the candidate section run backwards, the age trigger is skipped and GET /mgmt/epochs/status reports clock_insane. The count trigger still applies.

Lifecycle

The leader exports a sealed epoch in the background, on the same lane as other maintenance artifacts, publishes the object and then commits the digests through Raft.

GET /mgmt/epochs?database=sales lists the catalog. On the serving node it also shows local progress of export, release barrier and local release: the attempts, the last error and whether work is running. Failed attempts are retried with increasing backoff, up to 10 minutes.

POST /mgmt/epochs/{seal_id}/verify checks the archive on the serving node. It returns verified, missing, corrupt, not_configured or not_committed. The result applies to that node only.

Hot release

curl -X POST http://leader:7474/mgmt/epochs/<seal_id>/release

Release is strictly all-member:

1. The leader asks every current member, learners included, for a receipt. Each member verifies the archive object completely (length, SHA-256, structure and identity) before it answers.
2. Only with a fresh positive receipt from every member does the leader propose the release. Apply re-checks the membership and that no tag or pin was created in between. A member that is offline holds the release up; there is no quorum shortcut.
3. Each member applies the release locally, after verifying the archive itself. It keeps a hot snapshot at the epoch’s upper boundary, indexes the range as released, then removes the range’s delta and snapshot bodies. Each step is durable before the next, so after a crash the node holds either the old hot state or a state the next attempt completes.

Releases go strictly in chain order; the released part is always a prefix of the history.

The release blocks (release_blocked, archive and hot data kept) when:

• a member reports that it cannot read the object (missing, wrong digest, wrong identity);
• a tag or pin lies inside the range (tags stay hot);
• receipts are missing for longer than the block period.

A blocked epoch needs a new POST …/release, or with release_after_days a new automatic attempt after another block period.

After a release:

• AS OF, patinadb.diff, snapshot diff and tag exports of released engrams hydrate a temporary, fully indexed store from the verified archive segment. At most two hydrations run per process; requests keep their deadline.
• Reads newer than the released range use the hot snapshot at its upper boundary and never touch the archive.
• A missing or unreadable volume fails the read with HISTORY_ARCHIVE_UNAVAILABLE (retryable). A damaged or mismatched object fails with HISTORY_ARCHIVE_INTEGRITY. A read is never answered from HEAD or with an empty result instead.
• CREATE TAG (also on HEAD when HEAD is the released boundary) and pins into released history are refused. Apply decides this against the replicated release, so a tag committed after the release is refused even before members have removed their local bodies; a tag committed first refuses the release instead.
• A fork whose fork point lies in released history is refused.
• CALL patinadb.squash and prepared squash jobs are refused for that database. A retention squash whose cutoff lies above the released range still runs.
• A retention cutoff inside the released range raises the history’s visibility floor instead of squashing. Reads below it fail with HISTORY_EXPIRED, even though the archive still holds them.

New members

Once anything is released, a new member is admitted only after it returns positive readiness receipts for every released epoch it may have to serve: those of each database’s current history version that retention has not expired entirely. A learner that joins from a physical snapshot backfills the released range’s metadata from the archive, not from peers, and the install restores the retention floor, so expired targets answer HISTORY_EXPIRED on the new member as well. For fully expired epochs the leader publishes a metadata-only artifact and commits its hash and length through Raft (protocol 20 required). A new member verifies this artifact before completing its chain, even after the full archive was collected. This does not restore expired query results or synthesize missing deltas (#1428).

For archives collected by older releases, a node with complete retained metadata can still publish the artifact. If neither the full archive nor an authoritative metadata source remains, backfill stays incomplete and retries; no missing ancestry is invented.

Archive collection

Nothing in the archive is deleted unless PATINADB_EPOCH_GC is set. The collector runs on the leader, at most once an hour, and removes only:

• staging files older than the grace period;
• objects no catalog entry references (leftovers of cancelled or superseded seals, duplicates), older than the grace period;
• with PATINADB_EPOCH_GC_BELOW_RETENTION also set: archives that lie entirely below their history’s retention floor, only after their replicated metadata artifact has been verified locally. Missing or corrupt metadata prevents full archive deletion. The metadata artifact remains referenced for the lifetime of its catalog record; orphaned metadata follows the ordinary grace period (#1428).

It never removes an object that a catalog entry of any incarnation (including an imported reference), an archive hold or a running export refers to. Licence loss or expiry never deletes anything. PATINADB_EPOCH_GC_GRACE_SECS sets the grace period (default 604800, seven days).

GET /mgmt/epochs/status shows the configured volume, trigger evaluations and the last collection pass.

Backups and restore

A ?history=true portable backup is self-contained by default: it re-embeds every released epoch from its verified archive segment. It restores on a cluster without any archive volume, and the restored history is entirely hot. Taking it needs a readable archive; without one the backup fails with HISTORY_ARCHIVE_UNAVAILABLE rather than produce a shortened history. History below a retention floor is not exported; the oldest visible engram becomes the restored history’s root.

GET /mgmt/snapshot?history=true&references_only=true keeps released epochs as references instead. The backup carries their catalog records, the hot snapshots at their boundaries and the retention floor, and places an archive hold on the referenced objects. List holds with GET /mgmt/epochs/holds and remove one with DELETE /mgmt/epochs/holds/{hold_id} once the backup is no longer needed.

A restore of a references_only backup:

• needs every member at protocol 18;
• verifies every referenced object on the serving node first. It is refused before anything is written when PATINADB_EPOCH_ARCHIVE_DIR is not set or an object is missing or damaged (HISTORY_ARCHIVE_UNAVAILABLE in the message);
• imports the epochs into the restored database’s new incarnation as read-only references to the source objects. They are never collected and never overwritten by the restored database.

references_only applies to GET /mgmt/snapshot JSON backups only. format=archive backups and asynchronous backup jobs always re-embed. An offline patinadb-admin archive backup has no archive reader, so it fails for a database with released history; take that backup online.

Archive layout and offline verification

<root>/v1/<incarnation>/g<history generation>/<seal_id>/segment-<sha256>.bin
<root>/v1/<incarnation>/g<history generation>/<seal_id>/manifest-<sha256>.json
<root>/.staging/

Object names are their SHA-256 digests. An existing object is never overwritten. A second export of the same seal, for example after a leader change, accepts an existing file only if it verifies to exactly the same content.

Verify an object offline, without a server:

patinadb-admin epoch-verify /srv/epochs/v1/.../manifest-<sha256>.json --replay

The command checks the manifest digest against its file name, the body length and digest, the record format and the parent chain. --replay also rebuilds the newest state of the epoch in a scratch directory (--scratch, by default the system temporary directory) and reports the number of vertices and edges.

Limitations

See Limitations.