Protocol semantics

This page defines how a receiver turns a set of delivered events into materialized state, and why the result does not depend on the order they arrived in.

1. The apply chokepoint

Every transport MUST deliver received events through one apply path performing these checks in this order:

1. Signature verification. COSE_Sign1 verification per authentication: resolve the protected kid in the trusted-peer keyring and verify with external_aad set to the local mesh_id. An event that fails — unknown kid or invalid signature — is dropped and surfaced, never held.
2. Clock-skew bound. Reject hlc_physical_ms more than 300 000 ms in the receiver’s future.
3. Scope check. Events outside the receiver’s syncable scope are dropped and logged, not raised as errors.
4. Idempotent insert keyed on the derived event_id — BLAKE3 of the received bytes. Duplicates are no-ops.
5. HLC observe. Merge the event’s HLC into the local clock.

Order matters at two points. Signature verification is first so that no later check consumes attacker-controlled input it has not authenticated. And step 3 drops rather than raises specifically so that a hostile peer cannot wedge the apply loop by sending one out-of-scope event: an exception there would halt processing of everything behind it, which converts a filtered event into a denial of service.

A single apply path is itself normative. Two paths mean two places for these checks to drift apart, and the checks are the security boundary.

2. Materialization and conflict resolution

Events are materialized in HLC order, grouped per (site_id, schema, table).

2.1. Per-column last-writer-wins, bounded by the tombstone clock

Each (schema, table, row_key, column) carries a column clock. A column value is written only if its event’s HLC — including the site_id tiebreak — beats both the stored column clock and the row’s tombstone clock.

Column clocks advance only for writes that are actually applied. A rejected write leaves the clock untouched, so a losing event cannot raise the bar for a later legitimate one.

Resolution is per column, not per row. Two nodes editing different columns of the same row while partitioned both keep their edit; whole-row LWW would discard one arbitrarily.

2.2. Causal-length existence

Row existence is tracked separately from column values, and the highest-HLC existence event wins. A delete tombstones the row; a later, higher-HLC upsert resurrects it.

A resurrection SHOULD be surfaced to the application rather than applied silently, because a row reappearing is usually either a genuine re-creation or a sign of a clock problem, and the application is the only layer that can tell which.

2.3. The tombstone clock

Each row additionally tracks the highest-HLC delete ever applied to it, independent of whether that delete won existence.

A delete — winning or losing — MUST clear (set to null) every column whose clock it beats, and advances the tombstone clock if it exceeds it.

This is what makes a delete’s column-clearing effect independent of arrival order. A column write below the tombstone is rejected whether it arrives before the delete or after it. Without a separate tombstone clock, “delete then late-arriving write” and “late-arriving write then delete” produce different states, and the protocol loses convergence.

2.4. Resurrection rebuild

When an upsert resurrects a tombstoned row, the receiver MUST rebuild the row’s columns from its retained event log. For each column, the value is supplied by the highest-HLC upsert that:

• carries that column, and
• beats the tombstone clock, and
• passed the apply checks, and
• is not below the eviction horizon.

This restores concurrent post-tombstone writes that arrived while the row was absent. Without the rebuild, a write that arrived during the tombstoned interval is lost even though it causally succeeds the delete — a cross-site delivery-order defect that is invisible in single-node testing and only appears when two peers interleave a delete and a write.

2.5. Convergence

With the preceding rules, materialized state is a pure function of the delivered event set. Any delivery order permitted by delivery — per-peer in-order, arbitrary cross-peer interleaving, at-least-once duplication — converges to the same state.

This rests on exactly one precondition: a node MUST NOT reuse an HLC across distinct events. The tick discipline in the data model guarantees it. An implementation that reuses an HLC — by restoring a clock from a stale backup, for instance — breaks convergence, and no other rule on this page compensates.

Implementations SHOULD verify convergence by property-based testing over randomized delivery orders rather than by example, because the failure modes live in interleavings a hand-written test is unlikely to pick.

2.6. Held and dropped groups

Condition	Behaviour	Watermark
Destination table missing	Group is held and retried in a later cycle	Does not advance past it
Signature verification fails	Event is dropped and surfaced	Advances past it
row_key names no destination column	Event is dropped	Advances past it

The middle row deserves its rationale, because holding would seem safer and is not. Events are attributed by their signed kid, not by the connection that delivered them — so if verification failures held the attributed site’s group, a forger could stall the victim’s legitimate stream by sending garbage carrying the victim’s kid. A forgery must cost the forger, not the node it impersonates.

The last row is an adversarial-key defense: a peer that sends events keyed on non-existent columns must not be able to stall the watermark forever. No destination mutation runs, and the watermark advances.

Held-versus-dropped is the distinction to get right. Holding is for conditions that may resolve — a table that has not been created yet. Dropping is for conditions that will not, and for anything a hostile peer could otherwise weaponize into a stall.

3. Delivery

3.1. Cursors

A sender maintains one durable cursor per peer per transport path. Cursors advance only after a successful push. A failed push leaves the cursor untouched, so the same batch is re-offered on the next cycle.

Delivery is therefore at-least-once, and correctness rests on receiver idempotency rather than sender exactly-once. That is the cheaper and more robust half of the trade: idempotency is a local property a receiver can guarantee alone, whereas exactly-once delivery requires agreement the transport cannot provide.

There is no retry-backoff protocol. The retry unit is the next sync cycle.

3.2. Applied watermarks

A receiver advances a local applied watermark per contiguous applied chunk, and reports it to peers. A watermark is an HLC value per (site_id, schema, table) meaning “everything from this site for this table at or below this HLC has been applied”.

Watermarks MUST be monotonic and MUST fail closed: the absence of a watermark means not-applied, never applied. A receiver MUST NOT infer progress it cannot demonstrate.

Watermarks are an acknowledgement optimization, not a correctness mechanism. A sender that ignores them entirely and re-offers from its cursor is still correct, because apply is idempotent. Their purpose is to let a sender stop re-offering data the peer demonstrably has, and to let an operator see how far behind a peer is.