How it works

This page walks the mechanism end to end. It is descriptive; where it simplifies, the specifications are authoritative.

The running example is four machines under one trust authority, named by role rather than by form factor — the mechanism does not care whether a collector is a phone or a field sensor:

Node	Role
W	workstation — full access, does compaction and queries
C	collector — produces data, intermittently connected
S	server — stores everything, reads only part of it
A	archive — stores everything, reads nothing

Domains: who can read what

Machines are grouped into a small number of cryptographic domains. A domain is an MLS group whose membership is exactly the set of machines allowed to decrypt that domain’s data.

flowchart LR
  accTitle: Domain membership across the four nodes
  accDescr: The control and general domains contain the workstation, collector and server. The restricted domain contains only the workstation and collector, so the server cannot read it. The archive belongs to no domain at all.
  CONTROL["CONTROL<br/><small>mesh control messages</small>"] --- W
  CONTROL --- C
  CONTROL --- S
  GENERAL["GENERAL<br/><small>most data</small>"] --- W
  GENERAL --- C
  GENERAL --- S
  RESTRICTED["RESTRICTED<br/><small>S cannot read it</small>"] --- W
  RESTRICTED --- C
  A["A &mdash; no domain"]

Two things are worth noticing.

The archive A is in no domain at all. It stores ciphertext and holds no keys. The server S also stores RESTRICTED objects, but is not a RESTRICTED member, so it cannot read them either. This is the storage-is-not-reading rule: the set of machines that hold bytes and the set that can decrypt them are tracked separately, and neither is derived from the other.

The specification recommends starting with one data domain and adding a narrower one only when a machine genuinely exists that must not read something. The governing question is “which machine must NOT read this” — until there is an answer, extra domains are pure overhead.

Keys: three levels, deliberately

flowchart TD
  accTitle: The three-level key hierarchy
  accDescr: MLS group secrets distribute the domain key-encryption key, which is an independently random application key rather than an MLS secret. That key wraps a fresh per-object data key for every object.
  mls["MLS group secrets<br/><small>managed by MLS, forward-secret, disposable</small>"]
  kek["domain KEK generation<br/><small>an independently random 32-byte application key</small>"]
  dek["per-object DEK<br/><small>a fresh random key for every single object</small>"]
  mls -->|distributes| kek
  kek -->|wraps| dek

The middle level exists because MLS is designed to forget things. MLS rotates its secrets constantly and deletes old ones — exactly right for messaging, exactly wrong for data you intend to read in five years. So the domain key is not an MLS secret. It is an ordinary random application key that MLS is used only to distribute confidentially.

The practical consequence: routine MLS activity never touches your data keys. A new domain key generation is minted only on a membership-security event — a removal, a suspected compromise, or an addition that must not gain history.

Writing an object

The collector accumulates 100,000 measurement rows destined for RESTRICTED.

flowchart TD
  accTitle: Writing an object
  accDescr: Rows are encoded to Parquet, encrypted with a fresh random data key and an encrypted footer, then content-addressed by the BLAKE3 root hash of the encrypted bytes. Four artifacts follow: the persisted ciphertext, a signed immutable manifest, a sealed access record carrying the data key, and the published catalog entry.
  rows[rows] -->|encode| parquet[Parquet]
  parquet -->|"encrypt with a fresh random DEK, encrypted footer"| obj[encrypted Parquet object]
  obj -->|content-address| sid["storage_id = BLAKE3 root hash of the encrypted bytes"]
  sid --> persist[persist the ciphertext locally]
  sid --> manifest["sign an immutable ObjectManifest<br/><small>what this object is</small>"]
  sid --> access["seal an AccessRecord<br/><small>the DEK, wrapped under the domain key</small>"]
  sid --> catalog[publish the catalog entry]

Three separate artifacts, and the separation is the point.

• The object is immutable encrypted bytes. Its identity is the hash of those bytes, so any replica can verify it without trusting whoever sent it.
• The manifest is signed by the producing machine and never changes. It records provenance plus the pruning statistics a query planner needs — row counts, time ranges, min/max values — so a query can skip an object without opening it.
• The access record carries the object’s DEK wrapped under the current domain key generation. It is replaceable.

That last property is what makes key rotation affordable. Rotating a domain key replaces access records — a few dozen bytes each — and never rewrites a data file. The object bytes, its hash, and its provenance signature all stay identical. See rewrapping.

Once published, the object may be copied to S and A under ordinary placement policy, neither of which needs to be able to decrypt it to store it.

Reading

A query resolves in catalog order, not storage order:

1. The local planner reads decrypted manifests from its catalog and prunes objects that cannot match — wrong time range, wrong partition, wrong value range.
2. For survivors it resolves storage_id to the access record, unwraps the DEK with the domain key it holds, and hands the key to the query engine.
3. The engine scans the encrypted Parquet in place.

Because the footer is encrypted, an unauthorized holder cannot read even the schema or the column statistics — which is precisely why the pruning metadata lives in the encrypted catalog rather than in the file.

Finding each other

Everything above assumes C can reach S. Nothing in the mesh assumes a flat network to make that true.

Machines are addressed by public-key identity rather than by location, so a node’s name does not change when its address does — C moving from office wifi to a mobile link is the same peer throughout. SOP/1’s preferred live substrate, iroh, runs over QUIC: it attempts the fastest direct path first, hole-punches through NATs where it can, and falls back to an encrypted relay when no direct path exists. SSP/1’s default binding is mutually authenticated QUIC with raw public keys, deliberately the same family, so a mesh running both layers carries one transport stack.

Two consequences worth stating. A relay in the path carries ciphertext it cannot read, so falling back to one is a routing event and not a trust decision. And the transport only finds the path — delivery, retry and convergence remain SSP/1’s, which is why C can be unreachable for a week without anything breaking.

Syncing mutable state

Objects carry the measurements. But a mesh also holds state that genuinely changes – the collector’s configuration, a device profile, catalog metadata — and two machines can edit it while partitioned. That is the other half of the system, and it belongs to SSP/1 rather than to the object plane.

Every change is a signed event: which machine, which table, which row, which columns — stamped with a hybrid logical clock that gives all events a total order without any coordinator.

sequenceDiagram
  accTitle: Concurrent edits converging without coordination
  accDescr: While partitioned, the workstation sets a device profile's location and the collector sets its schedule. On reconnect they exchange signed events, and both converge to a row containing both edits, because merge is per column.
  participant W as W (workstation)
  participant C as C (collector)
  Note over W,C: partitioned
  W->>W: set profile.location = "north shed"
  C->>C: set profile.schedule = "hourly"
  Note over W,C: reconnected
  W->>C: signed event: location
  C->>W: signed event: schedule
  Note over W,C: both hold location = north shed, schedule = hourly

Convergence is deterministic because merge is per column, last writer wins, with the order coming from the clock rather than from arrival: any set of events produces the same state on every machine, however delivery interleaves. Deletes tombstone a row rather than erasing it; a later, higher-clock edit can resurrect it — surfaced to the application, never silent — and a resurrected row recovers edits that arrived while it was tombstoned. The merge rules carry the details.

Delivery is deliberately dumb. Each machine re-offers from a durable cursor until the peer’s acknowledged watermark moves past it: at-least-once delivery plus idempotent apply, so a duplicate costs nothing and a loss heals on the next cycle.

When two machines are online together and the connection supports it, delivery does not have to wait for the cycle: in realtime mode a machine pushes events as they are produced — same message, same rules — and the batch cycle keeps running behind it as the correctness backstop. Live state moves in round-trip time, and nothing new has to hold for that to be safe, because a lost push is simply re-offered by the next cycle.

This same plane carries the mesh’s own control records — vouches, role assignments, transport bindings — so trust changes travel with exactly the guarantees of the data they govern.

Going offline, and coming back

The collector loses connectivity. It keeps recording and keeps forming objects; the DEKs sit under machine-local protection, unpublished.

When it reconnects, one of two things is true.

If no membership-security transition happened while it was away, nothing interesting occurs: it catches up on ordinary MLS traffic, confirms the domain key generation it holds is still current, seals access records for its unpublished objects, and publishes. Routine MLS epoch churn does not invalidate it.

If a rotation did happen, it must obtain the current key generation before it can publish. Note that it does not re-encrypt anything — the objects were formed with their own DEKs, so only the small access records are created at publication time. This is publication-time wrapping, and it is why an offline producer never ends up stranded holding data encrypted to a stale key.

Revoking a machine

The collector is decommissioned, or stolen. The authority removes it.

flowchart TD
  accTitle: Revoking a machine
  accDescr: The node is removed from the trusted keyring, an MLS Remove and Commit advances the epoch with the node excluded, a new domain key generation is minted, and it is distributed only to the remaining members.
  keyring[remove from the trusted keyring] --> commit["MLS Remove + Commit<br/><small>new MLS epoch, C excluded</small>"]
  commit --> mint[mint a new domain KEK generation]
  mint --> dist[distribute it to the remaining members only]

The first step is SSP/1’s, and it does most of the work: every durable record resolves its signer through the keyring, so the removed machine’s events, vouches, transport bindings and live connections all fail closed through that one deletion. From that point C also cannot decrypt new objects and cannot publish new ones that peers will accept.

What it can still do is read whatever it already had. No protocol can reach into a machine and unlearn a key it legitimately received. The specification states this outright rather than implying otherwise, and the honest framing is that revocation is future-facing.

There is a related fail-closed tradeoff worth knowing: data created before removal but never published is rejected afterwards. The specification accepts that deliberately — the dominant removal cause is a lost or compromised machine whose unpublished data is untrusted anyway — and mitigates it by prompting for a sync before a planned decommission completes.

Adding a machine

A replacement collector joins. Trust comes first, and it is SSP/1’s ceremony: the new machine pairs directly with an existing one — a QR or short-authentication-string comparison confirmed by a human — or is vouched in by enough already-trusted machines, again with a human confirming and the default set to deny.

Cryptographic membership and data access then follow as separate steps: it joins the MLS group, and a policy decision determines what history it gets.

• Forward only — it receives the current key generation and nothing older, so objects predating it stay unreadable.
• Full history — it is provisioned with older key generations too, in a signed package delivered over the authenticated group channel.

The subtle rule is that these interact. While any member holds forward-only entitlement, an old object must not be re-wrapped under a key that member holds – otherwise the restriction silently evaporates. SOP/1 calls this entitlement-aware rewrapping, and it is why old key generations are retained rather than eagerly retired.

Deleting rows, and compaction

For tables that allow deletion, a delete appends an immutable delete object naming the row identities to hide. Because objects have no global order, a delete is a permanent tombstone for its identity: any observation carrying that identity is invisible regardless of arrival order. That avoids the trap where a late-arriving data object silently resurrects deleted rows.

Compaction merges many small objects into fewer large ones, dropping tombstoned rows along the way. It runs on a machine that can decrypt — the workstation here – produces new objects with fresh DEKs under the current key generation, and marks the old objects superseded. It is the expensive background operation in the system, and also the mechanism by which old key generations naturally age out, since every compacted object is re-keyed forward.

Coordination

Ordinary data writes require no coordination. Objects are added to a set; addition is commutative and idempotent; there is no sequence number to agree on. The mutable side is the same: events merge deterministically in any order, so no lock or leader exists there either.

Membership changes do require coordination, because two machines must not concurrently commit conflicting changes to the same MLS group. SOP/1 serializes those through a blind sequencer: a relay enforcing first-commit-wins per group using only the MLS framing fields that are already unencrypted, without being able to read or validate the commit itself.

Since a relay that can order messages can also lie about the order, every accepted commit is acknowledged with a signed, hash-chained receipt. Clients retain receipts and gossip their heads, so two conflicting receipts for the same epoch are detectable proof of misbehaviour. See sequencing and coordination.

Recovery

Recovering a dataset needs three things, and losing any one is fatal:

flowchart LR
  accTitle: The three components recovery requires
  accDescr: Recovery needs the ciphertext, the manifest and access-record catalog, and the applicable key history. Losing any one is fatal.
  ct[ciphertext] --- cat[manifest / access catalog]
  cat --- keys[applicable key history]

Ciphertext alone is unrecoverable noise. So the catalog is periodically snapshotted into an encrypted checkpoint object and replicated alongside the data, including onto blind replicas like A. A small signed recovery root names the current checkpoint and carries the access record needed to open it, and each machine’s protected key store retains a pointer to it.

Disaster recovery is then: read the recovery root pointer from the key store, fetch and decrypt the root, fetch the checkpoint it names, open it with retained key history, restore the catalog, resolve objects normally. If any of the three components is missing, recovery fails closed with a clear state rather than partially succeeding.