Placement, storage, compaction and purge

16. Placement and Replication

16.1. Scope is not placement

Scope defines which node may decrypt data.

Placement defines where ciphertext should physically exist.

Placement membership and decryption membership are formally distinct sets.

A node MAY be a SOP storage replica without being an MLS decryption member of the corresponding data domain.

Possession of ciphertext MUST NOT grant access to the domain KEK or object DEK. Blind in-fleet replication is an expected configuration, not an exception.

16.2. Example

flowchart TD
  accTitle: Decryption scope versus physical placement
  accDescr: For one table, two nodes may decrypt it while four hold copies. The home node and archive store blind encrypted replicas without being able to read them.
  table["one table"]
  table --> scope["scope<br/>(may decrypt)"]
  table --> place["placement<br/>(holds bytes)"]
  scope --> s1[laptop]
  scope --> s2[phone]
  place --> p1["laptop &mdash; full"]
  place --> p2["phone &mdash; 30-day hot window"]
  place --> p3["home-node &mdash; blind encrypted replica"]
  place --> p4["archive &mdash; blind encrypted replica"]

16.3. Blind replicas

Blind in-fleet replicas are normative SOP participants.

A blind replica MUST NOT receive the relevant domain KEK merely because it stores ciphertext.

Placement policy may therefore include nodes that are not members of the corresponding MLS data-domain group.

The authorization sets are:

storage_members(domain)
decryption_members(domain)

and neither set is inferred from the other.

16.4. Metadata-only knowledge

A node MAY know an opaque object exists without possessing its bytes or decryption keys.

17. Local Storage and Query Execution

17.1. At-rest invariant

Persistent user data MUST be encrypted.

This includes:

• PME objects;
• delete objects;
• SSP logs containing user data;
• OpenMLS sensitive state;
• retained application domain KEK generations;
• SOP manifests, access records, and catalogs;
• DuckDB persistent databases;
• WALs;
• query spill;
• compaction scratch;
• caches;
• snapshots.

17.2. OpenMLS storage

OpenMLS persists sensitive group state through a storage provider and relies on deletion of obsolete secret material for forward-secrecy properties.

The selected production storage provider MUST therefore:

• encrypt sensitive state at rest;
• honor deletion semantics as strongly as the platform allows;
• avoid unencrypted backups or journal copies;
• avoid debug logging of cryptographic material.

17.3. DuckDB

DuckDB is a local query engine, not a replicated database file.

For bulk SOP datasets, DuckDB SHOULD query encrypted Parquet directly where its PME support is compatible with SOP-PME-1.

17.4. Key injection

Object DEKs SHOULD be injected into the query engine only for the lifetime needed to read the object.

Long-lived global registration of large numbers of object keys SHOULD be avoided where a narrower reader integration is practical.

17.5. Temporary plaintext

A query path MUST NOT intentionally write decrypted SOP rows into an unencrypted temporary file.

If the chosen execution engine cannot satisfy this property for a workload, that execution path MUST NOT be used for fleet-private data.

17.6. Crash and debug artifacts

Production builds MUST NOT emit:

• decrypted rows;
• object DEKs;
• domain KEKs;
• MLS epoch secrets;
• decrypted manifests or access records;

into ordinary logs, crash bundles, or diagnostics.

OpenMLS debug features capable of exposing sensitive cryptographic data MUST be disabled in production.

18. Compaction

18.1. Purpose

Compaction converts many small data/delete objects into fewer query-efficient objects.

18.2. Confidential compaction

A compactor:

1. obtains authorized source DEKs;
2. decrypts source modules in memory or encrypted scratch;
3. applies logical deletes;
4. writes new Parquet;
5. encrypts with a new random DEK under SOP-PME-1;
6. computes a new ciphertext storage_id;
7. wraps the DEK under the current domain KEK;
8. publishes a supersession manifest and applicable access record.

Persistent plaintext intermediate files are prohibited.

18.3. Concurrent compaction

Multiple authorized nodes MAY compact the same logical source set and produce different encrypted output.

This is acceptable.

Correctness MUST NOT rely on deterministic ciphertext or identical Parquet encoding.

18.4. Compaction cost

Compaction is expected to remain one of the principal I/O amplification costs of SOP.

It SHOULD be preferentially scheduled on powered, storage-rich nodes.

18.5. Supersession verification and GC preconditions

A supersession claim is not self-certifying. An authorized but malicious or buggy compactor can publish an object E with supersedes[A, B, C] whose logical content silently omits live rows. The threat model (Section 23.1) explicitly includes malicious fleet peers.

Superseded source objects therefore MUST NOT become garbage-collection eligible except under one of two explicitly distinct modes:

verified GC
    a second authorized decrypting replica recomputes the logical state of
    E from the source set and publishes a signed verification attestation
    => adversarial integrity guarantee against a malicious or buggy
       compactor

single-verifier GC
    a configured retention timeout elapses under user-accepted policy
    => operational guarantee only; explicitly trusts the sole full replica

Time does not verify anything: an elapsed window MUST NOT be presented, in protocol or product language, as protection against a malicious compactor. If no second party can decrypt and recompute the source set, no party may be capable of raising a dispute.

The single-verifier mode is nevertheless coherent because of a dominance argument: when exactly one node can decrypt a domain, that node is the compactor, and if it is malicious it already holds plaintext access to everything in the domain — supersession-hiding is strictly dominated by the compromise already suffered. Single-verifier GC therefore protects against loss and bugs, not against the sole replica itself, and the product MUST label it accordingly.

Blind replicas SHOULD, as default placement policy, retain superseded ciphertext until a verified attestation exists; blind storage is cheap and this preserves the verified path whenever a second decrypting replica later appears.

Where a second decrypting replica exists but is slow or intermittently online, the dispute window remains useful as a bound on how long GC waits for it; a dispute suspends GC eligibility for the affected source set until the disagreement is resolved by recomputation or explicit user decision.

The record formats are decided in shape. A verification attestation is:

COSE_Sign1(
    {
        attestation_version,
        mesh_id,
        verifier_site_id,
        superseding_storage_id,
        superseded_set_hash,      # hash of the sorted superseded storage_ids
        compaction_horizon,
        logical_digest_method,
        logical_digest,
        issued_hlc
    }
)

Because Section 18.3 permits non-deterministic ciphertext and Parquet encoding, verification compares logical content, not bytes. logical_digest is therefore an order-independent keyed multiset digest — conceptually a combination (for example, sum in a suitable group) of keyed per-row hashes — so that compactor and verifier reach the same digest regardless of row order, row-group layout, compression, or encryption. The exact digest construction is the principal remaining implementation-open item and MUST be prototyped and fixed in Phase 4 before APPEND_DELETE compaction ships.

A dispute is a COSE_Sign1 record referencing the disputed supersession supersession manifest and carrying the disputer’s counter-digest. Any dispute is a user-surfaced security event.

The default dispute window is:

max(30 days, 2 x longest expected node-offline interval)

configured per fleet, so that a node offline for an extended period (for example, a long passage) can still observe and dispute a supersession before its sources become GC-eligible.

19. Garbage Collection, Revocation, and Purge

19.1. Revocation is future-facing

MLS removal prevents a removed node from learning new group epoch secrets.

It cannot revoke keys or plaintext already learned.

19.2. Domain key rotation

After a membership-security transition, newly published objects MUST be wrapped under the new KEK generation.

Objects that remain readable to the current membership MAY have their DEKs rewrapped under the new generation according to retention and risk policy, subject to the entitlement constraint of Section 7.6: rewrapping MUST NOT grant a FORWARD_ONLY member access to objects predating its admission boundary.

19.3. Rewrapping does not erase old knowledge

If a removed node already had an older domain key and old wrapped DEK, merely adding a new wrapper does not make the old ciphertext inaccessible to that node.

Cryptographic revocation of historical objects therefore requires eliminating all still-valid old wrappers from managed storage and relies on the removed node not having copied the DEK or plaintext.

19.4. Logical deletion

Logical deletion removes data from current query state but does not imply physical or cryptographic erasure.

19.5. Physical purge

Purging rows embedded inside a PME object requires creating a replacement PME object without those rows and removing the old ciphertext from managed replicas.

19.6. Cryptographic erasure

Where a DEK is unique to data no longer retained, destruction of every managed wrapped DEK can make retained ciphertext computationally inaccessible to managed systems.

This guarantee is invalid if an authorized or compromised node previously copied the unwrapped DEK or plaintext.

19.7. Honest product language

The product MUST NOT claim that revocation or purge can make an already compromised authorized node forget information it previously obtained.