Object model and table classes

12. SOP Object Model

12.1. External object record

Untrusted object infrastructure SHOULD see only:

storage_id
coarse byte-size information
opaque bytes

and, where unavoidable, opaque transport routing information.

12.2. Immutable ObjectManifest

The authoritative semantic description of an object is the immutable producer-signed ObjectManifest.

It contains claims about what the object is, not who may currently decrypt it:

manifest_version
mesh_id
object_kind
storage_id
table_schema
table_name
schema_fingerprint
producer_site_id
producer_mls_epoch
created_hlc
row_count
exact_byte_size
partition_values
pruning_statistics
key_ref
supersedes[]
row_identity_filter_ref?

partition_values MUST be encoded using Delta Lake partition-value string conventions, so that the Phase 5 checkpoint projection (Section 21) copies rather than translates them.

Fields whose meaning is expected to change through KEK rotation, historical entitlement, or table-to-domain migration MUST NOT appear in the ObjectManifest.

12.3. Producer manifest and mutable access records

The durable ObjectManifest MUST be deterministic CBOR protected with COSE_Sign1 using a signing key bound to the producer’s SSP identity:

ObjectManifest =
    COSE_Sign1_producer(deterministic_cbor_manifest)

object_manifest_hash =
    BLAKE3(exact ObjectManifest COSE_Sign1 bytes)

object_manifest_hash is an audit/binding hash over the exact signed manifest encoding. It is distinct from storage_id, which remains exclusively the iroh-blobs hash of the encrypted Parquet object.

The signature is immutable producer provenance. MLS sender authentication is transport/session provenance and does not replace it.

Current decryption authority is represented separately by one or more AccessRecord objects defined in Section 7.5. An access-authorized current member MAY issue a replacement or additional AccessRecord without changing or re-signing the producer manifest.

A portable catalog entry therefore consists conceptually of:

ObjectCatalogEntry {
    manifest = ObjectManifest
    access_records = [ AccessRecord, ... ]
}

The catalog MUST persist the manifest only inside encrypted local storage, encrypted catalog checkpoints, or an authenticated confidential transport. The producer signature does not make semantic metadata safe to expose in plaintext.

Acceptance requires both:

1. a valid producer ObjectManifest whose storage_id matches the object; and
2. a valid applicable AccessRecord whose object_manifest_hash matches the exact signed manifest bytes and whose issuer was authorized to grant that access.

The SOVM COSE profile MUST bind at least the following producer-manifest fields against substitution or cross-context replay:

manifest_version
mesh_id
producer_site_id
storage_id
object_kind

Access-specific binding requirements are defined in Section 7.5.

12.4. Mandatory pruning metadata

For data objects, the encrypted catalog’s ObjectManifest MUST contain planner-level pruning metadata sufficient to reject non-matching objects without opening their PME footers.

At minimum:

• row count;
• object logical time range where the table has a time dimension;
• min/max for partition columns;
• min/max for table-configured primary pruning columns;
• schema fingerprint.

These manifest statistics remain encrypted at rest and are decrypted into the local authorized catalog.

The local query planner SHOULD use manifest/catalog pruning as the primary file- selection path. PME footers SHOULD normally be opened only for objects that survive catalog pruning.

12.5. Object immutability

A published object’s exact PME bytes MUST NOT change under the same storage_id.

A correction creates a different object.

12.6. Producer and holder

The producer is the node that authored the logical object.

A holder is any authorized node or blind storage service retaining the ciphertext.

Possession does not imply authorship or authorization.

12.7. Catalog checkpoints and RecoveryRoot

Data recovery requires three components, not one:

flowchart LR
  accTitle: The three components recovery requires
  accDescr: Recovering a dataset requires the encrypted bytes, the manifest and access-record catalog, and the applicable key history. Any one missing makes recovery fail closed.
  ct["PME ciphertext"] --- cat["ObjectManifest / AccessRecord catalog"]
  cat --- keys["applicable KEK history"]

Ciphertext without an applicable AccessRecord is unrecoverable regardless of retained KEKs; the PME key_ref can help correlate forensic material but cannot substitute for missing access state. Manifest and access metadata are therefore first-class durability concerns, not incidental catalog state.

An implementation MUST periodically snapshot the encrypted ObjectManifest/AccessRecord catalog for each domain as a catalog checkpoint object: a content-addressed encrypted object placed on durable and blind replicas alongside data objects under ordinary placement policy.

The checkpoint has a bootstrap dependency — it is itself encrypted and needs an access record that cannot exist only inside the checkpoint it unlocks. SOP therefore defines a small encrypted and signed SOVM recovery-root object:

RecoveryRoot = COSE_Encrypt0(
    key = K_domain_generation,
    protected = {
        alg = A256GCM,
        content_type = sovm/sop-recovery-root-v1
    },
    plaintext = COSE_Sign1_authorized_issuer(
        {
            recovery_root_version,
            mesh_id,
            opaque_domain_id,
            kek_generation,
            recovery_generation,
            checkpoint_storage_id,
            checkpoint_manifest_hash,
            checkpoint_access_record,
            previous_recovery_root_id?,
            issuer_site_id,
            issued_hlc
        }
    )
)

recovery_root_id =
    iroh_blobs_hash(exact RecoveryRoot COSE_Encrypt0 bytes)

checkpoint_access_record is a standalone Section 7.5 AccessRecord for the catalog-checkpoint object’s DEK. checkpoint_manifest_hash supplies the binding needed to validate and open that access record without making the checkpoint’s sensitive manifest public.

previous_recovery_root_id forms an append-only recovery-root history over the exact encrypted root bytes. As with SOP data objects, the identifier is the exact iroh-blobs BLAKE3 root for those bytes; no parallel recovery-root hash namespace is defined. The RecoveryRoot is encrypted because checkpoint identifiers, generations, issuer metadata, and domain linkage are themselves privacy-sensitive.

The node’s protected recovery/key store MUST retain at least:

opaque_domain_id
recovery_root_id
kek_generation

for the latest known root. The encrypted RecoveryRoot bytes SHOULD be replicated through the SSP control plane and configured durable archive adapters.

Storage implementations MAY pin or index recovery_root_id using local iroh-blobs tags, filesystem names, object-store keys, or other adapter-specific locators; none of those mechanisms is protocol state. Where the storage substrate is content-addressed, recovery_root_id SHOULD be the retrieval key.

Disaster recovery reduces to: obtain the latest known recovery_root_id and KEK generation from the protected key store; fetch and decrypt the RecoveryRoot; fetch the named checkpoint by checkpoint_storage_id; use the embedded access record plus retained KEK history to recover the checkpoint DEK; decrypt the catalog; resolve all remaining objects normally.

A stale but valid RecoveryRoot recovers a consistent older catalog. If an archive adapter can enumerate later candidate roots, their previous_recovery_root_id chain MUST validate back to the trusted stale root before a later root is accepted. Gate G10 exercises this path.

13. Append-Only Tables

13.1. Ingestion

Rows are accumulated into an in-memory or encrypted staging batch.

Plaintext persistent staging files are prohibited.

13.2. Publication pipeline and atomicity

flowchart TD
  accTitle: The append-only publication pipeline
  accDescr: Logical rows are encoded to Parquet, encrypted under a random per-object key with a random AAD prefix, size-padded, then imported into iroh-blobs to obtain the content hash. From there five things happen before the catalog entry is published.
  rows[logical rows] --> encode[Parquet encode]
  encode --> encrypt["PME encrypt using random DEK + random AAD prefix"]
  encrypt --> pad["Padmé-size valid PME object"]
  pad --> hash[iroh-blobs import / hash]
  hash --> persist[durably persist ciphertext locally]
  hash --> protect[durably protect unpublished DEK locally]
  hash --> verify[verify current publication authorization]
  hash --> sign[COSE-sign immutable ObjectManifest]
  hash --> access[create COSE AccessRecord under current required domain KEK]
  persist --> entry[publish ObjectCatalogEntry]
  protect --> entry
  verify --> entry
  sign --> entry
  access --> entry

The ciphertext bytes MUST be durably persisted locally before its catalog entry is published.

Catalog-entry publication MUST be idempotently retryable.

A peer that receives a valid manifest/access catalog entry but cannot currently fetch its object bytes from any advertised or discovered holder MUST retain the catalog entry in a pending_bytes state. The entry is not poisonous and MUST NOT be treated as proof of permanent data loss.

Locally durable ciphertext that never reaches catalog-entry publication is an orphan object and MAY be garbage-collected after a safe unpublished-object retention interval.

13.3. Initial object sizing

The initial edge-oriented target is 16-64 MiB of final encrypted object bytes, before Padmé rounding.

This is a starting policy rather than a protocol invariant. It deliberately leans smaller than generic warehouse guidance because SOP must balance:

• mobile refetch cost;
• purge rewrite cost;
• compaction cost;
• memory pressure;
• query scan efficiency;
• iroh range/resume behavior.

Writers SHOULD also choose row-group sizes based on expected selective-scan columns and available memory. G6 may revise the default object range.

13.4. Convergence

APPEND table state is the union of all valid authorized non-superseded data objects known to the node.

Object addition is commutative and idempotent.

No global transaction number is required.

14. Append-and-Delete Tables

14.1. Delete objects

Ordinary deletion appends an immutable delete object.

The delete object MUST itself be encrypted at rest and authenticated.

14.2. Semantics: permanent identity tombstones

A delete object identifies stable logical row identities.

Delete semantics are order-free. SOP deliberately has no data sequence numbers, so Iceberg-style “applies only to strictly earlier data” semantics cannot be provided and are not imitated. Instead an equality delete is a permanent tombstone for its logical identity:

delete(id)
    => any observation carrying id is invisible,
       regardless of object arrival order

A data object published or discovered after the delete — including an offline-created object surfacing late — does not resurrect the identity. Tombstone application is commutative and idempotent, so APPEND_DELETE state remains a deterministic function of the replicated object set with no ordering protocol.

This composes with the existing rules: deleted observation identities are never reused (Section 14.7), and any table requiring delete-then-reinsert belongs in MUTABLE/SSP semantics. Delta materialization (Section 21.5) is unaffected, since deletes are materialized at checkpoint time regardless.

14.3. Representation

The representation is decided: SOP/1 supports exactly one delete encoding — encrypted Parquet objects containing equality-delete keys. Multiple encodings would multiply reader complexity for no v1 benefit.

Position deletes and deletion-vector encodings are viable in SOP — object immutability makes row positions stable — and are held as a Phase 4+ optimization, adopted only if G6 shows the equality anti-join is a real bottleneck after filter pruning (Section 14.5).

14.4. Permanent tombstone knowledge

Permanent identity tombstones are permanent logical knowledge. Expected node-offline duration is an operational parameter and MUST NOT be used as a correctness bound for tombstone expiry.

Delete objects need not be retained forever. Compaction MAY replace many delete objects with a more compact encrypted tombstone-set object or another normatively equivalent representation, but the logical set of deleted identities MUST remain represented for as long as the table’s identity namespace remains capable of accepting late data.

Therefore, when a compaction at horizon H physically removes tombstoned rows and absorbs corresponding delete objects:

1. the absorbed tombstone identities MUST remain represented in a durable encrypted tombstone set;
2. every newly published or newly discovered data object MUST be checked against applicable permanent tombstone knowledge before its rows enter queryable state, regardless of created_hlc or elapsed wall time; and
3. garbage collection MAY remove superseded delete-object bytes only after the replacement tombstone representation satisfies the same durability and GC-verification rules as other SOP state.

A future protocol MAY define an explicit table/identity-namespace retirement operation after which older data objects are categorically inadmissible. Only such a protocol-level closure may permit final deletion of the corresponding tombstone knowledge.

This makes APPEND_DELETE strictly order-free: an arbitrarily late object cannot resurrect a deleted identity.

14.5. Delete pruning and row-identity filters

Delete-object manifests MUST include an encrypted membership filter over delete keys and SHOULD include min/max delete-key bounds where meaningful.

Delete keys inside the delete object SHOULD be sorted in canonical key order.

A delete-side filter alone cannot prove that a data object is disjoint from the delete set; it answers “might this key be deleted,” not “does this object contain any deleted key.” Object-level skipping therefore uses, in order:

1. partition and min/max disjointness from mandatory manifest pruning metadata (Section 12.4); and
2. where enabled per table, an optional keyed static membership filter over the data object’s stable row identities, tested against the delete keys.

data object manifest:    row_id_filter     (optional, per-table policy)
delete object manifest:  delete_id_filter   (mandatory)

definitely disjoint  =>  skip the delete anti-join for this object

Row-identity filters are static-set filters over immutable objects, so binary-fuse or xor filters SHOULD be preferred over Bloom filters for size. Because a filter over a large object can reach megabytes at tight false-positive targets while manifests are designed to be small catalog records, implementations MUST apply at least one of:

• a coarse false-positive target (a few percent suffices to skip most anti-joins); or
• storing the filter as a small companion object referenced from the manifest, fetched only when min/max pruning is inconclusive.

Filters are keyed or stored only within encrypted metadata, consistent with Section 23.6.

When deletes can apply, sorted delete keys SHOULD permit an efficient merge anti-join where the local engine supports it.

14.6. Query state

current_state =
    union(live data objects)
    minus
    effective delete set

14.7. Re-insertion

For naturally immutable observations, a deleted row identity MUST NOT be reused; permanent tombstone semantics (Section 14.2) depend on it.

If semantic resurrection is required, the table SHOULD use MUTABLE/SSP semantics.

14.8. Delete acknowledgement

A user-visible promise that data is “deleted everywhere” MUST distinguish:

• logical deletion applied;
• managed ciphertext physically removed;
• relevant managed key material destroyed;
• acknowledgement received from required peers.

15. Mutable Tables and SSP/1

SOP does not generalize SSP’s conflict algebra.

MUTABLE tables continue to use:

flowchart TD
  accTitle: The mutable-table path, unchanged from SSP/1
  accDescr: A logical mutation becomes an SSP change event, ordered by hybrid logical clock, merged by per-column last-writer-wins, with delete and resurrection semantics on top.
  mut[logical mutation] --> ev[SSP event]
  ev --> hlc[HLC ordering]
  hlc --> lww[per-column LWW]
  lww --> del[delete / resurrection semantics]

Revision 8 recommends migrating future encrypted SSP control/event messaging to MLS where appropriate, but that transport encryption change MUST NOT alter SSP’s semantic event model.

Persistent SSP state containing user data MUST be encrypted at rest.