Parquet modular encryption profile

11. Parquet modular encryption profile

11.1. PME is the canonical SOP data encryption format

APPEND data objects MUST use a defined Parquet Modular Encryption profile unless a future object type explicitly specifies another format.

11.2. Initial profile: SOP-PME-1

The proposed initial profile is:

Profile: SOP-PME-1

footer: encrypted
all data columns: encrypted
footer key: per-object random DEK
column key: same per-object random DEK
plaintext footer mode: forbidden
semantic key metadata: forbidden
module AAD: empty
positional binding: storage_id content addressing, mandatory
size padding target: Padmé
column field IDs: stable Parquet field IDs, mandatory
key_metadata: random opaque 16-byte key_ref, always written

Every column in a SOP-PME-1 object MUST carry a stable Parquet field ID, assigned at table creation and never reused after a column is dropped. Field IDs cost nothing to write, cannot be retrofitted into immutable published objects, and are required for the Delta Lake column-mapping projection (Section 21) as well as any future Iceberg compatibility.

Using one DEK for all columns matches the current DuckDB encrypted-Parquet capability and avoids requiring column-specific key management in SOP/1.

11.3. AAD prefix

SOP-PME-1 objects carry no AAD prefix, and no module AAD at all. A writer MUST NOT emit one: a SOP-PME-1 object is defined by the empty-AAD construction and MUST be readable by any conforming reader holding only the object’s DEK.

This is a measured constraint of the selected writer rather than a preference. DuckDB rejects an aad_prefix argument at the binder and emits an empty AesGcmV1 structure — neither aad_prefix nor aad_file_unique is written, and every module authenticates against the empty string. No configuration of the writer produces a different result, and an AAD prefix cannot be added after the fact because it is an input to every module’s authentication tag.

The blob plane reaches the same construction by choice rather than by constraint: SOP-BLOB-1 also uses no AAD (construction), resting context binding on the fresh per-object DEK, content addressing, and the signed manifest/access-record chain.

The two formats are not equivalent in what they retain, and an implementer MUST NOT reason from one to the other:

• SOP-BLOB-1 keeps positional binding inside the AEAD. Its nonce is a pure function of the chunk index and carries a final-chunk flag, so transplant, reordering and truncation are authentication failures even for a decryptor holding nothing but a DEK and a byte stream.
• SOP-PME-1 keeps none. PME places positional binding entirely in the AAD, so with an empty AAD a module is bound to neither its module type nor its row group, column or page ordinal. An adversary with write access to untrusted storage may move a module to another position in the same object, or into a different object encrypted under the same DEK, and every tag still verifies.

Content addressing is therefore SOP-BLOB-1’s second defense against ciphertext substitution and SOP-PME-1’s only one. Section 11.10 states the obligation that follows.

11.4. Full metadata protection

The encrypted-footer mode MUST be used.

Plaintext Parquet schema, statistics, row counts, and key-value metadata MUST NOT be intentionally exposed to untrusted storage.

11.5. Key metadata

PME key_metadata MUST NOT contain semantic names such as:

health-key
finance-2026
alice-location

The decision is mode 2, strengthened: every SOP-PME-1 object MUST carry a random 16-byte opaque key_ref in PME key_metadata, and no reader may depend on it. Key resolution in the query path is always storage_id -> ObjectManifest + current AccessRecord -> DEK, with storage_id recomputable from the bytes themselves. The always-written key_ref makes stray ciphertext objects self-indexing for forensics and for the group-recovery path (Section 22.10) at zero leakage cost, while “never load-bearing” removes the option branch from reader implementations.

key_metadata is plaintext in FileCryptoMetaData and lies outside every module’s authentication tag. A writer whose Parquet library exposes no option for the field MUST therefore write it into the encoded object itself, as a step of the writer rather than as a repair applied afterwards. SOP-BLOB-1 satisfies the same requirement the same way, writing its key_ref as a cleartext trailer after encryption (construction); the container differs, the property does not. Section 11.12 fixes when that step runs relative to padding and content addressing.

11.6. Key wrapping outside the Parquet object

SOP SHOULD keep the wrapped DEK outside the encrypted Parquet file in the encrypted SOP access record.

This permits domain KEK rotation and DEK rewrapping without modifying the Parquet object’s bytes.

11.7. Canonical storage identity

For a PME object, storage_id is normatively the exact iroh-blobs content hash:

storage_id = iroh_blobs_hash(exact_encrypted_parquet_bytes)

iroh_blobs_hash is the 32-byte BLAKE3 root hash used by iroh-blobs for that exact blob.

SOP MUST NOT maintain a second BLAKE3 namespace or independently defined variant. The byte string accepted by iroh-blobs is the canonical object byte string and its iroh-blobs hash is the SOP storage_id.

This alignment gives SOP iroh-blobs’ incremental verification, resumable downloads, and verified range transfer without a second integrity layer or hash-conversion step.

The unkeyed hash of plaintext Parquet MUST NOT be exposed as a storage identifier.

11.8. Padmé object-size padding

SOP-PME-1 adopts the Padmé padding function (section 4.4 of [PADME]; the specification’s one padding function) as the target-length function for privacy-sensitive object-size padding.

Every SOP-PME-1 object MUST reach its Padmé target exactly. There is no fail-open: a writer that cannot reach the target MUST NOT publish the object. Padding MUST preserve a standards-compliant PME file, and arbitrary bytes MUST NOT be appended after the Parquet footer.

That last requirement is a read-path prerequisite as well as a format rule. A verified read begins by establishing the object’s length, and it does so by verifying the object’s final bytes against storage_id and reading the Parquet footer’s length and magic out of them (Section 11.10). Padding appended after the footer would leave nothing at the end of the object for that step to key off, and a reader would be back to taking a claimed length on trust — a value every later step of the read derives from.

The encoding is a reserved key-value entry in the encrypted Parquet footer metadata whose value is a run of zero bytes. The entry lies inside the encrypted footer rather than after it, so untrusted storage cannot distinguish padding from content.

The requirement is unconditional because the target is always reachable, which is a measured property of the encoding rather than an assumption. Encoding the object once with an empty padding value gives a length L; adding n filler bytes to the value yields exactly L + n until the Thrift length prefix on the value grows, at 128 bytes and again at 16 384. The writer therefore solves for n in closed form against Padmé(L) and rewrites the footer once. It does not search, and it does not iterate.

Three properties keep that closed form true, and each is normative:

• The entry is present on every object, including an object needing no padding. Introducing the entry only when padding is wanted costs the width of a fresh key-value pair the moment it appears, which puts every target nearer to the object than that width out of reach. Carrying it unconditionally folds the cost into L and leaves every non-negative delta reachable. This is the same instinct as the always-written key_ref of Section 11.5: a field that is always there removes an option branch from both writer and reader.
• The key is sovm:sop-pad-v1, optionally followed by one or more - characters, and nothing else. A key name is itself footer metadata, so it MUST NOT be derived from the object’s content, table, domain, or size. The optional tail is the second length knob: the two sizes immediately above the varint steps are unreachable by value length alone, and one extra key character reaches them. A writer MUST be able to vary both lengths; one knob does not cover the space.
• Readers MUST ignore any footer-metadata key carrying that prefix. Nothing in Parquet enforces that a metadata entry is semantically ignored, so the obligation belongs on the reader and is stated here rather than assumed.

The unpadded exact byte count remains encrypted catalog metadata; untrusted storage observes only the padded ciphertext size.

11.9. Optional logical deduplication

If plaintext-level deduplication is later required, it MUST use a keyed fleet/domain-private logical fingerprint and MUST NOT be exposed to untrusted infrastructure.

SOP/1 does not require plaintext-level deduplication.

11.10. Integrity layers

PME provides authenticated encryption of Parquet modules. It does not provide object integrity, for two independent reasons:

• A module that is never read is never authenticated. Per-module AEAD checks only the modules a query actually decrypts, so a projection that skips a column chunk never evaluates that chunk’s tag and corruption there is invisible to that query.
• A module is not bound to its position. Under SOP-PME-1’s empty module AAD (Section 11.3) a valid module stays valid in the wrong slot, or inside a different object sharing the DEK.

SOP content addressing supplies both missing properties. storage_id is a BLAKE3 root hash over the object’s exact bytes (Section 11.7), and BLAKE3 is a tree hash: a byte range carries the interior hashes on its path to that root, so the range authenticates against storage_id on its own. That path includes a leaf’s position in the tree, which is what supplies the second property — a module cannot be moved to another slot, or into another object sharing the DEK, because its bytes would have to hash to the same root at the same offset.

Verification is therefore available incrementally, per byte range, and a reader MUST NOT be required to materialise the complete object to satisfy the requirement below. A projection over three column chunks of a large object verifies those three ranges and the footer, not the object.

A reader MUST verify storage_id over an object’s bytes before releasing any plaintext derived from them. This is a correctness requirement, not an optimization: no cache-hit path, local-holder path, already-fetched path or other fast path may skip it. Where bytes arrive through the transfer layer’s incremental verification (Section 11.7), verifying each range before use satisfies the requirement; an implementation that obtains bytes by any other route MUST verify them itself.

The requirement binds the interface the bytes are read through, not the object. A content-addressed store commonly offers both a verifying and a non-verifying range read over the same object, and the non-verifying one is typically the ergonomic seekable reader a Parquet reader is most naturally handed. Bytes obtained that way carry no binding to storage_id: corruption of the local store after a verified download is invisible to them, and the object being content-addressed does not make the read verified. A conforming reader MUST NOT obtain object bytes through an interface that does not authenticate them against storage_id.

An implementation SHOULD enforce that by construction — not exposing a non-verifying read across the SOP reader boundary at all — rather than by convention, because a read through the wrong interface fails silently at every layer above it. A transport binding names which of its own read interfaces satisfy this clause; the iroh binding does so for SOVM-IROH.

Forensic and recovery paths inherit the same rule. A decryptor holding only a DEK and a byte stream cannot establish that a SOP-PME-1 object is intact — unlike the blob plane, where the AEAD alone fails closed (cryptographic notes). Recovery of PME objects MUST therefore resolve bytes through storage_id, and the opaque key_ref of Section 11.5 correlates stray ciphertext without ever authenticating it.

Producer authenticity remains separate and MUST be bound to the encrypted SOP producer manifest or publication message.

A reader MUST NOT satisfy a query result, a predicate, or an access decision from Parquet footer metadata alone. Row counts and column statistics are metadata: they are not required to be exact, and an engine MAY answer from them without decrypting any data page. Any value released to a caller, and any predicate that gates release, MUST be evaluated over materialised column data. Statistics MAY be used to skip work — subject to Section 12.4 — never to produce an answer.

This bounds what a statistics-derived value may be used for; it does not bound what an authorized reader may learn. A reader entitled to the object may materialise every column and compute the same bound exactly. What is forbidden is releasing, or deciding against, a number the format never promised was a number in the data.

A range-verified read establishes the integrity of the ranges it verified, not of the object. A reader MUST NOT report an object as verified, or treat unread regions as authenticated, on the basis of a projection over it.

11.11. Performance

PME performance overhead MUST be benchmarked on target nodes.

The protocol MUST NOT silently fall back to unencrypted Parquet to meet a performance target.

11.12. Writer step order

Three requirements of this section interact through object length and object identity, so the order of the writer’s steps is normative:

1. Parquet encode with stable field IDs (Section 11.2).
2. PME encryption under the object’s DEK (Section 11.1).
3. Write the opaque key_ref into FileCryptoMetaData (Section 11.5).
4. Size-pad to the Padmé target (Section 11.8).
5. Import into iroh-blobs; the returned hash is storage_id (Section 11.7).

Steps 3 and 4 MUST NOT be exchanged. Writing key_ref changes the encoded length of FileCryptoMetaData, so padding sized before it overshoots the Padmé target by the width of the field.

Steps 3 and 4 MUST both precede step 5. storage_id is defined over the exact final object bytes, so any mutation after the import yields an object whose content address no longer matches — which Section 11.10 obliges every reader to reject, and which no later repair can undo for an already-published object.

Step 3 is safe at that position because FileCryptoMetaData is plaintext, is laid out after the row groups in encrypted-footer mode, and is covered by no module authentication tag. The column offsets recorded in the encrypted footer address bytes that precede it and are unaffected by its width.