Parquet modular encryption profile

11. Parquet Modular Encryption Profile

11.1. PME is the canonical SOP data encryption format

APPEND data objects MUST use a defined Parquet Modular Encryption profile unless a future object type explicitly specifies another format.

11.2. Initial profile: SOP-PME-1

The proposed initial profile is:

Profile: SOP-PME-1

footer: encrypted
all data columns: encrypted
footer key: per-object random DEK
column key: same per-object random DEK
plaintext footer mode: forbidden
semantic key metadata: forbidden
AAD prefix: random unique per object
AAD prefix storage: permitted only as opaque random bytes
size padding target: Padmé
column field IDs: stable Parquet field IDs, mandatory
key_metadata: random opaque 16-byte key_ref, always written

Every column in a SOP-PME-1 object MUST carry a stable Parquet field ID, assigned at table creation and never reused after a column is dropped. Field IDs cost nothing to write, cannot be retrofitted into immutable published objects, and are required for the Delta Lake column-mapping projection (Section 21) as well as any future Iceberg compatibility.

Using one DEK for all columns matches the current DuckDB encrypted-Parquet capability and avoids requiring column-specific key management in SOP/1.

11.3. AAD prefix

Each SOP-PME-1 file MUST use a fresh random AAD prefix.

The prefix MUST contain no semantic table, user, domain, timestamp, or partition information.

Because the value is random and non-semantic it MAY be stored in the Parquet aad_prefix field. An implementation MAY instead omit it from the file and supply it through the protected SOP catalog entry if reader integration makes that preferable.

This supplements PME’s internally generated module/file identifiers and provides defense in depth against future profile changes or accidental DEK reuse.

11.4. Full metadata protection

The encrypted-footer mode MUST be used.

Plaintext Parquet schema, statistics, row counts, and key-value metadata MUST NOT be intentionally exposed to untrusted storage.

11.5. Key metadata

PME key_metadata MUST NOT contain semantic names such as:

health-key
finance-2026
alice-location

The decision is mode 2, strengthened: every SOP-PME-1 object MUST carry a random 16-byte opaque key_ref in PME key_metadata, and no reader may depend on it. Key resolution in the query path is always storage_id -> ObjectManifest + current AccessRecord -> DEK, with storage_id recomputable from the bytes themselves. The always-written key_ref makes stray ciphertext objects self-indexing for forensics and for the group-recovery path (Section 22.10) at zero leakage cost, while “never load-bearing” removes the option branch from reader implementations.

11.6. Key wrapping outside the Parquet object

SOP SHOULD keep the wrapped DEK outside the encrypted Parquet file in the encrypted SOP access record.

This permits domain KEK rotation and DEK rewrapping without modifying the Parquet object’s bytes.

11.7. Canonical storage identity

For a PME object, storage_id is normatively the exact iroh-blobs content hash:

storage_id = iroh_blobs_hash(exact_encrypted_parquet_bytes)

iroh_blobs_hash is the 32-byte BLAKE3 root hash used by iroh-blobs for that exact blob.

SOP MUST NOT maintain a second BLAKE3 namespace or independently defined variant. The byte string accepted by iroh-blobs is the canonical object byte string and its iroh-blobs hash is the SOP storage_id.

This alignment gives SOP iroh-blobs’ incremental verification, resumable downloads, and verified range transfer without a second integrity layer or hash-conversion step.

The unkeyed hash of plaintext Parquet MUST NOT be exposed as a storage identifier.

11.8. Padmé object-size padding

SOP-PME-1 adopts the Padmé padding function from Section 4.4 of [PADME] as the target-length function for privacy-sensitive object-size padding.

Padding MUST preserve a standards-compliant PME file. Arbitrary bytes MUST NOT be appended after the Parquet footer.

The preferred encoding is a reserved semantically ignored padding value in encrypted Parquet footer metadata, sized such that the final encrypted file is the smallest valid PME representation meeting the Padmé target length.

If the selected writer cannot produce such padding while preserving exact DuckDB/Parquet interoperability, the implementation MUST fail the size-padding capability rather than emit a malformed Parquet file. G5 tests this property.

The unpadded exact byte count remains encrypted catalog metadata; untrusted storage observes only the padded ciphertext size.

11.9. Optional logical deduplication

If plaintext-level deduplication is later required, it MUST use a keyed fleet/domain-private logical fingerprint and MUST NOT be exposed to untrusted infrastructure.

SOP/1 does not require plaintext-level deduplication.

11.10. Integrity layers

PME provides authenticated encryption of Parquet modules.

SOP content addressing additionally verifies identity of the complete stored ciphertext object.

Producer authenticity remains separate and MUST be bound to the encrypted SOP producer manifest or publication message.

11.11. Performance

PME performance overhead MUST be benchmarked on target nodes.

The protocol MUST NOT silently fall back to unencrypted Parquet to meet a performance target.