Add design specs for the provider architecture and permission model epic - #986
Draft
aram356 wants to merge 7 commits into
Draft
Add design specs for the provider architecture and permission model epic#986aram356 wants to merge 7 commits into
aram356 wants to merge 7 commits into
Conversation
Five specs covering the work proposed in PR #838 (issues #777-#782), written so the next implementation pass has a normative behavioral contract: - Pluggable providers: identity lifecycle contract (mint/recognize/hash/ tombstone), trait minimalism, adapter parity, validation table - Permission model: signal precedence (opt-out over TCF), fail-closed jurisdiction resolution, policy file validation, decision-matrix testing - Migration and rollout: behavior-preservation matrix, ID stability vectors, loud-failure requirements, operator recipes - Client-cycle EC resolve endpoint: threat model and prerequisites; on hold until its open questions get an issue - Integration response-header hook: #782 contract with ordering and collision policy, ships only with a real consumer
Self-review of the five specs (own pass plus an adversarial fresh-eyes pass) surfaced fixes applied here: - Policy location reversed per maintainer decision: the permission policy is a [permissions] section of trusted-server.toml flowing through the config-store pipeline, not a build-time-embedded YAML. Overrides name acquisition rules directly instead of +/- sigils, and a rules.default entry separates resolved-but-unlisted countries from geo default_country. - Removed a circular requirement: geo and device providers are inputs to permission resolution and cannot be gated on its output; the enforcement gate is EC-only and the decorative required_permissions declarations are dropped from those traits. - Fixed the identity-stability guarantee: the EC id has a random per-mint suffix, so known-answer vectors pin the deterministic 64-hex prefix, recognition of existing cookies, and hash-prefix semantics instead of full identifiers. - Split the KV key contract into the verbatim graph-row key and the deliberately-colliding hash prefix that IP-cluster trust depends on. - Declared previously silent behavior changes in the migration matrix: global opt-out honoring (including tombstones) and the fate of non-regulated countries, with a preserving recipe for the latter. - Sequenced the geo neutral-default flip into the permission model PR so no intermediate step zeroes EC issuance under the current fail-closed gate. - Resolved smaller contradictions: withdrawal triggers made exhaustive (including denied-baseline and policy-edit cases), jurisdiction consistency requirement now covers both legacy lists with explicit exceptions, ISO rule-key validation made decidable, response-header reserved surface defined at cookie-name granularity for Set-Cookie, and the host-signals provider's removal made explicit with config rejection.
aram356
marked this pull request as draft
July 31, 2026 08:02
…ssion specs Blocking findings from review of PR #986, all addressed: - Raw EC egress is now a first-class enforcement point with a mandatory egress inventory (user.id, derived request IDs, page bids, proxy/click forwarding, identify, pull/batch sync, graph access): bidstream egress requires both purposes, first-party identity operations require store-on-device, revocation is exempt, and no-provider mode never vacuously allows an existing cookie to egress. - The behavior-preserving recipe carries the complete policy table plus a delta; a partial permissive-default-only policy is called out as the trap it is, and the exact recipe text becomes a CI fixture run through the decision matrix. - The EC permission gate is split: it covers minting and identity use only; parse, canonicalization, and tombstoning always run, with a spy-provider test - a blanket gate would block withdrawal in exactly the state an opt-out produces. - Provider switching gets active-writer/legacy-readers semantics ([ec] legacy_providers) so old identities keep resolving and stay withdrawable; unmatched cookies never egress. - The lifecycle contract now distinguishes the canonical graph key (provider-owned canonicalization, equivalent envelopes collapse) from the cluster prefix, which must be a literal byte prefix of the graph key because cluster sizing is a KV prefix listing; cluster support is an optional capability with an explicit degradation policy. - Device gating rationale corrected: only geo is circular; device is ungated by decision (security classification authorized by operator selection), with the boundary stated - uses beyond security classification need a vocabulary extension and a gate. - Withdrawal triggers made consistent (TCF refusal withdraws under requires_signal or denied), and a withdrawal-durability contract added: tombstones first, cookie expiry only on success, browser-side durable signals as the retry queue, fault-injection tests. - A signal-normalization matrix is now required (dual-TCF conflict modes, expiry, proxy mode, KV fallback, exact GPP fields), and malformed-but-present records fail closed for acquisition instead of degrading to absent. - Auction jurisdiction class is an explicit per-group regime attribute (gdpr / us-privacy / none), never inferred from purpose flags, and a first-class enforcement point. - The no-geo acknowledgment guard now keys on any enabled jurisdiction consumer, not only EC-provider selection. - Policy validation additionally requires rules.default when the section is present, rejects empty sections and case-insensitive duplicate keys, and canonicalizes default_country. - Resolve endpoint: exact Origin-allowlist membership or session-bound CSRF token (Sec-Fetch-Site demoted to defense-in-depth), real replay mitigation (session nonce or one-time consumption), and bounded-input requirements with 413 boundary tests. - Response hook: structured mutation operations that core validates and attributes (no raw header-map access), framing/hop-by-hop headers reserved, and a normative response-eligibility matrix. - Each spec now carries an explicit divergence table against its issue (#778, #779, #782) so there is one acceptance contract. - Non-blocking clarifications folded in: geo lookup-failure residual declared and metered, deterministic entropy required in conformance tests.
…and rollout gaps Blocking findings from the second review of PR #986: - Signal taxonomy gains a grant-signal class (TCF consent, explicit GPP non-opt-out, US Privacy present-and-not-opted-out including N/A) so a requires_signal US rule reproduces today's no-signal-blocks / explicit-non-opt-out-grants behavior, which the two-class model could not express; migration matrix gains rows 3a-3c and the example US group changes to requires_signal. - Auction dispatch gets a normative regime matrix (gdpr / us-privacy / none across consent, opt-out, malformed, expired, absent states), the compiled fallback gains regime = gdpr, and blocked dispatch means no outbound request at all. - The normalization matrix now states outcomes instead of subjects: restrictive/permissive synthesize per purpose, newest selects whole records, expired records are absent entirely, valid-beats-malformed within a family, KV fallback is live-wins with TTL-bounded staleness and an exempt consent-state lookup, mirror mode loses to request records, and GPP fields are enumerated per section. - The egress inventory is a concrete path -> permission table: proxy / click / Testlight forwarding assigned (both purposes, declared as new hardening in split row 11a/11b since those paths are ungated today), identify and pull/batch sync classified as partner exchange (both purposes), and S2S sync authorized by stored provider/version-tagged provenance re-validated against current policy. - Withdrawal drops the false atomicity claim: revocation families are idempotent independent writes, readers fail closed on any present member, and fault-injection covers the Nth-write failure. - Equivalence is provider-declared via fixtures (hmac: hex prefix case-insensitive, suffix case-preserved) instead of a universal case rule; the legacy HMAC grammar is formally reserved as the hmac namespace so verbatim row keys and provider namespacing coexist; global cookie-safe identifier bounds added; non-cluster providers get defined dedupe and redaction. - Legacy readers get full semantics: first-match parse with namespace-overlap validation, recognizing provider's permissions govern, provider/version-tagged provenance, transactional linking rewrite with dual revocation, and provider = "none" as an explicit stateless state that keeps revoke-only legacy readers. - Graph store required at startup when any provider can mint or read; a minted identity is not active until its row commits; a runtime failure matrix covers provider, graph, cluster, rewrite, and geo/device runtime failures. - Rollout gains a dual-read release (N+1 accepts both config shapes, N+2 rejects loudly) since no config is accepted by both current main and a rejecting binary; the preserving recipe becomes one committed valid TOML fixture (the prose delta reopened [permissions.rules], which is invalid TOML); metrics extended with retirement thresholds. - Resolve endpoint defines same-identity no-op / different-identity rejection and an atomic single-key reservation tying replay consumption to graph persistence. - Hook ordering becomes core -> integrations -> inviolable cache/privacy invariant pass (an appended cookie plus replaced public Cache-Control can no longer produce a shared-cacheable cookie response); generic ops reject Set-Cookie; per-integration operation limits and erroring-mutator semantics defined; every eligibility row tested. - Stale device-circularity wording removed from the permission spec; region-form default_country (US/CA) restored; the source-agnostic permission-source requirement of #777/#779 explicitly deferred in the divergence table; fail-closed and most-protective labels qualified with their stated exceptions.
…fecycle repairs Blocking findings from the third review of PR #986: - Grant evidence is now regime- and permission-scoped: gdpr rules accept only TCF consent for the specific purpose, us-privacy accepts TCF or explicit GPP/USP non-opt-out, none accepts any grant class - closing the hole where sale_opt_out=false in France would have authorized identity and partner egress with no TCF. Opt-outs and refusals stay regime-agnostic. - The graph schema change gets an expand-contract rollout: reader/ preserver release (unknown fields preserved through read-modify-write), fleet-convergence gate, then writer activation, with schema versions, lazy backfill, and mixed-version tests. - S2S batch-sync authority is a full recompute of both permissions from stored per-permission, time-bounded evidence (grant basis, timestamp, jurisdiction, policy revision, provider/version) - failing closed on denied, tightened baselines without acceptable stored evidence, expired evidence, or regime-rejected grant sources. Legacy pre-epic rows are hmac-v0 with no grant evidence and fail closed until lazily backfilled. - Pull sync split from batch sync: pull is browser-request-scoped and keeps using the live P1/P4 decision plus revocation state; only batch is provenance-authorized, and its gate is declared hardening (new matrix row 11c; row 11a corrected). - Withdrawal centers on a family revocation record: a stable family ID in every member row, one record written first that is simultaneously the durable intent, the sibling-discovery mechanism, and the fail-closed marker; member tombstones become cleanup; degraded-graph mode fails S2S closed while writes fail; the healthy-graph residual is declared. - Legacy rewrite is confirmed by presentation: both linked rows stay live until a later request presents the new cookie (the server cannot observe Set-Cookie acceptance); linked rows share the revocation family. - The normalization matrix now preserves actual current semantics: whole-record selection by combined P1/P4 eligibility for restrictive/ permissive, LastUpdated with freshness threshold and restrictive tie-break for newest, proxy mode skips decoding, one-valid/one-expired row added, and GPP section fields enumerated normatively. - The auction matrix regains the raw-signal arm: a decodable TCF record applies the gdpr dispatch rule in every regime, so a P1 refusal on US or non-regulated traffic still blocks dispatch. - Provider namespaces become declarative descriptors core can prove pairwise disjoint at startup (opaque parse cannot be); version rotation gets a schema (versions entries, mint_version, newest-first parse, retirement rules). - Client resolve reservations get pending/committed/failed states, lease takeover, retention through token expiry, deterministic graph idempotency, adapter CAS capability - and duplicates never receive Set-Cookie unless the reservation is session-bound, closing the idempotent-replay fixation hole. - The hook invariant pass preserves any pre-hook private/no-store classification (cookieless personalized HTML cannot be made publicly cacheable) and strips CDN directives; panics are declared forbidden and fatal on wasm32-wasip1 (panic=abort - recovery was unimplementable); a cumulative final-response header budget with deterministic rejection order added. - Rollout ordering corrected to strictly reader-first with a convergence gate (the previous either-order claim was false against binaries that reject the new shape); mixed old/new config shapes rejected; rollback sequencing defined; one preserving fixture per adapter since device/geo selections are capability-gated. - Device-provider authorization reconciled with persisted use: new rows stop carrying fingerprint-derived buyer-facing fields (declared change); a field-level graph contract table is a required implementation deliverable. - Non-blockers folded in: adapter-capability matrix, assigned- subdivision region validation, retirement quiet period no shorter than max cookie/row lifetime plus skew, expanded telemetry, and the stale recognize/hashed/eligibility terms corrected.
…cs, and distributed contracts P0: legacy identities can now enter the family-revocation protocol. Rows lacking a family ID derive one deterministically from (record kind, provider namespace, canonical graph key), so a first-post-upgrade withdrawal is discoverable by every future reader even if the writer crashes before touching the v1 row; random IDs are called out as recreating the orphan the design exists to eliminate. Withdrawal never depends on backfill; tests pin the first-request-is-withdrawal and crash-between-writes cases. P1 groups: - US signal fields get a normative field x value x permission x destructive table (4.5): sale maps to P1+P4 and is destructive, sharing and targeted-advertising map to P4 only and never destroy identity, USP carries no targeted field, absent/N-A grants nothing, state sections override national, opt-out beats grant across sections. The regime evidence table now defers to this mapping. - Normalization is a six-state machine (valid-grant, valid-refusal, opt-out, malformed-present, expired, absent) wired into precedence and the decision matrix (malformed blocks the granted baseline); proxy mode keeps a syntax pass so malformed is distinguishable, blocks record-derived grants, and is declared as a change from today's fail-open skip. - Conflict resolution is deterministic: whole-record selection over the (P1, P4) tuple ordered lexicographically P1-first (split-purpose records decided), newest uses LastUpdated with the freshness threshold and falls back to restrictive; expiry drops sources before conflict resolution - a declared change from today's conflict-first ordering. - The auction raw-signal arm triggers on raw TC-string presence or a GPP section-2 hint before decoding, so malformed raw TCF still blocks dispatch outside GDPR regions. - Stored provenance ages: authoritative timestamp + valid_until per evidence class, re-presentation does not reset age, and every live resolution atomically replaces the full per-permission snapshot so a later refusal clears older positive authority; rewrite provenance is the fresh live resolution, partner mappings keep original expiries. - A per-record-class consistency matrix: replay reservations need linearizable CAS with fencing (Durable-Object-class on Cloudflare, not Workers KV), family revocation records need strong reads plus a declared bounded visibility lag with read-failure failing closed, identity rows may be eventual; retention outlives every dependent lifetime (today's 24h tombstone TTL explicitly does not carry over). - The US policy enumerates US/<state> rules for configured privacy states with country-level US non-regulated, preserving Wyoming-class traffic; regionless-geo degradation is declared; the states-list consistency test is region-shaped. - The graph field contract is normative in-spec (providers 6.3): every v1 and new field with purpose, source, gating permission, TTL, rewrite, and revocation treatment - including discontinuing fingerprint-derived buyer-facing fields; releases unified as N/N+1/N+2 with semantic (not byte) unknown-field preservation, a hard rollback floor at N+1 after writer activation, and stated mixed-version expectations. - Identity boundaries are structural: core constructs physical graph keys with record-kind/provider/version prefixes (legacy hmac verbatim excepted) and an AuthorizedIdentity newtype - constructible only after parse, permission, graph, and family checks - is the only type outbound serializers accept. - Legacy rewrite aliases to one canonical row via fenced CAS (no dual-write divergence), with confirmation by presentation and a finite retirement deadline. - Client-cycle: session binding is required for production schemes (one-time consumption demoted to defense-in-depth; at-most-once only as an explicitly recorded posture with orphan cleanup); reservations carry owner hash and monotonic lease epochs with fenced transitions; owner-hash retry re-emits the cookie so lost responses do not orphan rows; resolve checks the family revocation record and loses races to revocation. - The hook snapshots all pre-hook cache restrictions (origin-supplied included) and allows only equal-or-stronger privacy; Content-Encoding and Content-Range join the reserved surface; the test set covers origin-private and core-private cookieless HTML, cache hits, Vary, every CDN directive, and body encoding. P2/P3: mint 'cookie write' clarified as scheduled-on-final-response with egress eligibility at graph commit; degraded-health is a per-instance in-memory state machine with hysteresis; HMAC versions resolve from row provenance (untagged = hmac-v0), not parse; client limits are exact (65,536-byte body, content-type allowlist, 256-byte identifier, 128-byte reservation key, per-code statuses); migration matrix gains rows 3d-3g; fixtures include the graph-store config; every rollout metric ships with threshold, window, and action; batch-sync's coverage dip is operationalized with the provenance-coverage metric; FR default relabeled a protective opt-in fallback; device-selection authorization qualified to the opt-in fingerprint provider; the illustrative policy example is labeled as such.
…ncy eligibility, and rollout closure
P1 fixes:
- Opt-out signals split into destructive (GPC, sale, USP - withdraw) and
non-destructive (sharing, targeted-advertising - revoke P4 only, never
tombstone) subclasses assigned by the 4.5 mapping, resolving the
4.2-vs-4.5 contradiction.
- Proxy mode performs minimal opt-out extraction (the 4.5-mapped fields
and USP only) so globally authoritative opt-outs are never suppressed;
still no record-derived grants; declared as a change from today's
opt-out-blind skip.
- GPP applicability is an ordered algorithm with a pinned jurisdiction ->
section-ID map (usnat 7, usca 8, usva 9, usco 10, usut 11, usct 12):
applicability from resolved jurisdiction, state-over-national per
field, restrictive aggregation; foreign and non-applicable sections
contribute nothing; N/A preserved as not-opted-out (declared,
correcting the earlier contributes-nothing rule).
- TCF conflict selection reverts to today's algorithm - P1-and-P4
conjunction comparison with standalone winning equal conjunctions
(including split-purpose) - replacing the invented lexicographic tuple
and keeping the Preserved label honest.
- S2S freshness is a per-evidence-class contract: TCF ages by
LastUpdated, GPP/USP by first-seen with an equality digest
(re-presentation keeps original first-seen), baseline grants re-derive
from the current policy revision; clock-skew clamping.
- Degraded-mode protection is declared local-only with the cross-instance
residual quantified (bounded by user return latency, metered), instead
of implying fleet-wide fail-closed.
- Family revocation records require a strongly consistent primitive;
Workers KV is explicitly ineligible ('60 seconds or more' is not a
bound); alias/rewrite records join reservations in the linearizable
CAS class and rewrite_legacy is rejected without it.
- The trait now really returns graph_key_suffix (a round-4 batch loss),
core owns the 6.3 physical key grammar (id/, alias/, fam/, rwx/,
resv/ prefixes + reserved legacy grammar) with wire schemas and TTLs
per record class.
- HMAC version attribution really resolves from immutable row provenance
(also a round-4 batch loss); parse identifies namespace only.
- AuthorizedIdentity is scope-parameterized (GraphOps vs PartnerEgress)
so a P1-only identity cannot reach an ORTB serializer.
- The provider contract gains acquisition modes (ServerMint /
ClientResolve carrying resolve_from_client and the JS module id); the
resolve endpoint enforces the provider's full required_permissions.
- Revocation-wins at the resolve endpoint holds because the family check
runs through the linearizable class client-cycle already requires.
- Rewrite is a persistent fenced transaction: pinned target on retry,
reconciliation of updates that won the old-row CAS, orphan GC by
absent transaction, and fenced alias retargeting keeping chains
single-hop.
- Release protocol: rollback is binaries-first (N+2 -> N+1 keeping the
new config); N+1 rejects provider/version selections it cannot encode;
a pre-N+1 graph-store readiness step plus matrix row 12 covers
graphless HMAC deployments (breaking, declared).
- The abstract capability list becomes a concrete adapter matrix
(Fastly/Axum/Cloudflare/Spin) with honest cells - including that
Cloudflare supports platform geo country-only (the migration text
claiming it rejects platform geo was wrong) and that no CAS-class
primitive is currently wired anywhere but the dev adapter.
- Integration cookies enter the permission model: registration-declared
names with purpose and retention, persistent cookies gated on
store-on-device, session cookies as the narrow exemption.
- Cache monotonicity is a defined lattice (no-store > no-cache > private
> public, shrink-only ages, snapshot-gated stale directives, protected
Vary union) in the contract, not the tests.
P2/P3: persisted-KV consent now flows through the full pipeline
(declared change); provenance transition and mid-replacement fault tests;
the recipe renamed minimal-divergence with its unavoidable divergences
enumerated; batch-sync coverage is a gated stage with thresholds and a
pause action; policy-revision activation defined (stamped revisions,
bounded mixing, no tombstone resurrection); representation surface
extended (Content-Type, ETag, Last-Modified, Accept-Ranges, digests);
append restricted to list-valued headers; exact budgets and snapshot
read semantics; reservation namespacing and per-state ownership
conflicts; media-type matching ignores parameters; the pass-through test
wording fixed; and a product-decision sign-off list (9 items) added to
the migration spec for explicit maintainer ratification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Five design specs for the provider/permission epic (#777–#782), written after review of PR #838 so the next implementation pass has a normative behavioral contract instead of discovering design decisions inside a 67-file diff. They live in
docs/superpowers/specs/per the existing convention. Where a spec contradicts PR #838 — or one of the source issues (each spec ends with an explicit divergence table) — the spec is authoritative.pluggable-providers-designpermission-model-design[permissions]intrusted-server.toml(not build-time YAML), enforced-vocabulary rule, three-class signal taxonomy with regime-scoped grant evidence, normative normalization matrix preserving current conflict-mode/expiry/proxy semantics, exhaustive withdrawal triggers with a family revocation record, concrete raw-EC egress inventory (path → permission table), stored-provenance S2S authority with legacy fail-closed, regime-based auction dispatch matrix including the raw-TCF arm, fail-closed jurisdiction resolution with an acknowledged static-jurisdiction escape hatchprovider-migration-rollout-designclient-cycle-ec-resolve-designintegration-response-header-hook-designKey decisions encoded
trusted-server.toml, validated atts config pushand startup; absent policy falls back to a compiled-in everything-requires_signal,regime = "gdpr"posture.user.id, derived IDs, page bids, forwarding, identify, sync), not only EIDs; batch sync is authorized by stored provenance recomputed against current policy, and legacy rows fail closed until backfilled.host-signalsandclient-fixedproviders are deliberately not carried over; configs selecting them fail loudly.The specs have been through three review rounds (an adversarial fresh-eyes pass plus two maintainer reviews); the commit history records each round's findings and fixes.