diff --git a/CHANGELOG.md b/CHANGELOG.md index ab47dda8a..de3ef5300 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,393 @@ ## Unreleased +- Recover broad behavioral questions beginning with `Why`: treat the + interrogative as question scaffolding while preserving quoted/qualified + subjects and literal symbol lookup. Refresh disposable query responses. + +- Keep inadmissible exact source targets unresolved instead of + falling back to unrelated same-named declarations, including JavaScript and + TypeScript callable parameters. Rebuild disposable current graphs with AST + cache34; historical graphs and language qualification remain unchanged. + +- Add opt-in prospective developer graph-precision diagnostics with exact + conditional finite-population bounds. Preserve legacy formats and all existing + qualification gates; arithmetic does not establish design acceptance or quality. + +- Raise the Rustls minimum and locked dependency to 0.23.46 for upstream fixes + RUSTSEC-2026-0346, RUSTSEC-2026-0347, RUSTSEC-2026-0348 and RUSTSEC-2026-0349. + Other dependency advisories remain tracked separately. + +- Recover bounded helper candidates for eligible broad questions about one or + more operations through exact native calls, with Low/unverified navigation + paths. Preserve + requested depth and ambiguity; expose separate indexed/helper lookup budgets + in raw discovery2 and ranked-view7, including after presentation pruning. + Single-operation questions retain distinct reachable alternatives using bounded + lexical operation hints; hints do not establish intent or behavior. +- Show a retained, source-owned helper preview when no question-matching excerpt + exists. Label it behavior unverified, keep matching terms/ranking unchanged, + and refresh disposable quotation responses. Unmetered ranked-view4 remains. + +- Preserve typed exact-name and ID matches when structural discovery becomes + a ranked answer. Keep identity labels separate from relevance and coverage; + refresh disposable ranked identity responses. + +- Recognize `rebuilt`/`rebuild` and `written`/`write` spellings in broad + questions while preserving literal identities, operation distinctions and + existing confidence guards. Refresh disposable ranker responses. + +- Normalize broad-query comment matches by their matching owned passage when + the retained comment field is complete, preventing unrelated comments from + diluting that evidence. Keep admission, uncertainty and exact-name tiers. + +- Prefer distinctive source quotations after requested-action coverage, using + existing corpus statistics while preserving exact anchors, ownership, token + boundaries and candidate relevance. Refresh disposable quotation responses. + +- Recognize bounded, corpus-supported forward spellings in broad questions, + retaining literal topic guards, explicit identity and Low/unverified mapping + disclosure. Keep canonical query groups separate and refresh ranker responses. + +- Prefer retained source-quotation windows covering more distinct requested + terms, so an early generic match does not hide later question evidence. + Preserve exact UTF-8 source bytes, coordinates and quotation limits. Known + owned comments cannot supply code-window selection evidence. Candidate + ranking and relevance are unchanged; refresh disposable excerpt responses. + +- Restrict automatic deeper call context to High-relevance selected roots; + partial matches keep first-hop context. Explicit context inspection retains + witnessed paths without promoting their relevance. Refresh context responses. + +- Continue broad-query raw discovery context along exact source-witnessed calls + within the requested depth, with explicit state, fan-out, frontier and output + bounds. Count rejected adjacency records and lookahead against relationship + work limits. Related callees remain inspection context; candidate ranking and + relevance are unchanged. Refresh disposable responses with a new context epoch. + +- Derive unconfigured Python module names from verified source-root layout: + bare roots use relative modules; package roots retain their package prefix. + Checkout names no longer hide valid imported calls in bare roots. Retain + configured import-root precedence, ambiguity and exact call occurrences. + Retain source identities for script/group paths such as `billing-service`; + escape literal dots and percent signs inside components so they cannot + impersonate import separators. Package-entry links, non-regular metadata and + invalid package names fail explicitly when no configured roots apply. Python producer 2 remains + `Qualifying`; AST cache 33 and layout-bound fingerprints refresh current + evidence without rewriting historical graphs. + +- Retain up to 64 source-anchored comments per callable, allowing broad queries + to find explanations beyond an eight-comment prefix. Keep independent field, + file and artifact limits and explicit partial coverage. Source-query profile 8 + and AST cache 32 refresh current evidence; historical profiles remain unchanged. + +- Normalize native path separators in TypeScript source lookup keys, preserving + nested relative imports and directory-index selection on Windows. Keep exact + file identity, extension priority and source-root containment checks. + +- Preserve unresolved local export names before target resolution and require a + version1 source export census for typed imports and known-empty star branches. + Keep recovered, unsupported and legacy export surfaces unknown. Validate the + new optional root evidence and include it in fact-digest profile4; AST cache31 + refreshes current disposable facts. Independent qualification is pending. + +- In development, make typed TypeScript receiver imports traverse export + obligations before selecting a target. Preserve exact target files and unknown + branches, enforce public export visibility and explicit-export precedence, + and charge bounded shared/cyclic export graphs to the receiver work budget. + Unsupported namespace transitions remain unresolved; qualification is pending. + +- Bound aggregate TypeScript argument proofs before cloning or index retention. + Count repeated nodes and UTF-8 text with shared per-file validation rules. + On valid aggregate overflow, preserve declarations and eligible exact + relationships through explicit degraded partitioning and omission counts; + reject invalid per-file payloads before projecting partial results. + +- Bind TypeScript merge-absence evidence to explicit census capability version1. + Missing legacy metadata cannot certify absence, even with an old true flag; + unknown present versions are errors. TypeScript5 / JavaScript4 and AST cache31 + rebuild current disposable facts without changing historical realizations or + promoting development producers. Profile4 digests include the capability. + +- In development, require bounded observed TypeScript module/reference load + coverage before preserving nominal identity through `Awaited`; missing or + augmenting dependencies remain unresolved. Canonical fact digests include + the new typed source-load census. Bind optional owning load proofs to exact + kind and occurrence; keep type-package evidence out of ordinary module-symbol + lookup. Combined-patch review and gates are pending. + +- Preserve source/type identity through supported TypeScript `Awaited` wrappers + using explicit bounded nominal inventories and project merge certificates. + Keep unsupported thenables and incomplete/merged types unresolved. Extend + evidence/2 with TypeScript class/interface inventory metadata and optional + batch merge coverage; strict readers must upgrade. Fact-digest profile3 + includes the new certificate and typed argument proofs. Combined TypeScript + producer/cache identities are TypeScript5 / JavaScript4 / AST cache31; + independent qualification remains pending. + +- Return a must-use omission report from the Rust universal materialization + API so analysis limits cannot silently discard their completeness status. + Rust integrations must consume the report and preserve degraded status for + partial output; node/edge buffer updates are not transactional. + +- Retain nominal type-reference occurrences for Rust struct and struct-like + enum-variant literals, with exact owners/source anchors and shared conservative + cross-file resolution. Support concrete impl `Self`, aliases and generic + nominal paths without inventing constructor calls or field access semantics. + Keep block-local imports, aliases and impl receivers in their actual lexical + scope, preserving distinct declarations for same-named block aliases. + Distinguish repeated same-line declarations by source span and retain receiver + type provenance across body imports, field/index access and local bindings. + Keep generic parameter identity and trait-bound declaration context through + local shadows; follow bounded simple receiver aliases without guessing. + Preserve explicitly deferred receiver nodes across languages instead of + rewriting them to a uniquely named but unproven source method. + Resolve complete generic trait-bound sets and supertraits project-wide, + preserving mixed local/imported ambiguity and exact source occurrences. + Add typed `rust_generic_bounds` evidence/2 constraints and + `rust-generic-bound-dispatch` provenance; older strict evidence readers must + upgrade. Bound substitution chains to16 steps and merged maps to64 entries. + Require explicit trait-member inventory completeness across the full bound + hierarchy; macros and unmodeled transformations cannot prove method absence. + Keep non-type generics out of type-receiver constraints. Incremental digest + profile2 includes completeness and namespace evidence. + Rust development producer4 remains `Qualifying`; AST cache29 refreshes + disposable facts; historical realizations and the graph v1 shape stay unchanged. + +- Reject mixed incremental source-evidence reuse when code was clipped or its aggregate capacity may be exhausted; recover through the complete cached/fresh path. +- Retrieve bounded literal-only callable vocabulary in Python, Rust, Go and + TypeScript/TSX without an identifier seed. Preserve exact raw source slices, + callable ownership, code omissions, and independent lexical/parameter + coverage. Keep dependent matches Low. Source-query profile7 and AST cache27 + distinguish the new AST-order evidence; immutable profiles1–6 keep their + original capabilities and statistical meaning. Graph facts are unchanged. + +- Require independent topic evidence for broad decision-predicate questions. + Retain ambiguous block/permit nouns through exact native parameter sites or + actual class/struct declaration names, with whole source anchors and Low + relevance. Disclose partial/legacy role evidence only for affected questions. + Preserve literal topic recall, exact identity, graph facts and existing + operation equivalence; ranker21 refreshes disposable response caches. + + +- Retain separately bounded native parameter-binding source sites with explicit + complete, partial or unsupported coverage. Preserve exact callable ownership, + existing lexical priority and profiles1–5 semantics. Source-query profile6 + and AST cache26 distinguish the new evidence; no graph facts or retrieval + scores change solely from these role annotations. Preserve refreshed source + digests when conservative update reuse falls back to the full extraction path, + including cached edits with an unrelated cache miss. + +- Retrieve broad questions whose vocabulary occurs only in bounded AST-owned + code text, including error strings and literal keys. Keep code-dependent + matches Low and source-cited, exclude nested callable context, and avoid + counting identifier/comment words again. Source-query profile5 preserves + profiles1–4 with their original retrieval/statistical meaning; AST cache26 + and ranker18 refresh disposable current state. Preserve preferences supported + by complete identifier/comment fields when only code windows are clipped; + keep that clipping visible. Graph facts are unchanged. + +- Recall conventional `get`/`set` method spellings for broad read, lookup, + getter and setter questions. Require separate literal owner context, disclose + the translations and keep dependent candidates Low without an implementation + preference. Preserve exact identity and graph facts; ranker16 refreshes + disposable response caches. Preserve literal field provenance so namespace, + signature or prose words cannot silently promote a hinted operation. Keep + relationship, owned-identifier and source-body dependency attribution literal + so hinted names cannot raise source-dependent relevance. + +- Preserve declared native source-query evidence beside each developer-captured + Compass graph. Bind the sidecar to its graph and frozen input, reject omitted + or drifting sidecars, and isolate repositories' fixed filenames. Correct old + graph-only measurements with a new capture; native query behavior is unchanged. + +- Let developer query comparisons predeclare an exact native no-match exit + contract. Preserve raw status/output, budgets, genuine failures and legacy + scoring; distinguish valid abstention from execution failure only in new + frozen experiments. This does not establish a comparative quality result. + +- Add an explicit developer-only profile for recapturing exposed development + questions with linked historical metadata, exact question selection and newly + reviewed source definitions. Preserve old outcomes and heldout blinding; + calibration remains permanently ineligible for comparative superiority. + Recheck capture provenance after provisional scoring before completion. + +- Add opt-in version-2 developer query-evaluation registrations with predeclared + groups of equivalent implementation witnesses. Preserve exact fact ownership, + relationship evidence, legacy scoring and original/lookup panel bindings. + Group recovery is reported separately from per-witness graph/source recall; + no new comparative quality result is claimed. + +- Recall common code abbreviations for broad parameter, dictionary, seconds and + nanoseconds questions even when the unabbreviated word appears elsewhere in + the index. Require separate literal owner context, disclose translations and + keep dependent matches Low. Exact names, explicit identities and graph facts + retain their contracts; ranker15 refreshes disposable response caches. + +- Prefer cohesive compound identifiers over scattered word mentions for broad + questions such as `load-only` field setup. Require separate owner-topic + evidence, retain Low inspection relevance and bounded real source citations, + and preserve exact identities. Code-based inspection preferences now reject + missing, empty and native-ellipsis-clipped signatures. Ranker14 refreshes + disposable response caches; graph facts and recall postings are unchanged. + +- Require frozen, independently reviewed exposure lineage and verified history + artifacts for heldout developer comparison captures. Reject known source or + linked-mechanism reuse across renamed tasks, question rewrites and commits; + preserve legacy diagnostic inputs and the existing numeric quality gates. + +- Recall indexed `item`/`element` and `copy`/`clone` alternatives when ordinary + broad-query words `entry` or `snapshot` have no indexed literal match. Require + separate source-owned literal topic evidence for lexical-only candidates; + keep dependent matches Low and disclose each translation. Preserve exact + identities and existing literal candidate admission. Refresh disposable + response caches; graph facts are unchanged. + +- Prefer callable-local operation identifiers with separate topic evidence for + broad single-operation inspection. Exclude signature-overlapping identifiers, + compound/negative/temporal conditions and mismatched document/test intent from + this preference. Preserve uncertainty and exact identity; version disposable + response caches as ranking semantics change. Graph facts are unchanged. + +- Include retained source citations on compact discovery seed lines before + relationship context, keeping selected candidates inspectable on the first + page without changing retrieval order or cursor semantics. + +- Preserve documentation coverage warnings in local SQLite queries, matching + JSON queries. Recover older snapshots through bounded diagnostic reads and + invalidate cached answers that could omit these warnings. Retain graph + coverage warnings when discovery finds no matching seed. + +- Score ranked code quotations without borrowing terms from retained overlapping + comments. Preserve the precise comment when it supplies the matching evidence. + Under tight text budgets, shorten or omit lower-ranked quotes before clipping + the first candidate's inspection evidence. Candidate ranking is unchanged. + +- Exclude Python lambda keyword tokens from auxiliary callable scopes. This + corrects scope counts and can recover source evidence at the scope quota; + semantic graph records are unchanged. + +- Add separately labelled nested anonymous-callable inspection context to + already retrieved implementations. Native AST parent facts preserve ownership + barriers and actual syntax anchors; no graph calls or execution are inferred. + Direct windows keep priority within shared limits. Source profile4, ranked + view4 and AST cache24 version the change; readers retain older profiles. + + +- Bound disposable JSON query-index validation. Retain the full integrity check; + slow eligible file checks can use a size-bounded, byte-identical inspection + image, while WAL and mismatched images keep original-file validation. Report + startup-budget exhaustion as a typed timeout without rebuilding the index. + Cancel and join operation-local SQLite interrupt timers before reusing the + connection, avoiding coarse progress-callback overruns in integrity walks. + +- Retain one exact callable-local identifier occurrence per spelling for body + retrieval, with explicit omitted-occurrence counts. This reduces repeated-site + pressure on bounded publication while preserving independently collected code + windows. AST cache24 rebuilds disposable source-query evidence. + +- Development ranked queries retain bounded AST-owned code windows for + comment-free implementations. Ranked JSON uses `compass.query.ranked-view/3`; + immutable legacy views and source-query profiles remain readable. Code windows + have independent quotas and do not change ranking or graph relationships. + + +- Keep a bare existing node ID as the sole ranked inspection target, excluding + unrelated matches for identifier prefixes such as `sha256`. Raw discovery + and optional structural context retain their contracts. + +- Preserve real source postings when their terms equal store capability markers. + Store publication uses separate capability records and retains legacy reads; + indexing Compass's own capability constant signatures no longer conflicts with + empty markers. Graph records and relationship evidence are unchanged. + +- Add bounded source-owned comment/documentation quotations to opt-in ranked + answers. Ranked JSON uses version 2 and retains version 1 reader compatibility; + text shortens/omits quotes before candidate rows to honor the output budget. + Candidate ranking, graph relationships and structural confidence are unchanged. + Omit clipped quotation windows that no longer contain matching query terms. + + +- Retain direct variable-initializer function definitions in source ranges and + TypeScript/TSX body-query evidence using exact AST owner/parent facts. Preserve + original binding provenance, stable identities and relationship occurrences; + reject ambiguous ownership. Source-query profile2 reads alongside immutable + profile1 artifacts. AST cache20 rebuilds disposable entries. JavaScript3 and + TypeScript4 remain `Qualifying` pending independent source audit and promotion. + +- Preserve non-file AST declaration/occurrence ranges during fact-neutral edits, + including declarations ending at EOF. Refresh existing file-node digest/size + metadata from file inventory. Refresh file-scoped container envelopes only + with current AST ownership and original declaration provenance; restore Kotlin + package updates after metadata-tail truncation while preserving explicit + namespace bounds and cold-build/edit-restore equivalence. + Reject non-callable body owners and report + unknown source-query schema/profile versions through typed error codes. + +- Apply broad implementation-search test preferences to colocated test/spec + files and typed test/fixture roles. Preserve explicit test intent and both + candidate sets; isolate cached answers with ranker/9. + +- Select ECMAScript member scopes by their structural parent instead of hashed + child IDs. Preserve accessor ambiguity and declared field/method targets across + filename changes, and recognize JavaScript class-field name anchors rather + than replacing them with constructor writes. JavaScript3 and TypeScript4 remain `Qualifying`; AST cache20 + rebuilds disposable facts without rewriting historical graphs. + +- Add a native, graph-bound callable-body query index for Python, Rust, Go and + TypeScript/TSX. Retain bounded AST identifiers/comments with source citations, + shared JSON/store recall and statistics, and Low inspection relevance when + body evidence supplies missing context. Publish coherent immutable artifacts, + refresh both incremental paths, and invalidate caches by content/profile. + Comparative quality and release qualification remain under verification. + +- Require topic evidence beyond generic kind/language matches for broad + questions with additional subjects. Preserve simple listings, exact identity + and tentative candidates supported by real topic evidence. + +- Recall callable owners of matching local identifiers through bounded exact + AST containment. Preserve unique ownership, source citations and shared work + ceilings; label inferred query relevance as Low inspection evidence. + +- Retain more ranked candidates within the text budget by abbreviating lengthy + explanations before dropping an owner. Preserve uncertainty, citations and + the byte limit; the JSON projection and query ranking are unchanged. + +- Recall broad implementation candidates through bounded, verified outgoing + call/reference evidence and retain source citations. Mark matches that depend + on referenced identifiers as Low inspection leads. Fix plural acronyms such + as TTLs becoming missing symbol names; preserve explicit identity guards. + +- Add bounded AST leading documentation for development Rust3, Go2 and + TypeScript4 producers, retained as source-proven rationale resources. Require + the exact excerpt and full owner anchors before broad retrieval uses leading + prose. Preserve citation ranges, report omitted excerpts in graph/query + warnings, and invalidate extraction/publication/ranking caches. These producer + versions remain `Qualifying` pending source audit and release promotion. + +- Improve broad native retrieval with fielded names, signatures, aliases, + paths and source-owned prose, shared JSON/store term statistics and bounded + declaration recall. Mark partial candidates with missing owner-specific + context and retain exact lookup behavior. Unknown ordinary wording can return + Low candidates; absent explicitly named subjects remain guarded. Negative + qualifications return inspection candidates with an explicit unverified + condition warning rather than claiming that the condition holds. + Preserve temporal conditions as explicitly unverified while retrieving on + substantive terms; keep quoted and qualified names literal. + Disposable JSON indexes rebuild to format 4; older immutable stores give an + explicit update instruction rather than claiming full field coverage. +- Recover query-index builds after a process is terminated by using persistent + operating-system locks. Recheck index validity and replace invalid indexes + only while holding the build lock; a live competing builder yields an + actionable bounded-wait error. Keep the new cache namespace separate from + older presence-lock builders. +- Add opt-in `query --view ranked` and MCP `query_graph` with `view: "ranked"`. + Show up to five source-located candidates with relevance reasons, + separate identity/evidence labels, honest recall bounds and scope refinement. + Optional `--include-context` / `include_context` appends witnessed one-hop context. + Publish the additive `compass.query.ranked-view/2` projection; existing raw + formats and discovery cursor contracts retain their shapes. + ## 0.4.2 - 2026-10-02 - Add source-scoped architecture and community views, compact hotspot rankings, diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 6043437be..e190a992a 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -1,5 +1,981 @@ # Compass compatibility +## Exact source-binding resolution + +An explicit `exact_target_declaration_id` is authoritative for target identity. +When that declaration fails the candidate's constraints, resolution remains +unresolved instead of falling through to a same-spelled +lexical, module or external target. For example, a JavaScript/TypeScript call +through an injected parameter cannot become a call to an outer function with +the same name. This does not infer the parameter's runtime implementation. +Ordinary admissible exact targets retain their identities and provenance. +Missing declaration IDs continue to fail evidence validation explicitly. + +AST cache34 invalidates disposable current build state so `compass update ` +recomputes resolution. Graph/evidence schemas and immutable historical +realizations are unchanged. Existing language qualification is not promoted +by this correction; native controls and independent qualification are separate. + +## Prospective developer graph-precision bounds + +The opt-in developer precision design `compass.broad-graph-precision-design/2` +requires `populationBoundsProfile = "exact-hypergeometric-bonferroni/1"` and +emits `compass.broad-graph-precision-diagnostics/2`. Its added +`conditionalSamplingBounds` record uses +`compass.broad-graph-conditional-bounds/1`; endpoints are explicit rational +numerator/denominator pairs. Bounds are conditional on ideal uniform sampling +without replacement and truthful labels, not proof that those assumptions hold. +Design acceptance, semantic truth, independence and qualification remain false. +Existing certified-interval fields remain null; no noninferiority or joint +qualification decision is introduced. + +Design1/diagnostics1 retain their existing shape and meanings. Unknown majors or +bounds profiles fail explicitly. Both design versions bind the complete current +support-code hashes before capture and recheck them after assessment. Verify old +immutable experiments with their retained frozen code; never rewrite their design +or apply this new profile after observing outputs. This developer-only addition +does not change Compass's native runtime, graph/source formats or query ranking, +authorize any capture/reviewer, or weaken existing numeric qualification gates. + +## Callable comment recall + +Current updates publish `source-query/8-callable-comments` and rebuild disposable +AST cache 34. A callable can retain up to 64 source-anchored comments instead of +eight, so introductory bookkeeping comments are less likely to exclude later +implementation explanations. Selection remains source order and is bounded; +this does not index every comment in arbitrarily large callables. Each comment +remains limited to 4,096 bytes, each searchable field to 8,192 bytes/512 terms, +per-file retained identifier/comment text to 1 MiB, and artifact lexical text +to 16 MiB/65,536 sites. Exhausted limits retain explicit omission/coverage status. +Code-window and parameter limits are unchanged. + +Profiles 1–7 retain their capabilities, projection/statistics and eight-comment +validation limit. Older readers must upgrade to read profile 8; rebuilding current +artifacts does not rewrite historical realizations. Run `compass update ` +with the new binary. Comment retention uses the query contracts documented +below; source-artifact hashes separate response caches. Comments remain lexical +inspection evidence, not proof of behavior or new graph relationships. The cache 31 notes below +describe earlier development changes now superseded by cache 34. + +## In-progress TypeScript terminal inventories + +Development TypeScript5 / JavaScript4 and AST cache31 replace the prior +disposable producer/cache identities. They remain `Qualifying`; historical +realizations and the approved release-promotion record are unchanged. Run +`compass update ` with the new binary to rebuild current facts. + +The optional `typescriptMergeInventoryVersion` is1 for the supported complete +observed-load census contract. Merge absence requires this exact version and +`typescriptMergeInventoryComplete=true` in every TypeScript input. Older +missing-version facts remain readable but cannot certify absence, even when +an old boolean is true or the producer version is changed. Unknown present +versions fail validation explicitly. This capability is separate from producer +and cache identities; no default upgrades an absent legacy census. Profile3 +fact digests include the optional version distinctly. + +TypeScript class/interface facts can now carry `memberInventoryComplete` under +evidence/2. The bounded producer certifies supported direct members only after +matching their exact retained declaration anchors and primary member scope. +Certificates require an explicit ECMAScript module and no parser recovery, +heritage, constructors, computed/escaped member names, decorators, +ambient/namespace ancestors or unsupported nominal syntax. Other retained +nominals carry false. The accepted shape has an explicitly empty complete +base set. Inventory work shares the200000-step type-origin budget; exhaustion +returns a resource error without a successful partial batch. + +The optional batch field `typescriptMergeInventoryComplete` certifies an original +AST walk, independently of declaration projection. Nonempty scripts, parser +recovery, ambient declarations and namespaces cannot certify the absence of +augmentation. Missing/false metadata in any TypeScript input, or a TypeScript +source in the graph inventory without a corresponding certified batch, prevents the project +consumer from relying on a closed merged-type inventory. The syntax +walk charges each visited/frontier node before retention and enforces depth512. + +The typed project consumer unwraps intrinsic `Promise`/`PromiseLike` layers +individually in definition scope. A terminal `Awaited` value retains a nominal +identity only after expanding its exact actual/alias proof to a uniquely bound +TypeScript class/interface with complete project merge inventory, both nominal +certificates, no bases and no `then` member. Missing certificates, merged types, +inherited or unknown members and custom thenable result semantics are not +certified by this rule. Unsupported cases remain unresolved. This does not +prove runtime promise behavior or introduce a TypeScript compiler dependency. + +Older strict readers and validators restricted to Rust trait inventories must +upgrade before reading these TypeScript facts. Incremental fact-digest profile4 +includes the batch certificate and typed +argument proof trees, including explicit Unknown/absent differences. Exact proof +IDs can conservatively invalidate fact-neutral reuse when scope identities +change; correctness takes priority over that optimization. TypeScript5, +JavaScript4 and AST cache31 bind the combined development extraction changes; +no release promotion or historical rewrite is implied. + +The in-progress `typescriptMergeDependencies` census records typed module, +reference-path and reference-types loads with source anchors. Static imports, +re-exports, import-equals and literal import/require calls participate even when +an ordinary symbol import would be unused. Supported triple-slash directives +are recorded even in comment-only files. Unsupported directives, escaped names +or specifiers, nonliteral loads and parser recovery prevent certification. +Every observed load must reach supplied certified TypeScript source before +merge absence is used. Reference paths require the exact normalized source; +relative modules use exact owning project targets or bounded source candidates +with valid TypeScript source/declaration substitution within distinct extension +families. This is a bounded source profile, not compiler-wide resolution parity. Unavailable, invalid or +unknown source identities poison coverage across all resolution contexts and +cannot fall back to a lexical source. Literal census removes exactly the matching +outer delimiter and preserves interior quote/backtick characters. Bare +modules and reference-types require explicit project resolution metadata; +a same-spelled file is insufficient. +The optional owning edge attribute `typescript_source_load` is a version1 +`TypeScriptSourceLoadProof` containing the exact typed census dependency, +including every source-range coordinate. It binds a resolved target to one +observed load. Reference-types require this mode/occurrence-specific proof; +unanchored legacy module edges cannot satisfy them. A legacy module mapping +can cover only one matching module occurrence; repeated import/require loads +require their own proofs. Invalid/unknown versions fail explicitly and payloads +are inspected by reference before allocation. Cross-load proof matching shares +an explicit1048576-step limit. Missing owning reference-types resolution remains +unsupported; the census alone does not materialize a type package or invent +this proof. Source-load proof production/qualification is not claimed here. +The ordinary module-symbol index excludes reference-path/reference-types proof +edges; introducing type-package coverage must not add symbol-import targets. +Complete known alternatives can coexist; unknown source identities and targets +still prevent absence proof. A supplied augmenting or transitively +incomplete source also prevents the proof. Ordinary direct member lookup does +not require this merge-absence certificate. + +The census is limited before retention to4096 entries and1MiB of copied +specifier/source text per batch,1024 bytes per specifier and4096 per source +path. Project accounting checks65536 entries and16MiB before borrowed batch +cloning or owned indexing. Limit failures remain explicit. Profile4 digests +include every dependency field and ignore vector input order. Strict older +readers must upgrade for this new optional vector. Dependency closure review, +complete bounded export traversal and independent qualification remain open; +the combined development patch is not ready for release. + +Project typed-argument accounting counts repeated proof copies, Unknown nodes +and UTF-8 owner/reference/path bytes, including owners of present empty proofs. +It limits a resolver input to2000000 nodes and128MiB of copied proof text before +borrowed batch cloning or owned index retention, using the same per-batch +accounting as evidence validation. Aggregate auxiliary overflow takes the +existing explicit degraded exact-only partition path: declarations and eligible +exact relationships survive, omitted candidates are counted, and proof/load +payloads and merge certificates are discarded before partial indexing. Invalid +per-file payloads remain errors and cannot be hidden by an earlier aggregate +overflow. No new CLI or serialized limit configuration is introduced. + +## In-progress typed TypeScript export traversal + +Root TypeScript/JavaScript file/module declarations can carry optional +`ecmascriptExportInventory: {version: 1, complete: bool}` under evidence/2. +The producer checks original top-level export syntax against retained root +bindings on the shared200000-step source-work budget. Unresolved local export +names remain explicit binding obligations before target selection. Recovery, +non-module scripts, ambient/namespace exports, export assignments, unsupported +string-name/destructured export forms, and omitted projections cannot certify +completeness. Complete describes retained syntax, not available targets. +Missing/false legacy metadata cannot certify an empty star branch or a unique +typed import, independently of the separate merge-absence certificate. Mixed +complete/incomplete roots for one physical source remain incomplete. Unknown +present versions and certificates on other languages or nested declarations +fail validation. Strict older evidence readers must upgrade. Fact-digest +profile4 includes this state; AST cache31 refreshes disposable facts. Current +TypeScript5/JavaScript4 development identities remain unpromoted. + +Typed receiver proofs now retain per-binding export obligations and full target +source file identities before the ordinary module view can flatten alternatives. +An unavailable owning project target remains unknown across contexts and orders; +a same-stem file or directory index cannot replace that target. Without owning +metadata, the bounded native profile considers supported relative file/declaration +substitutions in file-before-directory order. Bare packages and custom suffixes +require owning project metadata. This profile does not claim compiler-wide +package/configuration or runtime-loader parity. + +Only explicit top-level export bindings supply public named imports. A private +same-named declaration, including a declaration exported only inside a namespace, +cannot become a public file export. Explicit local/named exports take precedence +over stars for that name. Missing named targets remain unknown rather than falling +back to a star. A known empty supplied source differs from an unavailable source. + +The typed root and nested argument import paths share fallible export traversal +with the receiver's16384-step work meter. Distinct traversal states are bounded +by the existing per-lookup candidate limit (default256), and depth is at most64. +Raw alternatives are checked before filtering. Repeated states are discovered +once per walk. Bounded fixed-point propagation computes each state’s export +closure on the same meter, then verifies every required physical target and +independent star-binding group before selecting a result. Empty/nonempty +alternatives within one binding remain unknown; a separate known-empty star +branch can be valid absence. +Limits stay explicit and cannot yield a usable partial winner or legacy fallback. +Namespace reexports that need an unmodeled namespace value/visibility transition +remain unknown; namespace imports of explicitly named exports remain supported. +These controls do not qualify the overall TypeScript consumer or a release. + +## Universal materialization status + +The Rust `UniversalResolutionIndex::materialize` API now returns a must-use +`UniversalMaterializationReport` instead of `()`. Its `omitted_candidates` +counts explicit resource-limit decisions whose relationships were not +projected. Callers must inspect this count and preserve degraded status when +publishing partial output. This is a reporting operation: it can append graph +records and update existing declaration attributes before returning the report. +It does not provide transactional rollback. There is no silent void wrapper. +Zero resource omissions does not certify semantic completeness: ambiguous and +unsupported source constructs still remain unresolved. The application build +path continues to expose resource omissions through its existing degraded +resolution report and `universal_resolution_partial` graph diagnostic. +Rust callers that used the old return type, including function pointers, must +update. No graph JSON or serialized report schema changes solely for this API. + +## Rust struct-literal references + +Rust development producer4 emits a type-reference occurrence for the nominal +path of a struct or struct-like enum-variant literal. The shared resolver selects +source-backed struct, enum-member or type-alias declarations; repeated literals +retain separate source anchors and enclosing declaration owners. Direct generic +literals anchor their nominal name; qualified paths retain their exact source +slice. Concrete impl `Self` +paths use the proven receiver; nonconcrete trait `Self`, parser recovery and +ambiguous targets do not invent a declaration. Conflicting imports retain an +unresolved occurrence with an `ambiguous-struct-literal-type` diagnostic. The +file-wide module/package name bucket cannot supply a literal target from an +unrelated nested module. Nominal scope lookup is separate from value-parameter +aliases and stops at Rust module boundaries. Each Rust block has its own lexical +scope while occurrences keep their enclosing declaration owner. Block-local +imports and type aliases are visible throughout that block, including before +the item statement, and cannot supply targets to sibling or outer blocks. +Same-named block aliases retain distinct exact declaration identities. Impl +receiver proof uses the same lexical scope chain. Initializer blocks are +collected before their imports can reach an enclosing scope. Retained receiver +types keep their declaration/inference context and source position across +parameters, typed lets, aliases, fields and indexed elements; a use-site import +cannot rebind an earlier annotation. Generic method lookup first identifies +the actual parameter at that origin, stopping at nominal/import shadows. Each +trait bound retains its own declaration context, including where clauses and +impl parameters; a later import cannot rebind it. Container bounds are not +copied onto same-spelled impl parameters. Field inference maps actual receiver +arguments to exact defining parameter IDs, preserving each argument's origin +through nested and indexed fields. Mappings admit at most64 arguments/bindings +and16 substitution steps; unsupported token forms or incomplete arguments keep +the uninstantiated field origin, without guessing a caller's same-named type. Simple nominal receiver aliases follow +their defining contexts for at most16 steps; cycles, compound/generic aliases +requiring substitution and unproven bounds remain unresolved. Repeated +same-named Rust declarations use source spans to distinguish current graph IDs, +including on one line. Unproven nominal paths retain an +unresolved occurrence with an `unresolved-struct-literal-type` diagnostic; generic +impl receivers cannot inherit an unrelated outer type. Literal-local recovery +checks exclude overlapping syntax errors without extending to the line tail. +Literals create `references`, +not constructor `calls`, and do not add field read/write claims. + +Explicitly deferred receiver nodes, in every language, are excluded from legacy +terminal-name stub rewrites. An unresolved invocation retains its placeholder, +source occurrence and inferred/deferred provenance instead of becoming a call +to a source method whose name happens to be unique. Existing published graphs +are unchanged; update with the new binary to apply the correction. +An external Rust callee such as `actix_web::web::get` keeps its own inferred +identity and call-site anchor when a local `near_matches::get` also exists. +The exact-only topology view counts that source-less callee as isolated; the +synthetic fixture bounds account for that distinct node. A native regression +also preserves the genuine local call and reversed-input equivalence. + +AST cache31 rebuilds disposable extraction facts. Run `compass update ` +with the new binary; existing immutable graphs and historical producer versions +are unchanged. Rust4 remains `Qualifying` pending independent qualification; +the approved release promotion record is unchanged. Graph v1's shape, +source-query profile7 and query-ranker21 stay unchanged. Evidence/2 adds the +typed constraint described below. + +Rust generic receiver calls now carry a typed `rust_generic_bounds` hierarchy +constraint under evidence/2: the exact type-parameter declaration ID, sorted +unique trait identities (at most64, each at most1024 bytes under default evidence +limits), and a completeness flag. Type parameters explicitly use the type +namespace. The producer does not select a method from its per-file inventory. +The shared resolver checks all source-backed bounds and supertraits as an +unordered union. Only one unique method from a complete inventory resolves; +missing/non-trait bounds remain unresolved, and competing or duplicate methods +remain ambiguous. Successful edges carry `rust-generic-bound-dispatch` +provenance. Existing resolver lookup budgets apply to traits and members. +Rust trait declarations separately carry optional `memberInventoryComplete`. +Only explicit `true` certifies the direct member inventory; missing or false +metadata cannot prove absence. Unexpanded trait-body macros, unsupported +member syntax and unmodeled attributes on the trait or enclosing syntax leave +it incomplete. The finite supported documentation/lint attributes do not alter +members. Direct-base coverage remains a separate fact. Const/lifetime generics +remain shadow barriers and cannot witness a type-parameter constraint. +Attribute summaries are cached per AST node and reused across at most64 +ancestors per trait. Inventory checks share a200000-visit budget per file; +exhaustion returns a typed resource error, never a complete prefix. AST cache29 +also excludes prior development cache28 facts from current reuse. +Incremental fact-digest profile2 includes this metadata and declaration +namespace, automatically invalidating prior disposable digest state. +Readers must understand this additive evidence/2 variant; older strict readers +reject it explicitly and must upgrade. Graph v1's shape is unchanged, but its +current edges and provenance can change. Immutable historical evidence keeps +its original meaning. Rust4/cache29 identify this development producer change. + +## Literal-seeded code evidence + +Current updates publish `source-query/7-literal-seeded-code` with AST cache31. +A string literal can seed a code window even when its callable has no other +identifier occurrence. The producer considers the whole literal and at most +eight enclosing AST nodes, retaining only safe complete source slices of at +most512 UTF-8 bytes. Each candidate subtree is limited to128 nodes within the +shared200000-visit code budget. Nested callables/types and parser recovery +cannot lend text to an enclosing owner. Only unique published callable owners +are indexed; anonymous scopes may instead supply existing nested inspection. + +Pinned literal kinds are Python `string` (including raw/formatted/bytes source +syntax), Rust string/raw-string literals (including byte-string source syntax), Go interpreted/raw strings, and +TypeScript/TSX strings and templates. These are raw source bytes, with no escape +decoding, interpolation execution or runtime-value claim. Rust character +literals, Go runes and TSX text do not independently seed this channel; an +existing identifier-owned code window may still quote surrounding source. +Defaults stay with their own callable. Class fields outside callables are +excluded. A nested Python decorator expression can belong to its lexical +enclosing callable; it is not attributed to the decorated inner callable and +does not establish runtime execution or ownership. + +The existing16-window/owner,256 KiB/file and8 MiB/32768-window aggregate code +limits remain. Per-file producer selection follows AST preorder: literal seeds +can consume a slot before a later identifier-derived code window. Later +aggregate clipping is deterministic: collection accounts whole files, while +publication accounts owners with nested contexts considered afterward. +Identifier/comment and +parameter retention remain separate. Already covered literals consume no new +slot or omission; unsafe, oversized or exhausted new windows disclose code +omissions. Code-dependent candidates stay Low inspection leads. Partial-topic +leads must disclose missing context; text occurrence does not prove behavior. + +Profiles1–6 retain their original capabilities and document frequencies, +including code vocabulary in5–6 and parameter roles in6. Unknown profiles fail +explicitly. Update with the new binary for literal-only recall; cache31 refreshes +disposable extraction state, while immutable historical artifacts remain +unchanged. Ranker21 and ranked JSON4 remain current; the source-artifact digest +separates responses for changed evidence. No query-time source reads, graph +relationships, provider calls or mandatory dependencies are introduced. + +## Broad query topic roles + +`query-ranker/21-typed-topic-roles` distinguishes finite decision-predicate +spellings from independent topic evidence during broad lexical admission. It +adds no operation equivalence or synonyms. Ambiguous block/permit noun wording +can retain a recalled exact callable through its typed profile6 parameter site, +or an exact AST Class/Struct through its own declaration name. Both remain Low +inspection leads with source anchors and no implementation/compound preference. +A declaration pointer does not establish behavior or implicitly recover methods. +Explicit identities retain their structural contracts. + +Binding terms use the same literal camel/underscore subword projection and +canonical morphology as ordinary identifier fields; raw query occurrence order +and multiplicity remain visible in the ambiguity explanation. Defaults, type +annotations, nested parameters, arbitrary callable names, paths and prose cannot +supply a typed binding. Positive retained partial sites remain usable with an +incomplete-evidence diagnostic; missing or legacy roles cannot prove absence. +Only role-sensitive questions receive binding availability warnings. Roles add +no DF/posting field or graph facts, and queries read no current source. +The ranker epoch refreshes disposable response caches automatically. + + +## Parameter-binding source evidence + +Parameter bindings were introduced by `source-query/6-parameter-bindings` and +remain available in current profile7. Each exact +callable owner has a separate ordered `parameterBindings` list with whole source +identifiers and byte/line/column anchors. Coverage is `complete`, `partial` or +`unsupported`; `omittedSites` counts known omissions only, with finite reasons +for unsupported syntax/spelling, recovery and resource limits. Unknown suffixes have no +invented omission count. This metadata is not another retrieval or document- +frequency field and does not prove behavior or create graph nodes or edges. + +The supported spelling shape is a whole Unicode XID identifier, with TypeScript +`$` and Rust `r#` forms. Escaped JavaScript spellings are explicitly omitted. +`unicode-ident` supplies bounded native Unicode checks; it was already a locked +transitive dependency and adds no runtime or network boundary. + +Bindings come from direct native AST parameter fields in Python, Rust, Go and +TypeScript/TSX. Destructuring and Rust receivers are conservatively partial. +Defaults, annotations, result names and nested callable parameters cannot supply +bindings to an enclosing owner. Generic lexical evidence receives its existing +budgets first. Parameter evidence uses spare traversal/text capacity, at most +64 bindings per owner, 256 UTF-8 bytes per whole identifier and 256 KiB per file, +within the existing file and artifact aggregate limits. Parameter omissions do +not make identifier/comment coverage incomplete. `_` remains a binding in Python +and TypeScript/TSX; Go blank identifiers and Rust wildcard patterns are excluded. + +Fact-neutral incremental publication uses prior owners only when a conservative +bound rules out shared text/site exhaustion. Every unchanged evidence file, +including files without published owners, is charged its full 1 MiB native text +allowance for this check. Current profile7 also charges each unchanged file its +full native code allowance:8192 windows (512 scopes times16) and256 KiB, including +zero-owner and conservatively deleted files. Fresh inputs count retained code +from all raw scopes, including unpublished scopes. Equality at32768 windows or +8 MiB is allowed; a possible excess uses the full cached/fresh extraction path. +Prior direct/nested code omissions and fresh code omissions reject mixed reuse, +as do previously aggregate-clipped roles. Without reason-specific code omission +provenance, even local code omissions conservatively refresh through the full +path. This recovers evidence when deletions release capacity and preserves +clean/update equivalence. The same guard protects direct mixed publication +before source artifacts are replaced. +This may reduce fast-path reuse; it changes no source, memory or artifact limit +and makes no update-latency claim. Successful early source reads retain their +exact digest/size through fallback, even when the full stage loads their newly +cached extraction. Later fresh reads supersede those digests; publication still +rejects any source-evidence/inventory disagreement. + +Profiles1–5 keep their original payload and search/statistical meaning, including +profile5 code vocabulary. They supply no parameter-role provenance. Older +readers reject profile6 explicitly. AST cache31 refreshes disposable extraction +state on `compass update `; use `compass update --force` if current +publication reports an older retained extraction. Historical realizations stay +immutable. The developer collector accepts profiles1–7; freeze new actors and +collect a new run rather than relabeling historical evidence. + +## Owned code vocabulary + +Profiles5–7 support owned code vocabulary. A separate +`body_code` field indexes bounded direct-owner code windows, including string +literals. Words already present in the owner's identifier/comment fields are +not counted again. Nested callable windows remain inspection-only. Matches +depending on new code text remain Low, disclose their origin and do not receive +an implementation or compound preference from that dependency. Text occurrence +is not proof of behavior. Exact/qualified identities and graph facts are unchanged. + +Immutable profiles 1–4 keep their original identifier/comment-only retrieval and +document-frequency meaning. Older readers reject profile 5 explicitly. Run +`compass update ` with the new binary for current code vocabulary; no +historical realization is rewritten. AST cache31 refreshes disposable per-file +profile metadata. Channel-completeness behavior introduced by +`query-ranker/18-channel-completeness` remains supported by the current ranker +described above. +Code-only clipping remains visible without suppressing preferences supported by +complete identifier/comment fields. Code-dependent matches retain their Low +band and cannot supply those preferences. +Existing code-window and per-field byte/term quotas remain in force; omitted +code limits recall explicitly. Ranked JSON remains version4 with the additional +matched-field label. No current source is read at query time. + +Retrieval covers retained windows; omitted, unsafe or unpublishable scopes can +still have no supporting evidence. +The developer collector/assessor now accepts profiles1–7. Freeze new actor +hashes and collect a new run for the current source profile; historical captures +remain unchanged. + +## Graph coverage diagnostic projections + +New current-store snapshots add point projections for all graph diagnostic +codes, retaining the existing first-occurrence lookup contract. This uses the +existing `diagnostic-code` metadata namespace and graph-index/1 layout; canonical +graph records, relationships and historical realizations are unchanged. +Readers recover missing projections from older immutable diagnostic supplements +within the standard 1,000-record/1 MiB scan limits. A truncated scan without a +match returns a limit error rather than asserting that no diagnostic exists. +Local query readers preserve this as a resource-limit error. Existing response +caches receive a new diagnostic-projection identity automatically, preventing +reuse of answers that omitted documentation coverage warnings. No manual cache +deletion or historical rewrite is required. +Discovery responses with no matching seed retain graph coverage warnings too; +an empty answer does not imply complete indexing. + +## Developer comparison-panel binding + +The explicit `python3 -m benchmarks.agent_query.broad_query_v2` developer command +supports capture-input/plan/run/normalization/judgment/review protocol2. Its +`--capture`, `--score` and `--check` paths do not silently fall back to version1. +The older developer commands retain their version1 implementations. New actor +freezes bind the actual version2 entry, normalizer, file-backed assessor, panel +and reporter, plus all owned evaluator source files. No historical artifact is +converted or regraded; use its frozen verifier or collect a new experiment. + +Version2 panels bind a definition digest in the union source registration and +both source registration digests in `compass.broad-query-panels/2`. Original, +lookup and union statistics reuse one immutable capture/normalization lineage; +shared native query/build costs count once. All three panels must pass their +existing qualification gates. Missing panels, unknown full precision, source +review, power or graph qualification leave the report ineligible. Report2 binds +the actual review-input bytes and complete actors; checking recomputes the full +assessment. Reports are no-clobber siblings of their input in the external +evidence root. Native commands, graph/storage formats and ranking are unchanged. + +Developer graph entries may freeze an optional `sourceQueryArtifact` reference +for Compass's fixed `source-query.json` sibling. Collection retains graph and +sidecar together in a separate directory per repository, checks the native +envelope/digest, and rechecks original and retained inputs before completion. +Existing but undeclared source sidecars now reject new collection. Canonical +review binds the copied sidecar to the plan and rejects undeclared retained +siblings. Old graph-only captures retain their layout and reduced retrieval +meaning; correct omitted evidence through a new frozen experiment. This does +not change native artifact schemas or add source facts to graph qualification. + +New developer captures may freeze an optional tool `queryExitContract` with +schema `compass.broad-query-query-exits/1`. It recognizes one explicitly declared +nonzero no-match code only with an exact bounded stdout line plus LF. The run +retains and binds this contract to its source-frozen plan. Timeouts, capture +limits, budgets and other errors retain their meaning; no-match is not a correct +answer to an answerable question. Legacy plans keep all-nonzero-fail semantics. +Use new actor hashes and a new experiment; original results are immutable. +Native Compass commands, graph formats and query ranking are unchanged. + +Version-2 developer registrations may opt into the bounded +`compass.broad-query-development-calibration/1` profile for explicitly exposed +development recaptures. It verifies a historical registration/run pair, exact +question selection and prospective independent oracle review; it never qualifies +for superiority. Capture/review inputs bind exactly two `calibrationEvidence` +references. Panels preserve profile metadata and exact restricted selection +maps. Unprofiled author-blinding and all heldout gates retain their contracts. +New runs add an `assessor` artifact reference; calibration runs require it. +Canonical review verifies its frozen hash and retained bytes. Legacy runs +without the profile and reference remain readable. Both retained actor copies +are rechecked after scoring before capture completion. +Freeze current capture/assessor hashes for a new run; original reports and +outcomes remain immutable. Native commands and graphs are unchanged. + +Developer source registrations may opt into +`compass.broad-query-registration/2` and +`compass.broad-query-essential-fact-groups/1` for predeclared alternative +implementation witnesses in broad answerable tasks. Every member retains exact +owner/source/relationship checks. All groups and individually essential facts +must be satisfied. Version 1 keeps its original behavior and rejects the new +group field; older assessors reject version 2. Panel/union bindings preserve +versions and group payloads. Existing reports remain immutable with their frozen +assessors. This changes only the developer comparison protocol; native commands, +graphs and query ranking are unchanged. See the review protocol for bounds, +independent semantic review and prospective freeze requirements. + +New heldout developer captures require the additive +`compass.broad-query-exposure-lineage/1` source-registration field described in +`benchmarks/agent_query/BROAD_QUERY_REVIEW.md`. Every task declares its split and +stable mechanism IDs. Hash-verified historical evidence and independently +confirmed history completeness are required before capture; known repository +or linked-mechanism exposure rejects confirmation, even with new question/task +IDs or commits. Original, lookup and union panels bind the same lineage. +Legacy registrations remain shape-valid and diagnostically readable, but cannot +qualify without lineage. Explicit development captures retain their existing +source-review requirement. This changes developer analysis/capture protocol, +not native product behavior. Freeze current normalizer/assessor hashes for a new +experiment; historical reports and their verifiers remain immutable. + +The developer benchmark adds `compass.broad-query-panels/1`, +`compass.broad-query-panel-review-input/1` and +`compass.broad-query-panel-review/1`. Unknown versions fail explicitly. Both the +original behavior panel and its declaration-lookup correction must pass the +existing canonical review gates; shared captures and costs are bound once. +The native product, graph formats and canonical review-report schema are +unchanged. Freeze the current assessor digests before collection. Historical +reports retain their original verifier provenance; changed analysis code +requires new reports and the corresponding pre-output protocol freeze. + +## Nested callable inspection context + +Nested inspection was introduced by `source-query/4-nested-callable-context`; +current evidence also supports profiles5–7. Ranked JSON uses +`compass.query.ranked-view/4`. Readers retain source profiles1–6 and ranked +views1–3 with their original meanings; older readers reject new versions. +AST cache24 introduced the native scope metadata; current cache31 refreshes +disposable metadata automatically. + +An already retrieved callable can expose `nested_callable_code` quotations from +an anonymous callable with explicit native AST ancestry. Each quotation carries +its exact `nestedScope` anchor, separate from its retained source slice. Named +nested functions, separately published callable bindings, ambiguous owners and +class/type boundaries cannot lend code to an enclosing owner. Lexical nesting +does not establish a call, execution or the requested behavior. + +Direct-owner windows receive capacity first. Direct and nested code share the +existing 16-window per-owner, 32,768-window and 8 MiB publication bounds; at most +eight nested scopes are retained per owner. Inspection omissions are explicit. +Incomplete nested evidence is regenerated on a fact-neutral update rather than +reusing a truncated prior projection. Ranked results still retain at most two +512-byte quotations per candidate within the first-page budget. Quote selection +uses the whole question; it does not change lexical terms, document frequencies, +candidate admission or node scores through quotations, or graph identities and +relationships. The separate scope-census correction below can recover evidence +in quota-limited files. No source +file is read at query time. + +Code-quotation priority excludes terms inside retained overlapping comment +anchors; those terms still support their separately labelled comment quote. +Under tight text budgets the first candidate's quote receives priority over +lower-ranked quotes, with omissions disclosed. JSON/source bytes, candidate +scores and graph relationships retain their existing contracts. + +## Disposable query-index validation + +JSON query-index format 4 and its cache keys are unchanged. Validation first +checks the original file with the existing full `PRAGMA integrity_check`. +After a two-second file probe, eligible rollback-mode indexes between 16 MiB and +768 MiB can be inspected through a serialized image. Its complete bytes must +equal the physical original, including EOF. The image is never published or +used as the query connection. WAL headers, size mismatches and unavailable +images retain original-file validation; they do not imply corruption. + +Metadata and integrity work share a 30-second validation budget. Expiry returns +`query_index_validation_timeout` with kind `Timeout` and leaves the cache in +place. A joined, operation-local SQLite interrupt timer complements progress +callbacks, so a long integrity operation cannot defer expiry until the next +coarse callback interval. The timer is cancelled before any subsequent SQL; +thread-start failure uses finer progress checks. Blocking filesystem operations +must still return before SQLite can observe interruption. A timeout must not be +treated as a corrupt index or an empty answer. The budget +does not cover loading the graph or constructing a missing index. Historical +graphs, source-query profiles, graph relationships and query response schemas +are unchanged. No manual cache deletion or data migration is required. + +## Callable initializer definition ranges + +JavaScript3 and TypeScript4 also publish the full definition range of a direct +variable-bound function. The binding's original AST anchor continues to identify +the node and its provenance. The resolver requires a uniquely owned function +scope parented by the binding scope; inherited callback scopes cannot substitute. +Multiple qualifying scopes remain ambiguous. No relationship is synthesized. + +Source-query profile2 uses the exact direct variable-declarator envelope for +TypeScript/TSX lexical body ownership. Immutable profile1 artifacts remain +readable with their original evidence; unknown profiles fail explicitly. +AST cache24 rebuilds disposable facts. These producers remain `Qualifying`; +the existing release-promotion decision is unchanged. + +## ECMAScript primary member scopes + +JavaScript3 and TypeScript4 select a unique owned child whose parent is the +declaration's binding scope for member lookup. Nested blocks and callable-type +scopes that inherit the same owner cannot win by hashed ID. Ambiguous accessor +fixtures remain unresolved; declared fields, methods and lexical shadowing keep +source anchors and occurrence evidence. JavaScript class fields read their +grammar's `property` name anchor rather than requiring TypeScript's `name` field. +These development producers are +`Qualifying` pending a new source audit and promotion decision. + +AST cache24 invalidates disposable extraction entries and current build state. +Run `compass update ` to obtain corrected current references. The +extraction/5 identity seed, evidence/2 and graph/1 formats are unchanged; published +historical realizations are not rewritten. + +## Broad source-evidence retrieval + +Current store snapshots record search capabilities in a separate term-index +namespace. Source terms equal to older capability-marker strings remain ordinary +searchable postings, including their complete node IDs and chunks. Readers still +accept historical empty-posting markers; new writers retain those legacy markers +where no real source posting occupies their key. Older readers may report the +affected capability unavailable on a marker collision and should be upgraded. +Unknown or malformed explicit capability versions fail rather than falling back +to a legacy marker. The additive records keep graph-index/1 and the graph's +canonical records, identities and relationships unchanged. Historical snapshots +are not rewritten. + +Broad native implementation questions now use a shared bounded field projection +of names, paths, existing signatures/types, aliases, kinds, roles, language and +source-owned document text. Each field is bounded to 8,192 UTF-8 bytes and 512 terms. Alias +enrichment reads at most nine relationships; more than eight aliases omits the +alias field and reports bounded recall instead of choosing an arbitrary subset. +Fielded aliases require nonempty, exact, nonheuristic provenance; ineligible raw +relationships still count toward the bound. Legacy strict alias lookup retains +its existing meaning. + +JSON query-index format 4 uses a separate cache key from older indexes and +rebuilds disposable format-2/3 indexes automatically. +New immutable snapshots publish additive field-frequency and declaration-channel +postings; older snapshots remain readable with an explicit incomplete-recall +warning. Run `compass update` on a current checkout to publish those postings. +Historical realizations are never rewritten. Graph identities, relationship +kinds/direction/multiplicity and exact-symbol command meanings are unchanged. +Owned-code lexical retrieval introduced `query-ranker/17-owned-code-lexemes`. +The current ranker is described above; response cache identities include its +profile and do not reuse results from older rankers. + +Broad retrieval can give an inspection preference to hyphenated query compounds +whose two to four normalized words occur contiguously in a callable's own name +or a single retained owner-local identifier. It considers at most eight unique +compounds; larger compound requests retain ordinary ranking without selecting +an arbitrary subset. It never joins separate identifier sites, uses prose or +namespace/path mentions as compound proof, or changes recall postings/counts. +Separate owner-field topic evidence outside the compound is required. Explicit +identities, document requests, mismatched test contexts, recognized negative or +temporal conditions and vocabulary-dependent candidates receive no compound +preference. Clipped field projections keep ordinary ranking. Dependent matches +remain Low, below exact identity tiers, with explicit unverified-behavior reasons. +Actual matching identifier anchors receive priority within the existing eight +body-source citation cap; omissions retain boundedness. This changes neither +source/graph schemas nor quotations' ownership. The cache epoch is +`compound-identifier/2`. + +Ordinary broad-query words `entry` and `snapshot` have directional vocabulary +fallbacks to `item`/`element` and `copy`/`clone`, respectively. A group expands +only when known corpus statistics show no indexed literal variant and an indexed +alternative exists. These statistics describe the index, not a source-wide or +query-scope absence proof. Queries with named contexts or explicit identities, +document requests and unavailable legacy statistics do not use this fallback. +At most four alternative-frequency probes share the existing work limits. +Lexical-only candidates depending on an alternative require separate literal +non-action topic evidence in an owner field; facets, paths, namespace prefixes +and generic document links cannot supply it. Existing explicit semantic-channel +admission remains separate. Dependent matches are Low, name the translations, +use the existing `synonym` concept-match method and receive no operation +preference. Known literal groups and exact identities retain their meaning; +expansion cannot veto a literal-only candidate that passed its original +supported-concept screen. Graph/source schemas and immutable artifacts are +unchanged. The additional response-cache epoch is `unindexed-vocabulary/1`. + +Ordinary parameter, dictionary, seconds and nanoseconds questions also consider +the code spellings `param`, `dict`, `sec` and `nsec`/`nano`. An indexed long form +does not suppress these alternatives in another owner. This fixed table adds at +most five frequency reads to the four zero-frequency paraphrase reads above; +candidate reads and probes keep their existing hard ceilings. The same explicit +identity, document-intent, legacy-statistics and separate literal-topic guards +apply. Dependent candidates remain Low, name each translation and receive no +operation/compound preference. These are lexical inspection leads, not asserted +equivalences, units or behaviors. Existing literal-only admission is preserved; +the response-cache ranker profile changes without a graph/index/schema rebuild. + +Broad ordinary `read`, `lookup` and `getter` wording additionally considers +`get`; `setter` considers `set`. These four directional method-spelling hints +apply even if the original word is indexed elsewhere and add at most four +frequency reads under the same hard work ceilings. All existing vocabulary +guards apply: unavailable statistics, explicit/named identities and document +requests retain literal behavior. Lexical-only dependent candidates require +separate literal owner-topic evidence; existing explicit semantic-support +admission is unchanged. Dependent candidates remain Low, name the translation +and receive no operation or compound preference. Literal-only admission remains +protected. Within a candidate, groups with a literal match keep their original +matching fields; a namespace, path, signature or prose mention cannot silently +authorize a hinted local name as a literal operation. The additional response +epoch is `vocabulary-literal-fields/2`. Literal dependency attribution is evaluated +at each original relationship, owned-identifier and body enrichment stage; a +hinted name cannot hide a source-dependent Low relevance guard. A hint does not establish method +behavior, data flow or equivalence. Graph/source/index +schemas and historical artifacts are unchanged; ranker16 refreshes disposable +response caches without a graph rebuild. + +For broad implementation questions with one recognized operation concept, +callable-local operation identifiers and separate topic evidence receive an +ordinal inspection preference. An operation mentioned only in prose or a +retained signature does not supply it. Identifier postings include parameters, so terms +also present in the retained signature are excluded from this code signal; +missing/empty signatures, native display signatures ending in `…`, or clipped +search-field projections retain ordinary ranking for code-based signals. +This is not an AST body-role classification: an identifier absent from the +retained signature is still only a lexical inspection signal. A callable's own name +remains a direct name signal. Multiple-operation, negative and temporal questions, +document requests and mismatched test contexts retain ordinary ranking. This +preference changes neither relevance bands nor structural confidence and never +establishes execution. Exact IDs/full names retain their higher tiers. Graph +facts, source schemas and read limits are unchanged; the response-cache context +epoch also invalidates intermediate experimental answers. + +Broad method recall also probes at most eight relationship-term postings, +hydrates at most 64 candidate owners, and examines at most 32 outgoing calls or +references per owner. This channel charges its decoded records and probes to the +existing shared hard ceilings, with a separate 6,400-read/96-probe sub-budget. +It requires nonempty exact AST node/edge evidence and a relationship site inside +the callable's source range. Only the target's bounded name field contributes; +its qualified name, path and prose do not. Additional `code_reference` matched +fields retain source citations and actual relationship kinds/direction. A +candidate that needs this field to match a concept is Low unless its ID or full +name matches exactly: using an identifier does not prove the requested behavior. +Property leaves receive the same implementation-search penalty as fields and +variables. Older immutable snapshots without broad relationship postings retain +direct-call recall with a specific update warning. Reached quotas explicitly +report bounded recall. Ordinary plural acronyms such +as APIs and TTLs normalize as prose; quoted and qualified identities keep their +original identity guards. No graph schema, identity or published record changes. + +Broad implementation ranking recognizes colocated `.test.`/`.spec.` files, +conventional Python/Go/Rust test filenames and typed test/fixture roles. The +existing soft context preference remains reversible by an explicit test request; +it does not exclude candidates or change exact/directed lookup semantics. +Response caches use the new ranker profile automatically. + +Broad questions require subject evidence beyond `kind` and `language` when +other subjects are requested. Generic facets cannot stand in for missing topic +evidence. Queries whose non-action subjects all match those facets retain their +listing behavior; exact identity and explicit semantic fallback remain intact. +Unfamiliar wording can still yield tentative candidates that have other topic +evidence. The 40% supported-concept threshold and work ceilings are unchanged. + +Broad local-declaration recall examines up to 32 matching parameter, variable, +field, property or constant names and four incoming `contains` edges per +declaration. It requires one unique callable parent with exact AST node/edge +evidence and enclosed source ranges. Truncated parent lists cannot establish +uniqueness. Only bounded terminal-name terms contribute `owned_identifier`; +namespace/path matches and nested callable declarations do not. Answer scopes +apply to the parent after uniqueness is checked. This channel reserves 288 +decoded records and 32 probes within the existing shared hard ceilings; reached +limits remain explicit. Source citations and actual containment witnesses are +retained. A candidate needing this field is Low unless its ID or full name +matches exactly. Containment establishes ownership, not requested behavior. + +A witnessed documentation link contributes an owner's prose only when the +source range establishes that it is enclosed in that declaration, or when the +versioned leading-documentation AST proof retains both the exact excerpt and +full declaration ranges in the same provenance item. General links +can suggest a candidate only when the source-backed passage contains its actual +name and the question's subjects. They never establish ownership or structural +confidence. Missing subject +evidence yields no supporting indexed evidence, with scope/limit caveats. +Partial candidates name missing owner-specific subjects or named contexts and +carry Low relevance; corpus-wide word occurrence does not establish that a +candidate implements the requested behavior. Source-enclosed prose remains +available as supporting evidence rather than duplicate implementation rows. +Standalone `not`, `never` and `without` retain an unverified condition: lexical +retrieval returns Low inspection candidates with an explicit warning. Inspect +source or use a structural query to verify the condition. Optional native LSA +remains explicit and independently bounded; +it adds approximate candidates without establishing structural confidence. + +Query-index builders hold a persistent operating-system lock through validation, +replacement and publication. Termination releases ownership; the coordination +file remains and its existence alone does not indicate a live builder. A +five-second retry budget bounds lock contention, with a typed actionable +`query_index_lock_timeout`; it does not bound graph loading or index construction. +Invalid indexes are removed only under the lock, after rechecking whether another +builder already published a valid result. Format 4 isolates this protocol from +older binaries that used file presence as ownership. Graph/history formats and +query ranking are unaffected by this recovery change. + +## Source-body query evidence (development implementation) + +The native parser now retains bounded callable-local identifiers and inline +comments for Python, Rust, Go and TypeScript/TSX in a reserved per-file extraction +extension, `compass.source-query.extraction/1`, with +`source-query/3-owned-code-windows`. It preserves original UTF-8 source slices, +byte positions, a source digest and explicit omissions. Nested callables have +separate owners; class/type boundaries block inheritance. Arbitrary string +literal contents are excluded. Repeated occurrence omissions are counted +separately from missing term coverage. Work is bounded to a 4 MiB source file, +200,000 AST visits, depth 512, 512 scopes, 256 names per scope, one exact site +per spelling, eight comments per scope, 4 KiB per comment and 1 MiB retained text +per file. Omitted repeated occurrences increment the occurrence counter without +claiming missing term coverage. Code windows are collected independently at +identifier visits, including repeated spellings. Readers retain support for +historical projections containing up to four sites per spelling. +Unsupported or out-of-profile evidence is absent or explicitly omitted. + +AST cache24 rebuilds disposable extraction entries to retain this metadata. +The semantic extraction/5 seed, Fact IDs and dependency graph contract do not +change for this query-only metadata. A typed `compass.source-query/1` auxiliary +artifact retains bounded source excerpts and corpus-statistic validation. Native +updates now stage it with the graph, bind it to the exact graph/file digests and +one unique exact-AST callable range, and include it in atomic publication and +required build-state seals. Per-file inventory supports both fact-neutral +incremental paths; invalid or aggregate-truncated prior artifacts require full +reconstruction. Query-only metadata does not enter structural fact digests; +existing declaration source hashes still track body-comment edits. + +Current JSON and local-store readers share the bounded body projection for +postings, document-frequency increments and scoring. Body recall probes at most +eight terms and hydrates at most 64 owners within existing shared work ceilings. +The in-memory index admits at most 262,144 postings and 65,536 terms; exceeding +these limits is an explicit error. Auxiliary input is capped at 64 MiB, 32,768 +owners/files, 65,536 source sites and 16 MiB retained text. Reached extraction or +publication limits remain explicit. Body-only relevance remains Low inspection +evidence; source occurrence does not establish the requested behavior. Response +and long-lived engine cache identities include auxiliary content identity. +Present corrupt, mismatched or unknown-major artifacts fail explicitly. Absent +older artifacts remain readable with a missing-body-recall warning. Detached +in-memory graphs and historical selectors do not borrow current source metadata. +The reader reports unsupported auxiliary versions with `UnsupportedSchema` +and `unsupported_source_query_schema` / `unsupported_source_query_profile`, +before interpreting current payload fields. A retrieved owner must remain an +exact-AST callable. Neutral edits preserve declaration/provenance ranges even +when a declaration coincides with the old EOF. File nodes and source-proven +file-scoped language containers can refresh their envelopes. A container requires +one directly owned full-file scope, its file-root parent, and matching original +AST declaration provenance; ambiguous scopes use full publication. This restores +Kotlin package-envelope refresh without widening explicit namespaces or callable +bodies to include outside comments. Historical realizations remain immutable. +Integration checks are underway; +comparative quality and release qualification remain unproven. + +## Leading documentation evidence (development) + +Rust4, Go2 and TypeScript4 producers are `Qualifying` pending a new source audit +and release promotion decision. Their optional `DeclarationFact.documentation` +field under evidence/2 carries one exact leading AST comment group. Rust source +consumers constructing `DeclarationFact` literals must initialize `documentation` +(normally `None`); serde readers still accept an absent field. Older strict +evidence readers must admit this field; absent metadata remains readable. +Excerpts retain raw UTF-8 source bytes (including the Rust line-comment newline) +up to 4096 bytes, with 1 MiB per file and 128 MiB per resolver index. + +Publication/3 emits `Resource(rationale)` nodes and `documents` edges under +`universal-leading-documentation-v1`. The existing provenance-anchor vector +retains the excerpt and its unique full declaration range; graph/1 has no new +wire fields. A claimed rule with a missing or inconsistent ownership proof is +invalid. Document content must retain exact nonheuristic evidence with its AST +excerpt anchor; the owner must retain an exact AST anchor inside its full source +range. A precise edge cannot promote contradictory node evidence. Generic +documentation links retain their existing meaning. The query +ranker consumes the shared proof and preserves source citations; it does not +infer ownership from names. Excerpt/attachment limits and ambiguous definition +ranges publish a `leading_documentation_omission_summary` warning consumed by +both JSON and store query paths. Structural declarations remain available. + +Extraction/5 and AST cache24 invalidate prior disposable extraction facts. +Rebuild current graphs to obtain this evidence; immutable historical graphs are +not rewritten. Python docstrings and the JavaScript producer are unchanged. +Native regression checks do not establish comparative query quality or producer +qualification. + +## Ranked implementation view + +`query-ranker/29-why-question-scaffolding` treats the ordinary interrogative +`why` as broad-question scaffolding, including sentence-initial `Why` and `WHY`. +It cannot become a required missing symbol or dilute content coverage. Quoted +and qualified identifiers retain their literal guards, and exact queries for +symbols named `Why` remain available. Disposable response caches refresh +automatically; graph/index formats, immutable artifacts and relevance meaning +are unchanged. This correction does not establish causal explanations from +lexical matches. + +`compass query QUESTION --view ranked` adds an opt-in implementation shortlist. +Text shows up to five ranked owners and keeps the whole answer within +`--text-budget`, including coverage, omission guidance and a narrowing action. +The renderer first tries shorter combined explanations before dropping a ranked +owner. It labels abbreviated reasons, retains relevance/identity/evidence labels +and source citations, and keeps Low matches explicitly unverified. The bounded +JSON projection retains its original reasons. If metadata and one whole candidate cannot fit, it returns an explicit error. +`--view ranked --format json` emits `compass.query.ranked-view/4`; this is a +separate projection, not a change to raw discovery JSON or Agent View. +Ranked view rejects discovery cursors, result envelopes, evidence mode, +legacy traversal controls and `agent-json`. Existing modes remain available. +When the entire question is an existing node ID admitted as a discovery seed, +the ranked shortlist contains that seed alone. Raw discovery and optional +structural context retain their existing contracts; words inside an ID do not +make unrelated lexical matches additional ranked targets. + +MCP `query_graph` accepts `view: "ranked"` and an optional `text_budget` (1–65536, +default 800). Its existing `result` field retains `compass.query.discovery/1`; +the optional `rankedView` field carries the new projection. Strict envelope +decoders must admit that field when requesting this view. Default requests +retain their existing Agent View. Unknown ranked-view major versions fail +explicitly in the shared decoder. + +Ranked context is opt-in through `--include-context` / `include_context: true` +(default false). It appends up to eight witnessed one-hop relationships with +their endpoint records, direction, occurrence identity and source evidence. +Context has its own omission fields and never changes candidate order; text +drops whole context rows before candidates to satisfy the same answer budget. + +Relevance bands are ordinal retrieval judgments, not calibrated probabilities. +Unknown ordinary query wording can yield Low candidates with missing-context +reasons instead of suppressing every result. Explicit names (including quoted +and underscore identifiers) remain guarded when absent from the index. +Negative wording is retained as an unverified condition: candidates are Low +with an `incomplete_coverage` diagnostic and an explicit qualification warning. +It never establishes a negative behavior from lexical matches. Unquoted +`before`, `after`, `during`, and `until` likewise remain unverified temporal +conditions, with Low candidates and an explicit diagnostic. These connectives +do not count as lexical subject evidence; quoted and qualified identifiers +keep their literal terms. These retrieval semantics remain supported by the +current ranker described above. Prior profile constants remain available and +disposable response cache identities change automatically. +Name identity, structural evidence, graph coverage and candidate recall bounds +remain separate. `responseTruncated` retains discovery's aggregate bound, +including candidate/source-field/context limits; it does not assert that a +traversal itself was cut short. A zero-result answer reports no supporting +indexed evidence; it does not claim that the behavior is absent. Historical +realizations remain immutable. + Concept recall adds optional `conceptMatches` to `compass.query/1` with closed methods `synonym`, `document_link`, and `semantic_lsa`. Strict consumers must update to the matching enum/field manifest and fingerprint. Absence means no @@ -557,6 +1533,10 @@ Discovery text pagination now uses node inventories. Older discovery cursors fail with an explicit version error; restart the query to obtain a new cursor. Natural discovery text now defaults to an 800-token page; `--text-budget N` and `--cursor` retain caller control. +Compact `SEED` lines now include their retained source anchor, when available, +before relationship context can consume the page. Missing anchors remain +unlocated. This additive text annotation preserves the version 3 entry order, +semantic positions and cursor contract; JSON and evidence-mode text are unchanged. The typed commands accept `--format agent-json`; default text is an answer-first presentation. MCP keeps `compass.mcp.tool-result/1` and adds the @@ -1740,3 +2720,295 @@ Graph, Program IR and immutable historical schemas are unchanged. Disposable impact/traversal cache headers `TRAILA04` and `TRAILT08` retain the pinned build commit; older caches are ignored and rebuilt. Missing Program IR exits with code 3 and recovery commands; unknown Program subcommands exit with code 2 before artifact access. + + +## Ranked source quotation projection + +The opt-in ranked answer projection now emits `compass.query.ranked-view/4`. +Native readers retain version 1 compatibility with absent quotations; version 1 +payloads containing the new field and unknown major versions fail explicitly. +Existing raw discovery JSON and cursor contracts are unchanged. + +At most two 512-byte immutable quotations are retained for each of the first five +candidates. They come only from source-query comments or proven raw leading +documentation. Normalized docstrings and general links retain their existing +retrieval pointers. Text may shorten/omit quotations to preserve candidate rows +inside the whole-response budget. This changes presentation, not ranking or +structural confidence. The response cache has a new projection epoch while +query-ranker/10, source-query schemas/profiles and immutable graphs are unchanged. + + +## Bounded code inspection windows (development) + +Source-query profile `source-query/3-owned-code-windows` adds optional +`codeWindows` and `omittedCodeWindows` per callable under `compass.source-query/1`. +New readers retain immutable profiles1/2; legacy profiles cannot claim these new +fields. Older readers reject profile3 and must update. AST cache24 rebuilds +disposable current extraction. Graph identities, relationships, extraction/5, +ranking profile10 and retrieval terms/statistics are unchanged. + +Each window retains the exact UTF-8 bytes of a small AST ancestor, excluding +nested callable/type subtrees and parser recovery. Selection walks at most eight +ancestors and 128 subtree nodes per attempt, with200,000 additional AST visits per +file, at most16windows of512bytes per owner and256KiB per file. Code text has +independent publication caps of8MiB and32,768windows; it cannot consume the +16MiB lexical budget. Omitted windows are counted separately. These are inspection +windows, not complete function bodies or proof of the requested behavior. + +Ranked output now emits `compass.query.ranked-view/4` to add the typed +`callable_code` excerpt kind. The reader accepts strict versions1/2; version1 +cannot contain excerpts, and version2 cannot contain code excerpts. Update +consumers before reading version3. Query projection epoch `ranked-source-excerpts/3` +invalidates disposable responses. Output retains at most two512byte excerpts +for each of the first five candidates, subject to the existing whole-answer +budget. Current source files are never opened during quotation selection. + +Python auxiliary scope extraction now requires a named AST node: the unnamed +`lambda` keyword token is not another callable. This corrects scope counts and +can recover later source owners when the per-file scope quota was previously +spent on keyword tokens. The change can affect source-field coverage and +retrieval scores in quota-limited files; it does not change semantic graph +records. AST cache24 rebuilds the affected disposable evidence. + + +## Python default source-root identity (development) + +Python producer 2 is `Qualifying`. Project-aware extraction without configured +Python import roots now derives module keys from verified root layout instead +of incidental checkout names. Bare roots use relative module names; regular +root `__init__.py`/`__init__.pyi` entries retain a valid package prefix for root +and nested modules. Configured import roots retain their existing precedence +and ambiguity. This is source identity, not a runtime `sys.path` guarantee. + +The two fixed package-entry metadata probes do not follow links. Unknown layout +or an invalid package prefix fails explicitly with a recovery message when no +configured roots apply. Non-identifier script/group paths such as +`billing-service/test-helper.py` retain a deterministic relative source identity. +Literal dots and percent signs within components are escaped as `%2E`/`%25`, +so they cannot impersonate directory/module separators or each other. These +source keys are not runtime importability certificates. Invalid absolute, +traversing or non-normalized paths fail explicitly, without checkout fallback. +Standalone extraction without project context retains +the previous path fallback. Current affected IDs and edges can change on rebuild; +immutable historical graphs, evidence/2 and graph-v1 shapes are unchanged. +AST cache 33 and root-layout fingerprint component version2 prevent reuse +across changed package context. No release-promotion decision is updated. + + +## Bounded broad-query call context (development) + +Native broad automatic discovery can continue outgoing `Calls` beyond its +existing first-hop neighbors, within `maxDepth`. Automatic discovery continues +only from selected roots with `High` relevance under the existing whole-question +ranker. Lower-relevance roots retain their first-hop context instead of amplifying +a partial lexical match. Explicit `discover_ranked_with_context(..., true)` callers +may inspect all selected roots' witnessed paths. The ordinal relevance band is +not a calibrated probability or proof of requested behavior. +Continuation requires exact +source-owned AST evidence for the caller, target and call occurrence; references +and inferred/heuristic relationships cannot establish another hop. Inferred +incoming broad questions keep their first-hop context. Explicit structural +requests retain the neighborhood already computed by structural discovery, +including its existing work count; optional context does not traverse them a +second time. Every context path remains a source relationship, not proof of +runtime behavior. + +Candidate admission, relevance, ranking and seed counts are unchanged. Raw +discovery/1 and ranked-view/4 shapes remain unchanged; the optional ranked +presentation still projects only its documented first-hop eight-edge context. +The response-cache identity adds `ranked-call-context/2` to refresh disposable +answers. No published graph or historical realization is rewritten. + +Both initial and continuation adjacency reads count filtered records and bounded +lookahead against `maxExpandedRelationships`. A tight work budget can therefore +retain fewer edges than before; truncation remains explicit. Continuation is +also bounded to32 expanded states,16 outgoing records plus one charged lookahead +per state and64 queued states, +subject to the request's existing node/edge/scope/deadline/cancellation limits. +These are work contracts, not comparative performance qualification. + +Expanded-relationship accounting counts full edge records. Store index-key scans +and bucket-head comparisons have separate finite bounds and can exceed that +count; it is not a physical I/O or latency limit. + + +## Ranked quotation window selection (development) + +Retained source sites up to4096 bytes now select a window covering the most +distinct canonical requested terms, counting each term once. Token-aligned and +context starts are considered in deterministic source order; equal coverage +keeps the earlier window. This scores terms, not independent behavior facts or +probabilities. Synonym variants remain lexical alternatives. Complete matching +tokens must fit; a clipped unrelated identifier prefix cannot justify a quote. + +The source is normalized once for original token spans and scanned with sliding +term counters. Internal quotation priority retains complete original-site token +membership through both action- and general-group sorting. A displayed clipped +prefix/suffix cannot become a new match by re-tokenizing the displayed slice. +This private metadata is discarded before the unchanged public quotation shape. +Combining marks stay within raw token spans while matching uses shared Unicode +normalization; displayed bytes are never normalized. Output remains an exact UTF-8 slice of the retained site, at most +512 bytes, with adjusted byte/line/column anchors. Known owned comment ranges +cannot supply code-window selection evidence, although original surrounding +comment bytes may remain in the displayed quote. Existing comment masking in +cross-quotation ranking remains. No current worktree source is read at query time. + +Only display evidence selection changes. Candidate admission, ordering and +relevance, source ownership, graph facts, ranked-view/4 and ranker21 are unchanged. +Disposable responses advance ranked-source-excerpts/4. No source rebuild or +historical rewrite is required; no real-quality or latency qualification is +established by these source bounds or synthetic controls. + +## Broad forward-spelling inspection (development) + +`query-ranker/22-forward-spellings` adds at most two forward spelling hints per +canonical content concept: `r`/`st` after a final `e`, otherwise `er`/`est`. +Only lower-ASCII concepts of4..32bytes participate; at most32 concepts generate +64 statistical lookups. This is a bounded lexical hint, not an adjective +classifier or a complete English morphology rule. Unknown words are never +reverse-stripped, and global/historical token normalization is unchanged. +Already canonical concept keys remain distinct, including non-fixed-point +spellings such as `book` and `booking`. + +Only corpus-supported alternatives enter the existing vocabulary channel. +Explicit identifiers, named context, documentation intent and unknown statistics +retain their literal-only policy. Without optional semantics, a dependent hint +requires separate literal owner-topic evidence. An independently positive opt-in +LSA score retains the existing semantic admission exception. In either case, +hint-dependent candidates remain Low, disclose their mapping, and receive no +implementation or compound preference; lexical resemblance establishes no +comparison condition or behavior. Exact identity retains its higher tier. + +Graph/source ownership, ambiguity, scopes, historical profiles and public JSON +shapes are unchanged. Ranker22 refreshes disposable responses without a source +rebuild; graph omissions still require an update. Candidate record limits remain +in force. Statistical lookups are bounded separately and are not included in +candidate-record counters or a physical I/O/latency guarantee. + +## Distinctive retained quotation ordering (development) + +Disposable responses advance to `ranked-source-excerpts/5`. Across each owner's +retained quotations, complete original-token action coverage remains first. +The next key is the smallest known positive document frequency among matched +query groups, followed by total concept coverage and existing code/source +ordering. Statistics reuse existing bounded retrieval groups; no new corpus +lookup is performed. Unknown, zero and legacy frequencies receive no rarity +preference, preserving prior ordering when all statistics are unavailable. + +Known owned comments remain masked out of code priority. A clipped prefix +cannot invent a term; exact source bytes/coordinates, the two-quote cap and +512-byte quote limit remain unchanged. This is display salience, not relevance, +behavioral evidence or a confidence probability. Candidate admission/order, +graph facts, ranker22, ranked JSON4 and context epoch2 are unchanged. No source +rebuild or historical rewrite is required solely for quotation ordering. + +## Comment-passage score normalization (development) + +`query-ranker/23-comment-passages` uses the smallest unique canonical token +count of a retained owned comment matching a concept's existing variants as +that concept's `body_comment` field length. Other fields and the existing BM25F +formula are unchanged. Unrelated retained comments cannot dilute an unchanged +complete matching passage. This is a lexical ranking heuristic; separate +comments need not jointly establish the requested behavior. + +Only exact AST-owned comments from the validated source artifact participate. +Recovered owners, missing source evidence or comment fields exceeding their +existing8192-byte/512-term bounds use the previous aggregate length. Fields with +more than64 sites, sites at their4096-byte retention ceiling or more than32 +query groups also retain the previous normalizer. Unrelated code/identifier +clipping does not itself change a complete comment field's normalization. + +Admission, topic guards, vocabulary disclosure, relevance labels, explicit +identity tiers, source anchors, graph facts and ranked JSON4 remain unchanged. +The public ranker profile refreshes disposable responses while preserving +quotation epoch5 and context epoch2. No graph rebuild or historical rewrite is +required solely for this query heuristic; omitted source evidence still needs +an update. Native controls and development observations do not qualify general +precision, latency or superiority. + + +## Irregular operation spellings in broad questions + +Ordinary broad retrieval keeps its original query groups and adds bounded +same-operation alternatives for `rebuilt`/`rebuild` and `written`/`write`. +Explicit named/quoted/qualified contexts stay literal. This adds at most two +variants per query; it does not make build/rebuild or write/rewrite equivalent. +Action/subject classification, admission predicates, confidence bands and exact +identity tiers retain their existing rules. Sent/send already uses the unchanged +canonical token profile. Strict lookup, graph indexes and historical tokens are +unchanged. `query-ranker/24-irregular-operations` refreshes disposable responses; +no graph rebuild or user migration is required for these query-side spellings. + +Ranked structural candidates now retain discovery's typed `exact_name` or +`exact_id` match instead of losing that identity metadata at projection. Lexical +seeds do not gain identity tags. Seed selection, ambiguity, direction, relevance +and coverage are unchanged; exact identity does not establish semantic relevance +or source completeness. `ranked-identity-match/1` refreshes disposable responses. +The existing ranked-view4 enum/schema is unchanged. + + +## Metered helper navigation and retained previews + +Eligible broad questions asking about one or more operations can add Low-relevance +helper candidates through exact native outgoing call occurrences. This channel +requires separately supported topic evidence, preserves ambiguity and identity, +and does not infer requested behavior from a caller or a call path. Explicit +identities, guarded contexts and stronger answers retain their existing routing. +Navigation follows the requested depth, capped at four; the default remains two. + +Operation-head selection is a bounded lexical navigation heuristic over the +existing classified query groups, not an English parser or proof of intent. +Filtering a noun-like modifier from the operation hints never supplies new root +topic credit. A single operation requires one source-witnessed target; multiple +operations still require two distinct helpers. Single-operation navigation keeps +at most eight best exact target alternatives per root and displays at most two; +multiple paths to one target retain one best path. Current-root and best-root +maps can coexist, so eight is not a global allocation/RSS bound. Diversity, +shortest path and stable occurrence IDs determine alternative order. More +same-operation targets do not by themselves improve root rank. Full-question +matching and Low/Approximate confidence remain. + +Metered responses use `compass.query.discovery/2` with +`stats.callLinkWork.schema = "compass.query.call-link-work/1"`. The new +`candidateProbes`/`candidateProbeLimit` pair counts additional helper-candidate +directional Calls API lookups, including empty/filtered/failed attempts, with a +shared ceiling of40 across roots. Failed requests still return their typed error. +The original `stats.candidateProbes` remains indexed recall only, capped at291; +`indexedCandidateProbes`/`indexedCandidateProbeLimit` mirrors that counter and +ceiling. These are two logical work ledgers, potentially331 probes combined, +not physical I/O, CPU, memory or confidence measurements. Work survives root-only +presentation caps and attempts that find no usable helper. Unmetered discovery +retains major1 without this field. Consumers must reject unknown/contradictory +profiles and verify the mirrored indexed count before accepting a response. + +The opt-in metered projection uses `compass.query.ranked-view/7`, even when no +helper survives; unmetered ranked output remains major4. `callLink` uses +`compass.query.call-link/1` and preserves root, ordered native occurrence IDs, +endpoints, selected source sites, attribution and the disclosed spelling hint. +It is approximate/Low navigation evidence. Prior unpublished experimental +linked majors5/6 are unsupported. Metered retrieval traces use major2; otherwise +trace major1 remains. `query-ranker/28-single-operation-navigation` and disposable +response epoch `ranked-call-link/7-classified-operation-heads` separate the +retrieval policy. Historical graphs/source profiles and graph identities do not +change solely because of this reader/query work. + +When a displayed helper has no query-matching quotation, it may retain one +`linked_callable_preview` excerpt from its immutable direct-owner code windows. +This kind requires ranked-view7 and a coherent helper path. It is rejected on +unlinked/legacy versions. Selection considers at most16 windows of512bytes and +64 owned comment ranges, preferring more non-comment tokens and source order +on ties. This display heuristic does not prove a body role or semantic importance. +Only the selected source/text is copied; no current source file is read. Preview +terms never affect candidate admission, rank, matched terms or relevance. Existing +matching quotations take priority. The first five candidates still retain at +most two512byte quotations each, inside the existing completed-answer budget. +Text labels previews as behavior unverified. Disposable quotation responses +advance to `ranked-source-excerpts/6-linked-preview`. + +CLI and MCP share the typed work and ranked contracts. Default text includes +labeled lookup counts; ranked JSON retains structured paths/work, and raw JSON +retains typed stats. The generic Agent View major1 keeps its existing shape and +raw identity plus an informational caveat; canonical work counters remain in the +raw response. Operational counts remain outside semantic result digests, while +the raw major distinguishes the profile. These contracts do not establish +independent query/graph superiority or a performance qualification. diff --git a/Cargo.lock b/Cargo.lock index 759468c80..3becc9af4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1425,6 +1425,7 @@ dependencies = [ "tree-sitter-html", "tree-sitter-md", "unicode-casefold", + "unicode-ident", "unicode-normalization", "unicode-properties", "url", @@ -1487,6 +1488,7 @@ dependencies = [ "sha2 0.10.9", "tempfile", "thiserror 2.0.20", + "unicode-ident", "unicode-normalization", ] @@ -5312,9 +5314,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.45" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "log", "once_cell", diff --git a/Cargo.toml b/Cargo.toml index c609451da..b0f7b6415 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -95,7 +95,7 @@ flate2 = "1.1.9" glob = "0.3.3" url = "2.5.8" roxmltree = "0.21.1" -rustls = { version = "0.23.42", default-features = false, features = ["ring", "std", "tls12"] } +rustls = { version = "0.23.46", default-features = false, features = ["ring", "std", "tls12"] } rustls-platform-verifier = "0.6.2" rustc-hash = "1.1.0" serde_yaml_ng = "0.10.0" @@ -103,6 +103,7 @@ scip = "=0.9.0" thiserror = "2" toml = { version = "1.1.3", features = ["preserve_order"] } unicode-normalization = "0.1" +unicode-ident = "1.0.24" unicode-casefold = "0.2" unicode-properties = { version = "0.1.4", features = ["general-category"] } tempfile = "3" diff --git a/MIGRATION.md b/MIGRATION.md index d0de600fd..0ccbb71fc 100644 --- a/MIGRATION.md +++ b/MIGRATION.md @@ -1,10 +1,341 @@ # Migrate from Graphify to Compass +Run `compass update ` with the new binary to rebuild current graphs under +AST cache34. Inadmissible explicit exact source targets now remain unresolved; +the resolver cannot substitute an unrelated same-named +function for an injected parameter. Historical realizations retain their +original meaning. No graph-schema or evidence-schema upgrade is required for +this correction. Missing target IDs still fail evidence validation. + +Broad-query consumers must accept and validate raw `compass.query.discovery/2` +and ranked `compass.query.ranked-view/7` when metered helper navigation is used. +Unmetered responses retain discovery1/ranked4. Upgrade strict readers before +enabling the new binary; do not strip unknown work fields or treat a limit as an +empty answer. The `callLinkWork` record separates indexed recall (291 probes) +from additional helper lookups (40); original `stats.candidateProbes` keeps its +indexed-only meaning. Work is retained even when presentation shows only a root. + +The new ranked excerpt kind `linked_callable_preview` is an explicitly unverified +retained helper-code preview, requiring a coherent call path and Low relevance. +It is not matching-query evidence or proof of behavior. The default depth stays +two; request `--max-depth 4` to inspect up to four helper hops when available. +Use `--view ranked` for paths and previews. No graph rebuild is required solely +for this query change; missing source/call evidence still needs `compass update`. +Single-operation questions now use the same metered contracts when eligible. +Their operation-name hints and alternatives are inspection aids, not verified +intent or behavioral answers. No new schema upgrade is required beyond those +above; disposable ranker responses refresh automatically. +Historical realizations and native graph/source ownership remain unchanged. +See COMPATIBILITY.md for exact bounds. + +Current builds use `source-query/8-callable-comments` and AST cache 34. Run +`compass update ` with the new binary to retain up to 64 callable comments +instead of eight. Upgrade readers for profile 8; profiles 1–7 retain their original +meaning and limits. Per-field byte/term and per-file/artifact limits still apply, +so large callables can remain partial. Comments are inspection pointers rather +than behavioral proof. Subsequent cache 31 notes describe earlier development +changes; cache 34 is the current disposable extraction identity. + +Python producer 2 also corrects source-root module identity when no import roots +are configured. Bare roots use repository-relative modules, independent of the +checkout directory name. A regular root `__init__.py` or `__init__.pyi` retains +the package namespace, including nested modules. Existing configured import +roots keep precedence and competing roots remain ambiguous. Graph IDs and edges +can change after rebuilding affected current graphs; historical realizations +remain immutable. The new producer remains `Qualifying`, pending promotion. + +Run `compass update ` to rebuild current evidence. If extraction reports +unavailable source-root package layout, fix linked/non-regular/unreadable package +entries or declare import roots in `pyproject.toml`; package directory names must +be valid Python identifiers. Default source identities retain non-identifier +script/group names such as `billing-service/test-helper.py`; they do not claim +runtime importability. Literal dots and percent signs inside components use +`%2E` and `%25`, preserving the distinction from actual directory/module +separators. Invalid absolute/traversing/non-normalized source paths still fail +explicitly. The layout participates in disposable cache +identity, including package-entry additions/removals. Standalone extraction +without project context retains its legacy path-based behavior. + +The new binary also extends raw broad-query discovery context through exact +outgoing calls within the requested depth, automatically from High-relevance +selected roots. Partial lexical roots keep first-hop context. Explicit native +context inspection can inspect every selected root's witnessed paths; this +changes neither ranking nor relevance. Source graph rebuilds are needed only +when the required relationships are missing; querying cannot invent them. +Callees remain context and keep the lexical shortlist unchanged. Ranked-view/4's +optional presentation retains its first-hop context contract. Adjacency work +limits now include filtered records and lookahead, so very small work budgets +can return fewer edges with an explicit truncation diagnostic. Disposable +responses refresh automatically; raw schemas and historical graphs are unchanged. + +The development export-proof repair adds `ecmascriptExportInventory` to root +TypeScript/JavaScript file/module declarations under evidence/2. Upgrade strict +readers for its explicit version1/complete contract; missing metadata stays +unknown. Unresolved explicit exports must retain their names and cannot fall +through to a star target. Run `compass update ` with the new binary: +AST cache31 and fact-digest profile4 rebuild disposable current state without +rewriting historical realizations. These certificates do not promote producers +or qualify comparative quality. + +The in-progress TypeScript evidence patch adds optional +`typescriptMergeDependencies` under evidence/2. Strict readers must upgrade. +The vector distinguishes module, reference-path and reference-types loads and +retains exact source ranges. Missing dependency coverage prevents merge-absence +proof, while unsupported load syntax leaves the batch uncertified. Disposable +fact-digest profile4 includes the census and its explicit version1 capability. +TypeScript5 / JavaScript4 and AST cache31 refresh current extraction identities. +Run `compass update ` with the new binary. Missing census-version metadata +remains legacy/unknown even if an old complete boolean is true. Unknown present +versions are errors; current readers must not infer capability from producer +version alone. + +Owning project integrations can represent a precise load resolution with the +optional `typescript_source_load` edge attribute (`TypeScriptSourceLoadProof`, +version1). It contains the exact typed census dependency and source range; +target source remains on the existing edge inventory/`target_file` interface. +Upgrade readers before consuming these edges: reference-types evidence cannot +be downgraded to an ordinary module import. Generic module mappings do not +certify type-package directives or distinguish repeated load occurrences. +Absent owning reference-types evidence remains unsupported; no new compiler or +automatic type-package loader is required or claimed. + + +Development evidence consumers must allow `memberInventoryComplete` on +TypeScript class/interface facts as well as Rust traits, and understand the +optional batch `typescriptMergeInventoryComplete` certificate. Missing/false +certificates cannot establish member or augmentation absence. TypeScript +`Awaited` terminal identity requires explicit bounded nominal/project proof; +unsupported thenable or merged types can remain unresolved. Update strict +readers/validators before consuming these facts. Fact-digest profile3 refreshes +disposable digest state automatically and includes merge certificates and exact +typed proof trees and the explicit census capability version. Producer/cache +integration is TypeScript5 / JavaScript4 / AST cache31; this is not a promotion +decision and does not rewrite historical realizations. + +Rust integrations calling `UniversalResolutionIndex::materialize` must now +consume its `UniversalMaterializationReport` return value. Check +`omitted_candidates` before publishing the mutated node/edge buffers: a nonzero +count means resource limits prevented relationships from being projected and +the output must be labeled degraded. The operation does not roll back those +buffers. Update callbacks/function pointers previously expecting `()` as well. +The normal Compass build path already propagates this state through its graph +diagnostics. A zero count does not establish that all source semantics resolved. + +Rust development producer4 records nominal type references from struct literals, +including concrete impl `Self` and struct-like enum variants. Run +`compass update ` with the new binary to rebuild disposable AST cache31. +Literal contacts are references, not constructor calls or field read/write +claims. Ambiguous/unproven targets remain unresolved. Historical graphs are +immutable, and Rust4 remains `Qualifying`; no release promotion is implied. +Repeated same-named Rust declarations now use source spans to distinguish their +current graph IDs. Resolve affected node IDs again after updating; historical +realizations retain their original IDs. Block imports cannot escape their +lexical scope or rebind parameter annotations used for receiver inference. +Explicitly deferred receivers across languages now remain unresolved rather than +being rewired by a unique terminal method name. Rebuild current graphs to remove +those guessed targets; historical graphs retain their original contents. +Generic Rust receiver calls now defer their complete trait-bound set to the +project resolver. Ambiguous local/imported bounds no longer produce a guessed +source call. Update evidence consumers to understand the additive evidence/2 +`rust_generic_bounds` constraint before reading newly extracted batches; older +strict consumers reject it. Current generic type-parameter fact identities +include the type namespace. Resolve affected identities again after updating. +New Rust traits carry `memberInventoryComplete` separately from base coverage. +Readers must preserve it; missing/false metadata cannot justify an empty member +set. Trait-body macros and unsupported attributes keep generic dispatch +unresolved. Current fact-digest profile4 refreshes automatically during update. + +Current updates use `source-query/7-literal-seeded-code`. Whole raw string +literals can now seed bounded callable-owned code windows without another +identifier. Parameter bindings remain separate from generic occurrences. Run +`compass update ` with the new binary; AST cache31 refreshes automatically. +If publication reports older retained extraction, run +`compass update --force`. Older binaries reject profile7 and must be +upgraded. Readers preserve profiles1–6 with their original lexical/statistical +meaning, including profiles5–6 code vocabulary and profile6 typed bindings. +Parameter roles are never synthesized for profiles1–5. +Historical realizations are not rewritten. Raw literal text is not a decoded +runtime value. Shared per-file producer capacity follows AST preorder and may +displace later code windows; deterministic aggregate clipping follows afterward; +code omissions remain visible without altering original lexical/role coverage. +Clipped code evidence and possible aggregate code-capacity transitions now use +the complete cached/fresh extraction path during incremental update. This may +reduce fast-path reuse; it does not rewrite historical artifacts or increase limits. + +Direct-owner code vocabulary remains searchable, with code-dependent matches +kept Low and source-cited. Parameter metadata adds no retrieval weight or graph +facts. Parameter-only omissions do not alter original lexical completeness. +`query-ranker/21-typed-topic-roles` refreshes disposable response caches for +broad noun/predicate admission. Run update for typed bindings when a role-sensitive +question reports missing evidence; historical snapshots retain their limited +meaning. Broad block/permit matches through typed parameters or exact class/struct +names remain Low inspection pointers. Nominal pointers do not inherit into +methods or establish behavior. Exact identities retain their contracts. + +The operation-vocabulary rollout introduced ordinary method translations. +Ordinary read/lookup/getter and setter wording can inspect conventional get/set +methods with explicit translations, separate literal context and Low relevance. +Literal-hit groups retain their original field and source-enrichment provenance; +hinted names cannot hide source-dependent Low relevance or silently promote an +operation through namespace/prose mentions. The disposable repair epoch advances +automatically; this needs no manual cache deletion or graph rebuild. +Ordinary parameter/dictionary/time-unit words can recall tentative abbreviated +code candidates with explicit translations and separate literal context. No +manual cache deletion, graph rebuild or source/index schema migration is needed +for this query-only change. Exact-symbol commands retain their contracts. + +Current updates rebuild AST cache31 and retain callable-body evidence for direct +TypeScript/TSX variable initializers. Their source ranges now cover the complete +definition; original binding identity and AST provenance stay distinct. Older +`source-query/1-ast-callable-lexemes` artifacts remain readable and historical +realizations are unchanged. New artifacts use profile7; older binaries must be +upgraded to read that projection. JavaScript4 and TypeScript5 remain `Qualifying` +pending source audit and release promotion. + Compass uses its own executable, output directory, environment variable, and sidecars. Its output root now preserves the familiar flat artifact shape so file-based workflows can transition while Compass's snapshot and store layout remains visible and clearly owned. +## Developer comparison registrations + +When recapturing native Compass graphs, declare any existing `source-query.json` +sibling through the graph entry's hash-bound `sourceQueryArtifact` reference. +Freeze new actor hashes before collection; the adapter now preserves each +[coherent graph pair](benchmarks/agent_query/BROAD_QUERY_CAPTURE.md#coherent-compass-graph-inputs). +Original graph-only captures remain immutable and cannot retrospectively claim +implementation-body recall. + +For a new developer capture whose pinned native CLI intentionally uses a +nonzero no-match exit, predeclare the optional typed +[query exit contract](benchmarks/agent_query/BROAD_QUERY_CAPTURE.md#native-query-exit-outcomes). +Verify its meaning from the pinned tool, freeze new capture/assessor hashes and +collect a new experiment. Existing reports retain their original verifier and +all-nonzero-fail interpretation; do not edit or silently regrade them. + +For recapturing already exposed development questions, use the explicit +version-2 [development calibration profile](benchmarks/agent_query/BROAD_QUERY_REVIEW.md#exposed-development-calibration). +Disclose author exposure, retain the linked original registration/run, freeze +the selected question mapping, and obtain new independent source review bound +to the exact prospective definitions. Supply the two `calibrationEvidence` +references and freeze new actor hashes. This is a new ineligible diagnostic run, +not a migration or regrading of historical reports. Heldout captures continue +to require source-first output-blind authoring and untouched-source lineage. + +For a new broad-question experiment requiring equivalent implementation +witnesses, opt into `compass.broad-query-registration/2` and predeclare reviewed +`essentialFactGroups` using the [review protocol](benchmarks/agent_query/BROAD_QUERY_REVIEW.md#equivalent-implementation-witnesses). +Keep separately anchored facts for each alternative; preserve flat essential +facts and exact/directed controls. Freeze new source/assessor/capture bindings +before collection. Version-1 registrations retain their original semantics and +must not receive the new field. Preserve old captures, reports and required +outcomes; this is not a post-output scoring migration or a native-product change. + +Before a new heldout comparison, freeze an independently reviewed exposure +history, explicit task splits/mechanism IDs and hash-bound historical evidence +using the [review protocol](benchmarks/agent_query/BROAD_QUERY_REVIEW.md). +Previously used repositories remain development inputs even when their commit, +task IDs or wording change. Legacy diagnostic registrations remain readable; +they cannot establish an untouched confirmation split. Freeze current capture +and assessor hashes; preserve existing reports with their original verifier. +Native Compass commands and graphs require no migration for this protocol. + +## Opt into ranked implementation answers + +Upgrade ranked JSON consumers to `compass.query.ranked-view/4` before using new +nested code quotations. `nested_callable_code` includes an exact `nestedScope` +anchor; it means lexical inspection context, not execution. Consumers must not +render it as direct-owner code or a graph call. New builds use source profile4 +and AST cache24; run `compass update ` to refresh disposable evidence. +Readers retain old source/ranked versions with their original meanings. + + +JSON query-index validation can now return `query_index_validation_timeout`. +Treat it as a startup timeout and retry or narrow the build scope. The cache is +preserved; no format migration or manual cache removal is required. + +Current updates retain one callable-local identifier site per spelling, counting +omitted repeats separately. AST cache24 refreshes disposable evidence +automatically; no manual cache removal is needed. Historical source artifacts +retain their original sites. Code windows still inspect repeated identifiers +independently of the lexical site budget. + +Use `compass query "where are request errors formatted?" --view ranked` for a +top-five shortlist. Existing query formats and cursors require no migration. +Request `--format json` with this view only when the consumer understands +`compass.query.ranked-view/4`; it has its own typed fields and rejects unknown +major versions. The shortlist has no discovery cursor: narrow with its suggested +source scope to examine other implementations. Do not combine it with +`--cursor`, `--result-envelope`, `--evidence`, `--format agent-json`, or legacy +traversal controls. + +Optional `--include-context` appends bounded one-hop witnessed relationships +after the shortlist; it is disabled by default. + +Run `compass update ` to publish fielded signature/prose/alias postings in +an existing current graph. Older immutable stores remain readable with a +limited-recall warning; historical realizations are not rewritten. Disposable +JSON query indexes rebuild automatically to format 4, isolating their new +operating-system lock from older presence-lock caches. Terminated builders no +longer require manual lock-file removal. Retained lock files alone do not mean a +builder is active. Broad questions with +negative wording may return Low inspection candidates with a warning that the +condition remains unverified. Inspect their source or use a structural query to +verify it. Unquoted `before`, `after`, `during`, and `until` are also reported as +unverified conditions rather than required lexical subjects. Quoted names and +qualified identifiers remain literal. Unfamiliar ordinary words reduce relevance and appear as missing +context; absent explicit names still guard retrieval. A Low partial candidate +does not establish the requested behavior. Disposable response caches use the +new `query-ranker/14-compound-identifier` profile automatically. Hyphenated +compound inspection matches remain Low with explicit reasons; code-based +preferences require a nonempty retained signature without native `…` clipping. +Exact identities and graph facts retain their meaning. No manual cache deletion +or graph rebuild is needed for this ranking-only change. + +Run `compass update ` to obtain the optional immutable `source-query.json` +artifact for callable-body identifiers and comments on Python, Rust, Go and +TypeScript/TSX. AST cache24 rebuilds disposable extraction entries automatically. +Body-dependent matches carry exact source locations and Low inspection relevance. +Old graphs keep their existing evidence with an explicit body-recall warning; +historical realizations are not rewritten. Keep the auxiliary artifact with its +matching graph snapshot. A present corrupt/mismatched artifact fails explicitly; +`compass update --force` reconstructs current source evidence. + +MCP clients opt in with `view: "ranked"` (and `include_context: true` for context). Read its projection from `rankedView` +and the raw discovery shape from `result`. Allow the new optional envelope field +and keep relevance separate from exact identity and structural confidence. + +Broad implementation questions can also recall methods through verified local +call/reference sites. A `code_reference` match is an inspection lead, with a +source citation and Low relevance when it supplies otherwise missing context. +It does not establish the requested behavior. No graph rebuild is required for +this query-only channel when existing relationship postings are available. + +## ECMAScript primary member scopes + +Run `compass update ` to rebuild JavaScript/TypeScript current graphs with +structurally selected member scopes. AST cache24 discards older disposable +entries automatically. JavaScript4 and TypeScript5 remain development producers +marked `Qualifying`; their updated identities do not alter the approved historical +promotion record. Evidence/2, graph/1 and historical realizations retain their +existing contracts. + +## Leading documentation evidence (development) + +Rust integrations constructing `DeclarationFact` values must initialize the new +optional `documentation` field (normally `None`). Serialized absent fields remain +readable. + +Run `compass update ` with the matching new binary to rebuild current +Rust/Go/TypeScript graphs with leading AST documentation. Strict readers of +`compass.languages.evidence/2` must allow the optional declaration +`documentation` object. Graph/1 retains its existing resource, relationship and +provenance fields; no historical realization is modified. Disposable extraction +facts rebuild under extraction/5 and AST cache24. The changed producer versions +remain `Qualifying` until a source audit and new promotion decision are complete. +Treat ranked results as source inspection candidates, not a verified behavioral +answer or a claim of superiority over another tool. + ## Scoped views and freshness No rebuild is required for new scoped views or hotspot ranking. Explicitly @@ -671,3 +1002,29 @@ Discovery text cursors now use version 3 because compact pages show relationship before declarations. Restart a query if a saved version 2 cursor is rejected. Natural discovery pages default to 800 tokens. Increase `--text-budget` or follow the printed `--cursor` to retrieve more of the same bounded answer. + + +## Consume ranked source quotations + +Clients of the opt-in ranked JSON projection must accept +`compass.query.ranked-view/4` and its bounded `excerpts` per candidate. The native +reader still accepts immutable version 1 results without quotations; unknown +major versions fail explicitly. No graph/history migration is required. Older +graphs can lack quotation evidence and retain their original retrieval scope. +Quoted comments/docs support source inspection, not verified behavioral answers. + + +## Ranked code excerpts (development) + +Consumers of opt-in ranked JSON must accept `compass.query.ranked-view/4` +and the `callable_code` excerpt kind. Compass still reads immutable ranked views1/2 +and source-query profiles1/2. Run `compass update .` with the updated binary to +retain code windows for current source; old historical artifacts remain unchanged. +Code windows do not change candidate ranking or create graph relationships. + +Python auxiliary scope extraction now requires a named AST node: the unnamed +`lambda` keyword token is not another callable. This corrects scope counts and +can recover later source owners when the per-file scope quota was previously +spent on keyword tokens. The change can affect source-field coverage and +retrieval scores in quota-limited files; it does not change semantic graph +records. AST cache24 rebuilds the affected disposable evidence. diff --git a/PERFORMANCE.md b/PERFORMANCE.md index b85cf8425..294f0a7c1 100644 --- a/PERFORMANCE.md +++ b/PERFORMANCE.md @@ -1,5 +1,116 @@ # Compass performance qualification +## In-progress comment-passage normalization bounds + +Query-local comment lengths inspect only already retained exact-owner data, +with no extra backend, posting or statistics reads. The optional normalizer +accepts at most64 comment sites,32 groups,8192 total bytes and512 unique terms; +sites at4096 bytes or any field overflow retain the prior normalizer. Each +comment is tokenized once and cancellation is checked per site. These bounds +are not a measured latency/RSS or independently qualified query-quality claim. + +## In-progress forward-spelling bounds + +At most32 canonical broad-query concepts generate two forward spellings each, +with at most64 additional bounded statistics lookups. Only supported spellings +enter existing posting and candidate quotas. These lookups are separate from +candidate-record counters; this is not a measured latency, precision or RSS +claim. Global source/index formats and their bounds are unchanged. + +## In-progress quotation window bounds + +Window selection tokenizes each already retained site of at most4096 bytes +once, then visits at most two candidate starts per matching token using sliding +term counters. Complete matching tokens count once per distinct canonical query +term; known owned comment ranges are excluded from code-window matching. +Cross-quotation salience reuses at most32 existing query groups with no added +statistics reads. Known positive group frequencies rank distinctive passages +after action coverage; unknown/zero frequencies supply no rarity preference. +Output remains at most512 bytes/quote and two quotes/displayed owner. These +finite source bounds are not a measured latency/RSS or query-quality claim; +fresh real-repository qualification is still required. + +## In-progress broad call-context bounds + +Raw broad automatic discovery retains first-hop neighbors, then may continue +exact outgoing Calls within requested max_depth for non-incoming questions. +Automatic continuation is restricted to High-relevance selected roots. Explicit +context inspection can continue every selected root; candidate order and bands +are unchanged. This avoids automatically amplifying partial lexical roots. +Continuation expands at most32 callable states, decodes at most16 outgoing +records plus one lookahead per state, and retains at most64 queued states. +It performs at most32 owner fetches and512 target fetches in that phase; +node/edge response limits can stop it earlier. BFS/DFS retain bounded best-depth +state so cycles cannot grow depth, while a shorter path may revisit a state. + +Raw examined records, including heuristic/rejected records and lookahead, +consume the same max_expanded_relationships allowance as first-hop context. +This meter counts full edge records; bounded per-kind index-key scans and +bucket-head comparisons can exceed it. It is not a physical I/O ceiling. +Scope checks, cancellation and deadlines remain active. Candidate recall keeps +its separate existing indexed-probe/read ceilings; continuation cannot promote +context nodes into candidates or improve their relevance labels. Ranker21 and +ranked-view4 are unchanged; a context epoch invalidates disposable responses. + +These limits and synthetic controls do not establish real-corpus latency/RSS, +query-quality gains, non-regression or a win over another tool. The original +fresh frozen independent graph/query and cost/performance qualification remains +required. Earlier one-hop measurements do not qualify this continuation. + + +## In-progress typed export traversal bounds + +Typed TypeScript receiver analysis shares its16384 work units with export +traversal. Every binding/presence/index probe, raw candidate and frontier +retention spends work. Each walk discovers at most the configured per-lookup +candidate count (default256) of distinct(language,source-file,export) states; +export depth is bounded at64. Duplicate DAG/cycle states do not expand again, +but reaching an existing state still spends the edge/probe work. Bounded +fixed-point passes and slot propagation also spend this meter before each +retention; per-state results are checked before any owning alternative is +discarded. An unfinished +walk returns an explicit resource decision before target selection. Complete +per-binding source alternatives are prepared during index construction; queries +consume them without scanning the entire project module map. + +These are development resource contracts. Focused synthetic boundary controls +and prior native baselines do not establish current real-corpus latency/RSS or +comparative performance; the original fresh qualification remains required. + +## Literal-seeded evidence bounds + +Profile7 considers a literal's whole subtree and at most eight ancestors. +Each scan considers at most128 nodes within the existing200000 code visits per +file. Windows remain at most512 bytes,16 per owner,256 KiB per file and +8 MiB/32768 windows per artifact. Covered seeds do not spend a second slot. +Within each file, producer AST preorder can displace a later identifier-derived +code window. Aggregate collection accounts whole files, and publication accounts +owners with nested contexts afterward. Original identifier/comment/parameter +retention has separate accounting. + +Native boundary tests establish limits, not latency or RSS. No performance +non-regression or comparative speed is claimed without a new frozen real-corpus +run. Earlier profile6 and older measurements do not qualify profile7. + + +## Parameter-evidence update paths + +Source-query profile7 preserves the existing per-file and aggregate text/site +caps introduced before profile6. A fact-neutral mixed update can reuse prior +evidence only when bounded +accounting rules out shared text/site pruning. Each unchanged evidence file is +charged its full 1 MiB native text allowance in this conservative eligibility +check, including zero-owner files. A possible limit transition, or previously +aggregate-clipped roles, uses the existing complete cached/fresh extraction +path. Current profile7 additionally charges each unchanged file8192 possible +code windows and256 KiB, counts actual fresh code from every raw scope, and +rejects prior/fresh code omissions. More than four unchanged evidence files +therefore exceed the conservative code-site allowance even if actual code is +small. Unchanged cache entries remain reusable on the full path. This can forgo +the narrow fact-neutral fast path; no latency improvement or non-regression is +claimed for this guard. Prior measurements do not qualify the new path without a fresh +source-frozen performance run. + Compass performance is measured against Compass-owned baselines. Qualification must never trade away graph correctness, deterministic output, resource bounds, or complete error reporting. @@ -26,6 +137,44 @@ review and evidence explaining the tradeoff. ## Natural-language answer cost qualification +JSON query-index validation uses a two-second original-file probe and a shared +30-second validation budget. Only eligible rollback-mode indexes from 16 MiB to +768 MiB may use a byte-verified in-memory inspection image after that probe. +Warm file checks normally avoid the image allocation. The image can temporarily +add up to 768 MiB plus SQLite working memory; eligibility is not a memory-use or +end-to-end latency qualification. Graph loading and missing-index construction +are outside this validation-stage deadline. + +An operation-local interrupt timer complements SQLite progress callbacks during +metadata and integrity SQL. It is cancelled and joined before the next operation; +if the thread cannot start, progress checks use an interval of one. This avoids +waiting for a coarse callback interval in long integrity walks. Interruption is +cooperative inside SQLite and cannot cancel a blocked filesystem call. No memory +mapping hint is added to validation. + +The 2026-10-17 bundled-SQLite stage diagnostics in Plan 029 used an exposed +537,133,056-byte Compass index. Two file checks on fresh APFS clone inodes hit a +60-second diagnostic deadline; two image checks completed in 3.883/3.518 seconds. +On the already warm inode, file checks took 0.643/0.647 seconds and image checks +0.773/0.751 seconds, with peak RSS about 5.8 MB and 545.7 MB respectively. These are +small stage diagnostics affected by hashing, shared extents and prior cache +state. They motivated the adaptive candidate; they are neither certified cold +measurements nor a production performance baseline or Graphify comparison. +Native end-to-end and platform qualification of the candidate remain required. + +Fielded broad retrieval retains the shared hard ceilings of 12,801 indexed +candidate records and 291 candidate probes, including reserved routing work. +Its search-stage counters cover posting, documentation-ownership and alias +reads; bounded intent-routing checks are reserved separately. Optional local +LSA has its own limits (32,768 nodes and 64 MiB of corpus data), uses the same +absolute query deadline, and is not included in those indexed-read counters. +Compare complete process wall time and peak RSS when measuring its cost. + +The new broad-query qualification workflow is described in +[`BROAD_QUERY_REVIEW.md`](benchmarks/agent_query/BROAD_QUERY_REVIEW.md). +Development traces and native regression tests establish individual fixes; +they do not establish a current-release superiority or performance claim. + The developer-side [agent query harness](benchmarks/agent_query/README.md) qualifies answer recall and output cost separately from runtime performance. Run `suite_natural.toml` on its five pinned repositories with the same question @@ -1554,3 +1703,31 @@ final verdicts. contains source and artifact pins, suite and binary hashes, the registration and amendment, every oracle observation and every cold/warm timing. These focused query measurements do not replace release-wide performance qualification. + +## Nested callable quotation bounds + +Source profile4 adds lexical context to already retrieved owners without adding +lexical postings, document-frequency contributions or node-score work. Native +preorder parent indexes allow a linear bounded ownership pass. Direct windows +receive publication capacity first; nested code uses the same 16-window owner, +32,768-window and 8 MiB totals, with at most eight nested scopes per owner. The +ranked shortlist keeps two512-byte quotes per candidate. Incomplete nested +context disables reuse of that auxiliary projection on a fact-neutral update; +it is regenerated through the existing build path. This can increase update +cost and must be included in end-to-end qualification. No latency, memory or +Graphify superiority claim follows from these limits. + +Python auxiliary scope extraction now requires a named AST node: the unnamed +`lambda` keyword token is not another callable. This corrects scope counts and +can recover later source owners when the per-file scope quota was previously +spent on keyword tokens. The change can affect source-field coverage and +retrieval scores in quota-limited files; it does not change semantic graph +records. AST cache24 rebuilds the affected disposable evidence. + + +The in-progress ECMAScript export census checks top-level syntax and retained +root bindings on the shared200000-step source-work budget. Its optional root +certificate has fixed size; no new export-name vector is cloned or retained. +Typed resolution requires a current complete census per visited source, and +aggregate fallback discards certificates when it discards binding evidence. +This is a resource contract, not measured real-repository cost or quality. diff --git a/SECURITY.md b/SECURITY.md index 1f0ce2529..e235fa64b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -90,6 +90,20 @@ of one size-bounded byte snapshot. Digest equality establishes content identity, not authenticity or semantic correctness. Disposable cache decoding is also byte-bounded; caches do not authorize access to their source graph. +Disposable JSON query-index metadata and full-integrity inspection have a shared +validation deadline. A slow file pager may use an inspection-only image bounded +to 768 MiB, pinned by a read transaction and compared against every original byte +and EOF. WAL-context images and physically different images are ineligible and +fall back to original-file inspection. The copy uses the same full integrity +operation on a writable in-memory handle required by FTS5; it grants no graph +trust and is never published. A validation timeout preserves the original cache +and returns a typed error rather than starting a replacement build. +SQL deadline timers use the safe SQLite interrupt handle, are scoped to one +operation, and are cancelled and joined before the connection is reused or +closed. A progress callback remains as a fallback, with finer checks if the +timer thread cannot start. Blocking filesystem calls may delay observation of +the deadline; the image retains its byte and size guards. + ## Document and OCR boundary PDF and OOXML files, XML relationships, compressed members, embedded images, @@ -138,6 +152,19 @@ include those excerpts as well as existing names and provenance. Apply the same repository disclosure policy to these graph artifacts. Corpus size/deadline failures remain explicit and do not publish a partially built semantic index. +## Local source-body query evidence + +Native extraction caches and the immutable `source-query.json` artifact can +retain parameter/local identifiers and bounded inline comments from indexed +source. Treat these as repository-derived source content when sharing output +folders. They remain local; no provider, credentials or source-file reads are +added to query execution. Current queries read only the bounded regular-file +artifact beside the selected graph, validate its schema/profile and digest +binding, and retain exact source coordinates. Hashes detect content mismatch; +they do not authenticate source authorship or establish behavioral correctness. +Detached/historical graph readers do not borrow current body evidence. Text +occurrence never creates graph dependency edges or structural confidence. + ## Local query and saved-output caches Completed-output budgets can save source excerpts, report content and diagnostics @@ -172,3 +199,61 @@ The marked root can be outside the current project; its path appears in the advisory recovery command and MCP freshness metadata. Shell quoting is display only. Source-root markers are local build metadata, not a trust assertion that arbitrary imported graph bytes describe that checkout. + + +Ranked answer quotations can disclose retained comment or documentation bytes +through local CLI/MCP output. They read only immutable, graph-bound evidence +with proven declaration ownership; query-time mutable source reads and generic +neighbor prose are excluded. Quotations remain untrusted data, with controls +and bidirectional formatting marks escaped in text output. Treat their contents +as source material, never instructions. Normalized legacy docstrings do not +qualify as verbatim source quotations. + + +## Local code inspection excerpts (development) + +Opt-in ranked answers may include exact code windows from the immutable +source-query artifact, including literals present in that code. Profiles5 and6 +also makes direct-owner code vocabulary searchable; nested context remains +inspection-only. These bytes are not +redacted summaries. Windows stay within one AST-owned callable, exclude nested +callable/type subtrees, and have independent extraction/publication/output caps. +The query never opens current working-tree files to supply missing text. Treat +these excerpts as untrusted source content when rendering; existing terminal +control/bidirectional escaping and JSON encoding apply. A source window is +inspection evidence and must not be interpreted as an instruction or a proven +runtime relationship. + +## Nested lexical source context + +Nested code quotations require an explicit bounded native AST parent chain and +a unique exact published enclosing callable. They retain the actual callable +syntax range, separately from variable definition envelopes. Class/type scopes, +named children and separately published callables are ownership boundaries. +Quotes remain untrusted source text; renderers preserve existing escaping and +output budgets. Context does not add graph records, calls, execution claims, +retrieval terms or network/model dependencies. Source profile4 and ranked view4 +make the distinction explicit; historical profiles retain their original meaning. + + +## Parameter-binding provenance + +Source-query profile6 additionally retains bounded whole parameter identifiers +with exact source anchors and separate coverage. Direct AST binding fields +establish the role during extraction; the artifact reader validates shape, +limits, owner containment and graph/source digest binding. Those checks cannot +independently prove syntax without the original source. A serialized role label +is not authentication or proof of runtime behavior. The new sites share the +existing local disclosure and untrusted-content boundaries, add no source reads +at query time and create no graph relationships. + + +## Literal-seeded source retention + +Source-query profile7 can retain a whole raw literal from callable syntax even +without an identifier seed. It uses the existing bounded local artifact and +quoted-output boundary. Raw text may include sensitive source content, just as +existing code windows do. It is not decoded, interpolated or executed. Only +safe AST-contained slices join unique graph owners; missing or unsafe evidence +is reported rather than invented. Query readers do not re-open current source +files, and this change introduces no credential or network boundary. diff --git a/benchmarks/agent_query/BROAD_QUERY_CAPTURE.md b/benchmarks/agent_query/BROAD_QUERY_CAPTURE.md new file mode 100644 index 000000000..d9e280458 --- /dev/null +++ b/benchmarks/agent_query/BROAD_QUERY_CAPTURE.md @@ -0,0 +1,448 @@ +# Frozen native query capture + +The explicit version2 developer entry point is +`python3 -m benchmarks.agent_query.broad_query_v2`. It connects collection, +normalization, printed-coverage/source scoring, immutable panel assessment, +report publication and recomputation. The older `broad_query_capture`, +`broad_query_review` and `broad_query_panels` commands remain protocol1-only; +neither path converts or silently mixes versions. A new comparison still needs +independent source/oracle acceptance, exposure review, prospective power and +the latest comparator verification before its pre-output freeze. Software +self-tests and source integrity do not supply those judgments. + +```bash +python3 -m benchmarks.agent_query.broad_query_v2 --self-test +python3 -m benchmarks.agent_query.broad_query_v2 --capture /external/capture-input.json --output /external/new-capture +python3 -m benchmarks.agent_query.broad_query_v2 --score /external/new-capture/review-input.json --output /external/new-capture/report-reviewed.json +python3 -m benchmarks.agent_query.broad_query_v2 --check /external/new-capture/report-reviewed.json +``` + +Capture/diagnostic scoring exits0 after coherent publication. Check exits0 only +when every union/original/lookup qualification gate passes,1 for a recomputed +ineligible report and2 for invalid/stale/mixed evidence. Reports must be new +siblings of their input; paths are never overwritten. CLI status includes a +bounded failure preview and total failure count; the complete ledger is in the +report. Native Compass runtime, commands and graph formats are unchanged. + +`broad_query_actors_v2.py` freezes all non-test Python sources under +`benchmarks/agent_query` and `benchmarks/performance/compass`, including imported +helpers and graph auditors. Its protocol descriptor binds the complete ordered +inventory and the actual version2 capture-entry, normalizer, assessor, panel and +reporter roles. Added, removed +or changed source files require a new pre-output freeze. Source discovery is +bounded by 16,384 entries, 512 directories and 512 Python files; retained code +is bounded by 64 MiB. Original and retained bytes are rechecked using the shared +bounded artifact reader and existing no-clobber bundle. Runtime callers must use +their own code root, never a root supplied by captured metadata. These integrity +components do not authenticate the execution or an independent source review. + +`broad_query_lineage_v2.py` stages finite, immutable review selections. It binds +every original task/tool/arm to exactly one `initialNormalizedRef` and one +`reviewedNormalizedRef`, plus the original run, raw stdout, source registration +and frozen actor digests. The initial reference must equal the original run's +reference, including its path. Initial normalization precedes run publication +and therefore never contains a circular run digest. A reviewed normalization +may correct suggestion IDs, owners and grouping without rewriting that run. +Transitive amendments, missing or reordered rows and mixed versions are rejected. + +The protocol2 `judgments` index has exactly `schema`, `runSha256` and +`observations`. Each ordered observation has exactly `task`, `tool`, `arm` and +`judgmentRef`. That reference selects a separate bounded judgments2 document +containing the same task/tool/arm, `runSha256`, +`reviewedNormalizedSha256` and the full reviewed claim/occurrence labels. +The existing 64 MiB JSON limit applies to each document; the shared 2 GiB and +65,536-artifact bundle limits still apply to the entire experiment. Embedding +all row labels in the global index is rejected. + +The loader reads raw streams, normalizations and judgments one observation at a +time. Consumers must exhaust its single traversal, discard each large body +after scoring, and call its final `recheck()` before publication. A late row +failure, partial traversal or final integrity failure prevents publication. +Rejected rows retain raw bytes and unknown assertion counts, with no ranked or +coverage prefix. This component checks lineage and partitions; source truth, +native costs, reviewer independence and qualification need the full assessor. +The lineage loader itself supplies no qualification or command dispatch. + +`broad_query_execution_v2.py` checks each retained command against the frozen +question, tool, graph, arm, isolated environment and measurement wrapper. It +recomputes byte costs and peak RSS from the original streams. The v2 shared +RSS parser checks decimal length before integer conversion and applies the +positive range up to 10^12 bytes after GNU KiB scaling. Missing, duplicate, +zero, malformed and out-of-range measurements stay unavailable. Its line +whitespace cannot consume another measurement line. Legacy v1 RSS parsing is +unchanged. Native failures, exact declared no-match +outcomes and byte-budget overruns retain their original costs without clipping. +`broad_query_observation_v2.py` connects that check to initial/reviewed coverage +and task scoring using one observation budget and the shared bundle counters. +Row limits and typed assertion collisions keep an explicit failure with unknown +full precision, while source or artifact integrity failures abort assessment. +Initial aggregate occurrence bounds are checked before allocating pending +labels. A nonrejected initial partition is checked even when the reviewed +normalization is rejected. Initial and reviewed failures remain separate: +reviewed material can supply diagnostics, but cannot erase an original parser +or limit rejection or promote that observation to success/formal precision. +Native capture truncation likewise leaves full precision, assertion count and +false-empty status unknown, even if an independent reviewer labels every +retained-prefix assertion. Known prefix judgments stay diagnostic; byte/time +costs and the failed observation remain present. +These layers require the full assessor to supply verified graph/source inputs +and do not authenticate execution, reviewer attestations or qualification. + +The internal `broad_query_assess_v2.assess_file_v2` connects the entire matrix to +the existing source/graph/tool and qualification gates. Its own runtime source +root supplies the actor check. It rechecks the actual review-input file and +original/retained inputs before returning. Current Git HEAD and cleanliness +are checked before and after; bounded temporary Git captures are created +alongside the external evidence directory and removed. Git optional locks, +fsmonitor and untracked caching are disabled for these checks. This verifies current +source state and supplied build records, not authentic historical execution. + +Capture2 runs must retain `captureInput`, a reference to the original validated +capture-input2 document. Its plan, original registration and source roots must +match the retained run. Each tool additionally retains an ordered `dependencies` +list of `{original, artifact}` records matching every file in its frozen +environment inventory. `original` is an absolute hash-bound input reference; +`artifact` is its contained retained copy. Original executables, launchers, +dependencies, release evidence, graph inputs, build streams and history metadata +are checked before and after assessment. Original inputs share a 65,536-file / +2 GiB bound; dependencies additionally share the existing 512 MiB bound. +Unique input/artifact count excesses reject before any payload read. Reads use +the remaining aggregate byte allowance; an oversized read may inspect one +sentinel byte but never hashes an excess payload. Known files are bounded by +their verified original length. Dependencies are bounded before original reads, +then retained copies are bounded by the verified original length/digest. +Release-reference presence as well as its digest must match the pre-output plan. + +The shared bounded subprocess runner treats process-group permission errors as +cleanup failures unless the child has been reaped and a separate group probe +confirms the group is absent. A live or inaccessible group cannot produce a +successful capture through that recovery path. + +Protocol2 summaries retain unknown precision at the aggregate, repository and +language levels. Unknown false-empty values are not averaged away. Complete +paired inputs reuse the existing repository/cluster weights, bootstrap and +finite-risk bounds exactly. Any unknown required matrix precision, including +in the default arm, makes formal precision comparison unavailable; no row is +dropped, imputed to zero or replaced with its diagnostic partial fraction. +The existing source review, exposure, task balance, prospective power, +independent graph qualification and subgroup gates remain required. This +assessor is invoked by the explicit version2 entry; the version1 path stays separate. + +`broad_query_collect_v2.collect_file_v2` internally connects collection to that +assessor. It preflights the frozen source review, complete actor inventory, +executables/dependencies, graph/build evidence, isolated environment and command +matrix before querying. +Truthfully exposed development calibration retains the existing explicit +exception to author output-blindness only after calibration history, source +definition and independent output-blind reviewer checks. It stays permanently +ineligible for superiority; non-calibration author blindness remains required. +It executes every task/tool/arm once in canonical order, +with no retries or free follow-ups. Raw outputs and costs remain present for +native failures, exact declared no-match outcomes, parser/line/JSON rejections, +missing RSS and byte-budget overflow. Initial normalizations and each separate +judgment document remain explicitly pending independent review. The provisional +report must retain the whole matrix and cannot establish superiority. + +The collector retains the original invocation file's exact bytes in +`captureInputInvocation: {original, artifact}`, separately from the canonical +validated `captureInput` snapshot. Both the collector and assessor bind that +file to the validated object and recheck its original path and retained copy. +The internal object-only `collect_v2` helper is diagnostic: missing invocation +file custody is an explicit qualification failure. + +Every retained file, including raw streams and provisional reports, counts +toward the shared artifact/byte bounds and is rechecked before completion. A +subprocess reserves capacity for both maximum-sized bounded streams plus one +maximum-sized JSON sidecar before launch (currently 96 MiB); this conservative +reservation may reject a near-capacity run before executing it. Actual retained +bytes, rather than the reservation, are charged to final storage. Source +HEAD/cleanliness and original/retained actors and inputs are checked before +queries, after collection and after provisional assessment. Mandatory integrity, +report or publication failures leave the directory without `COMPLETE.json`. +That marker confirms coherent pending-review capture only. This remains an +integration supplies coherent capture only; helper functions do not authorize +an experiment or establish independent source/answer qualification. + +`broad_query_panels_v2` computes original, lookup and union comparisons over +one immutable union capture. The union source registration freezes +`comparisonPanels: {schema: "compass.broad-query-panels-binding/2", +definitionSha256: ...}`. Capture-input2 additionally declares `panels` with a +definition reference and exactly `original`/`lookup` registration references. +Original references are absolute; the run retains contained copies. The +definition has schema `compass.broad-query-panels/2`, `frozenAt`, the complete +`actorSha256`, `registrations` mapping those two roles to SHA256 digests, and +`lookupPairs` with original/lookup task IDs. Digest-only role bindings avoid a +circular union-registration hash. Both panel registrations precede the +definition, which precedes the union source freeze and actual collection. + +All original broad tasks remain unchanged. Only named exact controls gain +canonical declaration lookup variants; source identities, clusters and shared +tasks must match exactly. The union contains the complete task union, with +the same sources, capture plan, cost arms, graph scope and policy. Each panel +keeps its own source-frozen cohorts/power and all existing qualification gates. +Combined eligibility requires original, lookup and union to pass. Missing +source-frozen panels makes version2 assessment ineligible. No run, initial +normalization, reviewed selection or raw stream is projected or rewritten. +Observations are scored once; shared query/build costs are charged once in the +union summary. Metadata/graph verification is sequential across panels, with +graph/source indexes released between them. + +`broad_query_report_v2` internally publishes and recomputes the version2 report +wrapper. Its exact fields are `schema: "compass.broad-query-review/2"`, absolute +`inputPath`, `inputSha256`, complete `actorSha256`, and `assessment`. The +collector's provisional report uses this wrapper too. New reports are siblings +of their review input in the external artifact root. Encoding is bounded +incrementally, then original/retained source, input and actor guards run before +exclusive publication. Rechecking recomputes the complete matrix and panel +assessments; stored metrics, numeric types, eligibility, actors and original +input/report bytes cannot be substituted. Pending or underpowered reports +remain diagnostic. Public version2 dispatch uses only the explicit entry above. +Historical +protocol1 runs/reports retain their original schemas and arithmetic. + +The staged components retain all printed occurrences in globally bounded +batches, exact byte-line coverage and typed source assertions. Initial +normalization always remains pending independent review. Rejection emits zero +normalized records with unknown actual assertion count; it never establishes +empty output. Incomplete semantic coverage or required judgments leaves full +precision unavailable. Diagnostic partial fractions cannot establish paired +precision noninferiority. These rules do not authenticate source truth or +reviewer independence. Historical protocol1 parsing, its 128-claim cap and its +scoring arithmetic remain unchanged; do not convert or regrade historical +captures. Freeze new actors for a new experiment when integration is complete. + +`broad_query_capture.py` is developer evaluation tooling. It reuses +`runner.run_bounded`, executes explicit preinstalled tools against explicit +prebuilt graphs, and never builds, installs, downloads, or changes source. +The collection adapter supports Darwin/Linux through the existing POSIX bounded +subprocess backend and rejects other hosts explicitly. Its self-test uses local +fake runners and small public-format graph fixtures: + +```bash +python3 -m benchmarks.agent_query.broad_query_capture --self-test +python3 -m benchmarks.agent_query.broad_query_capture \ + --capture /external/capture-input.json --output /external/new-capture-directory +python3 -m benchmarks.agent_query.broad_query_review \ + --check /external/new-capture-directory/report-provisional.json +``` + +No actual held-out captures belong in this checkout. The last command must +reject a provisional capture: completed collection is distinct from independent +source/output review, sufficient power, graph qualification, and a passing +comparative result. `COMPLETE.json` means the full matrix was captured and +provisional reporting succeeded; it does not mean the quality gates passed. +Failed runs leave an incomplete directory without that marker and preserve the +raw evidence. Output directories and files are never overwritten. + +Compact Compass `SEED` lines count as displayed declaration citations only when +they print a source location and sufficient owner identity that resolves uniquely +in that graph. Unlocated seeds and graph-only owners do not become candidates. +The seed's exact/approximate label makes no behavioral claim. A seed followed by +the same `NODE` citation counts once; both printed spans remain in the review +queue. Repeated citations of the identical resolved identity do not make a +relationship endpoint ambiguous; conflicting or unresolved citations still do. +This is display order, not an inferred retrieval rank. A changed normalizer needs +a new frozen experiment or an explicitly separate diagnostic analysis; retained +captures and primary reports are never silently replaced. + +## Freeze before execution + +The input has schema `compass.broad-query-capture-input/1`, `registration`, +`sourceRoots` (repository ID to absolute source root), and `plan`. External +input references are `{path: absolutePath, sha256: exactByteDigest}`. Every +registered source file, binary, dependency file, and graph is bounded and +verified before execution and rechecked afterward. Sources must have the +registered Git HEAD and a clean status before and after. Artifacts must live +outside both the product checkout and every source checkout. +Heldout registrations additionally require the frozen +[source/mechanism exposure lineage](BROAD_QUERY_REVIEW.md#source-and-mechanism-exposure-lineage) +and an exact `exposureEvidence` digest-to-external-reference mapping. The adapter +rejects missing history review, evidence drift and known confirmation reuse +before invoking any subprocess. Captured review inputs retain the verified +history bytes under contained references; original history inputs are rechecked +before completion. This verifies supplied records, not undisclosed exposure. +Executable symlink invocation paths are preserved, including Python virtual +environment prefixes. Hashing follows their current target, and post-capture +verification reopens the declared path so retargeting cannot evade the check. + +`plan` must equal `registration.capturePlan` exactly, and `plan.arms` must equal +`registration.arms`. Independent source review must already be confirmed and +source-first authoring attested, even when the panel is explicitly underpowered +and `qualificationEligible` is false. The explicit version-2 +[exposed development calibration profile](BROAD_QUERY_REVIEW.md#exposed-development-calibration) +allows prior author exposure with a verified historical registration/run pair, +an exact selected-question mapping and new independent source review. Its +`calibrationEvidence` has exactly those two hash-bound references; original +paths, retained copies and current actor files are rechecked after provisional +scoring and before completion. Newly captured runs bind the copied scorer through +`run.assessor`; profile reassessment requires that reference and verifies its +source-frozen digest. Both retained code copies receive the final publication +recheck. It remains permanently claim-ineligible. +Source-only pending registrations cannot accidentally launch either tool. + +Plan schema `compass.broad-query-capture-plan/1` contains: + +- `timeoutSeconds`: integer 1–300 per command. +- `normalizerSha256` and `assessorSha256`: exact hashes of the capture and + review Python modules, freezing parser and scoring code before collection. + The checker rejects captured commands/settings/binary or dependency hashes + that differ from this plan; reports from a changed assessor require a new + explicitly labeled analysis revision, never silent primary promotion. +- `tools`: exactly `compass` and `graphify`. Each has `version`, full `commit`, + `executable` reference, explicit `argvPrefix` array, `environment` reference, + `nativeOffline: true`, and recorded `settings` with `scopePolicy`, + `inferenceProfile`, `clustering`, and `queryInterface`. The environment is a + JSON object with `files` containing hash-bound dependency references and + optional version metadata. No environment variables or secrets are dumped. + The Graphify record additionally has a `releaseVerification` reference using + the review scorer's official-release contract. Python launchers may use an + explicit interpreter executable and module/script arguments; no runtime + installation or fallback is performed. +- `graphs`: one per repository/tool, each containing `repository`, `tool`, + `artifact`, `sourceCommit`, `sourceCleanBeforeAndAfter: true`, `sourceInventory`, + and `buildCapture`. Inventory JSON has `files` equal to the source registration + inventory. The existing bounded successful build capture is retained; this + adapter does not rebuild. Before/after source-clean build attestation is + required, and current source verification cannot retrospectively prove a + different build's cleanliness. Build JSON contains `argv`, `exit_code`, + `timed_out`, `output_limited`, `wall_ms`, `stdout_bytes`, `stderr_bytes`, + and hash-bound absolute `stdout`/`stderr` input references. The adapter copies + the original JSON and both raw streams, then publishes a rebased build record. + The assessor checks byte counts, the unchanged original fields/digests and + the graph/inventory hashes against the frozen inputs. Success flags alone + cannot stand in for a captured build. +- `arms`: exactly `{id: default}` and `{id: normalized, measure: utf8_bytes, + budgetBytes: INTEGER, overflow: fail}`. Default invokes native `query QUESTION + --graph GRAPH` with no renderer override. Normalized adds the native + `--text-budget` or `--budget` using floor(bytes/4), while actual UTF-8 output + bytes govern acceptance. Complete oversized output remains captured and the + row fails; output is never clipped into compliance. Opt-in Compass ranked + view is allowed only in an explicitly frozen normalized arm with + `compassView: ranked, configuredComparison: true`; resulting reports identify + the configured comparison, and cannot support a claim about native defaults. + Equal native token inputs do not imply identical estimates: Graphify uses + three Unicode characters per token. The separately frozen measured UTF-8 + ceiling applies to both tools without changing either native estimator. +- `environmentExecutable`: hash-bound system `env` executable. The runner uses + `env -i` with `childEnvironment.inherit: false`, frozen PATH `/usr/bin:/bin`, + and explicitly isolated `HOME`, all `XDG_*_HOME` paths, `APPDATA`, + `LOCALAPPDATA`, and `TMPDIR` contained under the fresh artifact directory. + `{artifactRoot}` is the sole path substitution. Unknown environment keys are + rejected. `GRAPHIFY_NO_AUTO_REFRESH=1` and `GRAPHIFY_QUERY_LOG_DISABLE=1` + explicitly keep query capture from refreshing the graph or writing query logs. + Optional Python bytecode/user-site/hash-seed and locale/timezone settings are + allowed only as frozen nonsecret values. No inherited model credentials or + user configuration are passed to either query process. +- `measurement`: an explicit hash-bound system `time` executable and + `style: darwin-l|gnu-v` for this host. Peak RSS is bytes (GNU kilobytes are + multiplied by 1024). Missing or ambiguous RSS remains unavailable and blocks + cost qualification. The wrapper is in every recorded argument vector. + +Collection order is task ID, default then normalized, tool ID. Every execution +is a fresh isolated CLI process with an unflushed OS cache. This is not a cold +cache claim. There are no adaptive retries, diagnostic query subprocesses, +free follow-ups, or task exclusions. Static graph inspection is separately +identified and supplies graph-stage diagnostics only. Wall time includes the +same measurement wrapper for both tools; stdout/stderr and limit/exit status +are retained separately. + +## Coherent Compass graph inputs + +When a native graph has a sibling `source-query.json`, its graph entry must +freeze `sourceQueryArtifact: {path, sha256}` alongside `artifact`. The input +path must be that exact sibling, and the file must be regular, not a symlink. +The collector rejects an existing but undeclared sibling before executing any +process. Without this artifact, native graph-only queries lack source-body +recall; graph JSON alone does not represent the full current query index. + +The collector retains each declared pair at +`graphs/-compass/{graph.json,source-query.json}`. Separate directories +prevent repositories from sharing the fixed sibling filename. Both query arms +receive that graph path. The sidecar is bounded at the native 64 MiB ceiling; +its schema must be `compass.source-query/1`, its profile one of the seven native +profiles1–7, and `graphDigest` must equal the exact graph-byte SHA-256. This +checks the envelope and binding; native Compass still validates the full +payload. A valid envelope does not establish source or graph quality. + +The run retains the reference, and canonical assessment checks its hash against +the frozen plan and its location against the actual graph path. Original and +retained bytes, regular-file status and undeclared sibling absence are rechecked +before completion. An undeclared retained sibling cannot silently affect the +native query. Legacy graph-only captures retain their old layout and meaning; +they do not gain body-evidence coverage retrospectively. Freeze new actor +hashes and collect a new run to correct an omitted artifact. This optional field +applies only to Compass and introduces no dependency on the comparator. + +## Native query exit outcomes + +A tool may optionally declare this exact `queryExitContract` in its frozen +capture plan. Collection validates it before launching any subprocess, copies +it into `run.tools`, and canonical assessment checks equality with the plan: + +```json +{ + "schema": "compass.broad-query-query-exits/1", + "noMatchExitCode": 4, + "noMatchStdoutLine": "No matching nodes found." +} +``` + +The source reviewer must verify the declared CLI outcome against the pinned +tool's implementation. This example describes Graphify 0.9.92's documented +query no-match outcome. The contract requires exactly these fields, a nonzero +exit code in 1–255, and a nonblank, control-free UTF-8 line of at most 4,096 +bytes. Recognition requires that exact code and exactly the encoded line plus +one LF byte. No substring, trimming, extra output or CRLF substitution is used. +Unknown versions, extra fields and malformed contracts fail explicitly. + +A recognized no-match is an abstention suggestion pending independent review. +It is still an incorrect answer to an answerable question. It cannot produce a +successful answered disposition. Timeouts, capture limits, output-budget +overruns and every other nonzero outcome remain failures in every denominator. +Raw exit status, stdout, stderr and costs are retained. Without this optional +contract, legacy interpretation is unchanged: every nonzero exit fails. + +Freeze new actor hashes and a new experiment before using this contract. +Historical captures and their original scoring remain immutable; this is not +permission to regrade an earlier run under changed semantics. Independent +answer/source/graph review and all existing qualification gates still apply. + +## Output normalization and review + +The normalizer has no registration/oracle input. It reads only the actual +first response and its tool's graph. Actual printed file/start/name anchors +must map to a unique graph declaration; collisions remain unresolved. A graph +neighbor absent from text is never credited. Printed qualified owners are +preserved. Missing receiver owners (for example `.method()`) require source +review rather than guessed ownership. Printed edge arrows are retained; +actual optional ` at=file:Lnn[-Lnn]` citations are separated from target labels +and retained as `printedSite` in the review queue. They contain no invented +source text. Initial connectivity claims keep occurrence arrays empty pending +independent source-aware citation review, even when source/graph metadata has +sites. Malformed citations never yield automatic connectivity claims. Identical +site-free connectivity claims are coalesced for both tools; every printed edge +line, duplicate, citation and exact byte span remains separately queued. Distinct +relations or directions are retained separately. This does not establish +column-level or complete call multiplicity. Display order is not retrieval rank. +Missing rank identities or rank gaps prevent automatic retrieval metrics. +Unexposed admission remains `null`. + +Every parsed declaration/relation and every unmatched text line has an exact +UTF-8 byte span/digest in `reviewQueue`. A declaration line asserts identity, +not an automatically inferred behavioral fact. No expected fact ID is assigned. +The initial typed claims are unjudged, and `normalizationComplete: false` +explicitly prevents qualification. An independent reviewer must inspect the +full raw response and pinned source, resolve every material ambiguity, preserve +unsupported claims, and check that no prose/context assertion has disappeared +from the precision denominator. AI review must be identified as AI review. +Tool labels can be blinded for review but distinctive output style can still +reveal the tool; disclose that limitation. + +Publish reviewed typed normalizations, a new run sidecar referencing them, +review judgments bound to those exact hashes, and a new review-input manifest +as new files. Preserve the original raw captures, original run and provisional +report. Set `normalizationComplete: true` only after that independent completeness +review. Then use the review scorer's `--score` to produce a new report and +`--check` to recompute eligibility. Programmatic checks verify integrity and +registered arithmetic; they cannot authenticate reviewer attestations or +establish source semantics automatically. diff --git a/benchmarks/agent_query/BROAD_QUERY_REVIEW.md b/benchmarks/agent_query/BROAD_QUERY_REVIEW.md new file mode 100644 index 000000000..03f94790e --- /dev/null +++ b/benchmarks/agent_query/BROAD_QUERY_REVIEW.md @@ -0,0 +1,684 @@ +# Source-fact broad-query review + +`broad_query_review.py` implements the developer-side scoring and integrity +checks for Plan 029. It does not run either product, install a competitor, infer +ground truth from grep, or qualify a source judgment by matching prose. Use the [frozen capture adapter](BROAD_QUERY_CAPTURE.md), which reuses +`runner.run_bounded`, to collect commands and retain their exact argument vectors, +exit/limit/timing fields and raw stdout/stderr in immutable artifact records. + +```bash +python3 -m benchmarks.agent_query.broad_query_review --self-test +python3 -m benchmarks.agent_query.broad_query_review \ + --validate-registration benchmarks/agent_query/broad_query_registration.json +python3 -m benchmarks.agent_query.broad_query_review \ + --score /external/evidence/input.json --output /external/evidence/report.json +python3 -m benchmarks.agent_query.broad_query_review --check /external/evidence/report.json +``` + +`--self-test` tests synthetic scorer contracts. Registration validation checks +shape, exact identities, source-file pins and internally coherent task oracles; +it accepts explicitly pending independent review. Neither command establishes +real-repository correctness. `--score` writes a new diagnostic report and exits +zero even when qualification is incomplete. `--check` recomputes the entire +assessment, rejects changed/fabricated/stale metrics, and exits zero only when +every registered qualification gate passes. Ineligible reports exit one; +malformed inputs or artifact-integrity failures exit two. Existing reports are +never overwritten. + +## Review and trust boundary + +### Source and mechanism exposure lineage + +A new heldout capture requires `registration.exposureLineage` with schema +`compass.broad-query-exposure-lineage/1`. A legacy registration without it stays +shape-valid and can produce an ineligible diagnostic report. It cannot launch a +new heldout capture or qualify through the current assessor. Explicit legacy +development captures retain their separate independent source-review gate. + +The inline lineage is part of the exact hashed source registration: + +- `frozenAt` and `historyCompleteThrough` equal the registration freeze; + `historyComplete` explicitly attests completeness, with an `author` identity. +- `independentReview` has `status: confirmed`, `reviewer`, `reviewerType` + (`human`/`ai`), `sourceAware: true`, `independentOfAuthor: true`, and UTC + `reviewedAt` no later than the freeze. The reviewer differs from both the + history and source authors. This audit is source-aware; it does not require + hiding historical tool identities. +- `repositoryAliases` contains explicit pairs of HTTPS repository URLs for + renames/mirrors. `mechanismLinks` contains pairs of stable mechanism IDs for + shared mechanisms, including mechanisms appearing in different repositories. + Both links are transitive. IDs use lowercase ASCII letters/digits and + `._:/-`; renaming an ID does not establish a new mechanism. +- `history` contains up to 8,192 events with unique `id`, UTC `observedAt` no + later than the independent review, `repositoryUrl`, full `commit`, + `mechanisms` (up to 32 IDs), `evidenceSha256`, and `kind`: one of + `source_development`, `graph_build`, `query_capture`, `answer_review`, or + `implementation_tuning`. Include all known prior exposure, not only exact + question repeats. Ordinary source-first oracle authoring for the new frozen + experiment is separate from prior development exposure. + +Every registered task declares `split: development|confirmation` and one to 32 +stable `mechanisms`. A heldout registration contains only confirmation tasks. +A confirmation task fails if any recorded history touches its repository +(across commits), the identical commit under another URL, or a linked mechanism. +Repository IDs and question wording do +not enter that decision. HTTPS origins normalize default ports, trailing dots, +`.git`/trailing slashes and GitHub host/path case; other renames require explicit +aliases. GitHub URLs must identify a repository root, not a tree/file page. +Escaped/ambiguous URLs fail. Aliased registered repositories or two URLs at the +same commit cannot count as two independent sources. No automatic semantic-equivalence claim is +made for mechanism IDs. + +Capture inputs supply `exposureEvidence`, mapping each distinct historical +digest to an absolute `{path, sha256}` reference. Exactly the registered digest +set is required. Before any subprocess, the adapter checks every evidence hash +and the lineage; each input is at most 64 MiB, with a 512 MiB aggregate history +limit. It copies these bytes into the external bundle and rechecks originals +before publishing completion. The review input carries the same mapping with +contained references; the assessor verifies it again. Original, lookup and union +panels retain identical lineage and mapped task split/mechanism assignments. + +The checker verifies the records supplied to it. It cannot discover omitted +experiments, authenticate reviewer identities, infer undeclared semantic reuse, +or turn an evidence digest into a true exposure claim. Independent history +completeness/source review is therefore a separate requirement, not a generated +pass label. Obtain independent split/history approval before materializing new +confirmation graphs. The final capture registration binds history through its +own freeze; preserve the earlier source-only review and build chronology +separately. The adapter operates on prebuilt graphs and cannot retrospectively +prove that ordering. A passing exposure check alone proves none of the +query-quality, statistical-power or graph-fidelity gates. + +### Required original and lookup panels + +`broad_query_panels.py` binds the linked declaration-lookup correction without +discarding its original named-behavior questions. Both canonical review reports +must pass their existing gates. A passing lookup report cannot substitute for a +missing or failing original report. The current source-author proposal contains +352 questions per panel, with 208 shared questions and 496 distinct questions. +Those drafts still require independent source/stratum review and prospective +power planning; these counts establish no quality result or statistical power. + +Before collecting, freeze a `compass.broad-query-panels/1` definition with: + +- `frozenAt`: a UTC timestamp after both source registrations and before capture; +- `assessorSha256`: `panel` and `canonical` SHA-256 values for + `broad_query_panels.py` and `broad_query_review.py`; +- `registrations`: `original` and `lookup` contained `{path, sha256}` references + to the complete, frozen source registrations; +- `lookupPairs`: explicit `{original, lookup}` task-ID pairs for every replacement. + +The binder preserves shared questions/facts, source inventories, source clusters, +cost arms, gate policy, graph scope and mapped metric cohorts. Lookup variants +must request their declaration identity. Source or rubric corrections require a +linked pre-output revision; the binder does not approve them automatically. + +Capture the union once in one isolated artifact root. Its source registration +must contain the exact union of both panels and the same capture plan, sources +and arms. After output review, project that union review input in place: + +```bash +python3 -m benchmarks.agent_query.broad_query_panels \ + --project /external/bundle/review-input.json \ + --definition /external/bundle/panels-definition.json +``` + +This writes `panels-original-input.json`, `panels-lookup-input.json` and their +run/review documents, then a `panels-projection.json` completion receipt. Existing +files are never overwritten. All projection documents remain in the union root +so its frozen home/cache/temp environment remains valid. Projection adds no +accepted source/output judgments and does not establish qualification. + +Score both projected inputs with the canonical `broad_query_review --score` +command. A `compass.broad-query-panel-review-input/1` document then binds a +`definition` reference and `reports.original` / `reports.lookup` references: + +```bash +python3 -m benchmarks.agent_query.broad_query_panels \ + --score /external/bundle/panel-input.json \ + --output /external/bundle/panel-report.json +python3 -m benchmarks.agent_query.broad_query_panels \ + --check /external/bundle/panel-report.json +``` + +Both reports are recomputed through the canonical checker. Tool, graph, arm and +environment captures must agree. Shared answers, normalization judgments and +costs must be identical; different task/tool/arm keys are charged separately even +when their stdout bytes happen to match. Query costs are deduplicated by that +key, and graph-build costs are listed once per repository/tool. The aggregate +report uses `compass.broad-query-panel-review/1`; `--check` exits 0 only when both +panels pass, 1 for an ineligible comparison and 2 for malformed or stale input. +As with the canonical checker, this verifies integrity and arithmetic, not the +authenticity of reviewer attestations or the truth of source judgments. + +Source meaning and answer meaning require independent review. A source author +freezes the questions, acceptable declarations, essential facts and expected +ambiguity/no-support outcomes before collection. A second independent reviewer +confirms source judgments and reviews anonymized output. This reviewer may be +human or AI, with its type and source-aware method disclosed. A reviewed typed +normalization maps actual output spans to graph-node identities and source +facts; it must distinguish retrieval rank from display/context order. The +normalizer is an evaluation adapter, not a tool-specific answer generator. + +The checker verifies pinned bytes, spans, unique graph identities, explicit +relations and sites, complete paired matrices, scorer arithmetic, costs and +statistical gates. It cannot authenticate a named reviewer, establish that an +attestation is truthful, or infer that a source line proves an arbitrary +semantic claim. A complete set of fabricated reviewer attestations is outside +its verification boundary. Reviewers must never label AI review as human +review. The report states that boundary explicitly. + +Development registrations, pending or non-independent reviews, unjudged assertions, +unreviewed candidates, post-output positive oracle additions, missing graph +qualification, and underpowered comparisons remain claim-ineligible. Correcting +an incomplete oracle requires preserving the original result and conducting a +new confirmation; a result cannot silently change its frozen gold labels. + +Native no-match exit interpretation is optional and must be frozen through the +tool's typed [query exit contract](BROAD_QUERY_CAPTURE.md#native-query-exit-outcomes). +The collector and assessor share the exact code/byte classifier; raw receipts +remain unchanged. A valid no-match can be an operationally successful abstention +while still failing an answerable task. Old plans without the field retain their +original all-nonzero-fail semantics. A changed contract requires a new experiment, +not a retroactive exception for an observed outcome. + +Native source-body evidence is a separate, hash-bound +[coherent graph input](BROAD_QUERY_CAPTURE.md#coherent-compass-graph-inputs). +Canonical assessment verifies the declared `sourceQueryArtifact`, native sibling +location, schema/profile, graph digest and frozen byte identity. Undeclared +retained siblings are rejected. Envelope integrity does not prove extraction +completeness or relevance. Legacy graph-only evidence remains graph-only; +correcting a missing sidecar requires a new frozen capture, not score promotion. + +## Source registration contract + +### Exposed development calibration + +Version-2 registrations may explicitly opt into +`compass.broad-query-development-calibration/1` to collect a new diagnostic run +after the source author has seen earlier outputs. This is permanently +superiority-ineligible. It does not turn an exposed question into confirmation, +regrade an old report, approve new source labels, or establish statistical power. +Unprofiled development and heldout captures retain the output-blind authoring +requirement. + +The `developmentCalibration` envelope has exactly these fields: + +```json +{ + "schema": "compass.broad-query-development-calibration/1", + "purpose": "paired_development_calibration", + "priorOutputExposure": true, + "priorRegistrationSha256": "<64 lowercase hexadecimal characters>", + "priorRunSha256": "", + "selection": [{"task": "current-id", "priorTask": "historical-id"}], + "selectionRationale": "Predeclared corpus, denominator and excluded-task scope." +} +``` + +Require `designation: development`, `qualificationEligible: false`, a named +`sourceReview.author`, and `sourceReview.blindToOutputs: false`. Unknown profile +versions/fields, version-1 opt-ins and conflicting eligibility flags fail. +Selection covers every current task exactly once, with no historical task +reused. It need not include every historical task. Disclose the selection +denominator and exclusions; selection is prospective only for the new run, +and is neither newly blinded nor necessarily representative. + +Capture input and review manifest contain `calibrationEvidence`, mapping exactly +the two digests to hash-bound artifact references (absolute inputs, contained +bundle references). The historical run must use the supported capture schema +and bind that exact original registration digest. Its recorded chronology must +satisfy `prior.frozenAt < priorRun.startedAt <= priorRun.completedAt < frozenAt`. +Every selected historical question has exactly one observation for each of the +two tools and two arms. Current/historical question, kind and expected disposition +must match exactly, as must repository URL, commit and complete pinned file +inventory. IDs can change. New oracle annotations may differ only after a new +independent source review; those annotations never replace old outcomes. + +The existing independent source-review requirements still apply. Additionally, +the reviewer attests `priorOutputsObserved: false` for substantive prior answers, +normalizations and per-task scores in the selected corpus. Metadata-only joins +are not answer observation. Freeze the review's UTC `reviewedAt <= frozenAt` +and `sourceDefinitionSha256`: SHA-256 of UTF-8 +`json.dumps({"repositories": repositories, "tasks": tasks}, sort_keys=True, +separators=(",", ":"), allow_nan=False)` using Python's default ASCII escaping. +This binds exact prospective facts/groups, rather than transferring approval +from a historical oracle. Disclose reviewer familiarity and method. Hashes do +not authenticate independence or truthful blinding; labels removed after the +reviewer knew an output's identity do not establish tool-label blindness. +New answers still require independent source-aware, anonymized output review. + +The adapter verifies and copies the two metadata documents before any process, +rechecks original paths and retained copies after provisional scoring, and +publishes no `COMPLETE` receipt on drift. Each document is bounded to 64 MiB, +128 MiB in aggregate. It does not recursively follow historical graph, answer, +normalization or report references, or accept calibration chains as the parent. +Keep the original report bundles and outcomes unchanged at their recorded +locations; cite them separately as historical provenance. This metadata pair +does not establish complete exposure history or independently authenticated +chronology. Do not give reviewers embedded answer payloads under a metadata-only +label. The assessor independently verifies the pair and selection, retains +diagnostic failures/metrics, and always rejects a superiority claim. +New runs also bind a contained `assessor` reference to the frozen scoring-code +digest. This reference is required for this profile. Canonical reassessment +verifies its bytes and frozen identity; legacy runs without the profile retain +their original no-reference behavior. Publication rechecks both retained capture +and assessor code copies, as well as their original paths, after provisional +scoring and immediately before completion. + +Original/lookup panels preserve equal profile metadata excluding `selection`. +Shared tasks have identical mappings; every panel's mapping must be the exact +union mapping restricted to that panel's task IDs, independent of list order. +The two evidence digests and checked bytes remain bound through projection and +recomputation. New lookup wording is unsupported by this narrow profile unless +that exact question already exists in the cited historical registration. + +Source witness text is one bounded UTF-8 line and may retain indentation tabs. +Newlines, carriage returns, NUL and escape controls remain invalid. Witness text +is preserved in the registration; other labels and identities still reject tabs. +Pinned file digests and source-line checks continue to bind the witness to its +source. Accepting indentation does not establish source meaning or independent +review. + +Schema: `compass.broad-query-registration/1`. The repository/source-preparation +workflow may add descriptive metadata. The core fields are: + +```json +{ + "schema": "compass.broad-query-registration/1", + "designation": "heldout", + "frozenAt": "2026-10-16T00:00:00Z", + "qualificationEligible": false, + "sourceReview": {"author": "source author", "blindToOutputs": true}, + "independentReview": { + "status": "pending", "reviewer": null, "toolLabelsBlinded": true + }, + "repositories": [{ + "id": "sample", "language": "python", "url": "https://example.invalid/sample", + "commit": "FULL_GIT_OBJECT_ID", + "files": [{"file": "src/store.py", "sha256": "HEX_SHA256"}] + }], + "tasks": [{ + "id": "save-task", "repository": "sample", "cluster": "save-mechanism", + "kind": "broad", "question": "where is persistent state saved", + "expected": "answer", + "relevant": [{"identity": { + "file": "src/store.py", "startLine": 10, "symbol": "save", "owner": "Store" + }, "grade": 3}], + "facts": [{ + "id": "save-fact", "essential": true, + "identity": {"file": "src/store.py", "startLine": 10, "symbol": "save", "owner": "Store"}, + "statement": "Save delegates persistence to the writer.", + "source": {"file": "src/store.py", "line": 12, "text": "return self.writer.flush()"} + }] + }] +} +``` + +The example uses placeholders and is not a valid qualification registration. +An identity has exactly `file`, `startLine`, `symbol`, and `owner`; the owner is +the reviewed lexical owner, or the empty string for a module-level declaration. +Paths are canonical repository-relative POSIX strings. Grades are 0–3; +grades 2–3 are relevant. Declaration file/start/name and owner must all agree +for answer scoring. Graph mapping uses the existing symmetric exact +file/start/terminal-name policy and requires a unique result. Source review +establishes lexical ownership; the normalizer cannot choose the graph node +whose name merely fits an expected answer. + +Task kinds are `broad`, `exact`, `directed`, `multihop`, `ambiguous`, and +`unsupported`. Expected outcomes are `answer`, `ambiguous`, and `abstain`. +Answerable tasks need positive identities and at least one essential fact (or a +version-2 essential group, below). +Ambiguous tasks need at least two source-reviewed positive identities. +Unsupported tasks have empty `relevant`/`facts` arrays and a `rationale` scoped +to the registered source inventory. They mean “no supporting indexed evidence +in this scope”, not proof that a behavior cannot exist anywhere. + +### Equivalent implementation witnesses + +New prospective registrations may opt into +`compass.broad-query-registration/2`. Version 1 retains its original conjunctive +fact requirements and rejects `essentialFactGroups`; older assessors reject +version 2. Unknown registration and group versions fail explicitly. Existing +source registrations, captures and reports are not migrated or rescored in place. + +A broad answerable task may add this versioned requirement: + +```json +{ + "essentialFactGroups": { + "schema": "compass.broad-query-essential-fact-groups/1", + "groups": [{"id": "expired-lookup", "anyOf": [["get-expiry"], ["peek-expiry"]]}] + } +} +``` + +Each alternative in `anyOf` is a complete witness set of fact IDs. For example, +`[["read", "validate"], ["alternate-read", "alternate-validate"]]` requires +both facts of one complete path; mixing `read` with `alternate-validate` fails. +Shared-prefix sets such as `[["read", "validate"], ["read", "reject"]]` +still require `read` and a complete alternative. Each referenced fact remains an +entry in `facts`, with its own exact +declaration identity, source witness and optional relationship/occurrences. Its +identity must be a frozen positive candidate. Group members have `essential: +false`; an individually essential fact cannot also be an alternative. A group +contains at least two alternatives with distinct sets of full declaration +identities (file, start line, symbol and lexical owner). Two methods in the same +class can qualify. Different complete paths through identical declaration sets +cannot be expressed by this contract; do not silently discard valid alternatives +to fit it. Group IDs are unique and distinct from fact IDs; members are known +and unique within each witness set. Facts may be shared between alternatives of +one group, but not between groups. Duplicate or subsumed witness sets fail. +The existing 64-fact bound applies; groups, alternatives and facts per set each +have a 64-entry bound, with at most 256 total fact references across alternatives +per task. The checker evaluates these sets directly without expanding combinations. +Empty groups/sets, extra group fields and unknown schemas fail. Exact, directed, +multihop, ambiguous and unsupported tasks cannot use groups. + +Success still requires every individually essential fact. It also requires +all supported, recovered facts of at least one alternative in **every** group. Claim matching never +substitutes a group member's owner for another fact's owner; relationship +direction and occurrence requirements remain exact. Unjudged/unsupported claims +and execution limits retain their original failure behavior. No fact can satisfy +two groups, and alternatives add no independent tasks or source clusters. + +The source author and independent reviewer must approve complete equivalence for +the question's requirement and quantifier, including branch qualifications and relevant alternative +implementations. The checker validates declared structure and evidence; it does +not infer behavioral equivalence from similar text or names. Group membership is +part of the hashed source task and registration, frozen before outputs. Any +oracle correction needs a linked prospective source review and new capture-plan/ +assessor hashes. Adding a group after seeing answers cannot rehabilitate a +held-out result. Original required panels/outcomes remain retained. + +The original/lookup panel binder preserves both registration versions and group +payloads, including through union projection. Scored rows with groups expose +`essentialFactGroupSchema`, `essentialFactGroups` (`id`, recovered member IDs, +zero-based `satisfiedAlternativeIndexes` in frozen input order and `satisfied`), +and `essentialFactGroupRecall`. All completed alternatives are reported. The per-witness source/graph ledger and +`sourceFactRecall` remain separate: recovering one of two equivalent witnesses +can satisfy a question with witness recall 0.5. Group recovery is not graph +precision, source-witness recall, independence or a qualification certificate. +Rows without groups retain their existing shape and scoring. + +A fact optionally includes a typed relationship: + +```json +{ + "relation": { + "source": {"file": "src/store.py", "startLine": 10, "symbol": "save", "owner": "Store"}, + "target": {"file": "src/writer.py", "startLine": 30, "symbol": "flush", "owner": "Writer"}, + "kind": "calls", "direction": "source_to_target", + "occurrences": [{"file": "src/store.py", "line": 12, "text": "return self.writer.flush()"}] + } +} +``` + +Both endpoint identities, relation kind, direction and the complete registered +occurrence set must agree. Repeating one occurrence cannot recover a different +missing one. Multihop tasks register each essential directed hop as a separate +fact; endpoints plus a list of relation names do not prove an intermediate +route. Native undirected artifacts explicitly report an unavailable direction +contract rather than an invented reverse-call error. + +## Evidence bundle contract + +All artifact references are `{ "path": "relative/path", "sha256": "64 lower-case hex digits" }`. +They resolve under the input manifest's directory; symlink escapes, `..`, +absolute artifact paths, digest drift, duplicate JSON keys and excessive +input sizes fail closed. Local `sourceRoots` are explicit external checkout +paths and are not product configuration. Keep raw captures and source content +outside the source repository under the source owner's access policy. + +Input schema `compass.broad-query-review-input/1` contains: + +```json +{ + "schema": "compass.broad-query-review-input/1", + "registration": {"path": "registration.json", "sha256": "HEX_SHA256"}, + "run": {"path": "run.json", "sha256": "HEX_SHA256"}, + "reviews": {"path": "reviews.json", "sha256": "HEX_SHA256"}, + "sourceRoots": {"sample": "/external/source/sample"}, + "graphQualification": [{ + "repository": "sample", + "manifest": {"path": "graph-audit-manifest.json", "sha256": "HEX_SHA256"} + }] +} +``` + +`graphQualification` supplies existing `compass.quality-audit/2` qualification +manifests. The checker calls the existing `run_audit` on the exact captured +Compass graph and source, preserving its 2,000-judgment, .995 precision, .99 +Wilson lower-bound and diversity/capability/source-coverage gates. It does not +trust an `eligible: true` summary from another file. Symmetric source-based +graph comparisons remain independent of this Compass production gate; neither +graph counts nor reproducing all competitor hypotheses replaces those audits. +For a multi-corpus production audit, supply one entry with `manifest`, relative +`graphRoot` and explicit external `sourceRoot` instead of `repository`. Its +corpus names, source roots, commits and graph hashes must match the captured +panel exactly. This preserves the existing aggregate 2,000-total and +400-per-corpus floors rather than requiring 2,000 judgments separately for +every repository. + +The run schema `compass.broad-query-capture/1` contains: + +- `registrationSha256` and UTC `startedAt`, after the source registration freeze. +- Exactly two `tools`, IDs `compass` and `graphify`, each with `version`, + `commit`, `nativeOffline: true`, recorded `settings`, and artifact references + `executable`, `environment`, and `normalizer`. Environment manifests must + retain the installed distribution/dependency/package hashes; a Python + launcher hash alone does not identify an installed competitor. + Settings explicitly name `scopePolicy`, `inferenceProfile`, `clustering`, and + `queryInterface`; missing settings prevent qualification. Record actual + differences rather than inventing equivalent internal feature flags. +- Graphify also has `releaseVerification`, referring to a record with + `version`, `commit`, UTC `checkedAt`, and + `source: "https://github.com/Graphify-Labs/graphify/releases"`. Pin the latest + release before collection, not divergent default-branch HEAD. +- One `graphs` entry per repository/tool: `repository`, `tool`, graph + `artifact`, `sourceCommit`, `sourceCleanBeforeAndAfter: true`, and + `sourceInventory`/`buildCapture` references. Both tools must receive the same + inventory hash for a repository. Retain parser omissions separately. + The successful bounded build record must explicitly have `exit_code: 0`, + `timed_out: false`, and `output_limited: false`. +- Exactly two `arms`, frozen identically in the registration: + `{"id":"default"}` and + `{"id":"normalized","measure":"utf8_bytes","budgetBytes":8000,"overflow":"fail"}`. + The byte budget is configurable before collection; it is a named proxy, + not a tokenizer. Normalized first-page scoring permits one capture and no + follow-ups. Overflow stays in the denominator as a failed execution; the + checker retains the complete raw response rather than clipping it. +- One `observations` entry for every task/tool/arm, containing `task`, `tool`, + `arm`, `normalized` reference, `captures` references (1–4), `condition`, + and `peakRssBytes`. Conditions are `fresh_cli`, `warm_cli`, `warm_mcp`, + `cold_cli`, or `unrecorded`. Missing measurements remain explicit and prevent + complete qualification. Interleave/randomize timing collection and retain + its order/setup metadata; this checker does not establish a latency win. + +Each capture is the existing bounded runner's field record: `argv`, +`exit_code`, `timed_out`, `output_limited`, `wall_ms`, `stdout_bytes`, +`stderr_bytes`, plus `stdout`/`stderr` artifact references instead of inline +text. Follow-up bytes and latency are charged in full. Nonzero exits, limits, +timeouts and over-budget normalized responses fail task success while remaining +in all cost and task denominators. +Broad and unsupported tasks must pass the exact frozen question as one argument +in each measured command. A tool-specific oracle-informed question rewrite is +rejected. Exact/directed controls may use the closest documented typed operation +with their preregistered source subjects; do not count known-ID preparation as +successful natural-language retrieval. + +## Typed response and judgments + +Each normalized response has schema `compass.broad-query-normalized/1`, matching +`task`, `tool`, and `arm`, plus: + +- `disposition`: `answered`, `ambiguous`, `abstain`, `error`, or `limit`. +- `rankingKind`: `retrieval`, `display`, or `unavailable`. A neighborhood's + arbitrary display order is not a ranked retrieval result. Retrieval metrics + are unavailable for display/context lists. +- `ranked`: candidate records `{identity,nodeId,firstPage,span?}`. A first-page + candidate must carry a raw first-capture `span`; it must uniquely resolve to + the recorded graph node at its exact declaration anchor. Ranks beyond the + first page may be retained from the documented typed ranked result/trace. +- `admission`: an instrumented array of admitted graph node IDs, or `null` if + the tool does not expose that stage. Missing instrumentation is unavailable, + never invented as a false admission failure. A candidate with an observed + rank must also be in an instrumented admitted set when one is supplied. +- `claims`: first-page fact assertions `{id,identity,relation?,span}`. Each span + is `{startByte,endByte,sha256}` over exact first-capture stdout bytes, with + `endByte` exclusive. Spans establish where the reviewed assertion appeared; + they do not establish its meaning. An independent reviewer must also confirm + complete normalization so omitted incorrect assertions cannot improve precision. + +Judgments schema `compass.broad-query-judgments/1` has `runSha256` and +`observations`. Each row repeats `task`, `tool`, and `arm`, binds +`normalizedSha256`, and has an `independentReview` object. Confirmed independent +source-aware review uses `status: "confirmed"`, `reviewerType: "human"` or +`"ai"`, a nonempty `reviewer`, `independentOfAuthor: true`, `sourceAware: true`, +and `toolLabelsBlinded: true`. The source author and independent source reviewer +must be distinct. Pending review is allowed as a diagnostic input and cannot +pass qualification. Reports disclose AI review and never call it human review. + +`claims` reviews contain `id`, `verdict: supported|unsupported|unjudged`, +`factId` (a frozen fact ID or null), and `reason`. A supported frozen fact +requires exact identity, relationship kind and direction agreement; a wrong owner or reversed +edge is unsupported even if the review labels it supported. A supported extra +assertion with null `factId` additionally needs a pinned `source` witness and +independent semantic review. Extra source files must be pinned before scoring. +Unknown fact IDs and duplicate semantic claims fail validation. + +`candidates` reviews contain `identity` and `grade` for displayed candidates +outside the frozen positive/negative identities. Unknown returned candidates +remain unjudged until reviewed. Newly discovered positive identities are +reported as post-output oracle corrections and prevent a held-out claim. +Changing a previously frozen grade is rejected. + +## Metrics and qualification + +An answerable task succeeds only when its first page identifies a relevant +declaration, answers every essential source fact and predeclared version-2 group, +and contains no unsupported +or unjudged assertion/candidate. An ambiguity-only list cannot pass an +answerable task. A valid ambiguity answer explicitly retains at least two +source-reviewed possibilities. An unsupported task succeeds through explicit +abstention with no competing asserted candidates/facts. + +The report retains the source → graph → admission → rank → first-page ledger +for every registered fact. Graph absence and missing instrumentation remain +different outcomes. End-to-end success includes every answerable task; +graph-present conditional statistics cannot hide extraction misses. + +Success@5 and MRR@10 use frozen grade ≥2 relevant identities and documented +retrieval ranks. nDCG@10 uses gains `2^grade - 1` and `log2(rank + 1)` discounts. +Source-fact recall is recovered frozen facts divided by all frozen facts. +Claim precision uses supported/(supported+unsupported) reviewed claims; +unjudged assertions are reported separately and prevent qualification. Its +paired noninferiority score assigns an empty answer zero per-task precision +(`precisionWithEmptyZero`) to prevent improving precision by abstaining on +hard tasks. This score is distinct from pooled claim precision and is labeled +accordingly. False-empty rate is explicit abstention or no candidates/claims +on answerable tasks. All summaries retain repository, language and kind strata. + +Only the normalized arm determines the registered query claim. The scorer uses +a deterministic hierarchical paired bootstrap: average paraphrases within +their source-task cluster, resample repositories, then resample source-task +clusters within each selected repository. Repositories have equal weight. +Seed and 1,000–10,000 resamples are frozen in `gatePolicy`. The result reports +the paired estimate, percentile 95% interval, repository/cluster counts, and +method. The reported interval is widened by a conservative exact finite-task +paired-risk bound: lower confidence for definite task-cluster wins minus upper +confidence for any task-cluster loss, with Bonferroni-adjusted Clopper–Pearson +bounds. Paraphrases remain one cluster; partially positive clusters receive no +definite-win credit. This bound assumes independently sampled source tasks +conditional on the fixed repository panel and requires equal independent +cluster counts per repository. It does not establish population repository +inference. In particular, 32 all-tie controls retain a nonzero upper bound on +unseen losses and cannot establish .02 noninferiority. A bootstrap's zero +empirical variance is never treated as certainty. Purposive corpus selection +still limits generalization even if the interval passes. + +The proposed gates cannot be weakened through the manifest: + +- At least eight repositories, four languages with at least two repositories + each. A single panel follows 10 broad / 4 exact-or-directed / 2 multihop / + 2 ambiguous / 2 unsupported per repository. A heterogeneous source-first + expansion may instead freeze `metricCohorts` and `samplingDesign`, retaining + every original task and using balanced independent cluster counts and language + allocation within each primary metric cohort. +- At least 80 independent broad source-task clusters; paraphrases do not add + independent sample size. +- Broad first-page success improves by at least .10 observed, with paired + 95% lower bound above zero. This does not assert that the true gain is ≥.10. +- Exact/directed task success and answerable per-task claim-precision score + have paired lower bounds strictly greater than −.02. A repository-level + answerable-success regression worse than .10 requires independent investigation. +- Independent source-aware source/normalization review, complete provenance/judgments, + common frozen cost arms, and existing independent graph qualification. +- A frozen `powerAnalysis`, including `targetPower` ≥.8, + `expectedBroadDiscordance`, `expectedControlDiscordance`, `designEffect` ≥1, + and `developmentEvidence` artifact reference. The approximate paired-binary + planning floor uses `(z_.975 + z_power)^2 * (discordance - effect^2) / + effect^2 * designEffect` for broad superiority and + `(z_.975 + z_power)^2 * controlDiscordance / margin^2 * designEffect` for + control noninferiority. These are prospective planning checks, not post-hoc + “achieved power”. Independently review the assumptions and retained development + evidence; the checker cannot establish their representativeness. + +The initial 160-task registration is not a promise that these gates can pass. +In particular, a .02 noninferiority margin can require far more independent +controls. Plan any balanced expansion before looking at confirmation outputs, +retain failures, and report an inconclusive result when uncertainty or power +is inadequate. Do not reclassify development reruns as a fresh holdout. + +## Occurrences, prospective cohorts and power + +Relationship source witnesses always retain the frozen occurrences. The +answer rubric is separately predeclared with `fact.requireOccurrences`: true +requires exact printed line witnesses; false permits omitted sites for a +connectivity-only question while still requiring exact endpoints/kind/direction. +Any actually asserted site must be source-supported. No absent printed site is +backfilled from source or graph. The initial panel uses false only for questions +that ask identity/connectivity/path rather than sites/counts, accepted before +outputs by the source author and independent reviewers. File+line witnesses do +not establish column-distinct call multiplicity. Native graph qualification and +its occurrence/provenance gates remain separate. + +`metricCohorts` has exactly `broadSuccess`, `exactDirected`, and +`claimPrecisionEmptyZero`, each with `taskIds`, `rationale`, +`minimumRepositories` (at least eight), and `minimumIndependentClusters`. +Broad success and precision use the same frozen broad tasks; control success +uses exact/directed tasks. Every captured task remains in diagnostics, even +when a new source-only control cohort is the prospective NI population. +`samplingDesign` explains selection, source dependence and original-task +retention. Paraphrases, shared implementations and repeated endpoints do not +create independent clusters. + +The normal-planning floor above is diagnostic and cannot by itself establish +power of the actual acceptance rules. The scorer additionally computes an exact +finite-control pass probability from independent-cluster N and the predeclared +adverse probability, crediting no definite wins. At N=200, adverse probability +.005 gives only about .367 probability of passing that finite NI bound. An +assumed equality alternative of zero requires hash-bound pre-output evidence; +zero losses observed afterward cannot supply the assumption. + +`powerAnalysis.acceptanceRuleSimulation` must reference a frozen +`compass.broad-query-power-simulation/1` artifact with `frozenAt`, an independently +confirmed `independentReview`, and `metrics` for all three comparison names. +Each metric has `assumptions`, `method`, hash-bound simulation `code` and +`results`, `independentClusters`, and `powerLowerBound`. The planned lower bound +must reach registered target power and the actual panel must reach planned +clusters. Simulations must model the actual hierarchical and finite-risk gates, +including per-task precision's nonbinary distribution. The checker verifies +these records and reviewer attestations; it does not authenticate reviewers or +assert that their prospective model represents a population. Missing planning +records keep the result ineligible rather than converting observed significance +into post-hoc power. + +Graph-quality claim scope may be frozen in +`graphQualificationRepositories` (at least eight registered repositories, +including every broad comparison repository). Added strict-query control +repositories are explicitly query-only and receive no graph-quality superiority +claim. Reports enumerate both groups and audited graph hashes. Without an +explicit scope, every registered repository requires graph qualification. +The native audit retains its existing 2,000-total and 400-per-corpus floors; +eight audited broad corpora therefore need at least 3,200 accepted judgments. diff --git a/benchmarks/agent_query/README.md b/benchmarks/agent_query/README.md index c0c20d487..3e8660a9c 100644 --- a/benchmarks/agent_query/README.md +++ b/benchmarks/agent_query/README.md @@ -4,6 +4,85 @@ in its suites compared with Graphify on the same pinned checkouts. It is developer-side tooling: Compass never runs it, and it never installs Graphify. +Plan029 broad-question capture and qualification use the explicit +`python3 -m benchmarks.agent_query.broad_query_v2` entry point. Its source-frozen +original/lookup/union panels, exact printed-assertion review, immutable lineage, +cost accounting and report checking are documented in +[`BROAD_QUERY_CAPTURE.md`](BROAD_QUERY_CAPTURE.md). Diagnostic/self-test success +does not establish comparative superiority. The older broad-query commands and +the suites below retain their original protocols. + +The proposed joint graph-and-query claim additionally needs a paired graph +comparison; the existing query checker only requires a Compass production +graph audit. `broad_graph_statistics.paired_graph_coverage` provides bounded +diagnostic arithmetic for paired source-fact recall, retaining unknown rows, +declared clusters and language/relation strata. It distinguishes micro recall +from the equal-repository/cluster-weighted difference. It always reports +`qualificationEstablished=false`: it does not verify source-fact digests, +artifact provenance, independent judgments, emitted-edge precision, power or +the final joint decision rule. Run its synthetic controls with +`python3 -m unittest benchmarks.agent_query.tests.test_broad_graph_statistics`. + +`broad_graph_evidence.assess_graph_coverage` connects those diagnostics to a +versioned frozen source-fact registration, both tools' graph/build artifacts, +and a digest-bound observation ledger. It verifies source-byte spans, source +inventories, recorded tool/profile bytes, raw build streams and cited graph +record hashes; it rechecks containment and input bytes after scoring. Missing +reviews and failed builds remain unknown. A graph without a directed contract +cannot certify one; contract unavailability is reported separately from a +missing relationship. Hashes and record references do not establish semantic +agreement, runtime authenticity or independent review. This layer also always +reports `qualificationEstablished=false`; precision, power and final joint +eligibility are still required. Its controls use synthetic files only: +`python3 -m unittest benchmarks.agent_query.tests.test_broad_graph_evidence`. + +`broad_graph_precision.assess_graph_precision` adds both tools' emitted-edge +sampling and precision diagnostics. A separate design freezes the quota, seed +and versioned SHA-256-counter/Floyd sampling method before collection. The +sample is recomputed from the exact graph bytes; labels cannot add records +outside it. Unknown labels remain in the sample denominator, and unselected +records remain in the identified population bounds. Empty populations have +undefined precision. Sample precision, a population-size-weighted point +estimate, and whole-population census precision are separate fields. Recorded +relation strata remain visible even when unsampled. Source witnesses and exact +record hashes bind judgments to evidence; they do not authenticate truth or +independence. No population confidence interval, precision noninferiority +decision or qualification is issued. Existing production precision/sample +floors remain separate requirements. Run synthetic controls with +`python3 -m unittest benchmarks.agent_query.tests.test_broad_graph_precision`. + +Prospective precision design2 additionally requires +`populationBoundsProfile: "exact-hypergeometric-bonferroni/1"` and emits +diagnostics2 with `conditionalSamplingBounds`. Design1 retains diagnostics1 and +its original null interval fields. The new arithmetic inverts finite-population +hypergeometric tails with integer counts, allocating a shared1/20 error allowance +across the two tails of all nonempty tool/repository frames. It sums population-count bounds, +then forms a simultaneous candidate-minus-comparator bound. Rational endpoints +avoid floating-point rounding; unreviewed labels widen the bounds, and failed +builds withhold a tool's entire interval. Empty populations remain undefined. +This follows the test-inversion approach described in +[A Note on Confidence Intervals for Proportions in Finite Populations](https://doi.org/10.1080/00031305.1987.10475484). + +These are conditional calculations under ideal uniform sampling without +replacement and truthful labelled records. The hash sampler's reproducibility +does not prove that randomization model, semantic correctness or reviewer +independence. The design must be frozen before capture and bind the bounds +module's hash; existing frozen experiments cannot be retrofitted. No new capture +or evaluator is authorized by this feature. Design acceptance and qualification +remain false, and `populationConfidenceInterval`/`precisionComparisonInterval` +remain null: conditional arithmetic is not a certified inferential or joint +decision. Per-recorded-relation confidence bounds are not issued. No original +production, power or noninferiority threshold changes. + +The calculation retains the 64-frame/8-million-record/65,536-sampled-record +limits and at most4096 sampled records per graph. It charges combination factors and +summation terms before arithmetic against a shared8,192,000-work ceiling; this +is a logical work count, not a CPU-time or RSS measurement. Limits reject the +whole calculation. Run the exact small-population coverage and integration +controls with `python3 -m unittest +benchmarks.agent_query.tests.test_broad_graph_bounds +benchmarks.agent_query.tests.test_broad_graph_precision`. + [`COVERAGE_PLAN.md`](COVERAGE_PLAN.md) tracks the broader real-repository question/evidence matrix, including ask, communities, clusters, and god nodes. The audit report distinguishes completed checks from surfaces still awaiting diff --git a/benchmarks/agent_query/broad_graph_bounds.py b/benchmarks/agent_query/broad_graph_bounds.py new file mode 100644 index 000000000..3bfe65fcb --- /dev/null +++ b/benchmarks/agent_query/broad_graph_bounds.py @@ -0,0 +1,200 @@ +"""Conditional exact bounds for fixed finite edge populations. + +These invert hypergeometric tails under ideal uniform sampling without +replacement and truthful labelled records. The portable hash sampler does not +prove that randomization assumption, semantic truth or review independence. +No design acceptance, noninferiority or joint qualification is issued here. +""" +from fractions import Fraction +from math import comb + +from benchmarks.agent_query import broad_query_review as query + + +SCHEMA = "compass.broad-graph-conditional-bounds/1" +PROFILE = "exact-hypergeometric-bonferroni/1" +TOOLS = ("compass", "graphify") +MAX_ARITHMETIC_WORK = 8_192_000 +MAX_FRAMES = 64 +MAX_TOTAL_POPULATION = 8_000_000 +MAX_TOTAL_SAMPLE = 65_536 +FRAME_FIELDS = frozenset(("repository", "tool", "population", "sampleSize", + "correct", "incorrect", "buildSuccessful")) + + +class _Work: + def __init__(self): + self.used = 0 + + def charge(self, amount): + if self.used + amount > MAX_ARITHMETIC_WORK: + raise ValueError("exact graph bounds arithmetic work limit exceeded") + self.used += amount + + +def _range_sum(population, successes, sample, start, end, work): + if start > end: + return 0 + # Two initial binomial coefficients cost at most sample elementary factors. + # Subsequent weights use an exact integer recurrence, not repeated combs. + work.charge(sample + end - start + 1) + weight = comb(successes, start) * comb(population - successes, sample - start) + total = weight + for observed in range(start, end): + numerator = weight * (successes - observed) * (sample - observed) + denominator = (observed + 1) * (population - successes - sample + observed + 1) + weight, remainder = divmod(numerator, denominator) + if remainder: + raise ValueError("inexact hypergeometric weight recurrence") + total += weight + return total + + +def _tail(population, successes, sample, observed, denominator, upper, work): + low = max(0, sample - population + successes) + high = min(sample, successes) + if observed < low: + return denominator if upper else 0 + if observed > high: + return 0 if upper else denominator + left_end = observed - 1 if upper else observed + left_size = left_end - low + 1 + right_size = high - left_end + # Compute the shorter tail; subtracting integer counts loses no precision. + if left_size <= right_size: + left = _range_sum(population, successes, sample, low, left_end, work) if left_size else 0 + return denominator - left if upper else left + right = _range_sum(population, successes, sample, left_end + 1, high, work) + return right if upper else denominator - right + + +def count_bounds(population, sample, correct, incorrect, *, family_size=1, _work=None): + """Return integer population-success endpoints, or None for an empty frame. + + Each of the family's two tails spends exactly 1/(40*family_size), for a + simultaneous confidence level of at least19/20 under the stated model. + Unreviewed sampled records are completed adversarially in both directions; + no missing-at-random assumption or dropped denominator is used. + """ + query._integer(population, "bound population", 0, 4_000_000) + query._integer(sample, "bound sample", 0, min(population, 4096)) + query._integer(correct, "bound correct", 0, sample) + query._integer(incorrect, "bound incorrect", 0, sample - correct) + query._integer(family_size, "bound family size", 1, MAX_FRAMES) + if not population: + return None + if sample == population: + return correct, population - incorrect + if not sample or (not correct and not incorrect): + return 0, population + work = _Work() if _work is None else _work + work.charge(sample) + denominator = comb(population, sample) + alpha_denominator = 40 * family_size + + lower = 0 + if correct: + low, high = correct, population - sample + correct + while low < high: + middle = (low + high) // 2 + accepted = _tail(population, middle, sample, correct, denominator, True, work) + if accepted * alpha_denominator >= denominator: + high = middle + else: + low = middle + 1 + lower = low + + upper = population + if incorrect: + observed = sample - incorrect + low, high = observed, population - sample + observed + while low < high: + middle = (low + high + 1) // 2 + accepted = _tail(population, middle, sample, observed, denominator, False, work) + if accepted * alpha_denominator >= denominator: + low = middle + else: + high = middle - 1 + upper = low + return lower, upper + + +def _ratio(value): + return dict(numerator=value.numerator, denominator=value.denominator) + + +def _interval(lower, upper): + return dict(lower=_ratio(lower), upper=_ratio(upper)) + + +def population_bounds(frames): + """Aggregate exact count bounds using captured population sizes, not quotas. + + A failed build withholds that tool's entire population interval. The + difference uses simultaneous marginal endpoints; it assumes no pairing or + independence between the tools. It is a conservative conditional bound, + not a noninferiority decision or an accepted experimental design. + """ + query._array(frames, "bound frames", MAX_FRAMES) + checked, identities, repositories = [], set(), set() + population_total, sample_total = 0, 0 + for raw in frames: + frame = query._object(raw, "bound frame") + if set(frame) != FRAME_FIELDS: + raise ValueError("conditional bound frame fields differ from version1") + repository = query._text(frame["repository"], "bound repository", 128) + query._choice(frame["tool"], TOOLS, "bound tool") + key = repository, frame["tool"] + if key in identities: + raise ValueError("duplicate conditional bound frame") + identities.add(key) + repositories.add(repository) + n, sample = frame["population"], frame["sampleSize"] + query._integer(n, "bound population", 0, 4_000_000) + query._integer(sample, "bound sample", 0, min(n, 4096)) + query._integer(frame["correct"], "bound correct", 0, sample) + query._integer(frame["incorrect"], "bound incorrect", 0, sample - frame["correct"]) + query._boolean(frame["buildSuccessful"], "bound build status") + population_total += n + sample_total += sample + if population_total > MAX_TOTAL_POPULATION or sample_total > MAX_TOTAL_SAMPLE: + raise ValueError("aggregate conditional bound population/sample limit exceeded") + checked.append(dict(frame)) + if identities != {(repository, tool) for repository in repositories for tool in TOOLS}: + raise ValueError("conditional bounds require both tool frames for each repository") + checked.sort(key=lambda frame: (frame["repository"], frame["tool"])) + family_size = sum(frame["population"] > 0 for frame in checked) + work, counts, by_frame = _Work(), {}, [] + for frame in checked: + result = None + if frame["buildSuccessful"]: + result = count_bounds(frame["population"], frame["sampleSize"], + frame["correct"], frame["incorrect"], family_size=max(1, family_size), _work=work) + counts[frame["repository"], frame["tool"]] = result + by_frame.append(dict(repository=frame["repository"], tool=frame["tool"], + lowerCount=result[0] if result is not None else None, + upperCount=result[1] if result is not None else None)) + tools, exact = {}, {} + for tool in TOOLS: + selected = [frame for frame in checked if frame["tool"] == tool] + population = sum(frame["population"] for frame in selected) + interval = None + if population and all(frame["buildSuccessful"] for frame in selected): + lower = sum(counts[frame["repository"], tool][0] for frame in selected if frame["population"]) + upper = sum(counts[frame["repository"], tool][1] for frame in selected if frame["population"]) + interval = Fraction(lower, population), Fraction(upper, population) + exact[tool] = interval + tools[tool] = _interval(*interval) if interval is not None else None + difference = None + if all(exact[tool] is not None for tool in TOOLS): + candidate, comparator = (exact[tool] for tool in TOOLS) + difference = _interval(candidate[0] - comparator[1], candidate[1] - comparator[0]) + return dict(schema=SCHEMA, profile=PROFILE, familyConfidence=_ratio(Fraction(19, 20)), + familySize=family_size, frames=by_frame, tools=tools, difference=difference, + arithmeticWork=work.used, arithmeticWorkLimit=MAX_ARITHMETIC_WORK, + qualificationEstablished=False, registeredDesignAccepted=False, + semanticTruthEstablished=False, independenceEstablished=False, + conditionalOn="ideal uniform sampling without replacement and truthful labelled records", + limitations=["Hash-stream reproducibility does not prove ideal randomization or independent review.", + "Counts describe emitted records in the captured frames, not source-fact recall or future repositories.", + "Bonferroni aggregation is conservative; power and the final joint decision remain separate."]) diff --git a/benchmarks/agent_query/broad_graph_evidence.py b/benchmarks/agent_query/broad_graph_evidence.py new file mode 100644 index 000000000..f1db94047 --- /dev/null +++ b/benchmarks/agent_query/broad_graph_evidence.py @@ -0,0 +1,296 @@ +"""Artifact-bound graph-coverage diagnostics; never independent qualification. + +This checks recorded bytes and source anchors. Review truth, edge-to-source +semantics, runtime authenticity, precision and prospective power remain separate. +No product, graph producer, model service or competitor installation is invoked. +""" +from collections import Counter +from pathlib import Path +import json +import re +import stat + +from benchmarks.agent_query import broad_graph_statistics as statistics +from benchmarks.agent_query import broad_query_review as query +from benchmarks.agent_query import community_tasks + + +INPUT_SCHEMA = "compass.broad-graph-coverage-input/1" +REGISTRATION_SCHEMA = "compass.broad-graph-coverage-registration/1" +CAPTURE_SCHEMA = "compass.broad-graph-coverage-capture/1" +REVIEW_SCHEMA = "compass.broad-graph-coverage-reviews/1" +RESULT_SCHEMA = "compass.broad-graph-coverage-evidence/1" +MAX_SOURCE_FILES = 65_536 +MAX_SOURCE_BYTES = 256 * 1024 * 1024 +MAX_RECORD_PROOFS = 64 +MAX_RECORD_PROOFS_TOTAL = 262_144 +MAX_RECORD_BYTES_TOTAL = 128 * 1024 * 1024 +STATUSES = ("present", "missing", "ambiguous", "unavailable_contract", "unreviewed") + + +def _canonical_bytes(value): + return json.dumps(value, sort_keys=True, ensure_ascii=False, + allow_nan=False, separators=(",", ":")).encode() + + +def canonical_sha256(value): + return query.sha(_canonical_bytes(value)) + + +def support_hashes(): + return {name: query.sha(community_tasks.read_bounded(Path(path), query.MAX_JSON_BYTES)) + for name, path in [("evidence", __file__), ("statistics", statistics.__file__), + ("query", query.__file__), ("reader", community_tasks.__file__)]} + + +def _fields(value, expected, where): + value = query._object(value, where) + if set(value) != set(expected): + raise ValueError(f"{where} fields differ from the version1 contract") + return value + + +def _regular(path): + if not stat.S_ISREG(path.stat().st_mode): + raise ValueError("graph evidence inputs must be regular files") + + +class _Artifacts(query.Artifacts): + def read(self, ref, maximum=query.MAX_JSON_BYTES): + ref = query._object(ref, "graph artifact reference") + path = (self.root / query._relative(ref.get("path"))).resolve() + if not path.is_relative_to(self.root): + raise ValueError("graph artifact escapes evidence root") + _regular(path) + return super().read(ref, maximum) + + def recheck(self): + # Repeat containment/type checks as well as hashes after assessment. + for path, digest in sorted(self.seen.items()): + self.read(dict(path=path, sha256=digest), query.MAX_GRAPH_BYTES) + + +def _tool_identity(tool, artifacts): + _fields(tool, ("id", "version", "commit", "profile", "executable", "environment", "configuration"), "graph tool") + query._choice(tool["id"], statistics.TOOLS, "graph tool ID") + for field in ("version", "commit", "profile"): + query._text(tool[field], f"graph tool {field}", 128) + for field in ("executable", "environment", "configuration"): + artifacts.read(tool[field], query.MAX_GRAPH_BYTES if field == "executable" else query.MAX_JSON_BYTES) + return {field: tool[field] for field in ("id", "version", "commit", "profile")} | { + field: tool[field]["sha256"] for field in ("executable", "environment", "configuration")} + + +def _span(span, files, *, declaration=False): + expected = ("file", "startByte", "endByte", "sha256") + _fields(span, (*expected, "symbol", "owner") if declaration else expected, "source anchor") + file = query._relative(span["file"]) + if file not in files: + raise ValueError("source anchor is outside the frozen file inventory") + data = files[file] + start = query._integer(span["startByte"], "source start byte", 0, len(data)) + end = query._integer(span["endByte"], "source end byte", start + 1, len(data)) + if query.sha(data[start:end]) != query._digest(span["sha256"], "source anchor SHA-256"): + raise ValueError("source anchor digest differs from pinned source bytes") + if declaration: + query._text(span["symbol"], "source symbol", 512) + query._text(span["owner"], "source owner", 1024, empty=True) + return file, start, end + + +def _sources(repositories, roots): + if set(query._object(roots, "source roots")) != {r["id"] for r in repositories}: + raise ValueError("source-root matrix differs from registration") + contents, inventory, identities = {}, {}, {} + total, count = 0, 0 + for repo in repositories: + _fields(repo, ("id", "commit", "files"), "source repository") + name = query._text(repo["id"], "repository ID", 128) + if name in contents: + raise ValueError("duplicate source repository") + if not re.fullmatch(r"[0-9a-f]{40}", query._text(repo["commit"], "source commit", 40)): + raise ValueError("source commit must be an exact Git object ID") + root = Path(query._text(roots[name], "source root", 4096)).resolve() + files = {} + for item in query._array(repo["files"], "source file inventory", MAX_SOURCE_FILES, 1): + _fields(item, ("file", "sha256"), "source file") + file = query._relative(item["file"]) + if file in files: + raise ValueError("duplicate source inventory file") + path = (root / file).resolve() + if not path.is_relative_to(root): + raise ValueError("source file escapes registered root") + _regular(path) + data = community_tasks.read_bounded(path, query.MAX_SOURCE_BYTES) + total += len(data) + count += 1 + if total > MAX_SOURCE_BYTES or count > MAX_SOURCE_FILES: + raise ValueError("aggregate graph source inventory limit exceeded") + if query.sha(data) != query._digest(item["sha256"]): + raise ValueError("graph source file digest mismatch") + files[file] = data + identities[name, file] = (root, file, item["sha256"]) + contents[name] = files + inventory[name] = repo + return contents, inventory, identities + + +def assess_graph_coverage(manifest, root): + """Bind paired manual observations without turning attestations into truth. + + A positive observation needs an exact raw edge-record digest in its pinned + graph. That proves a referenced record exists, not that its relationship, + owner, direction or occurrence agrees with source. This deliberately remains + diagnostic until the independent semantic/precision and joint gates exist. + """ + _fields(manifest, ("schema", "registration", "capture", "reviews", "sourceRoots"), "graph coverage input") + if manifest["schema"] != INPUT_SCHEMA: + raise ValueError("unsupported graph coverage input schema") + artifacts = _Artifacts(Path(root)) + registration = artifacts.document(manifest["registration"]) + _fields(registration, ("schema", "frozenAt", "supportSha256", "repositories", "tools", "facts", "resamples", "seed"), "graph registration") + if registration["schema"] != REGISTRATION_SCHEMA or registration["supportSha256"] != support_hashes(): + raise ValueError("unsupported or changed graph coverage analysis protocol") + frozen = query._timestamp(registration["frozenAt"]) + repositories = query._array(registration["repositories"], "graph repositories", statistics.MAX_REPOSITORIES, 1) + sources, repository_map, source_paths = _sources(repositories, manifest["sourceRoots"]) + tool_map = {} + for tool in query._array(registration["tools"], "frozen graph tools", 2, 2): + identity = _tool_identity(tool, artifacts) + if identity["id"] in tool_map: + raise ValueError("duplicate frozen graph tool") + tool_map[identity["id"]] = identity + if set(tool_map) != set(statistics.TOOLS): + raise ValueError("both graph tools must be frozen") + facts, anchors = {}, set() + for item in query._array(registration["facts"], "frozen source facts", statistics.MAX_SOURCE_FACTS, 1): + _fields(item, (*statistics.TEXT_FIELDS, "fact"), "frozen source fact") + for field in statistics.TEXT_FIELDS: + query._text(item[field], f"source fact {field}", 128) + repo = query._choice(item["repository"], repository_map, "source fact repository") + fact = _fields(item["fact"], ("source", "target", "occurrence", "requiresDirection"), "source relationship") + query._boolean(fact["requiresDirection"], "required direction contract") + positions = tuple(_span(fact[field], sources[repo], declaration=field != "occurrence") + for field in ("source", "target", "occurrence")) + key = (repo, item["id"]) + # Changing labels/owners/IDs cannot manufacture a second physical unit. + physical = (repo, item["relation"], positions) + if key in facts or physical in anchors: + raise ValueError("duplicate graph source fact or physical relationship occurrence") + anchors.add(physical) + digest = canonical_sha256(dict(relation=item["relation"], **fact)) + facts[key] = dict(item, sourceFactSha256=digest) + capture = artifacts.document(manifest["capture"]) + _fields(capture, ("schema", "registrationSha256", "startedAt", "tools", "graphs"), "graph capture") + if capture["schema"] != CAPTURE_SCHEMA or capture["registrationSha256"] != manifest["registration"]["sha256"]: + raise ValueError("graph capture is not bound to the frozen registration") + started = query._timestamp(capture["startedAt"]) + if frozen >= started: + raise ValueError("graph registration must precede collection") + captured_tools = [_tool_identity(t, artifacts) for t in query._array(capture["tools"], "captured graph tools", 2, 2)] + if sorted(captured_tools, key=lambda t: t["id"]) != sorted(tool_map.values(), key=lambda t: t["id"]): + raise ValueError("captured tool bytes/profile differ from frozen identities") + graphs, build_failures = {}, [] + for entry in query._array(capture["graphs"], "captured graph matrix", 64, 2): + entry = query._object(entry, "captured graph") + key = (entry.get("repository"), entry.get("tool")) + if key in graphs or key[0] not in repository_map or key[1] not in tool_map: + raise ValueError("duplicate or unexpected captured graph") + repo = repository_map[key[0]] + if entry.get("sourceCommit") != repo["commit"] or entry.get("sourceCleanBeforeAndAfter") is not True: + raise ValueError("graph source pin or before/after clean attestation differs") + if artifacts.document(entry.get("sourceInventory")).get("files") != repo["files"]: + raise ValueError("graph build inventory differs from source registration") + build = artifacts.document(entry.get("buildCapture")) + build_ok = query.verify_build_capture(build, artifacts.read) + if not build_ok: + build_failures.append(dict(repository=key[0], tool=key[1], + exitCode=build["exit_code"], timedOut=build["timed_out"], + outputLimited=build["output_limited"], wallMs=build["wall_ms"])) + graph = artifacts.document(entry.get("artifact"), query.MAX_GRAPH_BYTES) + # Existing bounded graph validation checks IDs, endpoints and metadata; + # its name/site lookup is intentionally not used as a semantic oracle. + query.GraphIndex(graph, key[1]) + graphs[key] = (entry, graph, build_ok) + if set(graphs) != {(repo, tool) for repo in repository_map for tool in statistics.TOOLS}: + raise ValueError("captured graph matrix is incomplete") + reviews = artifacts.document(manifest["reviews"]) + _fields(reviews, ("schema", "captureSha256", "reviewedAt", "observations"), "graph reviews") + if reviews["schema"] != REVIEW_SCHEMA or reviews["captureSha256"] != manifest["capture"]["sha256"]: + raise ValueError("graph reviews differ from the captured graph set") + if query._timestamp(reviews["reviewedAt"]) < started: + raise ValueError("graph review predates collection") + observed, record_hashes = {}, {} + proof_count, record_bytes = 0, 0 + for item in query._array(reviews["observations"], "graph observations", 2 * statistics.MAX_SOURCE_FACTS): + _fields(item, ("repository", "factId", "tool", "sourceFactSha256", "graphSha256", "status", "reason", "edges"), "graph observation") + key = (item["repository"], item["factId"], item["tool"]) + fact_key, graph_key = key[:2], (key[0], key[2]) + if key in observed or fact_key not in facts or graph_key not in graphs: + raise ValueError("duplicate or unexpected graph observation") + fact = facts[fact_key] + entry, graph, build_ok = graphs[graph_key] + if item["sourceFactSha256"] != fact["sourceFactSha256"] or item["graphSha256"] != entry["artifact"]["sha256"]: + raise ValueError("graph observation source-fact or graph digest differs") + status = query._choice(item["status"], STATUSES, "graph observation status") + query._text(item["reason"], "graph observation rationale") + edges = query._array(item["edges"], "graph record proofs", MAX_RECORD_PROOFS) + proof_count += len(edges) + if proof_count > MAX_RECORD_PROOFS_TOTAL: + raise ValueError("aggregate graph record proof limit exceeded") + if status != "present" and edges: + raise ValueError("non-present graph observation cannot assert supporting edges") + if status == "present" and (not edges or not build_ok): + raise ValueError("present graph observation requires bounded successful build and record proof") + if status == "present" and fact["fact"]["requiresDirection"] and not graph["directed"]: + raise ValueError("undirected graph cannot certify a directed contract; report unavailable_contract") + seen = set() + records = graph.get("edges", graph.get("links", [])) + for proof in edges: + _fields(proof, ("index", "sha256"), "raw graph record proof") + index = query._integer(proof["index"], "raw edge index", 0, len(records) - 1) + record_key = (*graph_key, index) + if record_key not in record_hashes: + data = _canonical_bytes(records[index]) + record_bytes += len(data) + if record_bytes > MAX_RECORD_BYTES_TOTAL: + raise ValueError("aggregate canonical graph record byte limit exceeded") + record_hashes[record_key] = query.sha(data) + if index in seen or record_hashes[record_key] != query._digest(proof["sha256"]): + raise ValueError("raw graph record proof is duplicated or differs from captured bytes") + seen.add(index) + observed[key] = (status, build_ok) + rows, statuses = [], {tool: Counter() for tool in statistics.TOOLS} + review_statuses = {tool: Counter() for tool in statistics.TOOLS} + for (repo, identifier), fact in sorted(facts.items()): + row = {field: fact[field] for field in statistics.TEXT_FIELDS} + row["sourceFactSha256"] = fact["sourceFactSha256"] + for tool in statistics.TOOLS: + status, build_ok = observed.get((repo, identifier, tool), ("unreviewed", graphs[repo, tool][2])) + review_statuses[tool][status] += 1 + statuses[tool][status if build_ok else "build_failure"] += 1 + # An unsuccessful build cannot establish graph quality, including + # absence. Preserve the capture and withhold arithmetic for it. + row[tool] = None if not build_ok or status == "unreviewed" else status == "present" + rows.append(row) + result = statistics.paired_graph_coverage(rows, resamples=registration["resamples"], seed=registration["seed"]) + for source_root, file, digest in source_paths.values(): + path = (source_root / file).resolve() + if source_root.resolve() != source_root or not path.is_relative_to(source_root): + raise ValueError("source file escapes registered root during recheck") + _regular(path) + if query.sha(community_tasks.read_bounded(path, query.MAX_SOURCE_BYTES)) != digest: + raise ValueError("source file changed during graph assessment") + artifacts.recheck() + if registration["supportSha256"] != support_hashes(): + raise ValueError("analysis support files changed during graph assessment") + return dict(schema=RESULT_SCHEMA, artifactBindingsVerified=True, sourceByteAnchorsVerified=True, + qualificationEstablished=False, statistics=result, + statusCounts={tool: dict(sorted(counts.items())) for tool, counts in statuses.items()}, + reviewStatusCounts={tool: dict(sorted(counts.items())) for tool, counts in review_statuses.items()}, + buildFailures=sorted(build_failures, key=lambda item: (item["repository"], item["tool"])), + artifacts=dict(sorted(artifacts.seen.items())), + limitations=["Manual source/edge semantics and review independence are not authenticated.", + "A raw record reference does not prove owner, target, occurrence or provenance agreement.", + "Failed builds remain unknown here; end-to-end failure/cost accounting is still required.", + "Emitted-edge precision, production floors, prospective power and joint eligibility remain unimplemented."]) diff --git a/benchmarks/agent_query/broad_graph_precision.py b/benchmarks/agent_query/broad_graph_precision.py new file mode 100644 index 000000000..68f903b66 --- /dev/null +++ b/benchmarks/agent_query/broad_graph_precision.py @@ -0,0 +1,257 @@ +"""Reproducible emitted-edge sampling and retained-denominator diagnostics. + +Sampling does not authenticate source judgments or establish independent units. +Version1 issues no population interval. Prospective version2 adds exact bounds +conditional on an accepted ideal sampling model and truthful labels; it does +not accept that model or issue a precision NI/joint qualification decision. +""" +from collections import Counter +import hashlib +from pathlib import Path + +from benchmarks.agent_query import broad_graph_evidence as evidence +from benchmarks.agent_query import broad_graph_statistics as statistics +from benchmarks.agent_query import broad_graph_bounds as bounds +from benchmarks.agent_query import broad_query_review as query +from benchmarks.agent_query import runner, path_audit + + +INPUT_SCHEMA = "compass.broad-graph-precision-input/1" +DESIGN_SCHEMA = "compass.broad-graph-precision-design/1" +DESIGN_SCHEMA_V2 = "compass.broad-graph-precision-design/2" +REVIEW_SCHEMA = "compass.broad-graph-precision-reviews/1" +RESULT_SCHEMA = "compass.broad-graph-precision-diagnostics/1" +RESULT_SCHEMA_V2 = "compass.broad-graph-precision-diagnostics/2" +SAMPLER = "sha256-counter-floyd/1" +MAX_SAMPLED_EDGES = 65_536 +MAX_POPULATION_EDGES = 8_000_000 +MAX_RANDOM_DRAWS = 4 * MAX_SAMPLED_EDGES +MAX_REJECTIONS = 16 +MAX_RECORDED_RELATIONS = 128 +MAX_WITNESS_BYTES = 64 * 1024 * 1024 +ISSUES = ("owner_identity", "target_identity", "relation_kind", "direction", + "occurrence", "provenance", "fabricated_target") + + +def support_hashes(): + return {"coverage." + key: value for key, value in evidence.support_hashes().items()} | { + name: query.sha(query.read_bounded(Path(path), query.MAX_JSON_BYTES)) + for name, path in [("precision", __file__), ("runner", runner.__file__), + ("pathAudit", path_audit.__file__), + ("populationBounds", bounds.__file__)]} + + +def sample_indices(population, count, namespace, *, draw_budget=None): + """Floyd sample using a versioned, portable SHA-256 counter byte stream. + + Rejection removes modulo bias; a fixed resource limit rejects pathological + draws instead of changing the sampling rule. The pseudorandom stream and + source/sampling independence assumptions still require design review. + """ + query._integer(population, "edge population", 0, 4_000_000) + query._integer(count, "edge sample size", 0, min(population, 4096)) + if count == population: + return list(range(population)) + key = hashlib.sha256(evidence._canonical_bytes(namespace)).digest() + counter, selected = 0, set() + budget = [0] if draw_budget is None else draw_budget + for upper in range(population - count + 1, population + 1): + limit = (2**64 // upper) * upper + for _ in range(MAX_REJECTIONS): + budget[0] += 1 + if budget[0] > MAX_RANDOM_DRAWS: + raise ValueError("precision sampling random-draw limit exceeded") + value = int.from_bytes(hashlib.sha256(key + counter.to_bytes(8, "big")).digest()[:8], "big") + counter += 1 + if value < limit: + selected.add(upper - 1 if value % upper in selected else value % upper) + break + else: + raise ValueError("precision sampler rejection limit exceeded") + return sorted(selected) + + +def _view(frames, selected, labels): + population = sum(frame["population"] for frame in frames) + counts = Counter(labels.get(key, "unreviewed") for key in selected) + complete = bool(selected) and counts["unreviewed"] == 0 and all(f["buildSuccessful"] for f in frames) + known_population = all(f["buildSuccessful"] for f in frames) + correct, incorrect = counts["correct"], counts["incorrect"] + weighted = None + if population and known_population: + contributions = [] + for frame in frames: + if not frame["population"]: + continue + keys = frame["sampleKeys"] + if not keys or any(labels.get(key, "unreviewed") == "unreviewed" for key in keys): + break + contributions.append(frame["population"] * sum(labels[key] == "correct" for key in keys) / len(keys)) + else: + weighted = sum(contributions) / population + return dict(capturedEdges=population, populationEdges=population if known_population else None, + sampledEdges=len(selected), correct=correct, incorrect=incorrect, + unreviewed=counts["unreviewed"], unsampledEdges=population - len(selected), + samplePrecision=correct / len(selected) if complete else None, + populationPrecision=correct / population if complete and len(selected) == population else None, + populationWeightedPointEstimate=weighted, + identifiedPopulationPrecisionBounds=[correct / population, 1 - incorrect / population] + if population and known_population else None, + populationConfidenceInterval=None, independenceEstablished=False) + + +def assess_graph_precision(manifest, root): + evidence._fields(manifest, ("schema", "coverageInput", "design", "reviews"), "graph precision input") + if manifest["schema"] != INPUT_SCHEMA: + raise ValueError("unsupported graph precision input schema") + artifacts = evidence._Artifacts(Path(root)) + coverage_input = artifacts.document(manifest["coverageInput"]) + coverage = evidence.assess_graph_coverage(coverage_input, root) + registration = artifacts.document(coverage_input["registration"]) + capture = artifacts.document(coverage_input["capture"]) + design = query._object(artifacts.document(manifest["design"]), "precision design") + version2 = design.get("schema") == DESIGN_SCHEMA_V2 + fields = ("schema", "frozenAt", "coverageRegistrationSha256", "supportSha256", "sampler", "seed", "quotaPerGraph") + evidence._fields(design, (*fields, "populationBoundsProfile") if version2 else fields, "precision design") + if (design["schema"] not in (DESIGN_SCHEMA, DESIGN_SCHEMA_V2) + or design["sampler"] != SAMPLER or design["supportSha256"] != support_hashes() + or (version2 and design["populationBoundsProfile"] != bounds.PROFILE)): + raise ValueError("unsupported or changed graph precision sampling protocol") + if design["coverageRegistrationSha256"] != coverage_input["registration"]["sha256"]: + raise ValueError("precision design differs from frozen graph source registration") + if not (query._timestamp(registration["frozenAt"]) <= query._timestamp(design["frozenAt"]) + < query._timestamp(capture["startedAt"])): + raise ValueError("precision sampling must be frozen after source registration and before collection") + seed = query._integer(design["seed"], "precision sample seed", 0, 2**32 - 1) + quota = query._integer(design["quotaPerGraph"], "predeclared edge quota", 1, 4096) + source_files, _, source_paths = evidence._sources(registration["repositories"], coverage_input["sourceRoots"]) + failed = {(entry["repository"], entry["tool"]) for entry in coverage["buildFailures"]} + frames, plan, relations = {}, {}, set() + draws, record_bytes, population_total = [0], 0, 0 + for entry in capture["graphs"]: + key = (entry["repository"], entry["tool"]) + graph = artifacts.document(entry["artifact"], query.MAX_GRAPH_BYTES) + records = query._array(graph.get("edges", graph.get("links", [])), "captured edge population", 4_000_000) + population_total += len(records) + if population_total > MAX_POPULATION_EDGES: + raise ValueError("aggregate precision edge-population limit exceeded") + indices = sample_indices(len(records), min(quota, len(records)), + dict(seed=seed, repository=key[0], tool=key[1], graphSha256=entry["artifact"]["sha256"]), draw_budget=draws) + if len(plan) + len(indices) > MAX_SAMPLED_EDGES: + raise ValueError("aggregate precision sample limit exceeded") + kind_counts = Counter() + for record in records: + kind = query._text(record.get("kind", record.get("relation")), "recorded edge relation", 128) + kind_counts[kind] += 1 + relations.add(kind) + if len(relations) > MAX_RECORDED_RELATIONS: + raise ValueError("precision recorded-relation limit exceeded") + keys = [] + for index in indices: + data = evidence._canonical_bytes(records[index]) + record_bytes += len(data) + if record_bytes > evidence.MAX_RECORD_BYTES_TOTAL: + raise ValueError("precision canonical-record byte limit exceeded") + sample_key = (*key, index) + kind = records[index].get("kind", records[index].get("relation")) + plan[sample_key] = dict(repository=key[0], tool=key[1], index=index, + graphSha256=entry["artifact"]["sha256"], recordSha256=query.sha(data), + recordedRelation=kind, directed=graph["directed"]) + keys.append(sample_key) + frames[key] = dict(repository=key[0], tool=key[1], population=len(records), + sampleKeys=keys, byRecordedRelation=kind_counts, buildSuccessful=key not in failed) + reviews = artifacts.document(manifest["reviews"]) + evidence._fields(reviews, ("schema", "coverageCaptureSha256", "designSha256", "reviewedAt", "observations"), "precision reviews") + if (reviews["schema"] != REVIEW_SCHEMA or reviews["coverageCaptureSha256"] != coverage_input["capture"]["sha256"] + or reviews["designSha256"] != manifest["design"]["sha256"]): + raise ValueError("precision review is not bound to its capture and sampling design") + if query._timestamp(reviews["reviewedAt"]) < query._timestamp(capture["startedAt"]): + raise ValueError("precision review predates collection") + labels, witness_cache, witness_bytes = {}, set(), 0 + issues = {tool: Counter() for tool in statistics.TOOLS} + for item in query._array(reviews["observations"], "sampled edge observations", MAX_SAMPLED_EDGES): + evidence._fields(item, ("repository", "tool", "index", "recordSha256", "verdict", "issues", "reason", "witnesses"), "precision observation") + query._text(item["repository"], "precision repository", 128) + query._choice(item["tool"], statistics.TOOLS, "precision tool") + query._integer(item["index"], "sampled edge index", 0, 3_999_999) + key = (item["repository"], item["tool"], item["index"]) + if key in labels or key not in plan: + raise ValueError("duplicate or non-sampled precision observation") + if query._digest(item["recordSha256"]) != plan[key]["recordSha256"]: + raise ValueError("precision observation differs from its captured raw record") + verdict = query._choice(item["verdict"], ("correct", "incorrect", "unreviewed"), "edge verdict") + query._text(item["reason"], "source precision rationale") + problems = query._array(item["issues"], "edge semantic issues", len(ISSUES)) + for issue in problems: + query._choice(issue, ISSUES, "edge semantic issue") + query._unique(problems, "edge semantic issues") + if (verdict == "incorrect") != bool(problems): + raise ValueError("only an incorrect edge verdict may name semantic issues") + if "direction" in problems and not plan[key]["directed"]: + raise ValueError("undirected contract cannot be scored as an invented direction error") + witnesses = query._array(item["witnesses"], "precision source witnesses", 16, 0 if verdict == "unreviewed" else 1) + for witness in witnesses: + evidence._fields(witness, ("file", "startByte", "endByte", "sha256"), "precision source witness") + file = query._relative(witness["file"]) + data = source_files[key[0]].get(file) + if data is None: + raise ValueError("precision source witness is not inventoried") + start = query._integer(witness["startByte"], "precision witness start", 0, len(data)) + end = query._integer(witness["endByte"], "precision witness end", start + 1, len(data)) + digest = query._digest(witness["sha256"]) + witness_key = (key[0], file, start, end, digest) + if witness_key not in witness_cache: + witness_bytes += end - start + if witness_bytes > MAX_WITNESS_BYTES: + raise ValueError("aggregate precision witness-byte limit exceeded") + evidence._span(witness, source_files[key[0]]) + witness_cache.add(witness_key) + labels[key] = verdict + issues[key[1]].update(problems) + by_tool = {} + for tool in statistics.TOOLS: + selected_frames = [frame for key, frame in sorted(frames.items()) if key[1] == tool] + selected_keys = [key for key in sorted(plan) if key[1] == tool] + by_repo = {frame["repository"]: _view([frame], frame["sampleKeys"], labels) for frame in selected_frames} + by_kind = {} + for kind in sorted(relations): + kind_frames = [dict(frame, population=frame["byRecordedRelation"].get(kind, 0), + sampleKeys=[key for key in frame["sampleKeys"] if plan[key]["recordedRelation"] == kind]) for frame in selected_frames] + kind_keys = [key for key in selected_keys if plan[key]["recordedRelation"] == kind] + by_kind[kind] = _view(kind_frames, kind_keys, labels) + by_tool[tool] = dict(overall=_view(selected_frames, selected_keys, labels), + byRepository=by_repo, byRecordedRelation=by_kind, + issueCounts=dict(sorted(issues[tool].items()))) + conditional = None + if version2: + conditional = bounds.population_bounds([ + dict(repository=frame["repository"], tool=frame["tool"], + population=frame["population"], sampleSize=len(frame["sampleKeys"]), + correct=sum(labels.get(key) == "correct" for key in frame["sampleKeys"]), + incorrect=sum(labels.get(key) == "incorrect" for key in frame["sampleKeys"]), + buildSuccessful=frame["buildSuccessful"]) + for _key, frame in sorted(frames.items())]) + for source_root, file, digest in source_paths.values(): + path = (source_root / file).resolve() + if source_root.resolve() != source_root or not path.is_relative_to(source_root): + raise ValueError("precision source witness escapes registered root during recheck") + evidence._regular(path) + if query.sha(query.read_bounded(path, query.MAX_SOURCE_BYTES)) != digest: + raise ValueError("precision source witness file changed during assessment") + artifacts.recheck() + if design["supportSha256"] != support_hashes(): + raise ValueError("precision support changed during assessment") + result = dict(schema=RESULT_SCHEMA_V2 if version2 else RESULT_SCHEMA, qualificationEstablished=False, + coverage=coverage, precision=by_tool, samplePlan=[plan[key] for key in sorted(plan)], + precisionComparisonInterval=None, sampling=dict(method=SAMPLER, quotaPerGraph=quota, + seed=seed, selectedRecords=len(plan), randomDraws=draws[0]), + artifacts=dict(sorted(artifacts.seen.items())), + limitations=["Recorded labels and source witnesses do not authenticate semantic truth or review independence.", + "Sample precision and a population-weighted point estimate are not population confidence bounds.", + "Unselected and unreviewed records remain in population bounds; empty populations have undefined precision.", + "Production qualification floors, prospective sampling/power acceptance and final joint eligibility remain required."]) + if version2: + result["conditionalSamplingBounds"] = conditional + result["limitations"].append( + "Exact sampling bounds are conditional arithmetic, not an accepted sampling design, semantic review or certified precision interval.") + return result diff --git a/benchmarks/agent_query/broad_graph_statistics.py b/benchmarks/agent_query/broad_graph_statistics.py new file mode 100644 index 000000000..8de3db3ca --- /dev/null +++ b/benchmarks/agent_query/broad_graph_statistics.py @@ -0,0 +1,125 @@ +"""Bounded paired source-fact coverage arithmetic, not graph qualification. + +Rows must come from a separately frozen, source-authored panel and reviewed +graph observations. This module cannot establish their semantics, provenance, +independence or precision. A positive interval never authorizes a quality claim. +""" +from collections import defaultdict + +from benchmarks.agent_query import broad_query_review as query + + +STATISTICS_SCHEMA = "compass.broad-graph-coverage-statistics/1" +MAX_SOURCE_FACTS = 65_536 +MAX_DECLARED_CLUSTERS = 8_192 +MAX_REPOSITORIES = 32 +MAX_STRATUM_VALUES = 128 +MAX_IDENTIFIER_BYTES = 16 * 1024 * 1024 +MAX_RESAMPLING_WORK = 8_192_000 +TOOLS = ("compass", "graphify") +TEXT_FIELDS = ("id", "repository", "cluster", "language", "relation") +ROW_FIELDS = frozenset((*TEXT_FIELDS, "sourceFactSha256", *TOOLS)) + + +def _counts(rows): + result = {} + for tool in TOOLS: + recovered = sum(row[tool] is True for row in rows) + missing = sum(row[tool] is False for row in rows) + unknown = len(rows) - recovered - missing + result[tool] = dict(sourceFacts=len(rows), recovered=recovered, + missing=missing, unknown=unknown, + microRecall=recovered / len(rows) if rows and not unknown else None) + return result + + +def paired_graph_coverage(rows, *, resamples=2000, seed=29): + """Keep unknown pairs and distinguish micro recall from the cluster estimand. + + Each row identifies one canonical source fact, not one emitted graph edge. + Repeated occurrences need distinct source-fact digests. Aliases/paraphrases + of one fact must not be submitted as additional units. Digest correctness + and appropriate cross-fact clustering require upstream source review. + """ + query._array(rows, "paired graph source facts", MAX_SOURCE_FACTS) + query._integer(resamples, "graph bootstrap resamples", 1000, 10_000) + query._integer(seed, "graph bootstrap seed", 0, 2**32 - 1) + validated, identifiers, facts, clusters, repositories = [], set(), set(), set(), set() + strata_keys = {field: set() for field in ("language", "relation")} + text_bytes = 0 + for raw in rows: + row = query._object(raw, "paired graph source fact") + if set(row) != ROW_FIELDS: + raise ValueError("paired graph source fact fields differ from the version1 contract") + for field in TEXT_FIELDS: + value = query._text(row[field], f"graph {field}", 128) + text_bytes += len(value.encode()) + digest = query._digest(row["sourceFactSha256"], "canonical source-fact SHA-256") + text_bytes += len(digest) + if text_bytes > MAX_IDENTIFIER_BYTES: + raise ValueError("paired graph identifier byte limit exceeded") + for tool in TOOLS: + if row[tool] is not None: + query._boolean(row[tool], f"{tool} recovered source fact") + key = (row["repository"], row["id"]) + fact = (row["repository"], digest) + if key in identifiers or fact in facts: + raise ValueError("duplicate graph source-fact ID or canonical digest") + identifiers.add(key) + facts.add(fact) + clusters.add((row["repository"], row["cluster"])) + repositories.add(row["repository"]) + if len(clusters) > MAX_DECLARED_CLUSTERS or len(repositories) > MAX_REPOSITORIES: + raise ValueError("paired graph cluster or repository limit exceeded") + for field, keys in strata_keys.items(): + keys.add(row[field]) + if len(keys) > MAX_STRATUM_VALUES: + raise ValueError("paired graph language or relation stratum limit exceeded") + validated.append(dict(row)) + # Bound the inherited bootstrap's repository and cluster draws before it + # allocates/resamples. Source-fact rows are grouped by that implementation. + cluster_counts = defaultdict(int) + for repository, _cluster in clusters: + cluster_counts[repository] += 1 + # A bootstrap can choose the largest repository at every outer draw. Sum + # of cluster counts alone underestimates work for an unbalanced panel. + maximum_draws = resamples * len(repositories) * (max(cluster_counts.values(), default=0) + 1) + if maximum_draws > MAX_RESAMPLING_WORK: + raise ValueError("paired graph resampling work limit exceeded") + validated.sort(key=lambda row: (row["repository"], row["cluster"], row["id"])) + missing_pairs = sum(any(row[tool] is None for tool in TOOLS) for row in validated) + complete = bool(validated) and missing_pairs == 0 + if complete: + paired = query.paired_interval([ + (row["repository"], row["cluster"], float(row["compass"]), float(row["graphify"])) + for row in validated + ], samples=resamples, seed=seed) + else: + sizes = defaultdict(set) + for row in validated: + sizes[row["repository"]].add(row["cluster"]) + paired = dict(estimate=None, lower=None, upper=None, + repositories=len(repositories), clusters=len(clusters), + balancedIndependentClusters=bool(sizes) and len({len(v) for v in sizes.values()}) == 1, + method="paired_repository_then_task_cluster_bootstrap/1", + reason="Empty source panel or unknown graph observations; no imputation or row exclusion.") + strata = {} + for field in ("language", "relation"): + groups = defaultdict(list) + for row in validated: + groups[row[field]].append(row) + strata[field] = {key: _counts(group) for key, group in sorted(groups.items())} + return dict(schema=STATISTICS_SCHEMA, rowsRetained=len(validated), + missingPairRows=missing_pairs, completePairs=complete, + declaredRepositories=len(repositories), declaredClusters=len(clusters), + coverage=_counts(validated), strata=strata, + pairedClusterWeightedRecallDifference=paired, + qualificationEstablished=False, + interpretation="Diagnostic arithmetic only. Micro recall and equal-repository/cluster-weighted paired recall are different estimands. Declared clusters do not prove independence.", + missingQualificationEvidence=[ + "independent source/graph judgments and source-fact digest verification", + "frozen source, graph, tool, configuration and capture bindings", + "symmetric emitted-edge precision and existing production graph floors", + "prospective power, sampling, strata and stopping-rule acceptance", + "paired graph decision rule and original/lookup/union query gates", + ]) diff --git a/benchmarks/agent_query/broad_query_actors_v2.py b/benchmarks/agent_query/broad_query_actors_v2.py new file mode 100644 index 000000000..32049fab9 --- /dev/null +++ b/benchmarks/agent_query/broad_query_actors_v2.py @@ -0,0 +1,167 @@ +"""Freeze the complete owned evaluator source set for protocol2 experiments. + +This checks supplied code-byte integrity, not reviewer independence or source +truth. It does not dispatch capture/assessment or authorize an experiment. +""" +import json +import os +from pathlib import Path + +from benchmarks.agent_query.broad_query_review import ( + MAX_JSON_BYTES, _digest, _json, _relative, read_bounded, sha, +) + +SCHEMA = "compass.broad-query-protocol-source/2" +BINDING_SCHEMA = "compass.broad-query-protocol-binding/2" +ROOTS = ("benchmarks/agent_query", "benchmarks/performance/compass") +ROLES = dict(capture="benchmarks/agent_query/broad_query_v2.py", + normalizer="benchmarks/agent_query/broad_query_normalize_v2.py", + assessor="benchmarks/agent_query/broad_query_assess_v2.py", + panel="benchmarks/agent_query/broad_query_panels_v2.py", + reporter="benchmarks/agent_query/broad_query_report_v2.py") +MAX_FILES = 512 +MAX_BYTES = MAX_JSON_BYTES +MAX_TREE_ENTRIES = 16384 +MAX_TREE_DIRECTORIES = 512 + + +def _encoded(document): + data = (json.dumps(document, indent=2, sort_keys=True, allow_nan=False) + "\n").encode() + if len(data) > MAX_JSON_BYTES: + raise ValueError("protocol source descriptor exceeds JSON byte limit") + return data + + +def _module(value): + relative = _relative(value) + path = Path(relative) + if (not relative.endswith(".py") or "tests" in path.parts + or not any(relative.startswith(root + "/") for root in ROOTS)): + raise ValueError("protocol actor is outside the owned evaluator source set") + return relative + + +def _descriptor(document): + if (type(document) is not dict or document.keys() != {"schema", "roles", "files"} + or document["schema"] != SCHEMA or document["roles"] != ROLES + or type(document["files"]) is not list or not 1 <= len(document["files"]) <= MAX_FILES): + raise ValueError("invalid protocol source descriptor") + files, names = {}, [] + for item in document["files"]: + if type(item) is not dict or item.keys() != {"module", "sha256"}: + raise ValueError("invalid protocol source file record") + module = _module(item["module"]) + if module in files: + raise ValueError("duplicate protocol source module") + files[module] = _digest(item["sha256"]) + names.append(module) + if names != sorted(names) or not set(ROLES.values()) <= files.keys(): + raise ValueError("protocol source order or required actor roles differ") + actors = {role: files[module] for role, module in ROLES.items()} + actors["protocol"] = sha(_encoded(document)) + return files, dict(sorted(actors.items())) + + +def freeze_protocol(root): + """Hash all non-test Python files under both fixed evaluator source roots. + + The descriptor contains file digests, not source text. Freezing the whole + source set includes imported helpers and graph auditors; adding/removing + even an unused owned source file requires a new pre-output freeze. + """ + root = Path(root).resolve(strict=True) + paths, pending = [], [] + for directory in ROOTS: + base = root / directory + if not base.is_dir() or base.resolve() != base: + raise ValueError("protocol source root is absent or follows a symlink") + pending.append(base) + entries, directories = 0, len(pending) + if directories > MAX_TREE_DIRECTORIES: + raise ValueError("protocol source directory limit exceeded") + while pending: + with os.scandir(pending.pop()) as children: + for child in children: + entries += 1 + if entries > MAX_TREE_ENTRIES: + raise ValueError("protocol source tree entry limit exceeded") + if child.name == "tests" and child.is_dir(follow_symlinks=False): + continue + if child.is_symlink(): + raise ValueError("protocol source tree cannot contain symlinks") + path = Path(child.path) + if child.is_dir(follow_symlinks=False): + if directories >= MAX_TREE_DIRECTORIES: + raise ValueError("protocol source directory limit exceeded") + directories += 1 + pending.append(path) + elif child.name.endswith(".py"): + if len(paths) >= MAX_FILES: + raise ValueError("protocol source file count exceeds512") + if path.resolve() != path or not child.is_file(follow_symlinks=False): + raise ValueError("protocol source must be a regular contained file") + paths.append((path.relative_to(root).as_posix(), path)) + files, total = [], 0 + for relative, path in sorted(paths): + data = read_bounded(path, MAX_BYTES - total) + total += len(data) + files.append(dict(module=_module(relative), sha256=sha(data))) + descriptor = dict(schema=SCHEMA, roles=dict(ROLES), files=files) + _, actors = _descriptor(descriptor) + return descriptor, actors + + +def verify_original_protocol(root, descriptor, actors): + _, expected = _descriptor(descriptor) + if type(actors) is not dict or actors != expected: + raise ValueError("frozen actor tuple differs from protocol source descriptor") + actual_descriptor, actual_actors = freeze_protocol(root) + if actual_descriptor != descriptor or actual_actors != actors: + raise ValueError("original evaluator source inventory or bytes drifted") + + +def retain_protocol(bundle, root, descriptor, actors): + """Retain hash-bound code copies using the existing no-clobber bundle.""" + verify_original_protocol(root, descriptor, actors) + copies = [] + for item in descriptor["files"]: + module = item["module"] + reference = bundle.copy("provenance/protocol/" + module, + dict(path=str(Path(root).resolve() / module), sha256=item["sha256"])) + copies.append(dict(module=module, artifact=reference)) + manifest = bundle.put("provenance/protocol-source.json", descriptor) + if manifest["sha256"] != actors["protocol"]: + raise ValueError("retained protocol descriptor encoding differs") + verify_original_protocol(root, descriptor, actors) + return dict(schema=BINDING_SCHEMA, protocolManifest=manifest, files=copies, actorSha256=actors) + + +def verify_retained_protocol(binding, artifacts, root): + """Verify retained and original actors with one shared bounded reader. + + The runtime caller supplies its own product code root, never a root selected + from an untrusted capture/manifest. Call before assessment and publication. + The caller additionally + rechecks all raw/source/run/normalization artifacts with this same reader. + """ + if (type(binding) is not dict or binding.keys() != {"schema", "protocolManifest", "files", "actorSha256"} + or binding["schema"] != BINDING_SCHEMA): + raise ValueError("invalid retained protocol binding") + descriptor = _json(artifacts.read(binding["protocolManifest"])) + files, actors = _descriptor(descriptor) + if binding["actorSha256"] != actors or binding["protocolManifest"]["sha256"] != actors["protocol"]: + raise ValueError("retained evaluator actor tuple differs") + copies = binding["files"] + if type(copies) is not list or len(copies) != len(files): + raise ValueError("retained protocol source partition differs") + verify_original_protocol(root, descriptor, actors) + total = 0 + for copy, (module, digest) in zip(copies, files.items()): + if (type(copy) is not dict or copy.keys() != {"module", "artifact"} + or copy["module"] != module or type(copy["artifact"]) is not dict + or copy["artifact"].get("sha256") != digest): + raise ValueError("retained protocol module order or digest differs") + data = artifacts.read(copy["artifact"], MAX_BYTES - total) + total += len(data) + verify_original_protocol(root, descriptor, actors) + return descriptor, actors diff --git a/benchmarks/agent_query/broad_query_assess_v2.py b/benchmarks/agent_query/broad_query_assess_v2.py new file mode 100644 index 000000000..fca447a2c --- /dev/null +++ b/benchmarks/agent_query/broad_query_assess_v2.py @@ -0,0 +1,147 @@ +"""File-backed protocol2 assessment for the explicit developer entry point. + +No graph query, provider or network is run. Bounded Git metadata checks verify +current source pins/cleanliness. Supplied review attestations remain explicit +inputs, not authenticated human/source judgments. +""" +from pathlib import Path +from tempfile import TemporaryDirectory +from contextlib import contextmanager + +from benchmarks.agent_query import broad_query_review as old +from benchmarks.agent_query.broad_query_inputs_v2 import BoundedArtifacts, verify_native_inputs +from benchmarks.agent_query.broad_query_lineage_v2 import load_review_lineage +from benchmarks.agent_query.broad_query_observation_v2 import score_bound_observation +from benchmarks.agent_query.broad_query_oracle import SourceReader +from benchmarks.agent_query.broad_query_protocol import BundleWork +from benchmarks.agent_query.broad_query_panels_v2 import read_panels, finish_panels + + +def _source_state(registration, roots, artifacts, spool): + reader = SourceReader(artifacts.root, spool) + for repository in registration["repositories"]: + root = Path(roots[repository["id"]]).resolve(strict=True) + options = ("--no-optional-locks", "-c", "core.fsmonitor=false", "-c", "core.untrackedCache=false") + if reader.git(root, *options, "rev-parse", "HEAD").strip() != repository["commit"]: + raise ValueError("source HEAD differs from the frozen source pin") + if reader.git(root, *options, "status", "--porcelain", "--untracked-files=all").strip(): + raise ValueError("source checkout is not clean") + + +@contextmanager +def _assessment_session(manifest, root): + """Assess the full frozen matrix and recheck every input before returning. + + All row payloads use one reader and are discarded after scoring. Every + required row remains in summaries/comparisons, including failed executions + and normalization rejections. Calling this internal function does not + authorize a comparison or prove reviewer independence. + """ + artifacts = BoundedArtifacts(Path(root)) + code_root = Path(__file__).resolve().parents[2] + lineage = load_review_lineage(manifest, artifacts, code_root) + registration, run = lineage.registration, lineage.run + roots = old._object(manifest.get("sourceRoots"), "source roots") + if roots.keys() != {repo["id"] for repo in registration["repositories"]}: + raise ValueError("source roots differ from the frozen source matrix") + for value in roots.values(): + if not Path(old._text(value, "absolute source root")).is_absolute(): + raise ValueError("source roots must be absolute") + # The evidence root must be outside the product and every source checkout; + # the temporary bounded Git captures live alongside it and are removed. + protected = [code_root, *(Path(value).resolve() for value in roots.values())] + if any(artifacts.root.is_relative_to(path) or path.is_relative_to(artifacts.root) for path in protected): + raise ValueError("assessment artifacts must be external to product and source checkouts") + with TemporaryDirectory(prefix="compass-review-v2-", dir=artifacts.root.parent) as directory: + spool = Path(directory) + if any(spool.is_relative_to(path) for path in protected): + raise ValueError("temporary audit captures must be outside source checkouts") + _source_state(registration, roots, artifacts, spool) + originals = verify_native_inputs(manifest, registration, run, artifacts) + panels = read_panels(registration, run.get("panels"), artifacts.read, run["protocolBinding"]["actorSha256"]) + started = old._timestamp(run["startedAt"]) + state = old._prepare_assessment(manifest, registration, run, artifacts, started, + actors=run["protocolBinding"]["actorSha256"]) + tasks, failures = state["tasks"], state["failures"] + if run.get("captureInputInvocation") is None: + failures.append("capture-input invocation file was not retained; object snapshots are diagnostic only") + if panels is None: + failures.append("source-frozen original/lookup comparison panels are missing") + graphs = {(entry["repository"], entry["tool"]): entry for entry in state["graph_records"]} + work = BundleWork() + scored, costs, reviewers, row_work, row_failures, reviewer_by_key = {}, [], set(), [], {}, {} + for observation in lineage.iter_observations(): + key = observation.key + task = tasks[key[0]] + review = observation.judgment.get("independentReview") + local_failures = old._review_failures(review, f"{key}: output normalization/facts") + reviewer_by_key[key] = str(review.get("reviewerType")) if type(review) is dict else "None" + reviewers.add(reviewer_by_key[key]) + graph_key = task["repository"], key[1] + result = score_bound_observation(observation, task, registration["capturePlan"], + graphs[graph_key], artifacts.root, state["graph_indexes"][graph_key], + state["sources"][task["repository"]], work) + row = result["row"] + if not row["normalizationComplete"] or not row["semanticCoverageComplete"]: + local_failures.append(f"{key}: output normalization/semantic coverage incomplete") + if not row["judgmentComplete"]: + local_failures.append(f"{key}: unjudged output or post-output source-oracle correction") + if row["evaluationRejected"]: + local_failures.append(f"{key}: evaluation rejected: {row['evaluationFailure']['code']}") + if not result["measurementComplete"]: + local_failures.append(f"{key}: peak RSS measurement unavailable") + row_failures[key] = local_failures + failures.extend(local_failures) + scored[key] = row + costs.append(result["cost"]) + row_work.append(dict(task=key[0], tool=key[1], arm=key[2], **result["work"])) + del observation + expected = {(task, tool, arm) for task in tasks for tool in old.TOOLS for arm in old.ARMS} + if scored.keys() != expected: + raise ValueError("assessment did not retain the full frozen matrix") + report = old._finish_assessment(manifest, registration, artifacts, started, state, + scored, costs, sorted(reviewers), protocol2=True) + # Panel source/tool/graph verification is sequential. Release the union + # indexes before creating another panel's indexes; scored rows remain. + state["graph_indexes"].clear() + state["sources"].clear() + if panels is not None: + combined = finish_panels(panels, manifest, registration, run, artifacts, started, + scored, costs, row_failures, reviewer_by_key, report) + report["panelAssessment"] = combined + report["eligibleForSuperiorityClaim"] = combined["eligibleForSuperiorityClaim"] + report["failures"] = combined["failures"] + report.update(protocol="compass.broad-query-review/2", actorSha256=run["protocolBinding"]["actorSha256"], + printedWork=dict(completedHashBytes=work.hash_bytes, completedLineLinks=work.line_links, rows=row_work), + sourceStateVerified=True, artifacts=dict(sorted(artifacts.seen.items()))) + guarded = False + + def recheck(): + nonlocal guarded + _source_state(registration, roots, artifacts, spool) + old._verify_sources(registration, roots) + originals.recheck() + lineage.recheck() + guarded = True + + # A publisher may encode first, invoke recheck immediately before its + # exclusive write, then leave the session without a second late check. + # Plain assessment receives its final check before returning a result. + yield report, recheck + if not guarded: + recheck() + + +def assess_v2(manifest, root): + with _assessment_session(manifest, root) as (report, _): + return report + + +def assess_file_v2(path): + """Bind the actual input file's bytes before and after full assessment.""" + path = Path(path).absolute() + data = old.read_bounded(path, old.MAX_JSON_BYTES) + report = assess_v2(old._json(data), path.parent) + if old.read_bounded(path, old.MAX_JSON_BYTES) != data: + raise ValueError("review-input manifest changed during assessment") + return report diff --git a/benchmarks/agent_query/broad_query_build_spec.json b/benchmarks/agent_query/broad_query_build_spec.json new file mode 100644 index 000000000..220fb2f41 --- /dev/null +++ b/benchmarks/agent_query/broad_query_build_spec.json @@ -0,0 +1,145 @@ +{ + "schema": "compass.broad-query-native-build-spec/1", + "status": "development_template_exposed_sources", + "scope": "These original8 sources and extra16 controls are exposed development inputs. Preserve their mandatory regression outcomes; this template does not define fresh confirmation data or establish graph/query superiority.", + "qualificationEligible": false, + "sourceRegistrationSha256": "9c5859dcb23b7aa29fdc0aa5e4610b48217da59c737ad81de746beb19f807d5c", + "graphQualificationRepositories": [ + "itsdangerous", + "marshmallow", + "httprouter", + "go-cache", + "log", + "humantime", + "p-queue", + "superjson" + ], + "sources": [ + { + "id": "itsdangerous", + "url": "https://github.com/pallets/itsdangerous", + "commit": "672971d66a2ef9f85151e53283113f33d642dabd", + "inventorySha256": "bba10a0daa9dce04d9ec598bb12fbeb4fa4f52878cd00c391035fd6065fc8d89" + }, + { + "id": "marshmallow", + "url": "https://github.com/marshmallow-code/marshmallow", + "commit": "d08471b17880c62b789971ada638707a11d3f8d4", + "inventorySha256": "65fea45e4d8294b46df43bfa8458594fc9442443cd70ed9c856876e3415286e0" + }, + { + "id": "httprouter", + "url": "https://github.com/julienschmidt/httprouter", + "commit": "484018016424d215c0b87c42f4c9b57d980fbd00", + "inventorySha256": "42a8e0fe1f185eccb4716816bf374797988d59ad1b84959ec6ac5a775681491f" + }, + { + "id": "go-cache", + "url": "https://github.com/patrickmn/go-cache", + "commit": "46f407853014144407b6c2ec7ccc76bf67958d93", + "inventorySha256": "7d69478248c3d2f5235a9ebfe26dd436b09bd5a815252918327584420456d67e" + }, + { + "id": "log", + "url": "https://github.com/rust-lang/log", + "commit": "27e3cf7a021dab43430a70cb2909c06ca8141f79", + "inventorySha256": "94b38f58a9cb84133faf2cb801e59279c25c15a0758fd05dd18ef2e639b2fc36" + }, + { + "id": "humantime", + "url": "https://github.com/tailhook/humantime", + "commit": "997ff566cc731e3ece5c792c8b83dd78e459f202", + "inventorySha256": "788d6afd828157348743ed7b49328d5a3e74fe4de3bfac8cb14279472e0d62b8" + }, + { + "id": "p-queue", + "url": "https://github.com/sindresorhus/p-queue", + "commit": "7f272ab5ff2254c0dc2537b7977e88ede1cd55e8", + "inventorySha256": "6bcc5040630c9bd7731e8595461e9ad6c8ba56c8230b97a55ed8621200ec3b99" + }, + { + "id": "superjson", + "url": "https://github.com/flightcontrolhq/superjson", + "commit": "aaa65e36e8e31da740265a56b27c965ca1c7b754", + "inventorySha256": "4870c799d1a94c9b29898bccd2feecdbdd674d88e5cf17098b543fd1e7e7664c" + } + ], + "requiresBeforeExecution": [ + "Root finishes native development/tests and selects the release-matching candidate binary", + "Verify exact binary SHA and version, installed Graphify0.9.92 package/interpreter/dependency hashes; recheck the official latest release at a new experiment freeze", + "Freeze merged source-reviewed task/cohort/power/settings manifest; no captures before independent review", + "Check all source roots clean and source inventory hashes unchanged; resolve templates as argv values, never shell interpolation" + ], + "limits": { + "timeoutSecondsPerBuild": 600, + "maxCapturedBytesPerStream": 16777216, + "maxWorkers": 1, + "outputLimitOutcome": "failure, never an empty successful graph" + }, + "childEnvironment": { + "inherit": false, + "PATH": "/usr/bin:/bin", + "HOME": "{isolatedHome}", + "XDG_CONFIG_HOME": "{isolatedHome}/config", + "XDG_DATA_HOME": "{isolatedHome}/data", + "XDG_CACHE_HOME": "{isolatedHome}/cache", + "XDG_STATE_HOME": "{isolatedHome}/state", + "APPDATA": "{isolatedHome}/appdata", + "LOCALAPPDATA": "{isolatedHome}/localappdata", + "TMPDIR": "{isolatedTmp}", + "GRAPHIFY_NO_AUTO_REFRESH": "1", + "GRAPHIFY_QUERY_LOG_DISABLE": "1" + }, + "builds": { + "compass": { + "argv": [ + "{compassBinary}", + "extract", + "{sourceRoot}", + "--code-only", + "--no-viz", + "--store", + "json", + "--max-workers", + "1", + "--out", + "{freshCompassOutput}" + ], + "graph": "{freshCompassOutput}/compass-out/snapshots/{publishedCurrentSnapshot}/graph.json", + "profile": "Native code-only typed JSON publication; optional provider/ProgramIR/network flags absent. JSON storage is declared for source graph auditing.", + "publishedSnapshotPointer": "{freshCompassOutput}/compass-out/current-snapshot", + "pointerValidation": "Read bounded current-snapshot; require one contained snapshot-* basename and coherent immutable graph/sidecars. Use existing runner._compass_graph validation; re-read publication identity before queries." + }, + "graphify": { + "argv": [ + "{graphifyPython}", + "-I", + "-B", + "-X", + "utf8", + "-m", + "graphify", + "extract", + "{sourceRoot}", + "--code-only", + "--max-workers", + "1", + "--out", + "{freshGraphifyOutput}" + ], + "graph": "{freshGraphifyOutput}/graphify-out/graph.json", + "release": "v0.9.92", + "sourceCommit": "86c57be8204673c2a1b9849a3292a7dd3ab373d8", + "profile": "Native local AST/code-only; optional semantic/provider/model-backed extraction excluded and labeled separately." + } + }, + "scopePolicy": "Registered tracked-source scope includes tests/examples. Record actual discovered file omissions at ledgerstage1; do not silently remove their tasks or mutate inputs. No-cluster/no-gitignore flags are not implicitly added; clustering/discovery retain declared native behavior.", + "outputsRequired": [ + "Bounded raw stdout/stderr plus argv/child nonsecret environment, exit/timeout/output-limit/walltime records", + "Graph and sidecar hashes and metadata; verify coherent publication before use", + "Source git status/commit/inventory before and after", + "Observe graph identity/multiplicity/provenance from source audits, not from another graph as oracle" + ], + "publication": "Fresh external per-repository output directories; fail if prior build artifacts exist. Generated graphs/caches must stay outside all source trees.", + "qualification": "No successful build or3200fact count is asserted. Source-sample feasibility and \u22652000accepted current graph judgments with required precision/Wilson bounds remain separate; insufficient small-corpus evidence stays unqualified." +} diff --git a/benchmarks/agent_query/broad_query_capture.py b/benchmarks/agent_query/broad_query_capture.py new file mode 100644 index 000000000..fda3026bd --- /dev/null +++ b/benchmarks/agent_query/broad_query_capture.py @@ -0,0 +1,648 @@ +"""Bounded, developer-only Plan 029 capture; never installs or builds a tool. + +A pre-output registration freezes the exact capturePlan. Queries use prebuilt +native graphs and explicit executables. Parsed spans are review suggestions, +not source facts: unresolved text and missing owners remain in a review queue. +Run ``python3 -m benchmarks.agent_query.broad_query_capture --help``. +""" +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import json +from pathlib import Path +import platform +import re +import sys +from typing import Callable + +from benchmarks.agent_query.broad_query_oracle import publish_exclusive +from benchmarks.agent_query.broad_query_review import ( + ARMS, TOOLS, Artifacts, GraphIndex, Identity, MANIFEST_SCHEMA, MAX_GRAPH_BYTES, + MAX_JSON_BYTES, NORMALIZED_SCHEMA, REVIEW_SCHEMA, RUN_SCHEMA, + _array, _boolean, _choice, _digest, _integer, _json, _object, _relative, + _review_failures, _text, _timestamp, _verify_sources, main as review_main, + sha, validate_registration, verify_build_capture, exposure_failures, + verify_calibration_evidence, calibration_review_failures, + query_exit_contract, query_outcome, + SOURCE_QUERY_FILE, MAX_SOURCE_QUERY_BYTES, source_query_reference, + validate_source_query, require_regular_source_query, require_absent_source_query, +) +from benchmarks.agent_query.community_tasks import read_bounded +from benchmarks.agent_query.runner import _terminal_symbol, run_bounded + +SCHEMA = "compass.broad-query-capture-plan/1" +INPUT_SCHEMA = "compass.broad-query-capture-input/1" +PENDING = dict(status="pending", reviewer=None, reviewerType=None, + sourceAware=False, independentOfAuthor=False, toolLabelsBlinded=False) +SAFE_KEY = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9_.-]{0,127}\Z") +GRAPHIFY_NODE = re.compile(r"^NODE (?P