Skip to content

docs: add advisor plugin category RFC (Draft) - #20

Merged
Scaxlibur merged 2 commits into
Scaxlibur:Devfrom
Epslion404:docs/advisor-plugin-rfc
Oct 5, 2026
Merged

Scaxlibur merged 2 commits into
Scaxlibur:Devfrom
Epslion404:docs/advisor-plugin-rfc

Conversation

@Epslion404

@Epslion404 Epslion404 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

This Draft RFC proposes an advisor category alongside the current V2 executable instrument plugins. The existing InstrumentDescriptor/factory/registry contract requires instrument models and capabilities; an advisor needs an independent invocation and validation contract. V1 metadata remains deprecated compatibility behavior.

The proposal defines a descriptor/factory with offline prepare and authorized execute stages, request and answer validation, and a first capture-triage task with a deterministic rule baseline. Core binds per-call consent to the complete prepared request, constrains run-scoped consent with an explicit template and limits, and records standalone calls and failures as well as successful run-associated calls. Probability, provider confidence, and per-question display thresholds have separate meanings; advisor output never controls an experiment or changes safety/quality gates.

This PR changes documentation only. Advisor APIs, commands, configuration, and artifacts remain proposed; the RFC remains Draft. V1 deprecation is also marked in the plugin Concept and Reference without removing runtime compatibility.

Validation: scoped strict documentation audit (0 errors, 0 warnings), generated Reference drift check, strict MkDocs build, and git diff whitespace check. Checks ran offline with respect to instruments; no instrument configuration or experiment data was changed.

Proposes a second plugin category alongside the existing instrument plugins, so
that System One style decision models can be plugged in without putting vendor
specifics into Core:

- introduce category = instrument | advisor, with its own entry point group
  (wavebench.advisor) and API version gate, instead of extending PluginKind
- keep the data-egress consent gate and the wavebench.decision.v1 artifact
  contract in Core, so every advisor gets the same preview/consent/audit surface
- ship a deterministic builtin rule_advisor so the category is testable offline
  and the "beat the deterministic baseline" comparison is executable
- record the six proposed decisions and the acceptance gates

Status is Draft. This commit changes documentation only; no Core behaviour.
@Epslion404 Epslion404 self-assigned this Sep 28, 2026
@Epslion404 Epslion404 added the enhancement New feature or request label Sep 28, 2026
Epslion404 pushed a commit to Epslion404/wavebench that referenced this pull request Sep 29, 2026
Brings the WaveBench x Jev draft track (seam v2, Core stage-1 skeleton, findings and planning files) under docs/project/design/advisor-draft so the work is reviewable in-repo; the RFC itself already arrives with the PR Scaxlibur#20 commit.
Epslion404 pushed a commit to Epslion404/wavebench that referenced this pull request Sep 29, 2026
Brings the WaveBench x Jev draft track (seam v2, Core stage-1 skeleton, findings and planning files) under docs/project/design/advisor-draft so the work is reviewable in-repo; the RFC itself already arrives with the PR Scaxlibur#20 commit.

Copy link
Copy Markdown
Owner

建议继续保持 RFC 为 Draft,补齐以下内容后再冻结实施合同。当前设计明确保留人工执行边界,但插件调用与外发授权还缺少关键约束。

  1. 以 V2 可执行插件为设计基线,明确 V1 已弃用。

    「核心实现现状」混用了两条路径:wavebench.drivers / wavebench.instrument.v1 是已弃用的 metadata 路径;wavebench.instruments / wavebench.instrument.v2 才是现有可执行插件路径。

    建议重写现状表及 PR 描述,以 V2 的 InstrumentDescriptor、factory、registry 和安装后校验解释新增 advisor 的需求。V1 仅放入兼容性说明:保持已有行为,不扩展、不作为 advisor 的实现基础。插件 Reference / Concept 同步标注弃用状态,本 PR 不承担 V1 运行时移除。

  2. 补充最小可执行契约。

    当前 AdvisorPlugin 只有描述字段,没有规定 Core 如何调用实现。建议明确 entry point 返回对象、factory 或调用协议、请求与答案类型,以及超时和失败语义。还应明确加载、构造和预览阶段不得发生网络调用。

    不需要预先搭建通用插件框架,但契约应足以让两个独立实现接受同一套 Core 验证。

  3. 将授权绑定到完整外发请求。

    目前白名单只覆盖 state.fields,但问题说明、候选项和评分标准也可能包含业务数据。预览记录了哈希,却未规定实际发送内容必须与预览一致。

    建议规定「完整请求生成 → 校验 → 预览 → 授权 → 发送」的流程:按次授权绑定最终请求摘要,发送前核对;run 级授权明确允许变化的内容及失效条件。transmits_off_machine 与 advisor.external_state 应有唯一判定来源,避免声明矛盾。与此同时,说明此机制依赖可信本地插件遵守契约,不提供任意 Python 代码的网络隔离。

  4. 明确概率、confidence 和阈值的语义。

    Jev 的 Choice / Score 返回 confidence,Noul 则只返回概率;confidence 也不等于最高选项概率。官方定义

    建议按答案类型定义 schema,并明确 accept/review 使用哪个指标、阈值是否按问题配置,以及缺失答案、未知选项、非有限数值和无效概率分布的处理方式。阈值结果只影响建议的展示与人工复核,不触发实验操作。

  5. 补齐独立调用和失败时的审计行为。

    advisor ask 示例没有指定 run,而 artifact 规则要求写入 run 目录、无 run 时不保存。建议明确目标 run 的指定方式,以及无 run、外部调用失败、响应校验失败时,授权记录和调用结果保存在哪里。

另建议先选定一个具体的 WaveBench 判断任务,提供输入、期望答案及规则基线的比较标准,再决定首版需要哪些 capability。rule_advisor 适合作为离线测试实现,但它本身不能证明通用类别抽象的必要性。

已核对当前实现与完整差异;两页定点严格文档检查为 0 错误、0 警告,当前 5 项 CI 全部通过。上述意见主要涉及契约完整性,现有检查尚不能验证这些设计约束。

@Epslion404

Copy link
Copy Markdown
Collaborator Author

已在 aba6f3f 补齐这次评审要求的契约,并重写 PR 说明;RFC 继续保持 Draft,本 PR 仍只修改文档。

  1. 以 V2 为设计基线。 现状表改为 InstrumentDescriptor、factory、wavebench.instruments registry 与安装后验证路径。wavebench.drivers / wavebench.instrument.v1 只作为已弃用的兼容 metadata 路径说明,并同步标注插件 Concept / Reference;不扩展 V1,不移除运行时。

  2. 补充最小执行协议。 明确 entry point 返回 descriptor 或无参数 descriptor factory、独立 advisor factory、prepare() / execute()、请求与响应形状、超时和失败语义。加载、构造和预览不得联网或访问仪器;规则实现与独立 fake provider 接受同一套 Core validator。worker 用于超时终止,不作为任意 Python 代码的网络沙箱。

  3. 授权绑定完整请求。 预览覆盖状态、问题、候选项、判据、模型、method、URL、语义 headers 和精确 body;按次授权绑定 call_id 与完整请求摘要,发送前重新核对。run 级授权明确允许变化的 state 路径、类型、范围、字节数、次数和有效期,其它变化使授权失效。本地 call_id 和审计时间不进入 provider body 或 run 模板比较。外发判定以 transmits_off_machine 为唯一来源,advisor.external_state 由 Core 派生。

  4. 区分概率、confidence 和展示阈值。 按 Choice / Score / Noul 定义答案校验;Noul 不伪造供应商 confidence。每个问题记录具体指标、定义版本及 accept/review 阈值;非法分布、非有限数、缺失答案和未知选项导致响应失败。阈值只控制建议展示和人工复核,不触发实验操作。

  5. 独立调用和失败均可审计。 advisor ask 必须指定已有 --run-dir 或独立 --audit-dir;请求与授权在发送前持久化。预览、拒绝、准备失败、执行失败、超时、取消、非法响应,以及发送后结果写盘失败,都有明确记录和退出语义,不自动重试。

首版任务收窄为已有采集报告的人工复核建议(advisor.triage),给出字段、封闭选项、规则基线、fixture 与效果比较指标。其它 capability 保留为后续候选;内置规则仅证明合同可执行,不能单独证明通用类别的必要性。

验证:改动页及直接 RFC 导航的严格文档检查 0 错误 / 0 警告;生成 Reference 检查、严格 MkDocs 构建、diff whitespace 检查通过。示例命令明确为提案语法;未实现 advisor 运行时,未连接仪器。

@Scaxlibur
Scaxlibur changed the base branch from master to Dev October 5, 2026 07:14
@Scaxlibur
Scaxlibur merged commit 902a178 into Scaxlibur:Dev Oct 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants