Skip to content

feat(config): cap reviewer effort per model tier - #552

Draft
zzwong wants to merge 1 commit into
mainfrom
feat/max-effort-tier-ceiling
Draft

feat(config): cap reviewer effort per model tier#552
zzwong wants to merge 1 commit into
mainfrom
feat/max-effort-tier-ceiling

Conversation

@zzwong

@zzwong zzwong commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Why

Agent catalogs declare an absolute effort that becomes the provider's reasoning-effort setting (--thinking on the Pi RPC adapter). A deployment wanting to bound spend on an expensive tier previously had one lever: edit the shared catalog. That changes the agent author's declared intent for every consumer, including ones who want the deeper review and are happy to pay for it.

cr already separates portable intent from deployment choice for models — an agent declares model_tier, the profile maps tiers to concrete models and sets a reviewer_model_tier baseline. Effort had no equivalent layer.

What

llm.max_effort, a per-tier ceiling that sits next to model_map:

llm_runtimes:
  pi-codex-monit-reviewer:
    model_map:
      large: openai-codex/gpt-5.6-sol
      medium: openai-codex/gpt-5.6-terra
    max_effort:
      large: medium

A tier absent from the map is uncapped. It is a ceiling only — an agent declaring low under a medium ceiling still runs at low.

The clamp lives in stagemodel.ResolveStageModel, which docs/architecture.md designates as the single path from profile preferences to a concrete model and effort, explicitly so "token costs and profile-level tier floors can be added without touching individual review stages."

Notable

A latent trap this surfaced. resolveReviewerRuntimeConfig discarded the resolver's effort and passed agent.Effort straight through. A clamp in the resolver alone would have silently missed reviewer agents — the only path that ever reaches the large tier, and the entire point of the feature. The resolved effort is now threaded through reviewerRuntimeResolution.

Init preservation. There is no init editor for max_effort, so hand-editing config.yml is the only way in. The runtime round trip (initLLMRuntimeDraftFromConfigexportConfig) rebuilds LLMConfig field by field, so without this the first later cr init touching the runtime would silently discard the ceiling — config parses fine, cap just gone. identityKey now includes the map too, so two runtimes differing only by ceiling no longer collide.

Deliberate bypasses. Four paths ignore the ceiling because each is an explicit selection of a concrete model or effort, and all four are documented in the README:

Path Why
--reviewer-effort operator override applied after resolution
--reviewer-model ModelOverride returns before the clamp
agent model_id no tier to key a tier-scoped ceiling on
cr benchmark run stages.reviewers.effort is required so candidates stay comparable

The model_id case is the one structural hole worth knowing about: a catalog pinning both model_id and effort: high is outside a deployment's control entirely.

Testing

go build ./..., golangci-lint run, and go test ./... (2998 passed, 71 packages) all clean.

New coverage: the clamp, uncapped tiers, the never-raise rule, model_id staying uncapped with a ceiling configured (the pre-existing exact-model tests use profiles without MaxEffort, so they would have passed either way), config validation, MinEffort, and the init round trip.

Also exercised end to end against a real PR — a large-tier agent declaring effort: high resolved to gpt-5.6-sol at medium while medium-tier agents were untouched, in a live run that posted to GitHub.

Follow-ups, not in this PR

  • --reviewer-effort / --reviewer-model vs. the ceiling: escape hatch or absolute? Needs a contract decision, and --selection-effort is currently asymmetric (it feeds into the resolver and is capped).
  • max_effort support in the cr init wizard.
  • ModelMapRow JSON parity — text mode shows the ceiling, --json does not — plus extending the architecture guardrail test to cover config.ResolveMaxEffort.
  • rebaseReviewerCohort's incompatibility message should name the field that drifted; enabling a cap trips it on the first run against a saved cohort.
  • LLMConfig.normalized() and empty() do not know about MaxEffort; extending normalized() would let ResolveMaxEffort index directly instead of scanning.

Agent catalogs declare an absolute effort that becomes the provider's
reasoning-effort setting. A deployment that wants to bound spend on an
expensive tier previously had only one lever: editing the shared catalog,
which changes the agent's declared intent for every consumer.

Add llm.max_effort, a per-tier ceiling resolved alongside model_map:

  max_effort:
    large: medium

A tier absent from the map is uncapped. The cap is a ceiling only, so an
agent declaring low under a medium ceiling still runs at low.

The clamp lives in stagemodel.ResolveStageModel, the single documented path
from profile preferences to a concrete model and effort, so every stage picks
it up without per-stage changes.

Reviewer resolution previously discarded the resolver's effort and passed
agent.Effort straight through, which would have left reviewers - the only
path that reaches the large tier - silently uncapped. The resolved effort is
now threaded through reviewerRuntimeResolution.

cr init has no editor for max_effort, so the runtime round trip is extended
to preserve it: dropping the field would silently discard a hand-written
ceiling on any later init pass. identityKey now includes the map so two
runtimes differing only by ceiling no longer collide.

Four paths intentionally bypass the ceiling because each is an explicit
selection of a concrete model or effort: --reviewer-effort, --reviewer-model,
agent model_id, and benchmark suites, where stages.reviewers.effort is
required so candidates stay comparable. README and docs/architecture.md name
all four.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant