feat(config): cap reviewer effort per model tier - #552
Draft
zzwong wants to merge 1 commit into
Draft
Conversation
Agent catalogs declare an absolute effort that becomes the provider's
reasoning-effort setting. A deployment that wants to bound spend on an
expensive tier previously had only one lever: editing the shared catalog,
which changes the agent's declared intent for every consumer.
Add llm.max_effort, a per-tier ceiling resolved alongside model_map:
max_effort:
large: medium
A tier absent from the map is uncapped. The cap is a ceiling only, so an
agent declaring low under a medium ceiling still runs at low.
The clamp lives in stagemodel.ResolveStageModel, the single documented path
from profile preferences to a concrete model and effort, so every stage picks
it up without per-stage changes.
Reviewer resolution previously discarded the resolver's effort and passed
agent.Effort straight through, which would have left reviewers - the only
path that reaches the large tier - silently uncapped. The resolved effort is
now threaded through reviewerRuntimeResolution.
cr init has no editor for max_effort, so the runtime round trip is extended
to preserve it: dropping the field would silently discard a hand-written
ceiling on any later init pass. identityKey now includes the map so two
runtimes differing only by ceiling no longer collide.
Four paths intentionally bypass the ceiling because each is an explicit
selection of a concrete model or effort: --reviewer-effort, --reviewer-model,
agent model_id, and benchmark suites, where stages.reviewers.effort is
required so candidates stay comparable. README and docs/architecture.md name
all four.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Agent catalogs declare an absolute
effortthat becomes the provider's reasoning-effort setting (--thinkingon the Pi RPC adapter). A deployment wanting to bound spend on an expensive tier previously had one lever: edit the shared catalog. That changes the agent author's declared intent for every consumer, including ones who want the deeper review and are happy to pay for it.cralready separates portable intent from deployment choice for models — an agent declaresmodel_tier, the profile maps tiers to concrete models and sets areviewer_model_tierbaseline. Effort had no equivalent layer.What
llm.max_effort, a per-tier ceiling that sits next tomodel_map:A tier absent from the map is uncapped. It is a ceiling only — an agent declaring
lowunder amediumceiling still runs atlow.The clamp lives in
stagemodel.ResolveStageModel, whichdocs/architecture.mddesignates as the single path from profile preferences to a concrete model and effort, explicitly so "token costs and profile-level tier floors can be added without touching individual review stages."Notable
A latent trap this surfaced.
resolveReviewerRuntimeConfigdiscarded the resolver's effort and passedagent.Effortstraight through. A clamp in the resolver alone would have silently missed reviewer agents — the only path that ever reaches thelargetier, and the entire point of the feature. The resolved effort is now threaded throughreviewerRuntimeResolution.Init preservation. There is no init editor for
max_effort, so hand-editingconfig.ymlis the only way in. The runtime round trip (initLLMRuntimeDraftFromConfig→exportConfig) rebuildsLLMConfigfield by field, so without this the first latercr inittouching the runtime would silently discard the ceiling — config parses fine, cap just gone.identityKeynow includes the map too, so two runtimes differing only by ceiling no longer collide.Deliberate bypasses. Four paths ignore the ceiling because each is an explicit selection of a concrete model or effort, and all four are documented in the README:
--reviewer-effort--reviewer-modelModelOverridereturns before the clampmodel_idcr benchmark runstages.reviewers.effortis required so candidates stay comparableThe
model_idcase is the one structural hole worth knowing about: a catalog pinning bothmodel_idandeffort: highis outside a deployment's control entirely.Testing
go build ./...,golangci-lint run, andgo test ./...(2998 passed, 71 packages) all clean.New coverage: the clamp, uncapped tiers, the never-raise rule,
model_idstaying uncapped with a ceiling configured (the pre-existing exact-model tests use profiles withoutMaxEffort, so they would have passed either way), config validation,MinEffort, and the init round trip.Also exercised end to end against a real PR — a large-tier agent declaring
effort: highresolved togpt-5.6-solatmediumwhile medium-tier agents were untouched, in a live run that posted to GitHub.Follow-ups, not in this PR
--reviewer-effort/--reviewer-modelvs. the ceiling: escape hatch or absolute? Needs a contract decision, and--selection-effortis currently asymmetric (it feeds into the resolver and is capped).max_effortsupport in thecr initwizard.ModelMapRowJSON parity — text mode shows the ceiling,--jsondoes not — plus extending the architecture guardrail test to coverconfig.ResolveMaxEffort.rebaseReviewerCohort's incompatibility message should name the field that drifted; enabling a cap trips it on the first run against a saved cohort.LLMConfig.normalized()andempty()do not know aboutMaxEffort; extendingnormalized()would letResolveMaxEffortindex directly instead of scanning.