Repository navigation
Limit suggested checks to 3 failures per run - #101
Merged
Merged
Conversation
Prevent infinite loops of failing suggested checks by tracking failure count and stopping suggestions after 3 attempts while allowing spec retries to continue.
…add missing tests - Move `ctx.step_get(..., "suggest_failures", 0)` inside `if cfg["suggest_extra_checks"]:` — reviewer: disabled path must not touch the key per spec - Assert `step_set` writes sequence `[1, 2, 3]` in counter-increments test — reviewer: persisted sequence was untested - Add test for counter unchanged at 3 when commit fails — reviewer: commit-failure invariant was missing - Add test for counter unchanged at 2 when static verify fails — reviewer: static-verify-failure invariant was missing - Assert `step_get` not called for `suggest_failures` in disabled-mode test — reviewer: disabled path must not read the key
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a failure counter to prevent the suggester LLM from being invoked indefinitely on broken suggested checks. After 3 consecutive failures within a single run, the suggested-checks block is skipped entirely, allowing spec retries to proceed directly to the commit step.
The counter is tracked in step context (
suggest_failures), incremented when suggested checks fail, and cleared on commit success. Static verify hooks are unchanged. This prevents hallucinated checks from exhaustingmax_retrieswhile maintaining the ability to retry the spec itself.Test plan
Added four new tests in
tests/draft/test_code_spec.py:test_suggest_failures_counter_increments_and_blocks_at_limit— verifies the counter increments from 0 → 1 → 2 → 3 across three failing suggest attempts, and asserts that neither_suggest_checksnor_run_suggested_checksis called once the limit is reached.test_suggest_failures_counter_unchanged_on_commit_failure— confirms the counter remains at 3 when the commit step fails (pre-commit hook), and the suggester is not called on that attempt.test_suggest_failures_counter_unchanged_on_static_verify_failure— confirms the counter is unchanged when static verify fails before reaching the suggest block.test_suggest_failures_not_written_when_suggest_disabled— confirms the counter key is never read or written whensuggest_extra_checks=False.Additionally verified: the counter is cleared to 0 on commit success.