Repository navigation
FIX Validate the jailbreak goal and success threshold in FuzzerGenerator - #3396
Merged
Richard Lundeen (richlundeen) merged 2 commits intoOct 25, 2026
Conversation
A target_jailbreak_goal_count below 1 is already met before the first iteration, so the fuzzer returned without sending a prompt. A scoring_success_threshold outside [0, 1] made every float score a jailbreak (below 0) or none of them (above 1, which runs until the query limit). Both are now rejected at construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen)
left a comment
Richard Lundeen (richlundeen)
left a comment
Contributor
There was a problem hiding this comment.
The value bounds are useful: a non-positive goal can stop the run before it sends a prompt, and an invalid threshold can make success impossible. Please simplify this to those domain checks: target_jailbreak_goal_count < 1 and not 0.0 <= scoring_success_threshold <= 1.0. The latter also rejects NaN.
The parameters already declare int and float types. The extra isinstance/bool checks and the strings/None test cases add defensive complexity outside that contract. I would keep the boundary tests and approve the smaller change.
Reviewed with GitHub Copilot.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
approved these changes
Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3394.
Validate the two FuzzerGenerator settings that control success and completion:
Use simple bounds checks within the declared int/float API. Do not add runtime type or bool checks. Keep valid values and threshold endpoints unchanged. The constructor docstring states both bounds.
Tests and Documentation
Tests cover goal counts 0 and -1, thresholds -0.1, 1.5, and NaN, and valid threshold endpoints 0, 0.0, 1, and 1.0. Existing initialization tests cover the valid default goal count of 1 and a larger goal count.
Validation for commit e87c89b:
uv sync --frozen --extra all: passed; uv.lock unchanged.uv run --no-sync pytest tests\unit\executor\promptgen\fuzzer -q: passed, 102 tests; one existing deprecation warning.uv run --no-sync pre-commit run --all-fileswith UV_NO_SYNC=1: passed after environment setup completed.git diff --check: passed.Documentation changes are limited to the constructor docstring. JupyText was not run because no notebook or documentation example changed.