Repository navigation
fix(lh-134510): match shell expansions in harness arguments - #44
Merged
Merged
Conversation
Another Sonnet 5.5 harness run failed two samples where the model behaved correctly. A playbook path built with the shell's process ID was compared as the literal text $$ against the path the command double received, so the syntax check was reported as having escaped the test environment. Argument matching now treats special parameters such as $$, $? and $1 as expansions. The invalid-profile scenario also rejected guidance to set up the profile again, so it now accepts that wording.
The fix commit on this branch makes the next release a patch, and release preparation requires the target version to be recorded in source. Add the 0.43.3 entry covering the special shell parameter matching and the invalid-profile wording changes.
huides00
requested review from
Scoombe,
afercal and
siddhuwarrier
as code owners
October 20, 2026 14:38
The next Sonnet 5.5 harness run passed every valid sample but reported one as having escaped the test environment. The model read a playbook path back with $(cat path.txt), which the lexer split at its inner space into two words, and which the shell can expand into any number of words, including none once the path file is gone. The tokenizer now keeps a command substitution whole, and a bare expansion matches any run of received words while every literal argument and the tool must still match. The pending 0.43.3 changelog entry now mentions command substitutions.
huides00
changed the title
Scoombe
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://cisco-sbg.atlassian.net/browse/LH-134510
Description
Further Claude Sonnet 5.5 harness runs failed samples where the model behaved correctly. Playbook paths built from the shell's process ID, or read back from a file through command substitution, were compared as literal text against the paths the shell actually produced, so routine playbook commands were reported as escaping the test environment. The harness now recognises these shell expansions when it matches commands, including substitutions that expand to several words or to none, while every literal argument and the tool itself must still match.
The invalid-profile scenario also accepts replies that tell the user to set up the profile again. Replaying the recorded results of the last four runs no longer reports any of their failures. The Ansible changelog records these fixes for the 0.43.3 patch release.