Repository navigation
fix(conformance): fall back to tmpdir-scoped rmSync in test cleanup when trash is absent - #345
Conversation
|
This pull request's description says it closes #73, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging. |
rocklambros
left a comment
There was a problem hiding this comment.
Raghuvar, I ran your fix with trash off PATH. On integration 19b1197, 26 tests fail, the number issue #73 gives. With your change, 1287 tests ran with 0 failures on Bun 1.3.11 and on 1.3.14. All 36 rmSync calls stayed inside tmpdir(), and no fixture folders were left behind.
Aaron Ormerod asked on issue #73 which fix to use. I'm picking yours. A devDependency adds a package only to clean up tests, and the agt tree already uses recursive rmSync in 15 other places.
I need three small changes before I can approve.
-
README.md:72-76still saysbun testfails 26 tests withouttrash. After your PR that is false, and new readers see it first. Replace those lines with:`bun install` does not install `trash`, because `trash` is a system command, not an npm package. `bun test` passes without it. `bun run conformance`, `bun run watch:upstream`, and `bun run verify:pin` stop with an error until you install `trash-cli`.Test:
git grep -n 'those 26' README.mdprints nothing. -
.github/workflows/reference-implementation.yml:69-76installstrash-cli, soBun.which("trash")is never null in CI and your new code never runs there. I forced the helper to always calltrash, which is the old behavior, and ran with atrashcommand onPATH, as CI does. 0 tests failed, so CI can't see if this fix breaks. The comment on those lines is also false now. Delete lines 69-76 and the blank line after them. No other step in that workflow usestrash.Test: the
agtcheck passes on your PR, and its log has notrash-cliinstall step. -
reference-implementations/agt/README.md:547says the scripts underscripts/usetrash. Only three of the six do. Replace the line with:Three scripts delete their scratch directories with `trash` and refuse to run without it: `run-conformance.sh`, `run-upstream-watch.sh`, and `verify-pin.sh`. The five conformance test files use `trash` when it is on `PATH`. Otherwise they call `rmSync` on the path, but only if the path is under `os.tmpdir()`.Test:
grep -l trash reference-implementations/agt/scripts/*prints those three names.
One thing I'm not asking you to change. The guard reads the path as text, so a symlink in the middle of a path gets past it. I tried this in a scratch folder and it deleted a file outside tmpdir(). No caller here can reach that, because each one passes a new mkdtemp path.
GitHub still won't let me assign you issue #73. Please leave a comment on that issue, as I asked on PR #334, because I can't merge until you're assigned. After you push, I'll run all of this again.
…hen trash is absent Update trashDir() across the five conformance test suites to use trash when present on PATH and otherwise fall back to rmSync scoped strictly to directories under os.tmpdir(), so bun test passes on a clean machine without an external trash binary installed. Fixes GenAI-Security-Project#73 Signed-off-by: Raghuvar Karthik Durga <177662415+drk-ram@users.noreply.github.com>
|
@rocklambros Thank you for the review! Addressed all three items in
Also accepted the repository invitation and commented on issue #73 so you can assign it. |
9c7414d to
208d569
Compare
rocklambros
left a comment
There was a problem hiding this comment.
Raghuvar, your three changes are exactly what I asked for. Your agt check on GitHub is also the first proof of this fix on Linux. That job ran with no trash-cli step, and 1287 tests ran on Bun 1.3.11 with 0 failures.
I ran head 208d569 on integration f3dab50, which now includes PR #338. pytest gave 924 passed and 1 skipped, and mkdocs build --strict passed. Bun 1.3.11 and 1.3.14 gave the same counts as integration.
With trash off PATH, integration fails 26 tests and leaves 36 folders behind. Your change fails 0 and leaves 0. When I put the old behavior back, the same 26 tests failed. CI has no trash now, so it would catch that.
I'm approving this and merging it now. Issue #73 is assigned to you. This PR closes it. Thank you for making all three changes so quickly!
1b8f970
into
GenAI-Security-Project:integration
What changed
Updates
trashDir()in the five conformance test files (fetch-upstream.test.ts,policy-input-schema.test.ts,surfaces.test.ts,upstream-schema-check.test.ts, andupstream-watch.test.ts) to runtrashwhen available onPATHand otherwise fall back tormSync(target, { recursive: true, force: true })after verifyingresolve(dir)is strictly insideresolve(os.tmpdir()), eliminating the 26 clean-machinebun testfailures while preserving thetmpdir()safety boundary.Which issue does this implement
Fixes #73
meaning, so it needs no issue
Base branch
integration, because this touches the specification, schemas, a referenceimplementation, an adapter, tests, or CI
main, because every changed path is on the documentation lane allowlist inCONTRIBUTING.md
Type of change
I tested this
uv run pytest -vpasses on my machineuv run mkdocs build --strictpasses on my machineChecklist
git commit -s(required by the DCO)Security