Skip to content

fix(conformance): fall back to tmpdir-scoped rmSync in test cleanup when trash is absent - #345

Merged
rocklambros merged 1 commit into
GenAI-Security-Project:integrationfrom
drk-ram:refimpl/trash-fallback-73
Oct 31, 2026
Merged

rocklambros merged 1 commit into
GenAI-Security-Project:integrationfrom
drk-ram:refimpl/trash-fallback-73

Conversation

@drk-ram

@drk-ram drk-ram commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Updates trashDir() in the five conformance test files (fetch-upstream.test.ts, policy-input-schema.test.ts, surfaces.test.ts, upstream-schema-check.test.ts, and upstream-watch.test.ts) to run trash when available on PATH and otherwise fall back to rmSync(target, { recursive: true, force: true }) after verifying resolve(dir) is strictly inside resolve(os.tmpdir()), eliminating the 26 clean-machine bun test failures while preserving the tmpdir() safety boundary.

Which issue does this implement

Fixes #73

  • This is an editorial correction (typo, grammar, link, formatting) with no change in
    meaning, so it needs no issue

Base branch

  • integration, because this touches the specification, schemas, a reference
    implementation, an adapter, tests, or CI
  • main, because every changed path is on the documentation lane allowlist in
    CONTRIBUTING.md

Type of change

  • Specification change (schema, hooks, events, AgBOM)
  • Reference implementation or adapter
  • Documentation
  • Tooling or CI
  • Governance (licensing, security policy, contributor docs)

I tested this

  • I synced my branch with the base branch before opening this
  • uv run pytest -v passes on my machine
  • uv run mkdocs build --strict passes on my machine

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

  • This change has no security impact

@github-actions

Copy link
Copy Markdown

This pull request's description says it closes #73, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging.

@acs-board-reconciler acs-board-reconciler Bot moved this from Needs triage to In progress in ACS Project Tracker Oct 29, 2026

@rocklambros rocklambros left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raghuvar, I ran your fix with trash off PATH. On integration 19b1197, 26 tests fail, the number issue #73 gives. With your change, 1287 tests ran with 0 failures on Bun 1.3.11 and on 1.3.14. All 36 rmSync calls stayed inside tmpdir(), and no fixture folders were left behind.

Aaron Ormerod asked on issue #73 which fix to use. I'm picking yours. A devDependency adds a package only to clean up tests, and the agt tree already uses recursive rmSync in 15 other places.

I need three small changes before I can approve.

  1. README.md:72-76 still says bun test fails 26 tests without trash. After your PR that is false, and new readers see it first. Replace those lines with:

    `bun install` does not install `trash`, because `trash` is a system command, not an npm
    package. `bun test` passes without it. `bun run conformance`, `bun run watch:upstream`,
    and `bun run verify:pin` stop with an error until you install `trash-cli`.
    

    Test: git grep -n 'those 26' README.md prints nothing.

  2. .github/workflows/reference-implementation.yml:69-76 installs trash-cli, so Bun.which("trash") is never null in CI and your new code never runs there. I forced the helper to always call trash, which is the old behavior, and ran with a trash command on PATH, as CI does. 0 tests failed, so CI can't see if this fix breaks. The comment on those lines is also false now. Delete lines 69-76 and the blank line after them. No other step in that workflow uses trash.

    Test: the agt check passes on your PR, and its log has no trash-cli install step.

  3. reference-implementations/agt/README.md:547 says the scripts under scripts/ use trash. Only three of the six do. Replace the line with:

    Three scripts delete their scratch directories with `trash` and refuse to run without it: `run-conformance.sh`, `run-upstream-watch.sh`, and `verify-pin.sh`. The five conformance test files use `trash` when it is on `PATH`. Otherwise they call `rmSync` on the path, but only if the path is under `os.tmpdir()`.
    

    Test: grep -l trash reference-implementations/agt/scripts/* prints those three names.

One thing I'm not asking you to change. The guard reads the path as text, so a symlink in the middle of a path gets past it. I tried this in a scratch folder and it deleted a file outside tmpdir(). No caller here can reach that, because each one passes a new mkdtemp path.

GitHub still won't let me assign you issue #73. Please leave a comment on that issue, as I asked on PR #334, because I can't merge until you're assigned. After you push, I'll run all of this again.

…hen trash is absent

Update trashDir() across the five conformance test suites to use trash
when present on PATH and otherwise fall back to rmSync scoped strictly
to directories under os.tmpdir(), so bun test passes on a clean machine
without an external trash binary installed.

Fixes GenAI-Security-Project#73

Signed-off-by: Raghuvar Karthik Durga <177662415+drk-ram@users.noreply.github.com>
@drk-ram

drk-ram commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

@rocklambros Thank you for the review! Addressed all three items in 208d569:

  1. Updated README.md:72-76 to state that bun test passes without trash while bun run conformance, bun run watch:upstream, and bun run verify:pin still require trash-cli.
  2. Removed the trash-cli global install step from .github/workflows/reference-implementation.yml:69-76 so CI exercises the rmSync fallback.
  3. Updated reference-implementations/agt/README.md:547 to name the three scripts (run-conformance.sh, run-upstream-watch.sh, verify-pin.sh) and the five conformance test files.

Also accepted the repository invitation and commented on issue #73 so you can assign it.

@rocklambros rocklambros left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raghuvar, your three changes are exactly what I asked for. Your agt check on GitHub is also the first proof of this fix on Linux. That job ran with no trash-cli step, and 1287 tests ran on Bun 1.3.11 with 0 failures.

I ran head 208d569 on integration f3dab50, which now includes PR #338. pytest gave 924 passed and 1 skipped, and mkdocs build --strict passed. Bun 1.3.11 and 1.3.14 gave the same counts as integration.

With trash off PATH, integration fails 26 tests and leaves 36 folders behind. Your change fails 0 and leaves 0. When I put the old behavior back, the same 26 tests failed. CI has no trash now, so it would catch that.

I'm approving this and merging it now. Issue #73 is assigned to you. This PR closes it. Thank you for making all three changes so quickly!

@rocklambros
rocklambros merged commit 1b8f970 into GenAI-Security-Project:integration Oct 31, 2026
6 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Done in ACS Project Tracker Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Bug] The test suite requires an undeclared trash binary and fails on a clean machine

2 participants