Skip to content

fix: pip-audit says when the network failed instead of passing or failing - #67

Merged
fatherlinux merged 4 commits into
mainfrom
fix/pip-audit-network-honesty
Nov 18, 2026
Merged

fatherlinux merged 4 commits into
mainfrom
fix/pip-audit-network-honesty

Conversation

@fatherlinux

Copy link
Copy Markdown
Member

What was wrong

pip-audit 2.9.0 has two network failure modes, and the wrapper reported both as verdicts:

  • Read timeout → "FAILED". PyPIService.query catches ConnectTimeout but lets ReadTimeout, connection resets and 5xx HTTPError escape as tracebacks. Every one exits 1, which is also the status for "found a vulnerability". On the busy Exedore runners the 15s default read timeout trips against a PyPI that answers in under a second (e.g. ashigaru run 44228385707).
  • Lookup refused → "Passed". A 404 from pypi.org/pypi/<pkg>/<ver>/json becomes SkippedDependency and pip-audit exits 0. If an egress path answers 404 for everything, every package is skipped and the scan "passes". The wrapper only printed a NOT AUDITED note. I believe this is the push-side "Passed", but I couldn't confirm it: the gateway refused the ashigaru log.

The fix (t_pip_audit in sdlc)

  • --timeout 60 (was pip's 15) and --progress-spinner off.
  • If the output matches a known network-failure signature (from pip-audit, requests, or uv fetching pip-audit) and the exit is non-zero, it retries twice with 10s then 30s backoff.
  • If uv.lock says a package comes from pypi.org and pip-audit reports it "not found on PyPI", that is a failed lookup, not an unauditable package. It is retried the same way. Local path and vendored dependencies (trentina's petit-plus) still pass with the NOT AUDITED note, as before.
  • A failure that outlasts the retries raises CannotRunError, which exits 2, COULD NOT RUN: neither a pass nor a finding.
  • A real vulnerability is still a 1, reported at once with no retry.

Why could-not-run fails CI

Exit 2 fails sdlc ci and sdlc push, the same as every other check here. I did not route it through the CodeQL-style DEFERRED path. A green CI that audited nothing is the pip-audit … || true problem this check was written to end. The retries absorb the transient timeouts that caused the reported failures (about 40s of extra wall time in the worst case before giving up), so what's left is a genuine outage, and that should stop a merge and say why. If persistent outages turn out to block pushes in practice, deferral can be added later as a separate decision.

Tests

tests/test_pip_audit_resilience.py (also added to the container workflow's unit-test loop):

  • read timeout → retry → clean pass, with the backoff and the --timeout argument checked
  • read timeout on every attempt → could-not-run after 3 attempts with the backoff growing
  • each other network signature (connect timeout, redirect loop, reset, proxy, TLS, 5xx, max retries, uv fetch failures) → retried, then could-not-run
  • proxy 404 for a PyPI-locked package → could-not-run, not a pass
  • local path dependency not found → pass on the first attempt
  • real vulnerability → 1, no retry
  • missing or malformed uv.lock → 2, pip-audit never runs

Ran locally: ruff 0.17.1 check and format, Gourmand 1.0.0, Gatehouse 0.26.0 (findings addressed in the diff), host contract, test_plan.py, test_codeql_resilience.py, and the new test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R7Y1jckWHhAgXg2WBwkWbb

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 1 issues (1 critical)

Full review.

4 finding(s) dropped: their evidence is not in the diff.

Comment thread sdlc
"run",
"--locked",
"--all-extras",
"--with",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL (Constitution): The pip-audit invocation does not pin an exact version. The constitution requires every tool in the image to be pinned to an exact version so consumers receive a fixed set of tool versions.

Suggestion: Pin pip-audit to an exact version in the uv invocation, for example with --with pip-audit==X.Y.Z.

Evidence:

"--with",
            "pip-audit",
            "pip-audit",

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 406f7a4

--with pip-audit=={BAKED["pip-audit"]}, which is the Containerfile's PIP_AUDIT_VERSION (2.9.0). With no baked version it is a 2. Both cases are tested.

Josui and others added 2 commits November 18, 2026 03:54
…ling

pip-audit 2.9.0 exits 1 for a PyPI read timeout, the same status as a
vulnerability, and an egress path answering 404 makes it skip every
package and exit 0. Raise --timeout to 60, retry a network failure twice
with 10s/30s backoff, and report one that outlasts the retries as exit 2
(could not run). "Not found" for a package uv.lock takes from PyPI counts
as a failed lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jckWHhAgXg2WBwkWbb
@fatherlinux
fatherlinux force-pushed the fix/pip-audit-network-honesty branch from f62a5a3 to 4ac95c2 Compare November 18, 2026 03:55

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 1 issues (1 critical)

Full review.

1 finding(s) dropped: their evidence is not in the diff.

Comment thread sdlc
"run",
"--locked",
"--all-extras",
"--with",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL (Constitution): The pip-audit tool is fetched without an exact version pin. The constitution requires every tool to be pinned to an exact version; pip-audit is not one of the explicitly permitted consumer-pinned seams.

Suggestion: Pin pip-audit to an exact version in the uv run --with specification, and ensure the installed version is reported alongside the pin.

Evidence:

"--with",
            "pip-audit",
            "pip-audit",

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 406f7a4

Same fix as the thread above: pinned to the image's baked pip-audit version, and an unknown version is a 2.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since 4ac95c2 (incremental).

Check pip-audit's "Found N known vulnerabilit" summary before classifying
anything: it prints only after every lookup returned, so a finding beside
a version PyPI no longer has, or beside uv's own fetch noise, stays a 1.
A run that skipped every registry package installed in the venv is a
failed lookup, which closes the exit-0 path behind a non-PyPI mirror.
Test fixtures now use pip-audit 2.9.0's real columns layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jckWHhAgXg2WBwkWbb

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since 406f7a4 (incremental).

2 finding(s) dropped: their evidence is not in the diff.

@fatherlinux
fatherlinux merged commit cfce0e1 into main Nov 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant