Skip to content

Exit 5 for content never obtained, one paid solve per page, readable outputs, canary without excuses - #9

Merged
jehrr merged 2 commits into
mainfrom
fix/audit-2026-10-14
Oct 15, 2026
Merged

jehrr merged 2 commits into
mainfrom
fix/audit-2026-10-14

Conversation

@jehrr

@jehrr jehrr commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes from the 2026-10-14 audit (base main = 6524017, same SHA as the one audited). The live site breakage (F1) is in a separate PR, #8.

Changes

Finding What changed Evidence
F2 exit codes finish_run: a zero-row run returns 4 only when its stop reason shows the listing is empty (no_results, listing_exhausted, no_new_listings, completed). Every other reason returns 5, including reasons added later, because the list names what proves emptiness rather than the failures. EXIT_API_ERROR moves into output_writer. Called for each reason in a new check. Before: all returned 4. After: timeouts, content_unparsed, page_never_painted, worker_lost_page, api_error and an unknown reason return 5. Rows followed by a failure still return 6.
F3 modes _atomic_write chmods the temp file before os.replace: a new file gets 0o666 & ~umask, an existing file keeps its mode. Check under umask 022/002/077, plus a 0664 file kept as 0664, plus the sidecar mode.
F4 CSV LF added to the formula leads. write_csv returns its escape count, and csv_cells_escaped goes into the sidecar under the run's own keys. Check
F5 solve budget page_flow.SolveBudget (SOLVES_PER_PAGE = 1) is created once per page outside the retry/rotation loop and claimed before solve() in all three engines. A stubbed-solver drive of Playwright _fetch_one_page with a 3-exit pool and an always-challenged page: 3 solves on main, 1 here. An AST check (read off disk, so it runs without the drivers) covers all three engines.
F6 status Playwright and pyppeteer pass goto()'s status into classify. A captcha reload clears it. Selenium passes status_code=None explicitly. The Scraper API reads http_code. The same drive now classifies the page as "blocked (HTTP 403)". An AST check covers every classify call.
F7 / F8 Scraper API waitFor is now an object; state is validated; x-debug goes through redact_secret_patterns. The docstring is corrected. A mocked requests.post check. No paid calls made.
F9 citations Removed from code, .gitignore, tests.yml, CHANGELOG (released 0.2.1 section: citations only) and the v0.2.1 release notes (two citations removed, otherwise byte-identical). New check no shipped file cites a private instruction file, built from assembled tokens. Check
F10 scan Skipped directories are matched on path.relative_to(REPO).parts. 0 files scanned is now a failure. Check: a clone under …/legacy/clone with a planted key is caught; an empty tree fails.
F13 --fingerprint is refused with exit 2 in pyppeteer and Selenium, and in Playwright together with --cdp-endpoint. Run in each engine's venv.
F15 The fingerprint-refusal check skips when Playwright has no Chromium. It runs with a private HOME. The local failure turned out to be a fingerprint cached in ~/.cache answering instead of the dead proxy, not only the missing binary. Playwright venv: 471 passed, 0 failed.
Canary (instructions §12) Exit 3 is now always an error, because the site serves this URL to GitHub runners (all 16 red runs from 2026-09-29 to 2026-10-14 reached the page and parsed it). Exit 5 is now an explicit error. The HAS_PROXY notice path is gone. No secret is required; FLIPPA_PROXY stays optional. Canary dispatch on this branch: see below
Metadata Description now says "2Captcha Scraping Browser API" and drops the "embedded-JSON parsing" and "Turnstile solving" claims. gh repo edit

Controls (fault planted in a copy, suite run, named check required red)

All 13 went red on the expected check:

  • exit mapping
  • chmod removed
  • LF lead removed
  • sidecar count removed
  • Selenium budget guard neutralised (the first version of that check stayed green against this; the check was rewritten to require an if not budget.spend(): return ahead of solve())
  • budget moved inside the loop
  • pyppeteer classify without status
  • http_code back to status
  • waitFor back to a string
  • x-debug unredacted
  • absolute-path scan
  • zero-file guard removed
  • a citation re-added

The harness ran with PYTHONDONTWRITEBYTECODE=1, on copies with no __pycache__.

Suites

venv passed failed skipped
Core only 437 0 9 (engine-absent)
Playwright 471 0 1 (local capture)
pyppeteer 441 0 8
Selenium 441 0 8

Not changed

Conflicts

CHANGELOG.md (both PRs add [Unreleased]), and possibly smoke_test.py and canary.yml. Whichever PR lands second needs a rebase.

🤖 Generated with Claude Code

jehrr and others added 2 commits October 15, 2026 21:52
…able output files

- A zero-row run exits 4 only when the listing is shown empty; a timeout,
  an unpainted or unparsable page, a lost worker or an API error is 5.
- One captcha-solve allowance per page in every engine, checked before the
  purchase and not reset by retries or exit rotation (3 solves -> 1).
- Playwright/pyppeteer pass the goto() status to the classifier; the
  Scraper API reads http_code, sends waitFor as an object, redacts x-debug.
- Outputs and sidecar get 0o666 & ~umask (existing files keep their mode).
- CSV guard escapes a leading LF; csv_cells_escaped in the sidecar.
- Credential scan uses paths relative to the repo and fails on 0 files.
- --fingerprint refused where it is not applied.
- Canary: exit 3 and 5 are errors; the site serves GitHub runners.
- No citations of private instruction files; a suite check guards it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… urllib3

pip-audit had been red since 2026-10-05 on two new urllib3 1.x advisories
in the pyppeteer lock, PYSEC-2026-4175 and PYSEC-2026-4177. Both are fixed
only in urllib3 2.8.0; pyppeteer 2.0.0 still requires urllib3<2 and 1.26.20
is the last 1.x release, so no compatible pin clears them. They are ignored
by ID beside the five already listed; any other advisory still fails.

The README now says what that means for someone running the pyppeteer
engine: requests in that virtualenv shares the vulnerable urllib3, so the
proxy preflight and captcha API calls do too; prefer Playwright with proxy
credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/audit-2026-10-14 branch from 472b304 to a83ac5c Compare October 15, 2026 21:53
@jehrr
jehrr marked this pull request as ready for review October 15, 2026 21:55
@jehrr
jehrr merged commit e6f4820 into main Oct 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant