From 007b8e4158a6cb70b98f3f809486353a760d79c8 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 21 Oct 2026 11:25:37 +0000 Subject: [PATCH 1/2] Count proxy traffic per run and stop at --proxy-budget-mb (default 10 MB) A browser this repo launches behind --proxy / --proxy-file now counts the bytes it sends through the proxy (Playwright: request.sizes(); pyppeteer and Selenium: CDP Network.loadingFinished.encodedDataLength plus request sizes; the pre-check HEAD too). Once the total passes --proxy-budget-mb no further listing, worker or product page is opened, the browser aborts what it still tries, the products read so far are kept, and the run exits 6 (5 with none) with stop_reason proxy_budget_exceeded. The sidecar records proxy_bytes, proxy_budget_bytes and proxy_budget_exceeded, and the run ends with one "Proxy traffic: X.X MB of Y MB budget" line. Nothing is metered over --cdp-endpoint or without a proxy. The production-like canary passes --proxy-budget-mb 10, writes its traffic to the job summary and fails when the budget was hit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8 --- .github/proxy_traffic_summary.py | 71 +++++ .github/workflows/canary.yml | 11 +- .github/workflows/tests.yml | 18 +- CHANGELOG.md | 20 +- Dockerfile | 2 +- README.md | 38 ++- output_writer.py | 18 ++ playwright_scraper.py | 27 +- proxy_meter.py | 447 ++++++++++++++++++++++++++++ proxy_pool.py | 4 +- puppeteer_scraper.py | 11 + pyproject.toml | 2 + selenium_scraper.py | 16 + smoke_test.py | 487 +++++++++++++++++++++++++++++- tests/test_proxy_meter.py | 491 +++++++++++++++++++++++++++++++ 15 files changed, 1652 insertions(+), 11 deletions(-) create mode 100644 .github/proxy_traffic_summary.py create mode 100644 proxy_meter.py create mode 100644 tests/test_proxy_meter.py diff --git a/.github/proxy_traffic_summary.py b/.github/proxy_traffic_summary.py new file mode 100644 index 0000000..20add55 --- /dev/null +++ b/.github/proxy_traffic_summary.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +"""Canary step: one line about a run's proxy traffic for the job summary. + + python3 .github/proxy_traffic_summary.py