Lost rows fail the run, one paid solve per page, atomic formula-safe output

.dockerignore

@@ -1,6 +1,7 @@
 .git/
 .github/
 .env
+.env.*
 *.log
 *.key
 secrets*
@@ -11,7 +12,9 @@ __pycache__/
 .venv/
 venv/
 env/
-binance_rows.*
+makemytrip_hotels.*
+canary*.json
+*.page[0-9]*
 *_debug.html
 *_debug.png
 live_results/

.env.example

@@ -62,8 +62,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here
 # Selenium cannot use this: chromedriver's `debuggerAddress` takes a bare
 # host:port with nowhere to put a password. Playwright and pyppeteer can.
 #
-# The credentials in a profile expire in about a day, so never paste a
-# working one anywhere that is committed.
+# The credentials in a profile do expire (how soon varies), so never paste
+# a working one anywhere that is committed.
 MAKEMYTRIP_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-in-pid-{profileId}:{password}@cb.2captcha.com:9222
 
 # --- Proxy ----------------------------------------------------------------
@@ -89,7 +89,12 @@ MAKEMYTRIP_PROXY=http://{user}:{password}@xx.proxy.2captcha.com:2334
 #
 # Only the query in the address is read; the flags (--city, --checkin,
 # --checkout, --adults ...) do the same without a URL.
-MAKEMYTRIP_URL=https://www.makemytrip.com/hotels/hotel-listing/?checkin=11102026&checkout=11112026&city=CTGOI&locusId=CTGOI&locusType=city&country=IN&roomStayQualifier=2e0e
+#
+# Shipped commented out, on purpose. A URL set here is a default for a run
+# given NO query flags; a run that types --city, --checkin and the like
+# ignores it. The dates in an example go stale, so put your own in when you
+# uncomment it.
+# MAKEMYTRIP_URL=https://www.makemytrip.com/hotels/hotel-listing/?checkin=11102026&checkout=11112026&city=CTGOI&locusId=CTGOI&locusType=city&country=IN&roomStayQualifier=2e0e
 
 # Deliberately NOT here: an output prefix. `--out` already carries a
 # non-empty default, so the loader, which only fills UNSET values, would

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -18,8 +18,9 @@ body:
           was measured to work is the Scraping Browser API with a
           `country-in` profile (`--cdp-endpoint`). Say in the report which
           exit you used and what the sidecar's `stop_reason` was.
-        - **Exit 5 naming HTTP 401?** The Scraping Browser profile expired;
-          its credentials last about a day.
+        - **Exit 5 naming HTTP 401?** The Scraping Browser endpoint's
+          credentials were refused (expired, revoked or mistyped); copy it
+          again from your 2Captcha dashboard.
         - **Exit 4, zero rows?** The site answered "No Hotels Found" for
           that city and those dates. That is a correct answer. See
           [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md).

.github/canary_assert.py

@@ -3,7 +3,7 @@
 
 A separate script rather than a heredoc in canary.yml, and it reads only the
 run's own output files: a workflow that imports the repo's modules inline is
-red on the first push and invisible to every local run (CLAUDE.md §26).
+red on the first push and invisible to every local run.
 
 The thresholds are measured: a three-page Goa run through the Scraping
 Browser returned 90 rows, 90 priced, all in INR, all RECOMMENDED_HOTELS, on

.github/ci_checks.py

@@ -285,7 +285,7 @@ def tracked_files():
 # into pip's vendored code and flags a 32-hex string in `_elffile.py` as
 # key-shaped. Correct about the string, wrong about the file — and it is the
 # FIRST thing a new user sees from `python3 smoke_test.py`. A guard people
-# have to argue with is one they learn to suppress (CLAUDE.md §22).
+# have to argue with is one they learn to suppress.
 #
 # Deliberately NOT a narrowing of what gets scanned: an untracked file is
 # still read, because a key pasted into a scratch file beside the scripts is

.github/workflows/canary.yml

@@ -4,21 +4,26 @@
 # Two jobs, because the site has two answers and both are claims the README
 # makes:
 #
-#   datacentre        no secrets, daily. A GitHub runner is a datacentre
-#                     address, and from one (a netcup VPS, 2026-09-24) Akamai
-#                     refused every client, real Chromium included, by
-#                     dropping the connection. The job asserts the engine
-#                     REPORTS that correctly: exit 3, a refusal, never 1, 4
-#                     or 5. If the runner is served instead (exit 0), the
-#                     rows are checked as strictly as the other job's, and
-#                     the README's datacentre sentence is due a re-measure.
+#   datacentre        DATACENTRE REFUSAL EXPECTED. No secrets, daily. A
+#                     GitHub runner is a datacentre address, and from one
+#                     (a netcup VPS, 2026-09-24, and every scheduled run of
+#                     this job since) Akamai refused every client, real
+#                     Chromium included, by dropping the connection. The job
+#                     asserts the engine REPORTS that correctly: exit 3, a
+#                     refusal, never 1, 4 or 5. If the runner is served
+#                     instead (exit 0), the rows are checked as strictly as
+#                     the other job's, and the README's datacentre sentence
+#                     is due a re-measure. This job does NOT test extraction;
+#                     the next one does.
 #
-#   scraping-browser  the path measured to work: a Scraping Browser profile
-#                     with an Indian exit. Dispatch-only, not scheduled: a
-#                     profile's credentials last about a day, so a scheduled
-#                     job would be red every morning for a secret that had
-#                     expired, not for the site (CLAUDE.md §19). SKIPS with a
-#                     notice when the secret is not set (§11).
+#   scraping-browser  the path measured to work, and the one that reads real
+#                     rows: a Scraping Browser profile with an Indian exit,
+#                     on the same schedule. It FAILS, with an error naming
+#                     the secret, when MAKEMYTRIP_CDP_ENDPOINT is not set,
+#                     and it fails on a refusal, on "never obtained" (a 401
+#                     is a refused endpoint: set a fresh secret) and on too
+#                     few rows. A green canary therefore means a real
+#                     three-page listing was read and parsed.
 #
 #     gh secret set MAKEMYTRIP_CDP_ENDPOINT --repo 2scraper/makemytrip-scraper
 #     gh workflow run canary.yml --repo 2scraper/makemytrip-scraper
@@ -90,51 +95,48 @@ jobs:
           if-no-files-found: ignore
 
   scraping-browser:
-    if: github.event_name == 'workflow_dispatch'
     runs-on: ubuntu-latest
     timeout-minutes: 20
     env:
       MAKEMYTRIP_CDP_ENDPOINT: ${{ secrets.MAKEMYTRIP_CDP_ENDPOINT }}
     steps:
-      - name: Skip without a Scraping Browser endpoint
+      - name: Require a Scraping Browser endpoint
         if: env.MAKEMYTRIP_CDP_ENDPOINT == ''
-        run: echo "::notice::MAKEMYTRIP_CDP_ENDPOINT is not set, so the Scraping Browser run is skipped. It is the one path measured to get past Akamai from a datacentre; set the secret (a profile with country-in) and dispatch again."
+        run: |
+          echo "::error::The MAKEMYTRIP_CDP_ENDPOINT secret is not set, so nothing tested the site. It is the one path measured to get past Akamai from a datacentre; set it to a Scraping Browser endpoint with country-in in the login."
+          exit 1
       - uses: actions/checkout@v4
-        if: env.MAKEMYTRIP_CDP_ENDPOINT != ''
       - uses: actions/setup-python@v5
-        if: env.MAKEMYTRIP_CDP_ENDPOINT != ''
         with:
           python-version: "3.12"
       - name: Install
-        if: env.MAKEMYTRIP_CDP_ENDPOINT != ''
         run: |
           python -m pip install --upgrade pip
           pip install -r requirements.txt -r requirements-playwright.txt
-      # >= 3 pages, because with one page pagination is never exercised (§11).
+      # >= 3 pages, because with one page pagination is never exercised.
       - name: Run
-        if: env.MAKEMYTRIP_CDP_ENDPOINT != ''
         id: run
         run: |
           set +e
           python playwright_scraper.py --city Goa --pages 3 \
             --out canary --format both --dump-html canary_dump.json
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
       - name: Interpret
-        if: env.MAKEMYTRIP_CDP_ENDPOINT != ''
         run: |
           code="${{ steps.run.outputs.exit_code }}"
           echo "exit code: $code"
           case "$code" in
             0) ;;
             3) echo "::error::Refused through the Scraping Browser. That is the README's central claim failing; see the log for which refusal."; exit 1 ;;
-            5) echo "::error::Never obtained. A 401 in the log means the profile's credentials expired (they last about a day): set a fresh secret."; exit 1 ;;
+            4) echo "::error::Zero rows, reported as an EMPTY listing. Goa is never empty."; exit 1 ;;
+            5) echo "::error::Never obtained, or served and not parsed (see stop_reason in the log). A 401 in the log means the endpoint's credentials were refused: set a fresh MAKEMYTRIP_CDP_ENDPOINT secret."; exit 1 ;;
+            6) echo "::error::A partial run: a page failed after rows were gathered. See stop_reason in the log."; exit 1 ;;
             *) echo "::error::Exit $code."; exit 1 ;;
           esac
       - name: Assert
-        if: env.MAKEMYTRIP_CDP_ENDPOINT != ''
         run: python .github/canary_assert.py canary 60
       - uses: actions/upload-artifact@v4
-        if: always() && env.MAKEMYTRIP_CDP_ENDPOINT != ''
+        if: always()
         with:
           name: canary-scraping-browser
           path: |

.gitignore

@@ -19,7 +19,7 @@ makemytrip_hotels_*.*
 live_results/
 live_results.zip
 # ANY working directory a live run writes into, by KIND rather than by the one
-# name an older repo used (CLAUDE.md §24). Measured here: `live/` was not
+# name an older repo used. Measured here: `live/` was not
 # listed, and a `--dump-html` of a 3-page run wrote `canary_dump.json.page3`,
 # which `*.json` does not cover. It sat untracked and NOT ignored, one
 # `git add -A` from a commit, until the credential scan flagged the 32-hex
@@ -93,7 +93,7 @@ secrets*
 # recoverable only because that repo was private at the time. On 2026-09-17
 # a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC
 # repository, one `git add -A` from publication. Craigslist was fixed then
-# and the fix never reached anyone else, which is §16's rule unapplied: when
-# you fix one, check the siblings.
+# and the fix never reached anyone else: when you fix one, check the
+# siblings.
 .env*
 !.env.example

CHANGELOG.md

@@ -9,22 +9,80 @@ said so at the top of its release notes.
 
 ## [Unreleased]
 
-> **Behaviour change:** `--retries 0` is now a usage error (exit 2). It used
-> to be accepted and made the page-load loop unreachable, so nothing was
-> fetched and the engine reported the browser's own blank page as a result:
-> exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a
-> script of yours passes 0, pass **1**: the flag counts ATTEMPTS, so 1 means
-> "try once and do not retry".
+> **Behaviour changes.**
+>
+> * `--retries 0` is now a usage error (exit 2). It used to be accepted, and
+>   the page loop (`while attempt < args.retries`, after the landing) then
+>   never sent the listing request: the run fetched no page and ended with
+>   no rows. If a script of yours passes 0, pass **1**: the flag counts
+>   ATTEMPTS, so 1 means "try once and do not retry".
+> * A page the site SERVED with properties in it that parses to no rows is
+>   now a failed page, `stop_reason: parser_found_nothing`: exit 5 when
+>   nothing was gathered, exit 6 (`partial`) after earlier pages. It used to
+>   be exit 4 ("the listing is empty") on page 1, and a complete run with
+>   exit 0 ("the listing ended") on a later page.
+> * Negative values for `--delay`, `--retry-delay` and
+>   `--proxy-block-retries` are usage errors (exit 2) naming the flag.
+> * Output files (`.json`, `.csv`, `.meta.json`) are written to a temporary
+>   file beside the target and renamed over it. A new file gets the mode
+>   `open()` would have given it (`0o666 & ~umask`); a file that already
+>   exists keeps its own mode.
+> * CSV cells that begin with `=`, `+`, `-`, `@`, a tab, CR or LF get a
+>   leading apostrophe, so a spreadsheet reads them as text. JSON is
+>   unchanged, and the sidecar records `csv_cells_escaped`.
+> * `.env.example` ships `MAKEMYTRIP_URL` commented out, and a URL from the
+>   environment no longer blocks typed query flags.
 
 ### Fixed
 
-- **`--retries 0` fetched nothing and called it a result.** The load loop is
-  `range(1, retries + 1)`, so 0 never runs it and the engine judged
-  `about:blank`. Three third-party audits of this family ran exactly that and
-  reported a working scraper as broken. `--retries` now goes through
-  `page_flow.attempts` in every entry point; a check reads the source of each
-  one and calls the validator, so a new engine declaring the flag with a
-  plain int fails the suite.
+- **`--retries 0` fetched nothing and called it a result.** `--retries` now
+  goes through `page_flow.attempts` in every entry point; a check reads the
+  source of each one and calls the validator, so a new engine declaring the
+  flag with a plain int fails the suite.
+- **A served page whose rows the parser lost read as an empty or a finished
+  listing.** With every hotel's `id` renamed in a real Goa capture, the page
+  still classified as content holding 4 properties and parsed to 0 rows;
+  `--pages 1` exited 4, and a good page 1 followed by such a page 2 exited 0
+  with `status: complete`. Both are now parser failures (see above), and the
+  response is saved as `<out>_pageN_debug.html` for the issue.
+- **The README's first run failed after `cp .env.example .env`.** The
+  example shipped a real `MAKEMYTRIP_URL`, the engines applied `.env` before
+  checking for a `--url`/`--city` conflict, and `--city Goa` exited 2 with
+  "--url already carries the query". The line is now commented out, each
+  engine records whether `--url` was typed before `.env` is applied, and an
+  environment URL gives way to typed query flags. A typed `--url` with
+  `--city` is still refused.
+- **One page could buy three captcha solves, and a run by city name five,**
+  against `SOLVES_PER_PAGE = 1`: the allowance restarted on every landing,
+  and a refused page lands again after each relaunch. Measured with the
+  solver stubbed; real spend also needs `TWOCAPTCHA_KEY` and a captcha this
+  site has not been seen to serve. Every landing for a page now spends one
+  `page_flow.SolveBudget`, checked before each purchase and never reset by a
+  retry or a rotation, and the city lookup spends page 1's.
+- **A write that failed halfway destroyed the previous good output.** A
+  failure injected inside `json.dump` left `out.json` and `out.meta.json`
+  as 10-byte fragments of invalid JSON. Writes are now atomic per file.
+- **Formula-shaped text from listings reached CSV cells unescaped** (see
+  above).
+- **A CDP 401 was always explained as credentials that "last about a
+  day".** That lifetime was not measured and is not a constant. The message,
+  README, TROUBLESHOOTING, `.env.example`, the bug template and the canary
+  now say the endpoint's credentials were refused (expired, revoked or
+  mistyped).
+- `.dockerignore` named another repo's output prefix; it now lists this
+  repo's (`makemytrip_hotels.*`), `.env.*`, canary outputs and `.pageN`
+  dumps.
+
+### Changed
+
+- **The canary's Scraping Browser job runs on the schedule** instead of on
+  dispatch only, and FAILS with an error naming `MAKEMYTRIP_CDP_ENDPOINT`
+  when that secret is missing, instead of skipping green. The datacentre
+  job still expects Akamai's refusal (exit 3) and says it does not test
+  extraction; the Scraping Browser job is the one that requires real rows.
+- Comments, docs and the suite state their reasons in this repo's own words
+  instead of pointing at section numbers in notes the public cannot read. A
+  suite check keeps it that way.
 
 ## [0.1.1] — 2026-09-25
 

CONTRIBUTING.md

@@ -65,8 +65,8 @@ break, and each is loud or guarded:
    behind Akamai Bot Manager, and what was measured to get through
    (2026-09-24) is the Scraping Browser API with a `country-in` profile. If
    that stops being served, the refusal is reported as exit 3 with
-   `blocked_akamai` or `blocked_akamai-reset`, and the dispatch-only canary
-   job says so.
+   `blocked_akamai` or `blocked_akamai-reset`, and the canary's Scraping
+   Browser job goes red.
 
 If you are reporting a break, say which of those four it is, and attach the
 `--dump-html` output: the exact JSON the parser was given, on success as well
@@ -94,12 +94,13 @@ Then the rest of the presentation, in the order that matters:
    both halves. The daily job needs no secrets: it runs from a GitHub runner
    (a datacentre address) and asserts that Akamai's refusal is reported as
    exit 3, not as an empty or a crashed run, and if the runner is ever
-   served it checks the rows strictly instead. The second job is
-   dispatch-only and goes through a Scraping Browser profile (the
-   `MAKEMYTRIP_CDP_ENDPOINT` secret) to check a 3-page Goa run. A profile's
-   credentials last about a day, so refresh the secret before dispatching.
-2. The repo description, homepage and topics set (see the family notes on
-   what those should say).
+   served it checks the rows strictly instead. The second job runs on the
+   same schedule through a Scraping Browser profile (the
+   `MAKEMYTRIP_CDP_ENDPOINT` secret) and checks a 3-page Goa run. It FAILS
+   when the secret is missing or refused (a 401), so a green badge always
+   means real rows were read.
+2. The repo description, homepage and topics set: the words people search
+   for (the site, hotels, scraper, the engines), not a slogan.
 3. Only then the row in the org profile README — and check it with an
    ANONYMOUS request rather than your own logged-in browser. A row pointing
    at a private repo is a 404 for every visitor, which costs more trust than

README.md

@@ -187,9 +187,9 @@ One key, four separately-billed products ([2captcha.com](https://2captcha.com)):
 
 * **The Scraping Browser API** (`--cdp-endpoint`, `MAKEMYTRIP_CDP_ENDPOINT`)
   is what makes this repo work from a datacentre: a remote browser with an
-  Indian exit. One live connection per `pid`. A profile's credentials last
-  about a day; an expired one answers HTTP 401, which the engines report as
-  exit 5 naming the expiry.
+  Indian exit. One live connection per `pid`. A profile whose credentials
+  are refused (expired, revoked or mistyped) answers HTTP 401, which the
+  engines report as exit 5 naming the status.
 * **The Scraper API** (`scraper_api_client.py`) works here only routed
   through such a session, and then reads five properties per listing.
 * **Captcha solving**: no captcha was met on this site. The family's
@@ -254,9 +254,11 @@ no city.
 
 The [canary](.github/workflows/canary.yml) runs twice over. Daily, with no
 secrets, from a GitHub runner: a datacentre, so it asserts the refusal is
-REPORTED as one (exit 3, never a crash or an "empty listing"). On dispatch,
-through a Scraping Browser profile: a real 3-page Goa scrape, checked for
-prices, currency, duplicates and zero ratings.
+REPORTED as one (exit 3, never a crash or an "empty listing"); that job
+tests the refusal path, not extraction. On the same schedule, through a
+Scraping Browser profile: a real 3-page Goa scrape, checked for prices,
+currency, duplicates and zero ratings. That job fails when its secret is
+missing or refused, so the badge is green only when real rows were read.
 
 ---