Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e19b389
flight-cli: require carrier corroboration on the route+time award fal…
ak2k Jul 29, 2026
fb59713
flight-cli: resolve award route+time buckets instead of filtering them
ak2k Jul 30, 2026
2569c04
flight-cli: use arrival time as the award-match identity
ak2k Jul 30, 2026
a6657d9
flight-cli: stop re-querying airlines PointsPath doesn't support
ak2k Jul 30, 2026
62a4662
flight-cli: per-entry TTL on the unsupported-airline cache, + review …
ak2k Jul 30, 2026
4a8dac8
flight-cli: fix P0 — arrival must narrow within carrier stages, not b…
ak2k Jul 30, 2026
f6383e2
flight-cli: resolve the primary and matched-id keys too, not just the…
ak2k Jul 30, 2026
6a3530e
flight-cli: normalize seats.aero timestamps + wrap PP catalog HTTP er…
ak2k Jul 30, 2026
13c6564
flight-cli: fix 3 P0s found by cross-model (codex/gpt-5.6) adversaria…
ak2k Jul 30, 2026
6e1f2a6
flight-cli: resolve once over the union of all keys, not per-bucket
ak2k Jul 30, 2026
a38f608
flight-cli: use connection airports as the cross-provider journey ide…
ak2k Jul 30, 2026
a9ad051
flight-cli: one price-less Google row no longer discards the whole re…
ak2k Jul 30, 2026
49b7262
flight-cli: repair NID cookie seeding, silently broken by the fli 0.9…
ak2k Jul 30, 2026
f07645e
flight-cli: renderer must state the truth about its own row
ak2k Jul 30, 2026
f2ca073
flight-cli: date a pinned segment by when it departs, not when it lands
ak2k Jul 30, 2026
8335cab
flight-cli: --pick must index the rows the user actually saw
ak2k Jul 30, 2026
dbdb77e
flight-cli: fix typecheck in the --pick ordering test
ak2k Jul 30, 2026
8fa909a
flight-cli: diskcache-backed HTTP cache with a 15-minute TTL
ak2k Jul 30, 2026
709878a
flight-cli: support multi-city links instead of silently dropping legs
ak2k Jul 30, 2026
c074c5a
flight-cli: unambiguous cache key, honest tax currency, canonical cab…
ak2k Aug 1, 2026
6f69b8b
flight-cli: carry the stop limit into the Google Flights search URL
ak2k Aug 1, 2026
db43197
flight-cli: carry routing, extension and arrival-date into the Matrix…
ak2k Aug 1, 2026
02c1f8f
flight-cli: automate the Matrix SPA capture — no human step
ak2k Aug 1, 2026
48a7e36
flight-cli: create the PointsPath token file 0600 instead of widening…
ak2k Aug 2, 2026
fc32be1
flight-cli: fix four cli.py defects found by cross-model adversarial …
ak2k Aug 2, 2026
9d054ee
flight-cli: fix the remaining three cli.py findings + the null-price …
ak2k Aug 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ htmlcov/
# `research/*` not `research/` so git can still see inside the dir.
research/*
!research/extract_help_docs.py
!research/capture_matrix_spa.py

# Claude Code per-session agent state. Project-shared `.claude/settings.json`
# and `.claude/skills/` ARE checked in; per-machine override stays local.
Expand Down
55 changes: 55 additions & 0 deletions docs/memories/matrix_spa_url_state.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Matrix SPA URL state vs the /batch API

Captured 2026-08-01 by driving the real Matrix UI (patchright + real Chrome,
Advanced controls, one-way JFK→LHR with `Routing=BA+`, `Extension=MAXSTOPS 0`).

## The trap

The SPA's URL state and the `/batch` API use **different names for the same
values**. Guessing from the API side gets you a link the app ignores.

| Concept | `/batch` API (`wire.py`) | SPA URL state (`links.py`) |
|---|---|---|
| Routing language | `routeLanguage` | `routing` |
| Extension codes | `commandLine` | `ext` |
| Return-leg routing | (own slice) | `routingRet` |
| Return-leg extension | (own slice) | `extRet` |
| Arrival-date intent | `isArrivalDate: bool` | `departureDateType: "depart"|"arrive"` |

Round trip folds into ONE slice, which is why the inbound leg needs the
separate `*Ret` keys rather than a second slice.

## Presence is conditional

With no routing codes set the SPA **omits all four keys**; with any set it
emits all four (blank string for the unused ones). `_spa_routing_fields`
mirrors that, so our links stay byte-identical to the app's own in both cases
— the tracked fixtures in `tests/fixtures/matrix_url/` cover both shapes.

## Capture recipe

`research/capture_matrix_spa.py` does this **unattended** — re-run it whenever
the SPA changes:

uv run --with patchright python research/capture_matrix_spa.py

Headless is blocked by `waa-pa` bot attestation, so it drives real Chrome via
patchright, but needs no human. It records both surfaces at once: `page.url`
(decode the `search=` base64) and `page.on("request")` filtered to
`alkali`/`batch` (the API body).

Four form-driving traps, each of which silently leaves Search **disabled**:

1. Airports are an autocomplete — type, then **click the `mat-option`**.
`fill()` leaves the underlying model empty.
2. The date input has **no placeholder**; select it by
`input.mat-datepicker-input`.
3. The date must be typed with **`press_sequentially`**. `fill()` sets the
visible value but does not fire the events Angular's form model listens
for, so Search stays disabled with a date plainly showing — the most
misleading of the four.
4. `mat-input-*` ids are regenerated per render. Never select on them;
`input[placeholder="Routing"]` / `"Extension"` are stable.

Order matters too: pick airports **before** switching to One way, or the date
control isn't rendered yet.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ dependencies = [
"flights>=0.9", # fli — drives Google Flights for booking handoff
"fast-flights>=2.2", # tfs= URL encoder (Google Flights deep-link)
"rookiepy>=0.5", # reads Chrome cookies for `auth pp login --from-chrome`
"diskcache>=5.6", # HTTP response cache: per-entry TTL, LRU, SQLite/WAL
]

[project.optional-dependencies]
Expand Down
138 changes: 138 additions & 0 deletions research/capture_matrix_spa.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
#!/usr/bin/env python
"""Capture the Matrix SPA's URL state + /batch request body, unattended.

Why this exists: the SPA names things differently from the /batch API
(`routing`/`ext` vs `routeLanguage`/`commandLine`), so `links.py` cannot be
written from the API side — the field names have to be observed. See
docs/memories/matrix_spa_url_state.md.

Run: uv run --with patchright python research/capture_matrix_spa.py

Headless is blocked by Google's waa-pa bot attestation, so this drives real
Chrome via patchright. It still runs without supervision.

Form-driving notes, each learned the hard way:
* Airports are an autocomplete: type, then CLICK the mat-option. `fill()`
leaves the model empty and Search stays disabled.
* The date input has NO placeholder — select it by `input.mat-datepicker-input`.
* The date must be typed with `press_sequentially`; `fill()` sets the value
but doesn't fire the events Angular's form model listens for, so Search
stays disabled with a date visibly present.
* `mat-input-*` ids are regenerated per render — never select on them.
"""

from __future__ import annotations

import base64
import json
import pathlib
import sys
import urllib.parse

from patchright.sync_api import Page, sync_playwright

PROFILE = "/tmp/mx-capture-profile"
URL_OUT = pathlib.Path("/tmp/matrix_spa_state.json")
REQ_OUT = pathlib.Path("/tmp/matrix_batch_body.json")


def _decode_state(url: str) -> dict | None:
if "search=" not in url:
return None
q = urllib.parse.unquote(url.split("search=", 1)[1].split("&")[0])
q += "=" * (-len(q) % 4)
try:
return json.loads(base64.b64decode(q))
except Exception:
return None


def _pick_airport(pg: Page, index: int, code: str) -> None:
box = pg.locator('input[placeholder="Add airport"]').nth(index)
box.click()
box.type(code, delay=120)
pg.wait_for_timeout(1800)
option = pg.locator("mat-option, [role=option]").first
option.wait_for(state="visible", timeout=10_000)
option.click()
pg.wait_for_timeout(700)


def capture(origin: str, dest: str, date_mmddyyyy: str, routing: str, extension: str) -> int:
bodies: list[dict] = []

with sync_playwright() as p:
ctx = p.chromium.launch_persistent_context(
PROFILE, channel="chrome", headless=False,
viewport={"width": 1500, "height": 1000},
)
pg = ctx.pages[0] if ctx.pages else ctx.new_page()

def on_request(req):
if "alkali" not in req.url and "batch" not in req.url:
return
body = req.post_data
if body and ("routeLanguage" in body or "commandLine" in body):
bodies.append({"url": req.url.split("?")[0], "body": body})

pg.on("request", on_request)
pg.goto("https://matrix.itasoftware.com/", wait_until="domcontentloaded")
pg.wait_for_timeout(5000)

_pick_airport(pg, 0, origin)
_pick_airport(pg, 1, dest)

pg.locator("text=One way").first.click()
pg.wait_for_timeout(1500)

date_in = pg.locator("input.mat-datepicker-input").first
date_in.click()
pg.wait_for_timeout(600)
date_in.press_sequentially(date_mmddyyyy, delay=90)
pg.wait_for_timeout(800)
pg.keyboard.press("Tab")
pg.wait_for_timeout(1200)

pg.locator("text=Advanced controls").first.click()
pg.wait_for_timeout(1500)
pg.locator('input[placeholder="Routing"]').first.fill(routing)
pg.locator('input[placeholder="Extension"]').first.fill(extension)
pg.wait_for_timeout(600)

search = pg.locator('button:has-text("Search")').first
if search.get_attribute("disabled"):
print("Search still disabled — the form shape changed", file=sys.stderr)
ctx.close()
return 1
search.click()

state = None
for _ in range(90):
pg.wait_for_timeout(1000)
found = _decode_state(pg.url)
if found and (found.get("slices") or [{}])[0].get("routing"):
state = found
break
if found:
state = found
if bodies:
break

if state:
URL_OUT.write_text(json.dumps(state, indent=1))
print(f"URL state -> {URL_OUT}")
print(json.dumps(state["slices"][0], indent=1))
if bodies:
REQ_OUT.write_text(json.dumps(bodies, indent=1))
print(f"batch bodies -> {REQ_OUT} ({len(bodies)})")
ctx.close()
return 0 if state else 1


if __name__ == "__main__":
raise SystemExit(
capture(
origin="JFK", dest="LHR", date_mmddyyyy="09/01/2026",
routing="BA+", extension="MAXSTOPS 0",
)
)
26 changes: 24 additions & 2 deletions src/flight_cli/_gflight_ids.py
Original file line number Diff line number Diff line change
Expand Up @@ -434,8 +434,9 @@ def _seed_cookies_once(client: Any) -> None:
log.debug("gflight cookie cache past TTL; re-warming")
return
try:
jar = _cookie_jar(client)
for c in saved:
client._client.cookies.set(
jar.set(
c["name"],
c["value"],
domain=c.get("domain", ".google.com"),
Expand All @@ -445,6 +446,27 @@ def _seed_cookies_once(client: Any) -> None:
log.debug("could not seed gflight cookies: %s", e)


def _cookie_jar(client: Any) -> Any:
"""The session cookie jar, across fli client shapes.

fli <=0.8 exposed `Client._client`; 0.9 replaced it with a per-thread
`Client._session()`. Both hand back an object with the same `.set()` /
`.jar` interface. Reaching for the old attribute silently raised
AttributeError into this module's best-effort `except`, which turned NID
seeding AND persistence into no-ops — so every process started cold, and
the comments above put the cold-start empty rate at ~40% versus ~0% warm.

Raises AttributeError when neither shape is present, so a future upstream
rename fails loudly at the callers' `except` + debug log rather than
degrading silently forever.
"""
session = getattr(client, "_client", None)
if session is None:
# fli 0.9: per-thread session accessor replaced the old `_client` attr.
session = client._session() # pyright: ignore[reportAny]
return session.cookies # pyright: ignore[reportAny]


def _persist_cookies(client: Any) -> None:
"""Write the session's allowlisted Google cookies (NID) to disk after a warm
call, once per process, so the next invocation starts warm. Best-effort."""
Expand All @@ -458,7 +480,7 @@ def _persist_cookies(client: Any) -> None:
"domain": str(ck.domain or ".google.com"),
"path": str(ck.path or "/"),
}
for ck in client._client.cookies.jar # pyright: ignore[reportAny] # fli/curl_cffi untyped
for ck in _cookie_jar(client).jar # pyright: ignore[reportAny] # fli/curl_cffi untyped
if str(ck.name) in _PERSIST_COOKIE_NAMES
and _GOOGLE_DOMAIN_SUFFIX in str(ck.domain or "")
]
Expand Down
Loading
Loading