Repository navigation
feat(wayback): import archive history at archive.org's pace, finding changes instead of sweeping quarters - #391
Merged
Conversation
Once archive.org has throttled a client for long enough it stops answering 429 and refuses the TCP connection (seen as connection refused within 200 ms after two CDX calls 5 s apart). Both backends read a fetch error from the CDX index or the availability API as "no capture", and a refused replay as a skipped target. The index then fell back to up to seven availability probes per target, each refused too, so every target became skipped_no_capture, Save Page Now failed and the bulk runner marked the app done with no history. A resumed 201-app run raced through the queue that way, making about 170 refused requests per app and prolonging the block. A request to the index, the availability API or a replay that fails below HTTP is now Unavailable / WaybackUnavailableError, status 0 and no Retry-After: the transport's "fetch failed" (refused or reset), "terminated" (dropped mid-body), timeout and unresolved-host errors, with messages such as "archive.org refused the connection for CDX index". Both sides classify by message, so other fetch errors (a blocked URL, a body over the cap, a redirect loop) still fall back or skip, and aborts stay aborts. The bulk runner then backs off and pauses the queue as rate_limited, and the per-app route answers 503 archive_unavailable. The bulk runner's wait without a Retry-After rises from 30 s to 300 s, since reported blocks last about five minutes, and its cap from 120 s to 900 s. - Node: lib/wayback.ts (waybackTransportFailure), lib/historical-import.ts, lib/wayback-bulk-runner.ts. - Rust: core/src/scrape/wayback.rs (Unavailable::transport), core/src/scrape/history.rs, core/src/server/wayback_runner.rs. Oracles: the history case that used "fetch failed" for a quiet replay failure now uses a body over the cap, which behaves as before; seven history cases (index refused, timed out, unresolved, redirect loop still falling back, availability refused, replay refused and dropped) and two runner cases (a refusal backing off for the default wait, a refusal on the retry pausing the queue) are appended. New Node tests in tests/app/wayback-refused-connection.test.ts; Rust tests assert the error is flagged as throttling, not only its message. Negative controls: the old Rust classification failed exactly the six throwing history cases and both runner cases; the old 30 s default alone failed only backoff_is_bounded and the default-wait case; a Node classifier that never matches, with the old default, failed all seven new Node tests. Save Page Now, lookup_latest, the activity wording and the sampling strategy are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A real paused bulk run logged a backoff of 30000 ms for "archive.org rate-limited for availability API" after 21 s: the CDX index timed out (20 s), the import fell back to availability probes, and the first probe got a bare 429. The new test replays that: a timed-out index now throws "archive.org timed out for CDX index" with no availability probe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds three HistoryOptions fields the bulk runner will set: a held capture list in place of a CDX call, skip Save Page Now, and fail with INDEX_UNAVAILABLE instead of fanning out into availability probes when the index is unusable. Defaults keep today's behaviour; the history oracle is unchanged. docs/WAYBACK_IMPORT.md is the design and the working contract for the Rust-only redesign: packages, file ownership, API shapes and the rules for the agents building it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Wayback import is Rust-only from its redesign (docs/WAYBACK_IMPORT.md), so history-cases.json and wayback-runner-cases.json become Rust-owned regression fixtures and need a way to be updated without the Node extractors. With PT_BLESS=1 the two replays still run every case, then rewrite its outputs (calls, stream, rows, expected; the NDJSON frames ride in the expected body) from the actual results, leave its inputs alone, write the file back in the extractors' format and pass. A case written by hand with inputs only gets its outputs appended; the runner replay dumps the six recorded tables for it. Without the variable both replays compare exactly as before. The helper (core/src/bless.rs, test-only) also pins that both fixtures are in the form a bless writes, so blessing unchanged code moves no byte. Checked: a bless of the current tree rewrote 0 of 33 and 0 of 49 cases with git diff --exit-code clean, and a bless over fixtures with outputs removed or corrupted in three cases each (and reformatted) rewrote exactly those cases and restored both files byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Wayback import is Rust-only from its redesign (docs/WAYBACK_IMPORT.md): the Node rollback keeps the importer and bulk runner it had before, so re-recording history-cases.json and wayback-runner-cases.json from Node would undo every intended change. Drop the two core-parity steps that ran the extractors and diffed the fixtures, and delete extract-history-cases.mjs and extract-wayback-runner-cases.mjs. Both fixtures stay gated by the core crate tests and are updated with PT_BLESS=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/wayback/import-all's payload is Rust-only from the Wayback redesign (docs/WAYBACK_IMPORT.md): the core's state carries the redesigned runner's blob, while the Node rollback keeps the old runner and its v2 blob. Move it from READS to QUARANTINE with that reason, so the differ stops byte-comparing a body the backends no longer share and its HEAD probe still checks routing and the gate on both. read-parity.mjs selects manifest entries by route (its BATCH_1 list), not by this entry, and handoff.mjs reads only MUTATIONS, so neither depended on it. pnpm parity:manifest: 127 routes, all classified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core/README.md: a "Rust-owned fixtures" gate note (why two fixtures left the Node oracle, the bless commands, how to review a blessed diff and add a case), and the history and Wayback runner oracle paragraphs rewritten as Rust-owned fixtures, with the status GET now quarantined. AGENTS.md: the Wayback section opens with the redesign being Rust-only, the Node rollback keeping the pre-redesign importer and runner, how to bless, and the two fixtures still recorded from Node that cover the import's routes (imports-cases.json, operations-cases.json). docs/WAYBACK_IMPORT.md: P0's fixture handover marked done, with the exact bless commands and what P0 left Node-recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parse the v3 bulk state and the new NDJSON frames (phase, survey-app, survey-done, waiting, estimate), derive the run line, survey result, estimate, wait and app-level totals, describe a per-app result in label changes, and find the window a reconstructed change happened in. Every redesign field is optional: with none present each derivation returns null, so the Node rollback keeps today's card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run line, survey result, wait, estimate and app-level totals for Settings, label-change results and the busy-until time for the detail card, and the timeline's "Changed between" caption. Rewords the copy that described a quarterly sweep and drops the em dashes from the Wayback strings. zh.json by hand while Crowdin is disabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run line names its phase ("Checking the archive index: 37 of 201
apps", "Reading archived pages: 12 of 143 apps"), and the card shows
what the survey found, the time left, the wait while archive.org has
asked the import to slow down ("resumes at 14:32"), and totals in label
changes and pages read. The last run reads in apps checked, label
changes and apps with no archived pages, and a run the server resumed
after a restart now counts as the last run too.
The hook feeds every NDJSON frame through the pure reducer and merges
the v3 state from the poll. Without the new fields (the Node rollback)
the card renders the checkpoint tally exactly as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A throttled import (503 archive_unavailable) now says when to try again, from retryAfterMs or the Retry-After header. A change-finding result reads as label changes found, notes how many archived pages were read, says when changes found were already on the timeline, and says when archived history starts well after February 2021. A result without those fields reads as snapshot rows, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A wayback change row directly after an unchanged wayback row says the change happened between the two archived copies' dates. Adds a story with that pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-app import no longer reads one archived page per quarter. It reads a skeleton (the earliest capture after the February 2021 floor, the newest, and the capture nearest each cadence step back from today, yearly by default), then bisects wherever two neighbouring samples differ until each change sits between captures adjacent in the index or at most a week apart. Rows go in for the baseline, both sides of each change and the newest read. Stored wayback rows are samples: a skeleton date within the dedupe window of one costs nothing, and at the default cadence neither does a date between two stored rows with the same labels, so a re-run or a retry after throttling reads only what is new. A page without labels before any page with them is bisected like a change, so the baseline lands where labels first appeared rather than a year later. Unusable captures hand over to the next nearest, two reads per point. Without an index, one availability probe per skeleton date and no bisection. The planner (Sampler) does no I/O and is tested over synthetic and random timelines. The result adds reads, changes, labelVersions, firstCaptureMs, lastCaptureMs and windows; each target adds its phase. The history-cases.json replay still expects the quarterly sweep until the fixture is re-blessed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eight cases through import_app_history and the real transport: change rows on the captures that bracket each change in five years of daily captures (18 reads, not 22 a year); a stable app costing the six skeleton pages; a change reverting between skeleton reads going unseen at the yearly cadence and found at a quarterly one; unusable captures handing over to the next nearest; stored rows making a re-run read only the index; force re-reading without storing a capture twice; a throttle mid-bisection storing what it learned so the next run reads only the rest; and the no-index path probing each skeleton date once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Reading archived pages: 12 of 143 apps" is long enough to wrap at phone width, and the run line's flex row squeezed the app name after it into a sliver. The lead and the name now share one text flow beside the spinner; at desktop width the line looks as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Package P3 of docs/WAYBACK_IMPORT.md, Rust only. - Throttling waits instead of pausing: waitingUntil (Retry-After, five minutes without one, no upper cap) and waitReason are stored, a `waiting` frame replaces `backoff`, and the same app is asked again. A wait sleeps in one-second slices, so a pause stored by the PATCH ends it within a second and a cancel at once; a long wait rewrites the state every ten minutes so the health check never reads it as dead. Six throttles in a row with no app finished pause the queue with pauseCause rate_limited, as before. - Fresh runs survey first: one CDX listing per app, cached for a week as wayback.captures.<appId>, apps with no captures finished as noArchive, the rest ordered (never imported first, then most archived). Reading passes the cached listing with Save Page Now and the availability fallback off. - State blob version 3: phase, consecutiveThrottles, survey, estimate (from wayback_pacer_state.perMinute, 10 when absent), queue and totals in apps, reads and label changes. A v1/v2 blob resumes as a reading phase; a run stopped mid-survey or mid-wait resumes where it was. - Every boundary write carries over a pause or cancel the PATCH stored meanwhile, so a pause asked for while an app is read is no longer lost. - Activity rows in apps and changes; frames phase, survey-app, survey-done, waiting and estimate. - GET /api/wayback/import-all projects the v3 fields for a v3 blob only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scripted archive and a fake clock (the runner's waits advance it instead of sleeping): wait-and-retry, the five-minute default, the pause after six throttles and the count reset by a finished app, pause and cancel during a wait, a pause asked for mid-app, the survey's cache reuse and ordering, an unreadable index, a v2 queue resumed as reading, a run resumed mid-survey and mid-wait, the estimate, and the status route's v3 projection. Each test has its own run ids, since the cancel registry is keyed by them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A change-finding result with pages read but no label state among them (the first weeks of 2021, or pages that were not product pages) read "No label changes found", as if the labels had stayed the same. It now says none of the archived pages had privacy labels to read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Blessed with P0's PT_BLESS on a throwaway merge (outputs only), then reviewed case by case against the Node recording. Inputs edited by hand: - The newest capture is always a skeleton read now, so cases 0, 1, 8, 10, 11 and 23 get a reply for 2021-10-20 that keeps their story; in 9 and 31 its labels differ, so the throw still lands mid-bisection, after the rows learned so far are stored. Case 2 gets 2021-02-15 so force visibly re-reads a stored capture without storing it twice. - The install date is no longer a skeleton date. Its three cases now pin that: "the install date adds no read", "an index of one capture is read once", "skeleton dates before the first capture are dropped". "target walk from a month end" keeps its name and pins 31 May minus three months (3 March) through the skeleton at a quarterly cadence. - Appended: bisection narrowing a change to a week, the stretch before labels appeared, stored rows covering the skeleton between them, a throttle after two midpoints, and probes at a quarterly cadence. - Replies nothing asks for are dropped, so each case lists what it uses; a re-bless after the prune rewrote 0 of 38 cases. The eight cases that throw before any read are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Option::is_none_or is stable from Rust 1.82 and the core declares 1.77, which clippy's incompatible_msrv flagged. map_or(true, ..) as elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The no-index fallback is one probe per skeleton date now, not up to seven per target; held captures replay their own address; the probe walk records probed times rather than returning them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wait runs until the later of archive.org's Retry-After and the paced client's stored `wayback_cooldown_until`. A retry inside the cooldown would only meet the client's own synthetic 429, spend one of the six throttles the queue tolerates, and wait again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p2-change-finding-3205b9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p3-runner-53b722
The replies of every case whose requests moved are re-sequenced (the survey lists every app before any page is read, and a bulk run never asks Save Page Now), the hooks retargeted, and six cases renamed for what they now pin: a pause asked for mid-app stops after the app instead of being overwritten, a pause ends an archive wait, a cancel lands mid-survey, six throttles in a row pause the queue (five, then a refused connection, too) and a refused connection waits the default five minutes. Six cases are appended: a fresh listing reused, never-imported apps read first, a throttled read waited out and read again, a survey and a wait resumed at boot, and an unreadable index failing its app in the survey. The replay's waits take no time except in a case with a delayed hook, which has to land during one. Outputs blessed with PT_BLESS=1; every rewritten case reviewed against its Node recording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… moves Every case that reaches the archive has one capture or none, so the skeleton is one read or none: requests, the replies they are paired with, hook positions, write counts and stored snapshot rows are as Node recorded them. What moves is what the import reports: activity and audit lines no longer count 22 or 23 skipped quarter targets per app, the bulk state's totals follow, and streamed target frames carry a phase and report the read as sampled (the app-done result marks it imported). A re-bless rewrites 0 of 49. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P3's shapes are final: reading progress is totals.appsRead of the survey's appsWithCaptures, and the GET never carries the queue. The done + failed - appsNoArchive stand-in over-counted apps whose index listing failed: they finish as failed in the survey and are never read. That stand-in now only covers totals without appsRead. The stream counts apps with no captures and failed listings where they finish, on their survey-app frame (a failed one has captureCount null and an error), so they never count toward reading. The survey line says how many apps could not be checked, the closing line how many failed, and etaMs is read as the duration the runner reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
changes and labelVersions count the rows already on file, so an app with imported history whose index answered empty now reads as its changes rather than "archive.org has no archived pages for this app". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health check's lock heal reads the blob through `read_bulk_state`, so it needs no change for version 3: a waiting run an hour silent is left alone, and one seven hours silent is healed as before. A 25-minute wait rewrites the state at 10 and 20 minutes, so `updatedAt` never falls past the six-hour stale margin however long archive.org asks us to wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wires the pacer into each path that reaches archive.org: - sync_runner.rs: start_background files the server's pacer (read from app_settings) before the first request is served and drops it when the server stops (file_archive_pacer); the boot-time Wayback resume runs through it. - runner_writes.rs: POST and PATCH /api/wayback/import-all, so the buffered run, the streamed run and the run a resume spawns all share it (a spawned run takes the pacer with its shared fetcher). - imports_writes.rs: the per-app POST /api/apps/[id]/import-history; in a cooldown it answers 503 archive_unavailable with retryAfterMs, as for any throttle. - policy_store.rs: fetch_and_store, which covers the source's Wayback fallback (policy/source.rs, tier 3), the existing-capture lookup and Save Page Now, and run_follow_ups' deferred Save Page Now. The policy's own site passes straight through. Every replay sees the pass-through pacer, so all stay byte-identical (cargo test: every lib and integration test passes). Negative control: with an unregistered connection given a real pacer on an instant clock instead, the Wayback runner replay failed exactly its four throttled cases (throttled once, throttled twice, streamed and throttled, throttled then refused), whose retries met the synthetic 429; the only other failures were the two tests that pin the pass-through for an unregistered connection. The control was removed before the passing run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts apps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rter Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # core/tests/fixtures/wayback-runner-cases.json
…ned import Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P3's runner cases were blessed against the quarterly import; with P2's
change-finding import merged, the 14 cases that read pages move. Only
outputs changed: no case's calls moved, so every case still makes the
same archive requests in the same order. A stable fixture app now reads
one page, its target frame carries "phase", and its activity line counts
that read ("1 page read" where it said "0 pages read" next to an import).
history-cases.json and import-history-route-cases.json re-bless with no
change; the full core suite passes (432 lib tests plus integration).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…core/README CHANGELOG: one Changed entry for the redesign (one pace for archive.org, waits instead of stopping, index first, change-finding, app-level progress; the Node rollback keeps the old import) and one for the Rust-owned fixtures; the #386 Fixed entry notes that the Rust server now waits instead of retrying once. AGENTS.md: the per-app route's cases are Rust-owned now, and a paragraph on how the Rust import works (pacer, Sampler, the v3 runner), for whoever touches it next. core/README.md: the third bless command, the imports oracle's real count (162) and its moved cases, and the pinned pause the redesign fixed. ci.yml: the imports oracle comment no longer claims the per-app Wayback import. docs/WAYBACK_IMPORT.md: fixture paragraph and package status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage (P5) lets a person add up to three older apps.apple.com addresses for an app whose archived history starts late. Parse the per-app result's lookupUrl, historyStartsAt, historyStartsLate and alternateUrls, prefer the server's late-start verdict (it also weighs the release date) over the first-capture heuristic, and describe the card's older-addresses block. Check an address before it is posted (apps.apple.com, this app's id, not already checked, at most three). Count late-starting apps from app-done results during a stream, and read totals.appsHistoryLate when the runner reports one. Every field is optional; without them nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An import now lists an app's US-storefront page first: labels belong to the app, and archive.org crawls US pages far more than any other. The stored address is listed only when the US index is empty (or unusable), not when it is merely smaller, which would cost every non-US app a second listing; an address with more history can be added instead. Older App Store addresses a user adds for the same app id are stored as wayback.alt_urls.<appId> (at most three, each https://apps.apple.com/<cc>/app/<slug>/id<sameId>), and both the survey and the per-app import merge their captures in by timestamp, each replayed at its own address. POST /api/apps/[id]/import-history takes alternateUrls, validates and stores it before importing, and answers a bad entry with 400 {error, code: "invalid_alternate_url"}. The result adds lookupUrl, alternateUrls, historyStartsAt and historyStartsLate; survey queue entries and frames add lookupUrl and historyStartsLate. apps keeps no release date, so a start counts as late from the February 2021 floor alone. The survey's cache records the stored address it was taken for and each older address's timestamps, and is listed again when either changes. Backups leave the capture cache out and keep the older addresses, and every app delete forgets both keys, reading first so an app without them deletes with exactly the statements it always did: the Node-recorded backup, library, device, import and runner oracles all replay unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit tests for the address helpers (any storefront to its US page; an older address must be this app's apps.apple.com page; a capture names its address) and for the late-start rule, and six end-to-end cases through the stub archive, which can now answer each index request by the address it names: the US page leads; an empty US index falls back to the stored address; a smaller US index still leads (and here starts late); older addresses merge by timestamp and replay at their own address; an older address with an unusable index adds nothing; and a throttled older address is the import's error before any page is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The detail card's explanation, address input, stored list, remove control and the plain-words errors for an address that cannot be used, and the Settings line for apps whose archived history starts late. The start-date note now says the archive has no earlier copy of the page at this address, since older pages may sit under an old one. zh.json by hand while Crowdin is disabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the server says an app's archived history starts late, the card explains that a renamed app's older pages may sit under its old address and takes up to three older apps.apple.com addresses, posted as the whole alternateUrls list with a forced import; each stored address can be removed the same way. An address is checked before it is posted, and the route's invalid_alternate_url is said next to the input in plain words, with the last result left standing. Results from a route without coverage show no block. Also stops the card's copy keeping a 260px height when it stacks at phone width. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One muted line in the running import and in the last run, only when the count is above zero: "Archived history starts late for 5 apps. You can check them one by one on each app's page." Counted from the app-done results of a run this tab streams, or read from totals.appsHistoryLate when the runner reports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fake archive can list by address. Three cases: the survey lists the US page and an older address (the stored one is never asked), caches them with the stored address they were taken for, and replays the oldest pages at the old address; a cached listing is taken again when the stored address or the older ones change, or when it predates the US lookup for a non-US page, and reused when nothing changed; and deleting an app forgets its listing and older addresses, through the orphan sweep too, writing nothing for an app that has neither. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The P5 route stores only https://apps.apple.com/<storefront>/app/<slug>/ id<n> with a lowercase two-letter storefront, and refuses /app/id<n> or a storefront without a slug. The client check now mirrors that (canonical_app_store_address in core/src/scrape/wayback.rs), so those are caught before posting, and its message shows the full shape with the app's own id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p5-ui-68d9d2
The survey's wayback.captures.<appId> listings are a cache archive.org can answer again; the older App Store addresses a user typed (wayback.alt_urls.<appId>) stay in the export. Node's backup has no such cache, so the recorded backup cases cannot pin it; they replay unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p5-coverage-3205b9
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Blessed on the branch merged with the integration tip, whose fixtures already matched its code, so the diff is P5's alone and purely additive: every import result gains lookupUrl, alternateUrls, historyStartsAt and historyStartsLate (in the route's responses and the activity rows too); survey queue entries and survey-app frames gain lookupUrl and historyStartsLate; the capture cache gains storedUrl. No case's requests moved: every fixture app is a US page with no older addresses, so the lookup is the one listing it always was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Appended, inputs only, outputs blessed and reviewed. The route stores older addresses (two spellings of one collapse to it), lists the US page then the old address, reads the oldest pages at the old address and the newest at the US page, and asks Save Page Now; it refuses another app's address, more than three, and a non-list with 400 invalid_alternate_url before writing anything; and an empty list deletes the stored addresses before importing. The history fixture, whose stub can now answer each index request by address, pins the US page leading, the stored address used when the US index is empty, and an older address merged in with its own replays and change window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p5-coverage-3205b9
totals.appsHistoryLate counts finished apps whose import result says historyStartsLate, next to appsWithHistory and appsNoArchive, so it reaches GET /api/wayback/import-all and the run's activity row: the status read carries no queue, and the Settings card needs the count after a reload and in the last-run view. A missing flag reads as false. A queue resumed from before the count fills it from its finished, read entries, so a v1 or v2 blob (which never carries the flag) counts 0 and reads byte-identically through the status routes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Additive only, and no request moved: appsHistoryLate appears wherever the totals do (responses, frames, activity details, state blobs and their writes). It is 1 for each archived fixture app, whose captures start in July 2026, 0 for apps with no captures and for the entries a resumed v2 queue carried over. operations-cases.json is untouched and replays unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An Added entry for the US-first lookup, older addresses and the late-start hint; the AGENTS.md paragraph gains the coverage part (the listing rule, the alternate-address contract, which settings are cache and which are the user's); the contract doc records appsHistoryLate in the totals and the two known limits of the late-start check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A bulk Wayback import of 201 apps finished one app, then paused on rate limiting. Four things were wrong at once:
The summary also counted checkpoints, not apps: "2 imported, 5 no-op, 17 skipped" was one app's 24 dates.
What changes
Rust server only. The Node rollback (
BACKEND=node) keeps the pre-redesign importer, so nothing underlib/that the Node server runs changes.docs/WAYBACK_IMPORT.mdis the design and the contract the work was built to.core/src/scrape/archive_pacer.rs). Every request to archive.org, including the privacy policy's archive lookups, shares one pace per server: 10 a minute to start, between 2 and 15, halved by a throttle, plus one after 20 successes in a row. A 429, a 5xx or a refused connection starts a cooldown of the longer of Retry-After and 5 minutes doubling per throttle in a row, capped at an hour; during it, archive requests get a synthetic 429 and never reach the network.wayback_cooldown_untilandwayback_pacer_statecarry both across restarts. Replay harnesses get a pass-through pacer, which is what keeps every Node-recorded oracle byte-identical.core/src/scrape/history.rs). A pure planner reads a skeleton (the earliest and newest captures and one a year between them), then bisects wherever two neighbouring samples differ until each change sits between captures that are adjacent or at most 7 days apart. About 6 reads for an app whose labels never changed, about 6 more per change. Stored rows count as samples, so a re-run reads only what is new, and a throttled run keeps what it found. Each change is pinned to a window the timeline can show.core/src/server/wayback_runner.rs). A fresh run surveys the index first (one listing per app, cached for 7 days), finishes apps with no archived pages at once, and reads never-imported apps first. A throttle waits until the later of Retry-After and the cooldown, in 1 s slices so pause and cancel still land, and carries on; the sixth throttle in a row pauses the queue. Save Page Now and the availability-API fallback are off for bulk runs. The state blob isversion: 3; v1 and v2 blobs still resume and still project byte-identically throughGET /api/wayback/import-all. A pause pressed while an app was being read used to be overwritten by the runner's next state write; it now stops the run after that app.wayback.captures.<appId>is a cache (not in backups, deleted with the app);wayback.alt_urls.<appId>is the user's and stays in backups.Fixtures and parity
The import's replay fixtures are now Rust-owned regression fixtures rather than Node oracles:
history-cases.json(41 cases),wayback-runner-cases.json(55) and the newimport-history-route-cases.json(20, moved out of the Node-recordedimports-cases.json, whose extractor reserves their forwarded addresses so its remaining 162 cases re-record unchanged).PT_BLESS=1rewrites a fixture's outputs from the Rust replay and leaves its inputs alone; blessing unchanged code moves no byte. Their CI re-record steps and Node extractors are gone, andGET /api/wayback/import-allis quarantined in the parity manifest.operations-cases.jsonstays Node-recorded and unchanged: a v1 or v2 blob reads exactly as before.Verification
On the final tree (with
mainmerged in):cargo test --locked --manifest-path core/Cargo.toml: 449 library tests and every integration test pass. Re-blessing all three fixtures on the merged code rewrites nothing. Clippy's six warnings are pre-existing, in files this PR does not touch.pnpm lint(1050 files),pnpm typecheck,pnpm lint:i18n(5319 keys),pnpm test(1149 tests, 0 failures),pnpm parity:manifest, andpnpm audit --prodclean.Known limits
intervalMonthsstill makes the skeleton denser.historyStartsAtreads the archive's listing only, never stored rows, so an app imported before this change from a non-US page whose US archive starts later can be told its history starts late while its timeline shows older rows. Andappsstores no release date, so an app first published after mid-2021 is also offered the older-address input.Dependencies
Built on #386 (its two commits show here until it merges) and on
mainafter #389, which fixed the Next.js audit that failed CI'squalityjob on every branch.🤖 Generated with Claude Code