Skip to content

feat(wayback): import archive history at archive.org's pace, finding changes instead of sweeping quarters - #391

Merged
AdamXweb merged 60 commits into
mainfrom
feat/archive-org-imports-strategy-850ef6
Oct 8, 2026
Merged

AdamXweb merged 60 commits into
mainfrom
feat/archive-org-imports-strategy-850ef6

Conversation

@adamXbot

@adamXbot adamXbot commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Why

A bulk Wayback import of 201 apps finished one app, then paused on rate limiting. Four things were wrong at once:

  • Nothing paced the requests. Each app sent one CDX index call and then one full archived page per quarterly checkpoint, back to back, and the next app started straight away.
  • The retry landed inside the block. archive.org answers a burst with 429, then a block reported to last about five minutes, then refused TCP connections. The runner retried after 30 s and paused for the user.
  • A throttled archive read as an empty one. A refused connection or a timeout counted as "no capture" (fixed first, in fix(wayback): count a refused archive.org connection as throttling #386).
  • Most reads were wasted. Labels rarely change, but every app cost about 22 page reads, plus a Save Page Now request.

The summary also counted checkpoints, not apps: "2 imported, 5 no-op, 17 skipped" was one app's 24 dates.

What changes

Rust server only. The Node rollback (BACKEND=node) keeps the pre-redesign importer, so nothing under lib/ that the Node server runs changes. docs/WAYBACK_IMPORT.md is the design and the contract the work was built to.

  1. One pace for archive.org (core/src/scrape/archive_pacer.rs). Every request to archive.org, including the privacy policy's archive lookups, shares one pace per server: 10 a minute to start, between 2 and 15, halved by a throttle, plus one after 20 successes in a row. A 429, a 5xx or a refused connection starts a cooldown of the longer of Retry-After and 5 minutes doubling per throttle in a row, capped at an hour; during it, archive requests get a synthetic 429 and never reach the network. wayback_cooldown_until and wayback_pacer_state carry both across restarts. Replay harnesses get a pass-through pacer, which is what keeps every Node-recorded oracle byte-identical.
  2. Change-finding instead of a quarterly sweep (core/src/scrape/history.rs). A pure planner reads a skeleton (the earliest and newest captures and one a year between them), then bisects wherever two neighbouring samples differ until each change sits between captures that are adjacent or at most 7 days apart. About 6 reads for an app whose labels never changed, about 6 more per change. Stored rows count as samples, so a re-run reads only what is new, and a throttled run keeps what it found. Each change is pinned to a window the timeline can show.
  3. A runner that waits (core/src/server/wayback_runner.rs). A fresh run surveys the index first (one listing per app, cached for 7 days), finishes apps with no archived pages at once, and reads never-imported apps first. A throttle waits until the later of Retry-After and the cooldown, in 1 s slices so pause and cancel still land, and carries on; the sixth throttle in a row pauses the queue. Save Page Now and the availability-API fallback are off for bulk runs. The state blob is version: 3; v1 and v2 blobs still resume and still project byte-identically through GET /api/wayback/import-all. A pause pressed while an app was being read used to be overwritten by the runner's next state write; it now stops the run after that app.
  4. The UI counts apps and label changes. The Settings card shows the phase, an estimate, when a wait ends and the survey result; an app's card reports the label changes it found and when its archived history starts; the timeline says between which two dates a label changed. Every new field is optional, so the Node rollback still renders the old card.
  5. Coverage. The listing asks the US-storefront address first (labels belong to the app, and archive.org crawls US pages far more often), then the stored address when the US index is empty. When history starts more than 180 days after the February 2021 floor, the app's page says so and lets the user add up to three older App Store addresses (a renamed app's older pages live under its old address); the import merges their captures. wayback.captures.<appId> is a cache (not in backups, deleted with the app); wayback.alt_urls.<appId> is the user's and stays in backups.

Fixtures and parity

The import's replay fixtures are now Rust-owned regression fixtures rather than Node oracles: history-cases.json (41 cases), wayback-runner-cases.json (55) and the new import-history-route-cases.json (20, moved out of the Node-recorded imports-cases.json, whose extractor reserves their forwarded addresses so its remaining 162 cases re-record unchanged). PT_BLESS=1 rewrites a fixture's outputs from the Rust replay and leaves its inputs alone; blessing unchanged code moves no byte. Their CI re-record steps and Node extractors are gone, and GET /api/wayback/import-all is quarantined in the parity manifest. operations-cases.json stays Node-recorded and unchanged: a v1 or v2 blob reads exactly as before.

Verification

On the final tree (with main merged in):

  • cargo test --locked --manifest-path core/Cargo.toml: 449 library tests and every integration test pass. Re-blessing all three fixtures on the merged code rewrites nothing. Clippy's six warnings are pre-existing, in files this PR does not touch.
  • pnpm lint (1050 files), pnpm typecheck, pnpm lint:i18n (5319 keys), pnpm test (1149 tests, 0 failures), pnpm parity:manifest, and pnpm audit --prod clean.
  • Each package carried its own tests and negative controls; the P5 UI was driven through a browser harness at desktop and phone width (which also found and fixed a pre-existing 260 px gap under the history card at phone width).
  • No live request was made to archive.org at any point; every test runs on stubs and recorded replies.

Known limits

  • A label change that reverts between two skeleton reads is missed, as the quarterly sweep missed one inside a quarter; the per-app intervalMonths still makes the skeleton denser.
  • historyStartsAt reads the archive's listing only, never stored rows, so an app imported before this change from a non-US page whose US archive starts later can be told its history starts late while its timeline shows older rows. And apps stores no release date, so an app first published after mid-2021 is also offered the older-address input.
  • Rolling back to the Node image mid-run loses that run; the boot heal clears its lock.

Dependencies

Built on #386 (its two commits show here until it merges) and on main after #389, which fixed the Next.js audit that failed CI's quality job on every branch.

🤖 Generated with Claude Code

adamXbot and others added 30 commits October 8, 2026 00:06
Once archive.org has throttled a client for long enough it stops
answering 429 and refuses the TCP connection (seen as connection refused
within 200 ms after two CDX calls 5 s apart). Both backends read a fetch
error from the CDX index or the availability API as "no capture", and a
refused replay as a skipped target. The index then fell back to up to
seven availability probes per target, each refused too, so every target
became skipped_no_capture, Save Page Now failed and the bulk runner
marked the app done with no history. A resumed 201-app run raced through
the queue that way, making about 170 refused requests per app and
prolonging the block.

A request to the index, the availability API or a replay that fails
below HTTP is now Unavailable / WaybackUnavailableError, status 0 and no
Retry-After: the transport's "fetch failed" (refused or reset),
"terminated" (dropped mid-body), timeout and unresolved-host errors,
with messages such as "archive.org refused the connection for CDX
index". Both sides classify by message, so other fetch errors (a blocked
URL, a body over the cap, a redirect loop) still fall back or skip, and
aborts stay aborts. The bulk runner then backs off and pauses the queue
as rate_limited, and the per-app route answers 503 archive_unavailable.

The bulk runner's wait without a Retry-After rises from 30 s to 300 s,
since reported blocks last about five minutes, and its cap from 120 s to
900 s.

- Node: lib/wayback.ts (waybackTransportFailure), lib/historical-import.ts,
  lib/wayback-bulk-runner.ts.
- Rust: core/src/scrape/wayback.rs (Unavailable::transport),
  core/src/scrape/history.rs, core/src/server/wayback_runner.rs.

Oracles: the history case that used "fetch failed" for a quiet replay
failure now uses a body over the cap, which behaves as before; seven
history cases (index refused, timed out, unresolved, redirect loop still
falling back, availability refused, replay refused and dropped) and two
runner cases (a refusal backing off for the default wait, a refusal on
the retry pausing the queue) are appended. New Node tests in
tests/app/wayback-refused-connection.test.ts; Rust tests assert the
error is flagged as throttling, not only its message.

Negative controls: the old Rust classification failed exactly the six
throwing history cases and both runner cases; the old 30 s default alone
failed only backoff_is_bounded and the default-wait case; a Node
classifier that never matches, with the old default, failed all seven
new Node tests.

Save Page Now, lookup_latest, the activity wording and the sampling
strategy are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A real paused bulk run logged a backoff of 30000 ms for "archive.org
rate-limited for availability API" after 21 s: the CDX index timed out
(20 s), the import fell back to availability probes, and the first probe
got a bare 429. The new test replays that: a timed-out index now throws
"archive.org timed out for CDX index" with no availability probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds three HistoryOptions fields the bulk runner will set: a held capture
list in place of a CDX call, skip Save Page Now, and fail with
INDEX_UNAVAILABLE instead of fanning out into availability probes when the
index is unusable. Defaults keep today's behaviour; the history oracle is
unchanged.

docs/WAYBACK_IMPORT.md is the design and the working contract for the
Rust-only redesign: packages, file ownership, API shapes and the rules
for the agents building it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Wayback import is Rust-only from its redesign
(docs/WAYBACK_IMPORT.md), so history-cases.json and
wayback-runner-cases.json become Rust-owned regression fixtures and
need a way to be updated without the Node extractors.

With PT_BLESS=1 the two replays still run every case, then rewrite its
outputs (calls, stream, rows, expected; the NDJSON frames ride in the
expected body) from the actual results, leave its inputs alone, write
the file back in the extractors' format and pass. A case written by
hand with inputs only gets its outputs appended; the runner replay
dumps the six recorded tables for it. Without the variable both
replays compare exactly as before.

The helper (core/src/bless.rs, test-only) also pins that both fixtures
are in the form a bless writes, so blessing unchanged code moves no
byte. Checked: a bless of the current tree rewrote 0 of 33 and 0 of 49
cases with git diff --exit-code clean, and a bless over fixtures with
outputs removed or corrupted in three cases each (and reformatted)
rewrote exactly those cases and restored both files byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Wayback import is Rust-only from its redesign
(docs/WAYBACK_IMPORT.md): the Node rollback keeps the importer and bulk
runner it had before, so re-recording history-cases.json and
wayback-runner-cases.json from Node would undo every intended change.
Drop the two core-parity steps that ran the extractors and diffed the
fixtures, and delete extract-history-cases.mjs and
extract-wayback-runner-cases.mjs. Both fixtures stay gated by the core
crate tests and are updated with PT_BLESS=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/wayback/import-all's payload is Rust-only from the Wayback
redesign (docs/WAYBACK_IMPORT.md): the core's state carries the
redesigned runner's blob, while the Node rollback keeps the old runner
and its v2 blob. Move it from READS to QUARANTINE with that reason, so
the differ stops byte-comparing a body the backends no longer share and
its HEAD probe still checks routing and the gate on both.

read-parity.mjs selects manifest entries by route (its BATCH_1 list),
not by this entry, and handoff.mjs reads only MUTATIONS, so neither
depended on it. pnpm parity:manifest: 127 routes, all classified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core/README.md: a "Rust-owned fixtures" gate note (why two fixtures
left the Node oracle, the bless commands, how to review a blessed diff
and add a case), and the history and Wayback runner oracle paragraphs
rewritten as Rust-owned fixtures, with the status GET now quarantined.

AGENTS.md: the Wayback section opens with the redesign being Rust-only,
the Node rollback keeping the pre-redesign importer and runner, how to
bless, and the two fixtures still recorded from Node that cover the
import's routes (imports-cases.json, operations-cases.json).

docs/WAYBACK_IMPORT.md: P0's fixture handover marked done, with the
exact bless commands and what P0 left Node-recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parse the v3 bulk state and the new NDJSON frames (phase, survey-app,
survey-done, waiting, estimate), derive the run line, survey result,
estimate, wait and app-level totals, describe a per-app result in label
changes, and find the window a reconstructed change happened in. Every
redesign field is optional: with none present each derivation returns
null, so the Node rollback keeps today's card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run line, survey result, wait, estimate and app-level totals for
Settings, label-change results and the busy-until time for the detail
card, and the timeline's "Changed between" caption. Rewords the copy
that described a quarterly sweep and drops the em dashes from the
Wayback strings. zh.json by hand while Crowdin is disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run line names its phase ("Checking the archive index: 37 of 201
apps", "Reading archived pages: 12 of 143 apps"), and the card shows
what the survey found, the time left, the wait while archive.org has
asked the import to slow down ("resumes at 14:32"), and totals in label
changes and pages read. The last run reads in apps checked, label
changes and apps with no archived pages, and a run the server resumed
after a restart now counts as the last run too.

The hook feeds every NDJSON frame through the pure reducer and merges
the v3 state from the poll. Without the new fields (the Node rollback)
the card renders the checkpoint tally exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A throttled import (503 archive_unavailable) now says when to try
again, from retryAfterMs or the Retry-After header. A change-finding
result reads as label changes found, notes how many archived pages were
read, says when changes found were already on the timeline, and says
when archived history starts well after February 2021. A result without
those fields reads as snapshot rows, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A wayback change row directly after an unchanged wayback row says the
change happened between the two archived copies' dates. Adds a story
with that pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-app import no longer reads one archived page per quarter. It
reads a skeleton (the earliest capture after the February 2021 floor,
the newest, and the capture nearest each cadence step back from today,
yearly by default), then bisects wherever two neighbouring samples
differ until each change sits between captures adjacent in the index
or at most a week apart. Rows go in for the baseline, both sides of
each change and the newest read.

Stored wayback rows are samples: a skeleton date within the dedupe
window of one costs nothing, and at the default cadence neither does a
date between two stored rows with the same labels, so a re-run or a
retry after throttling reads only what is new. A page without labels
before any page with them is bisected like a change, so the baseline
lands where labels first appeared rather than a year later. Unusable
captures hand over to the next nearest, two reads per point. Without an
index, one availability probe per skeleton date and no bisection.

The planner (Sampler) does no I/O and is tested over synthetic and
random timelines. The result adds reads, changes, labelVersions,
firstCaptureMs, lastCaptureMs and windows; each target adds its phase.
The history-cases.json replay still expects the quarterly sweep until
the fixture is re-blessed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eight cases through import_app_history and the real transport: change
rows on the captures that bracket each change in five years of daily
captures (18 reads, not 22 a year); a stable app costing the six
skeleton pages; a change reverting between skeleton reads going unseen
at the yearly cadence and found at a quarterly one; unusable captures
handing over to the next nearest; stored rows making a re-run read
only the index; force re-reading without storing a capture twice; a
throttle mid-bisection storing what it learned so the next run reads
only the rest; and the no-index path probing each skeleton date once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Reading archived pages: 12 of 143 apps" is long enough to wrap at
phone width, and the run line's flex row squeezed the app name after it
into a sliver. The lead and the name now share one text flow beside the
spinner; at desktop width the line looks as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Package P3 of docs/WAYBACK_IMPORT.md, Rust only.

- Throttling waits instead of pausing: waitingUntil (Retry-After, five
  minutes without one, no upper cap) and waitReason are stored, a
  `waiting` frame replaces `backoff`, and the same app is asked again.
  A wait sleeps in one-second slices, so a pause stored by the PATCH
  ends it within a second and a cancel at once; a long wait rewrites the
  state every ten minutes so the health check never reads it as dead.
  Six throttles in a row with no app finished pause the queue with
  pauseCause rate_limited, as before.
- Fresh runs survey first: one CDX listing per app, cached for a week as
  wayback.captures.<appId>, apps with no captures finished as noArchive,
  the rest ordered (never imported first, then most archived). Reading
  passes the cached listing with Save Page Now and the availability
  fallback off.
- State blob version 3: phase, consecutiveThrottles, survey, estimate
  (from wayback_pacer_state.perMinute, 10 when absent), queue and totals
  in apps, reads and label changes. A v1/v2 blob resumes as a reading
  phase; a run stopped mid-survey or mid-wait resumes where it was.
- Every boundary write carries over a pause or cancel the PATCH stored
  meanwhile, so a pause asked for while an app is read is no longer lost.
- Activity rows in apps and changes; frames phase, survey-app,
  survey-done, waiting and estimate.
- GET /api/wayback/import-all projects the v3 fields for a v3 blob only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scripted archive and a fake clock (the runner's waits advance it
instead of sleeping): wait-and-retry, the five-minute default, the pause
after six throttles and the count reset by a finished app, pause and
cancel during a wait, a pause asked for mid-app, the survey's cache reuse
and ordering, an unreadable index, a v2 queue resumed as reading, a run
resumed mid-survey and mid-wait, the estimate, and the status route's
v3 projection. Each test has its own run ids, since the cancel registry
is keyed by them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A change-finding result with pages read but no label state among them
(the first weeks of 2021, or pages that were not product pages) read
"No label changes found", as if the labels had stayed the same. It now
says none of the archived pages had privacy labels to read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Blessed with P0's PT_BLESS on a throwaway merge (outputs only), then
reviewed case by case against the Node recording. Inputs edited by hand:

- The newest capture is always a skeleton read now, so cases 0, 1, 8,
  10, 11 and 23 get a reply for 2021-10-20 that keeps their story; in
  9 and 31 its labels differ, so the throw still lands mid-bisection,
  after the rows learned so far are stored. Case 2 gets 2021-02-15 so
  force visibly re-reads a stored capture without storing it twice.
- The install date is no longer a skeleton date. Its three cases now
  pin that: "the install date adds no read", "an index of one capture
  is read once", "skeleton dates before the first capture are dropped".
  "target walk from a month end" keeps its name and pins 31 May minus
  three months (3 March) through the skeleton at a quarterly cadence.
- Appended: bisection narrowing a change to a week, the stretch before
  labels appeared, stored rows covering the skeleton between them, a
  throttle after two midpoints, and probes at a quarterly cadence.
- Replies nothing asks for are dropped, so each case lists what it
  uses; a re-bless after the prune rewrote 0 of 38 cases.

The eight cases that throw before any read are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Option::is_none_or is stable from Rust 1.82 and the core declares 1.77,
which clippy's incompatible_msrv flagged. map_or(true, ..) as elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The no-index fallback is one probe per skeleton date now, not up to
seven per target; held captures replay their own address; the probe
walk records probed times rather than returning them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wait runs until the later of archive.org's Retry-After and the paced
client's stored `wayback_cooldown_until`. A retry inside the cooldown
would only meet the client's own synthetic 429, spend one of the six
throttles the queue tolerates, and wait again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p2-change-finding-3205b9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p3-runner-53b722
The replies of every case whose requests moved are re-sequenced (the
survey lists every app before any page is read, and a bulk run never asks
Save Page Now), the hooks retargeted, and six cases renamed for what they
now pin: a pause asked for mid-app stops after the app instead of being
overwritten, a pause ends an archive wait, a cancel lands mid-survey, six
throttles in a row pause the queue (five, then a refused connection, too)
and a refused connection waits the default five minutes. Six cases are
appended: a fresh listing reused, never-imported apps read first, a
throttled read waited out and read again, a survey and a wait resumed at
boot, and an unreadable index failing its app in the survey.

The replay's waits take no time except in a case with a delayed hook,
which has to land during one. Outputs blessed with PT_BLESS=1; every
rewritten case reviewed against its Node recording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… moves

Every case that reaches the archive has one capture or none, so the
skeleton is one read or none: requests, the replies they are paired
with, hook positions, write counts and stored snapshot rows are as Node
recorded them. What moves is what the import reports: activity and
audit lines no longer count 22 or 23 skipped quarter targets per app,
the bulk state's totals follow, and streamed target frames carry a
phase and report the read as sampled (the app-done result marks it
imported). A re-bless rewrites 0 of 49.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P3's shapes are final: reading progress is totals.appsRead of the
survey's appsWithCaptures, and the GET never carries the queue. The
done + failed - appsNoArchive stand-in over-counted apps whose index
listing failed: they finish as failed in the survey and are never read.
That stand-in now only covers totals without appsRead.

The stream counts apps with no captures and failed listings where they
finish, on their survey-app frame (a failed one has captureCount null
and an error), so they never count toward reading. The survey line says
how many apps could not be checked, the closing line how many failed,
and etaMs is read as the duration the runner reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
changes and labelVersions count the rows already on file, so an app
with imported history whose index answered empty now reads as its
changes rather than "archive.org has no archived pages for this app".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The health check's lock heal reads the blob through `read_bulk_state`,
so it needs no change for version 3: a waiting run an hour silent is left
alone, and one seven hours silent is healed as before. A 25-minute wait
rewrites the state at 10 and 20 minutes, so `updatedAt` never falls past
the six-hour stale margin however long archive.org asks us to wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adamXbot and others added 26 commits October 8, 2026 22:12
Wires the pacer into each path that reaches archive.org:

- sync_runner.rs: start_background files the server's pacer (read from
  app_settings) before the first request is served and drops it when
  the server stops (file_archive_pacer); the boot-time Wayback resume
  runs through it.
- runner_writes.rs: POST and PATCH /api/wayback/import-all, so the
  buffered run, the streamed run and the run a resume spawns all share
  it (a spawned run takes the pacer with its shared fetcher).
- imports_writes.rs: the per-app POST /api/apps/[id]/import-history; in
  a cooldown it answers 503 archive_unavailable with retryAfterMs, as
  for any throttle.
- policy_store.rs: fetch_and_store, which covers the source's Wayback
  fallback (policy/source.rs, tier 3), the existing-capture lookup and
  Save Page Now, and run_follow_ups' deferred Save Page Now. The
  policy's own site passes straight through.

Every replay sees the pass-through pacer, so all stay byte-identical
(cargo test: every lib and integration test passes). Negative control:
with an unregistered connection given a real pacer on an instant clock
instead, the Wayback runner replay failed exactly its four throttled
cases (throttled once, throttled twice, streamed and throttled,
throttled then refused), whose retries met the synthetic 429; the only
other failures were the two tests that pin the pass-through for an
unregistered connection. The control was removed before the passing run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts apps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	core/tests/fixtures/wayback-runner-cases.json
…ned import

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P3's runner cases were blessed against the quarterly import; with P2's
change-finding import merged, the 14 cases that read pages move. Only
outputs changed: no case's calls moved, so every case still makes the
same archive requests in the same order. A stable fixture app now reads
one page, its target frame carries "phase", and its activity line counts
that read ("1 page read" where it said "0 pages read" next to an import).
history-cases.json and import-history-route-cases.json re-bless with no
change; the full core suite passes (432 lib tests plus integration).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…core/README

CHANGELOG: one Changed entry for the redesign (one pace for archive.org,
waits instead of stopping, index first, change-finding, app-level
progress; the Node rollback keeps the old import) and one for the
Rust-owned fixtures; the #386 Fixed entry notes that the Rust server now
waits instead of retrying once.

AGENTS.md: the per-app route's cases are Rust-owned now, and a paragraph
on how the Rust import works (pacer, Sampler, the v3 runner), for whoever
touches it next. core/README.md: the third bless command, the imports
oracle's real count (162) and its moved cases, and the pinned pause the
redesign fixed. ci.yml: the imports oracle comment no longer claims the
per-app Wayback import. docs/WAYBACK_IMPORT.md: fixture paragraph and
package status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage (P5) lets a person add up to three older apps.apple.com
addresses for an app whose archived history starts late. Parse the
per-app result's lookupUrl, historyStartsAt, historyStartsLate and
alternateUrls, prefer the server's late-start verdict (it also weighs
the release date) over the first-capture heuristic, and describe the
card's older-addresses block. Check an address before it is posted
(apps.apple.com, this app's id, not already checked, at most three).
Count late-starting apps from app-done results during a stream, and
read totals.appsHistoryLate when the runner reports one. Every field is
optional; without them nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An import now lists an app's US-storefront page first: labels belong to
the app, and archive.org crawls US pages far more than any other. The
stored address is listed only when the US index is empty (or unusable),
not when it is merely smaller, which would cost every non-US app a
second listing; an address with more history can be added instead.

Older App Store addresses a user adds for the same app id are stored as
wayback.alt_urls.<appId> (at most three, each
https://apps.apple.com/<cc>/app/<slug>/id<sameId>), and both the survey
and the per-app import merge their captures in by timestamp, each
replayed at its own address. POST /api/apps/[id]/import-history takes
alternateUrls, validates and stores it before importing, and answers a
bad entry with 400 {error, code: "invalid_alternate_url"}.

The result adds lookupUrl, alternateUrls, historyStartsAt and
historyStartsLate; survey queue entries and frames add lookupUrl and
historyStartsLate. apps keeps no release date, so a start counts as late
from the February 2021 floor alone. The survey's cache records the
stored address it was taken for and each older address's timestamps,
and is listed again when either changes.

Backups leave the capture cache out and keep the older addresses, and
every app delete forgets both keys, reading first so an app without
them deletes with exactly the statements it always did: the
Node-recorded backup, library, device, import and runner oracles all
replay unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit tests for the address helpers (any storefront to its US page; an
older address must be this app's apps.apple.com page; a capture names
its address) and for the late-start rule, and six end-to-end cases
through the stub archive, which can now answer each index request by
the address it names: the US page leads; an empty US index falls back
to the stored address; a smaller US index still leads (and here starts
late); older addresses merge by timestamp and replay at their own
address; an older address with an unusable index adds nothing; and a
throttled older address is the import's error before any page is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The detail card's explanation, address input, stored list, remove
control and the plain-words errors for an address that cannot be used,
and the Settings line for apps whose archived history starts late. The
start-date note now says the archive has no earlier copy of the page at
this address, since older pages may sit under an old one. zh.json by
hand while Crowdin is disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the server says an app's archived history starts late, the card
explains that a renamed app's older pages may sit under its old address
and takes up to three older apps.apple.com addresses, posted as the
whole alternateUrls list with a forced import; each stored address can
be removed the same way. An address is checked before it is posted, and
the route's invalid_alternate_url is said next to the input in plain
words, with the last result left standing. Results from a route without
coverage show no block. Also stops the card's copy keeping a 260px
height when it stacks at phone width.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One muted line in the running import and in the last run, only when the
count is above zero: "Archived history starts late for 5 apps. You can
check them one by one on each app's page." Counted from the app-done
results of a run this tab streams, or read from totals.appsHistoryLate
when the runner reports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fake archive can list by address. Three cases: the survey lists the
US page and an older address (the stored one is never asked), caches
them with the stored address they were taken for, and replays the
oldest pages at the old address; a cached listing is taken again when
the stored address or the older ones change, or when it predates the
US lookup for a non-US page, and reused when nothing changed; and
deleting an app forgets its listing and older addresses, through the
orphan sweep too, writing nothing for an app that has neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The P5 route stores only https://apps.apple.com/<storefront>/app/<slug>/
id<n> with a lowercase two-letter storefront, and refuses /app/id<n> or
a storefront without a slug. The client check now mirrors that
(canonical_app_store_address in core/src/scrape/wayback.rs), so those
are caught before posting, and its message shows the full shape with
the app's own id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The survey's wayback.captures.<appId> listings are a cache archive.org
can answer again; the older App Store addresses a user typed
(wayback.alt_urls.<appId>) stay in the export. Node's backup has no such
cache, so the recorded backup cases cannot pin it; they replay
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p5-coverage-3205b9
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Blessed on the branch merged with the integration tip, whose fixtures
already matched its code, so the diff is P5's alone and purely
additive: every import result gains lookupUrl, alternateUrls,
historyStartsAt and historyStartsLate (in the route's responses and
the activity rows too); survey queue entries and survey-app frames gain
lookupUrl and historyStartsLate; the capture cache gains storedUrl. No
case's requests moved: every fixture app is a US page with no older
addresses, so the lookup is the one listing it always was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Appended, inputs only, outputs blessed and reviewed. The route stores
older addresses (two spellings of one collapse to it), lists the US
page then the old address, reads the oldest pages at the old address
and the newest at the US page, and asks Save Page Now; it refuses
another app's address, more than three, and a non-list with 400
invalid_alternate_url before writing anything; and an empty list
deletes the stored addresses before importing. The history fixture,
whose stub can now answer each index request by address, pins the US
page leading, the stored address used when the US index is empty, and
an older address merged in with its own replays and change window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-850ef6' into feat/wayback-p5-coverage-3205b9
totals.appsHistoryLate counts finished apps whose import result says
historyStartsLate, next to appsWithHistory and appsNoArchive, so it
reaches GET /api/wayback/import-all and the run's activity row: the
status read carries no queue, and the Settings card needs the count
after a reload and in the last-run view. A missing flag reads as false.
A queue resumed from before the count fills it from its finished, read
entries, so a v1 or v2 blob (which never carries the flag) counts 0 and
reads byte-identically through the status routes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Additive only, and no request moved: appsHistoryLate appears wherever
the totals do (responses, frames, activity details, state blobs and
their writes). It is 1 for each archived fixture app, whose captures
start in July 2026, 0 for apps with no captures and for the entries a
resumed v2 queue carried over. operations-cases.json is untouched and
replays unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An Added entry for the US-first lookup, older addresses and the
late-start hint; the AGENTS.md paragraph gains the coverage part (the
listing rule, the alternate-address contract, which settings are cache
and which are the user's); the contract doc records appsHistoryLate in
the totals and the two known limits of the late-start check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

# Conflicts:
#	CHANGELOG.md
@adamXbot
adamXbot requested a review from AdamXweb as a code owner October 8, 2026 12:36
@AdamXweb
AdamXweb merged commit f63e871 into main Oct 8, 2026
20 checks passed
@AdamXweb
AdamXweb deleted the feat/archive-org-imports-strategy-850ef6 branch October 8, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants