Repository navigation
feat(nara): child-record walks and digital-object downloads, like NARA's scripts - #104
Merged
Merged
Conversation
…A's scripts
Follows usnationalarchives/Catalog-API bulk-download-scripts:
- CATALOG_API_KEY accepted when OPENFOIA_NARA_API_KEY is unset.
- records search -s nara --ancestor <NAID> (q optional, sent as "*") and
--all/--max walk pages with searchAfter ("*", then the last hit's
sort[0]); every page is a quota-counted query; the planned count is
printed first when it exceeds 5.
- records fetch -s nara --list-objects previews digitalObjects (--json);
--objects downloads them to <data>/nara/<NAID>/<objectFilename>, skips
files present at the expected size, and ingests the rest through
DocumentIngester under the NARA-<NAID> request. Downloads are allowed
only from https://nara-media-001.s3.amazonaws.com or
s3.amazonaws.com/NARAprodstorage/, throttled 1 req/s, not quota-counted.
store_text_document now looks up the file-less text Document, so a text
re-fetch never overwrites an ingested object sharing the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
- --list-objects/--objects build the adapter from the configured egress policy (fails closed when Tor is set but down) and the download notice names the route. - Each object is persisted right after it is ingested, and a file on disk with no Document row under NARA-<naid> is ingested on re-run, so an interrupted run recovers instead of losing those records. - store_text_document attaches the text doc to an existing request that holds only ingested files instead of inserting a duplicate request number. - objectFilename is sanitized as a bare name (no urlparse), so '#', '?' and ';' no longer truncate names into collisions. - --objects exits 1 when every NAID failed; --json without --list-objects exits 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the NARA Catalog adapter do what NARA's own bulk-download scripts (usnationalarchives/Catalog-API) do, their way.
CATALOG_API_KEYaccepted as an alias forOPENFOIA_NARA_API_KEY(NARA's convention)records search -s nara --ancestor <NAID> [--all --max N]: every child record under a series, paged withsearchAfterlike NARA's scripts, one request a second, each counted against the local quota, with a "up to N queries" notice before a long walkrecords fetch <NAID> -s nara --list-objects: preview the digital objects without downloading (NARA's GenerateObjectURLs script)records fetch <NAID> -s nara --objects: download every scan into<data_dir>/nara/<NAID>/, resume by re-running, ingest each through the local extractor/OCR into the active case, persisted per file so an interrupted run recoversstore.pyreuses an existingNARA-<naid>request instead of raising on the unique constraintReviewed (Opus, read-only); the two high findings (Tor bypass, interrupted-run data loss) fixed in the second commit with tests. Suite: 491 passed, 20 skipped. Live NARA call still unverified (key pending).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y