fix: deduplicate shared downloads across test suites review - #1
Closed
rsanchez87 wants to merge 9 commits into
Closed
rsanchez87 wants to merge 9 commits into
rsanchez87 wants to merge 9 commits into
Conversation
rsanchez87
force-pushed
the
download-dedup-review
branch
from
September 29, 2026 08:39
61254ef to
a628279
Compare
rsanchez87
force-pushed
the
download-dedup
branch
from
September 29, 2026 08:42
95eb4f3 to
73204c5
Compare
rsanchez87
force-pushed
the
download-dedup-review
branch
2 times, most recently
from
September 29, 2026 09:03
77f668c to
5a8368f
Compare
rsanchez87
marked this pull request as ready for review
September 29, 2026 12:03
Download each unique source URL once for the whole selection of test suites and distribute it to every suite that needs it, instead of fetching shared archives once per suite. Sources are downloaded in parallel into resources/.cache/<md5(url)>/ and the cache is removed at the end of the run unless --keep is used. Download orchestration moves from TestSuite into a new fluster/download_manager.py module. TestSuite.download() remains as a thin wrapper for the generator scripts, and --mirror support is preserved.
Add filename_from_url() so query strings and fragments are stripped when naming files, and use it for the download destination and the shared download cache. Signed URLs like GCS/S3 links end with ?X-Amz-Signature=..., which broke extension checks and left odd filenames on disk. Ported from #338.
DownloadManager used retries**retries, so -r 5 meant 3125 download attempts per URL before failing. download() already retries internally with backoff; pass the user-provided count instead. Ported from #338.
When two selected test suites share a source URL but declare different checksums, abort instead of silently trusting whichever checksum was seen first. Also prefer a real checksum over __skip__ when both appear for the same URL. Ported from #338.
If a cached archive cannot be extracted (corrupt zip/tar or gunzip failure), delete it from the cache and report a clear error, so the next run re-downloads it instead of failing forever. A missing archive member is not treated as corruption and keeps the cache. Ported from #338.
Explain that sources shared by test suites are fetched once into resources/.cache/, that the cache is removed at the end of the run unless --keep is used, and that concurrent download runs against the same resources directory are not supported. Ported from #338.
- Expose DownloadManager.test_vector_source_path() and use it in gen_mpeg4_video.py instead of hard-coding the layout - Use download_test_suite() in TestSuite.download() (it was unused) - Type test suites as TestSuite instead of Any - Report which source failed when a download errors - Fix the file's copyright year (2026, not 2020)
rsanchez87
force-pushed
the
download-dedup-review
branch
from
September 29, 2026 12:18
5a8368f to
4f68683
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.