Skip to content

refactor: deduplicate shared downloads across test suites review - #395

Merged
rsanchez87 merged 2 commits into
masterfrom
download-dedup-review
Sep 30, 2026
Merged

rsanchez87 merged 2 commits into
masterfrom
download-dedup-review

Conversation

@rsanchez87

@rsanchez87 rsanchez87 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Based on the review of #390

cc @dabrain34

@rsanchez87

rsanchez87 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

AC verification:

  • AC1 — no regressions (hierarchy + dedup): fluster.py download for all suites ran on Linux and Windows; folder/file hierarchy is identical (0 missing entries between the two trees); the shared AV1-ARGON archive downloads once (Unique sources: 1).
  • AC2 — no execution regressions: 56/56 tests pass (unittest discover) and ruff / ruff-format / mypy --strict are green.
  • AC3 — Linux + Windows: both covered; Windows full download completed and the tree matches Linux.
  • AC4 — issue Downloading the AV1 test suites results in downloading multiple times a 6GB archive #309: resolved — the 6.5 GB archive shared by the 12 AV1-ARGON-* suites is downloaded once instead of once per suite.
  • Extraction performance: zips open once (single pass); benchmark with 3000 members: 0.23 s vs 16.5 s per-member reopen.

cc: @rubenrua @mdimopoulos

Comment thread fluster/download_manager.py Outdated
Comment thread fluster/download_manager.py
@rsanchez87
rsanchez87 force-pushed the download-dedup-review branch 2 times, most recently from 09754a0 to 94c6f9a Compare September 29, 2026 14:42
Comment thread fluster/download_manager.py Outdated
- Simplify download manager and zip extraction
- Use download_test_suite() in TestSuite.download() (it was unused)
- Type test suites as TestSuite instead of Any
- Report which source failed when a download errors
- Fix the file's copyright year (2026, not 2020)
@rsanchez87
rsanchez87 force-pushed the download-dedup-review branch from 94c6f9a to 11748d8 Compare September 30, 2026 08:35
@rsanchez87 rsanchez87 changed the title fix: deduplicate shared downloads across test suites review refactor: deduplicate shared downloads across test suites review Sep 30, 2026
@rsanchez87
rsanchez87 merged commit 61dbc5f into master Sep 30, 2026
5 checks passed
@rsanchez87
rsanchez87 deleted the download-dedup-review branch September 30, 2026 09:16
@dabrain34

Copy link
Copy Markdown
Contributor

Sorry about this misunderstanding with my thumb up. I was reviewing again and found some pitfalls such as the license in download_manager.py could be shared but no worries.

I can create another PR to fix the issues.

@rsanchez87

Copy link
Copy Markdown
Contributor Author

Sorry about this misunderstanding with my thumb up. I was reviewing again and found some pitfalls such as the license in download_manager.py could be shared but no worries.

I can create another PR to fix the issues.

Ok, sorry about the misunderstanding, too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants