CI: surface risky Dockerfile content to reviewers, add Copilot review, fix S3 upload#625
Merged
Conversation
…S3 fix Detection layers so a container PR never merges with unreviewed 'strange' content (e.g. #621's `curl | sh`): - validate.py: scan_dockerfile_risks() lifts the lines a human must sign off on into the PR comment as an advisory reviewer checklist (curl|sh, http://, ADD <url>, paste/shortener/bare-IP hosts, chmod 777, --insecure, odd base image). High-confidence embedded credentials (AWS/GitHub/Slack tokens, private keys) block the PR. Line-continuations are folded so a split `curl \<nl>| sh` cannot evade; download rules are scoped to RUN/ADD/COPY so an http:// homepage in a LABEL is not a false positive. 12 new unit tests. - pr-report.yml: render the checklist + a bioconda-style 'build & test locally' block (and point at the existing test-cmds.txt convention). - Copilot: .github/copilot-instructions.md + instructions/dockerfile.instructions.md so automatic Copilot review hunts the same risks on every container PR. - publish.yml: fix Singularity S3 upload on Ceph/RGW (aws-cli v2.23+ default checksums broke multipart; set *_CHECKSUM to when_required).
|
❌ BioContainers CI failed for Errors (must fix):
|
- Never echo a snippet from a line that also tripped a secret rule (no token leak into the public PR comment). - Detect credentials in `ENV KEY value` (whitespace) form, not just `KEY=value`. - Fold heredoc bodies (RUN <<EOF … EOF) into their instruction so a heredoc'd `curl | sh` is scanned under RUN scope. - Base-image check: parse all FROM stages (not just the first), skip `--platform=` flags, ignore references to prior build stages, dedupe. - pr-report: markdown-escape untrusted report strings (backticks + angle brackets) before rendering, so a fork-controlled report cannot inject formatting/HTML. - 8 new regression tests (37 total, all green).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Container PRs (e.g. #621) can add arbitrary
RUNcontent — the classic case is aremote script piped into a shell (
curl … | install.julialang.org | sh). A reviewerskimming 20 lines can miss it. This adds defense-in-depth detection so nothing
"strange" merges unreviewed, plus a bioconda-style local test path, and fixes the
Singularity → S3 upload that was silently failing.
What
1. Deterministic Dockerfile risk scan (
validate.py) — zero new deps.scan_dockerfile_risks()lifts the lines a human must consciously approve into the PRcomment as an advisory 🔍 Reviewer checklist:
curl … | sh/wget … | bash,ADD <url>, insecurehttp://, paste/URL-shortener/bare-IP hosts,
chmod 777,--insecure/--no-check-certificate, non-biocontainers/*base image.
curl \↵| shcan't evade); downloadrules are scoped to
RUN/ADD/COPYso anhttp://homepage in aLABELis not afalse positive. 12 new unit tests (30 total, all green).
2. Reviewer aids (
pr-report.yml) — renders the checklist and a bioconda-style"🧪 Build & test this container locally" block, pointing at the existing
test-cmds.txtconvention. Untrusted report fields stay sanitized (same path as errors/warnings).
3. Copilot review layer —
.github/copilot-instructions.md+.github/instructions/dockerfile.instructions.md(applyTo: **/Dockerfile) so automaticCopilot code review hunts the same risks on every container PR. (Requires enabling
automatic Copilot review + a Copilot subscription — repo owner action.)
4. S3 upload fix (
publish.yml) — the Ceph/RGW store rejects aws-cli v2.23+ defaultCRC32 checksums on multipart uploads (
InvalidRequestonCreateMultipartUpload), soevery
.sifover ~8 MB failed to upload. SetAWS_REQUEST/RESPONSE_CHECKSUM_CALCULATION= when_required. Confirmed the endpoint supports multipart via boto3; end-to-end re-testafter merge.
On #621
The scanner flags line 19 (
curl … | sh) in the reviewer checklist; Copilot would commenton it independently. Nothing blocks — it spotlights, so the human signs off explicitly.