Repository navigation
feat(summary): copy-number, traits, SV, ancestry and the other sections in Rust - #405
Merged
Merged
Conversation
og dev build cn-intervals, callability and sv-rule-regions read the pinned downloads from a folder and write bin/cn_loci.tsv's interval rows, assets/callability_genes.tsv and assets/sv_rule_regions.tsv, byte for byte as the Python builders did. The GTF is read as bytes with a hand scanner of its attributes, the cut sets are merged once per chromosome and searched by binary search, and og never downloads. The SV rule builder now keeps the first GENCODE record per HGNC id and skips _PAR_Y copies, as the callability builder does; the committed table does not change. Bead: pgp-bs9u.35
build-callability.sh and build-sv-rule-regions.sh keep their downloads and hand the folder to og dev build; their Python heredocs are gone. build-cn-intervals.py becomes build-cn-intervals.sh with the same --list, --cache and --check. The three share scripts/ci/lib/fetch-pinned.sh: a cached file with the right sha256 is read in place, a 403 or 404 fails at once, anything else is retried for about five minutes. The marker line of bin/cn_loci.tsv names the new script; no row changes. The cn-intervals case, the e2e data fill and the converter checks of tests/test_copy_number.py call the new script and og. Bead: pgp-bs9u.35
Red-first control, reverted in the next commit: callability keeps the _PAR_Y copies, the SV rule builder keeps the last record per HGNC id, and each cn-intervals piece ends one base short. Bead: pgp-bs9u.35
… --only b) ClinVar, the carrier list, PharmCAT, CPIC, pypgx, Cyrius, CPSR, the clinical filter with its ACMG SF tier, slivar and the cyp2d6 block, each input read once, beside the Python collector. No caller changes. Bead: pgp-bs9u.42
og summary collect --only c reads copy number, HLA, the polygenic scores, ancestry, structural variants, repeats, telomeres, runs of homozygosity, the haplogroups, the opt-in step tables, chrM, traits with the APOL1 G2 deletion, and the couple screen, with the parts of the post pass that change them. og roh summary writes step 11's summary table; step 11 and the ROH module run it, so the bands live in one const. Bead: pgp-bs9u.43
The clinical VCF, slivar's VCF and the hits were read whole before being split; now one pass streams each, the hits are no longer copied for the ClinVar section, and the summary's own data is freed before the JSON text is made. A table cut short reads as Python's EOFError. Bead: pgp-bs9u.42
The note now says "line L column C (char N)" in code points after universal newlines, and names a UTF-8 BOM, as json.load does. Bead: pgp-bs9u.42
…thon Ten cases of og summary collect --only b: every input in both layouts, failed and stale steps, unreadable inputs, a VEP output with and without its index, phased deletions in trans and in cis, and no assets. The oracle stages the helper modules the collector imports from the pinned commit instead of stubs. cb-notgz holds the one deliberate fix. Bead: pgp-bs9u.42
…rate map Bead: pgp-bs9u.42
Plant: the deletion's read-backed phase is ignored, so a variant in cis with a deletion counts as a second hit and a trans pair is only possibly biallelic. The two_hits unit tests and the cb-full parity case must fail. Bead: pgp-bs9u.42
…go red" The plant turned five builder tests red in Guard's rust job (run 43417543846); back to the right rules. Bead: pgp-bs9u.35
…y cases Adds the table, chrM and traits readers (APOL1 G2 by haplotype, read through the .tbi when there is one), og roh summary writing stdout without --out, 12 summary-collect c cases and 3 roh-summary cases, the c-bad2 golden for a plain-text chrM .vcf.gz, and the changelog. Bead: pgp-bs9u.43
Plants g2-shifted-not-matched so CI shows the unit test and the parity case catch it. The next commit takes it out. Bead: pgp-bs9u.43
… failed fetch_pinned downloads to a mktemp file beside the target, so two runs on one cache never write the same partial file, and gives curl a 30 s connect timeout and a 1 KB/s for 60 s floor, so a stalled server reaches the retry or the error instead of holding the build. Bead: pgp-bs9u.35
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 43 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Revert the acmg-biallelic-ignores-phase control. Keep cis variants out of the two-hit count and classify a phased trans pair as biallelic. Bead: pgp-bs9u.42
Keep the invented data rows separate from their column labels, as the other generated tables do. This preserves every fixture byte and keeps the personal-data scanner enabled without adding an exemption. Bead: pgp-bs9u.42
Removes the planted g2-shifted-not-matched fault. Shifted records count as G2 when they produce the same haplotype on the pinned context. Bead: pgp-bs9u.43
Separate the invented repeat values from their FORMAT headers without changing generated bytes. This keeps Personal data scanning enabled without allow-list changes. Bead: pgp-bs9u.43
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md # rust/crates/og-dev/src/lib.rs # rust/og/src/main.rs
Collaborator
Author
Independent review fixes
The widened file list includes these integration paths: |
…list zram0 Some runner images list /dev/zram0 as a disk of 0 bytes with no file system. The spare-disk step took it for the blank disk, and mkfs.ext4 stopped the leg before any case ran. The step now skips zram, ram and loop devices and any disk of 0 bytes.
…-r2b # Conflicts: # CHANGELOG.md
…-r2c # Conflicts: # CHANGELOG.md # docs/rust.md # rust/Cargo.lock # rust/README.md # rust/crates/og-summary/src/collect.rs # rust/crates/og-summary/src/collect/scan.rs # rust/crates/og-summary/src/collect/sections.rs # rust/og/src/main.rs # tests/parity/cases.tsv # tests/parity/inputs/summary-collect/oracle.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Rust summary collector could read seven sections. Copy number, traits, structural variants, ancestry, repeats and the other collector-owned sections still had only Python readers. Step 11 also kept two copies of its ROH summary awk program and masked bands.
og summary collect --only cnow reads those 32 sections. It uses the existing folder index, streams large inputs and reads independent sections on a bounded thread pool. Section errors stay inside their section. APOL1 G2 matches the haplotype on the pinned context, including shifted deletions, and uses the VCF index when available.og roh summaryreplaces the inline awk inscripts/11-roh-analysis.shandmodules/local/roh/main.nf. Both use the same Rust masked-band list as the summary reader. Its table is unchanged. The Python collector retains its list until the collector switches. A test checks that both lists agree.Decisions and output changes:
.vcf.gzis read by its magic bytes. Python marked that section unreadable. Thec-bad2golden pins this change. No output file, column or existing key is renamed or removed.Shared files changed for this lane:
rust/og/src/main.rsadds the ROH command.rust/Cargo.lockrecords og-summary's existing workspace dependencies.rust/README.md,docs/rust.mdandCHANGELOG.mddescribe these commands.tests/parity/cases.tsv,tests/parity/oracle.ref, the summary oracle wrapper and the new parity generators/golden cover this port.tests/test_collect_summary.pychecks the Rust masked-band list instead of the removed awk copies.How it was checked
g2_shifted_is_matchedandtwo_g2_records_are_not_read: 66 passed, 2 failed. Remaining work on that head was cancelled after the failure. Commit 93428ba removes the planted fault.cargo test -p og-summary --no-fail-fast, 72 passed. Rustfmt, clippy with-D warnings, and cargo doc withRUSTDOCFLAGS='-D warnings'passed.ORACLE_RUNNER=local scripts/ci/oracle-diff.sh summary-collect roh-summary: all 203 comparisons across 51 cases agree. These include 15 new group-c cases and 3 ROH cases.scripts/ci/oracle-diff.sh --self-test: all 15 harness controls and all 44 comparator controls passed. Linux shadow execution remains CI's check.tests/test_collect_summary.py: 501 checks passed with the rebuilt og. Shellcheck passed for the changed scripts. Personal data scanned all 2,396 tracked text files with no finding./usr/bin/time -l. Python readers: 0.31-0.35 s and 41.3-41.5 MB peak RSS. Rust collection: 0.30-0.31 s and 34.3-34.8 MB. This is a parity port, with no speed claim.7adf3d4a2e8c8261d5b663e2e8639d7e649ead9b: Guard, Rust, Lint, Nextflow, Docs, Container Test, Personal data.Test map: the collector-owned section checks are covered by the
c-full,c-nf,c-bad,c-bad2and G2 JSON subtree cases. CNVpytor sex filtering, Locityper primary calls and CFTR phase also have Rust unit tests. The G2 tests pin the context, shifted calls, duplicate calls, reference blocks and absent calls. ROH cases compare its table and stdout with the old awk, including band overlaps, the 5 Mb threshold and rounding.The container E2E cases
32-roh-11andbash-step-11-rohneed Linux CI: this box cannot mount the data-disk worktree in Colima. Their summary command is covered locally by the three ROH oracle cases. E2E and the later full-collector shadow run remain with the landing train. CodeRabbit returned a review-limit warning and no review.Bead implemented: pgp-bs9u.43.
Independent review fixes
30, nan, 40now make the section unreadable with empty data. The median usespycompat::stats::median. Infinity keeps Python's comparison rules:30, inf, 40has median 40, and30, -inf, 40has median 30. A non-finite median has no detection limit./usr/bin/time -l: before, 0.17-0.19 s and 153,059,328-153,092,096 bytes peak RSS. After, 0.01 s and 3,538,944-3,588,096 bytes. The summary JSON agrees exactly, with onlygenerated_utcmasked.cargo test -p og-summary --no-fail-fastreported 68 passed and 2 failed.chip_nan_depth_is_unreadableobservedok.unindexed_window_retains_only_overlapsobserved all 250,003 chr22 records. The fixes pass those tests. The original G2 planted fault and its revert remain linked above.The widened file list includes these integration paths:
docs/rust.md,rust/Cargo.lock,rust/README.md,rust/og/src/main.rs,tests/parity/cases.tsv,tests/parity/gen/roh-summary.sh,tests/parity/gen/summary-collect-c.sh,tests/parity/golden/summary-collect/c-bad2/bad2.json,tests/parity/inputs/roh-summary/oracle.sh,tests/parity/inputs/summary-collect/oracle.sh,tests/parity/oracle.refandtests/test_collect_summary.py.The final head merges main once to resolve GitHub's changelog conflict. Both lanes' entries are preserved. The combined summary and developer-tool tests pass (72 and 58 tests). Clippy passes with the repository's pinned Rust 1.93 toolchain.
Final-head CI runs on
a6f150d0a538c48df424512d3a3f53e8e70310ae: Guard, Rust, Nextflow, Lint, Docs, Container Test and Personal data. Personal data passed. The other fast checks are queued or running. E2E stays with the landing train.Rechecked the controls locally on the final head with
cargo +1.93.0 test -p og-summary --no-fail-fast. Planted median, G2 matching and window faults made all six relevant tests fail (66 passed, 6 failed). Restored both files byte for byte, then all 72 tests passed. These temporary controls were never committed or pushed.