feat(summary): ClinVar, PGx, ACMG SF and two-hit sections in Rust

CHANGELOG.md

@@ -4,6 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
 
 ## Unreleased
 
+- The Rust `og` binary reads the clinical sections of summary.json: `og summary collect --only b`. That covers the ClinVar screen (stars, call floor, gnomAD frequencies and rsIDs from step 13's indexed VEP output, the cDNA check, scan and deep intronic tables), the carrier list, PharmCAT, CPIC, pypgx, Cyrius, CPSR, the clinical variant filter with its ACMG SF v3.3 tier and the two-hit rule, slivar with its phased compound-het pairs, and the CYP2D6 block. Nothing calls it yet: the reports still read the summary that `bin/collect_summary.py` writes. The sections are the same as the Python's on 13 parity cases, compared as JSON with key order; a fourteenth holds the deliberate fix below. Quoted headers in SMN, interval and dosage tables preserve deletion evidence. Non-ASCII consensus contigs leave the collector running. Each input is read once: the ClinVar hits feed four sections, and the clinical VCF is read line by line in one pass for its record count and its HIGH-impact records. On a synthetic folder with 300 ClinVar hits, a 200,000-record clinical VCF and a 200,000-row slivar table, it takes 0.3 s and 51 MB, where the Python collector took 1.3 s and 135 MB. With 50,000 ClinVar hits (far more than a genome has) it takes 0.6 s but 460 MB, where the Python took 3.5 s and 113 MB, because og builds the whole summary as a JSON tree before it writes it. What differs from the Python:
+  - A clinical or slivar VCF named `.vcf.gz` that is plain text is read. The Python marked the section unreadable.
+  - A PharmCAT report that is not UTF-8 is still unreadable, but the note names the fault in Rust's words.
+
 - The three builders of the committed asset tables now run in the Rust `og` binary: `og dev build cn-intervals` (the interval rows of `bin/cn_loci.tsv`), `og dev build callability` (`assets/callability_genes.tsv`) and `og dev build sv-rule-regions` (`assets/sv_rule_regions.tsv`). They replace `scripts/ci/build-cn-intervals.py`, which is gone (the script is now `scripts/ci/build-cn-intervals.sh`), and the Python inside `scripts/ci/build-callability.sh` and `scripts/ci/build-sv-rule-regions.sh`. The scripts still download the pinned files, and the three tables are the same, byte for byte, as the committed ones. With the downloads in the cache, the callability table takes 2.5 s and 27 MB instead of 5.5 s and 39 MB, and the SV rule table 4.9 s and 30 MB instead of 10.5 s and 217 MB. What changes on purpose:
   - The three scripts share one download function, `scripts/ci/lib/fetch-pinned.sh`. A file the cache folder already holds with the right sha256 is read where it is, not copied. A 403 or 404 fails at once, including after an earlier retry. Other failures, stalled transfers included, get at most six retries, 30 s apart, within a five-minute retry window. `build-cn-intervals.py` tried three times, 30 s apart.
   - The marker line of `bin/cn_loci.tsv` names `scripts/ci/build-cn-intervals.sh`. No row changes.
@@ -19,6 +23,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
   - `versions.env` is read, not run in bash, and a line that is not `NAME="value"` stops the check with exit 2.
   - Two `*_IMAGE` variables with the same value are both named, joined by `|`. The Python named only the alphabetically last one. No two share a value today.
   - `scripts/ci/image-smoke.sh` probes a changed image's helpers only when `bin/og` is built, and says so when it is not. Guard's `helper-binaries` job still probes the helpers of every image on each pull request.
+
 - The Rust `og` binary has a first part of the summary collector: `og summary collect --only a`. It reads seven sections of summary.json (coverage, callability, sex_check, sample_qc, mosaic_sex_loss, read_count and variants), the run status, the manifest and each section's state. Nothing calls it yet: the reports still read the summary that `bin/collect_summary.py` writes. The sections are the same as the Python's on 33 parity cases, compared as JSON with key order. It walks the sample folder once instead of running a `glob` for each candidate path, and it reads step 33's table once for its two sections. On a synthetic 1,000,000-record VCF the variant count takes 0.05 s and 7 MB, where the Python collector took 0.9 s and 38 MB. What differs from the Python:
   - A file named `.vcf.gz` that is plain text is read. The Python marked the variants section unreadable.
   - A sample name with `*`, `?` or `[` is matched as written. The Python read it as a file pattern.

docs/rust.md

@@ -22,7 +22,8 @@ rust/
   crates/og-setup/      what setup.sh runs once: the STRchive catalog, zips, the genetic map
   crates/og-steps/      the step helpers: constraint-join, prs format, step 40's consumer export; og family (step 38); og pgx outside-calls|hla-concordance|pypgx-summary (steps 08, 32, 36); og paraphase (step 35b)
   crates/og-summary/    the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample;
-                        og summary collect --only a, the first seven summary sections (no caller yet)
+                        og summary collect --only a, the first seven summary sections (no caller yet);
+                        --only b, the clinical sections: ClinVar, carrier, PGx, CPSR, ACMG SF, two hits
   crates/og-pedigree/   the pedigree rules: families files, members, kinship, PARs
   crates/og-chip/       step 49, clonal haematopoiesis: og chip prepare, report, check-table
   crates/og-cn/         copy number from depth (steps 32, 35, 41, 46): og cn bed, call, intervals, check-bam, marker-regions, marker-read

rust/README.md

@@ -10,7 +10,7 @@ OwnGenome's own tools, ported from `bin/*.py` one at a time into one static bina
 | `crates/og-bam` | BAM reading as pysam does it (fetch, `reference_end`, `cigarstring`, `get_blocks`), on htslib linked statically, and the tools that read BAMs: the reads at a variant (`reads`: one CIGAR walk per read, SA pieces, filters per tool), `og cdna check`, `og cdna scan`, `og comphet phase`, `og bam contigs`, the CYP2D6 depth check and the RFC1 motif count |
 | `crates/og-pedigree` | the pedigree rules of step 38: families files, members, kinship, declared relationships, PARs |
 | `crates/og-qc` | `og qc sex-loss`: step 33's mosaic loss of Y or X, with CPython's medians, `fmean` and exact `variance` |
-| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the first sections of summary.json, `og summary collect --only a`, beside the Python collector, which still writes it |
+| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC) and `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar), beside the Python collector, which still writes it |
 | `crates/og-chip` | step 49, clonal haematopoiesis: `og chip prepare`, `og chip report`, `og chip check-table` |
 | `crates/og-cn` | copy number from depth, steps 32, 35, 41 and 46: `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions`, `og cn marker-read` |
 | `crates/og-bloodtype` | step 41, blood groups: `og bloodtype report`, `og bloodtype check-markers`, `og bloodtype systems` |

rust/crates/og-summary/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "og-summary"
-description = "The summary cluster: the step tools that turn a third-party tool's output into one of OwnGenome's tables (step 33's sample QC verdict, step 25's PRS table, step 20's chrM tables)"
+description = "The summary cluster: the step tools that turn a third-party tool's output into one of OwnGenome's tables (step 33's sample QC verdict, step 25's PRS table, step 20's chrM tables), and og summary collect, summary.json section by section"
 version.workspace = true
 edition.workspace = true
 license.workspace = true
@@ -9,6 +9,8 @@ publish.workspace = true
 [dependencies]
 pycompat.workspace = true
 og-io.workspace = true
+og-bam.workspace = true
+og-clinvar.workspace = true
 og-steps.workspace = true
 indexmap.workspace = true
 memchr.workspace = true

rust/crates/og-summary/src/collect.rs

@@ -7,7 +7,9 @@
 //! switches to it. Group `a` is the model and the first seven sections:
 //! coverage, callability, sex_check, sample_qc, mosaic_sex_loss,
 //! read_count and variants, with the run status, the manifest and each
-//! section's state.
+//! section's state. Group `b` is the clinical sections ([`group_b`]):
+//! clinvar, carrier, pharmcat, cpic, pypgx, cyrius, cpsr, clinical (with
+//! the ACMG SF tier and the two-hit rule) and slivar, and the CYP2D6 block.
 //!
 //! How it reads a folder: one walk of the folder ([`scan::FolderIndex`])
 //! instead of a `glob` per candidate path; each input read once (the
@@ -16,10 +18,20 @@
 //! sections are read. A reader's error stays in its section, which is
 //! then `unreadable` with a note that names the file.
 
+pub mod acmg;
+pub mod assets;
+pub mod carrier;
+pub mod clinical;
+pub mod clinvar;
+pub mod cpsr;
+pub mod group_b;
 pub mod model;
+pub mod pgx;
 pub mod scan;
 pub mod sections;
+pub mod text;
 pub mod time;
+pub mod two_hits;
 
 use crate::args::{self, Kind, Opt};
 use model::{GROUP_A, Manifest, RunStatus, Section, Spec, Summary};
@@ -29,7 +41,7 @@ use sections::{Callability, Coverage, Data, Mosaic, ReadCount, SampleQc, SexChec
 use std::io::Write;
 use std::path::Path;
 
-const USAGE: &str = "usage: og summary collect --sample SAMPLE --sample-dir SAMPLE_DIR --only a
+const USAGE: &str = "usage: og summary collect --sample SAMPLE --sample-dir SAMPLE_DIR --only {a,b}
                           --out OUT [--declared-sex DECLARED_SEX] [--assets ASSETS]
 ";
 
@@ -39,7 +51,7 @@ const OPTS: &[Opt] = &[
     Opt("--declared-sex", Kind::Value, false),
     Opt("--out", Kind::Value, true),
     Opt("--assets", Kind::Value, false),
-    Opt("--only", Kind::Choice(&["a"]), true),
+    Opt("--only", Kind::Choice(&["a", "b"]), true),
 ];
 
 /// What one collect reads.
@@ -75,12 +87,21 @@ pub fn cli(argv: &[String], out: &mut impl Write, err: &mut impl Write) -> u8 {
     // A part of the sections must never replace the sample's complete
     // summary.json, so there is no default path while `--only` exists.
     let target = a.text("--out").to_owned();
-    let summary = match collect(&input) {
+    let read = match a.text("--only") {
+        "b" => group_b::collect,
+        _ => collect,
+    };
+    let summary = match read(&input) {
         Ok(s) => s,
         Err(e) => return crate::finish(Err(e), err),
     };
+    // The sections' own data is freed before the text is made, and the
+    // tree before the file is written: a large ClinVar list is in memory in
+    // one form at a time.
+    let value = summary.to_json();
+    drop(summary);
     let body = match dumps(
-        &summary.to_json(),
+        &value,
         &Dump {
             indent: Some(1),
             ..Dump::default()
@@ -89,6 +110,7 @@ pub fn cli(argv: &[String], out: &mut impl Write, err: &mut impl Write) -> u8 {
         Ok(b) => b + "\n",
         Err(e) => return crate::finish(Err(e.to_string()), err),
     };
+    drop(value);
     let r = crate::write_file(&target, body.as_bytes());
     if r.is_ok() {
         let _ = writeln!(out, "Summary written: {target}");
@@ -195,6 +217,7 @@ pub fn collect(input: &Input) -> Result<Summary, String> {
         manifest,
         declared_sex,
         sections,
+        cyp2d6: None,
     })
 }