diff --git a/CHANGELOG.md b/CHANGELOG.md index b2cd0f587..3175932c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The Rust summary's CHIP reader marks a depth table with NaN unreadable instead of sorting it into a usable sensitivity estimate. Its unindexed APOL1 G2 reader keeps only records overlapping the deletion span and stops past that span on sorted input. + +- The Rust `og` binary now reads the rest of the summary's own sections: `og summary collect --only c`. It reads 32 sections of summary.json (couple, parascopy, paralogs, hla, prs, ancestry_fine, manta, delly, cnvpytor, sv_consensus, annotsv, expansions and their readings, telomere, roh, haplogroup, y_haplogroup, y_tree, y_str, consumer_export, bloodtype, lpa, vntyper, locityper, mito, traits with the APOL1 G2 call, archaic, ancient, viral, exomiser, chip, mei and mosaic). Nothing calls it yet: the reports still read the summary that `bin/collect_summary.py` writes. The sections are the same as the Python's on 15 new parity cases, compared as JSON with key order. The readers run on a thread pool, one per section. On a synthetic sample folder with every section and a 1,000,000-record VCF it takes 0.33 s and 34 MB, where the Python readers took 0.33 s and 41 to 59 MB. What differs from the Python: + - A chrM file named `.vcf.gz` that is plain text is read, as the variants section already does. The Python marked the mito section unreadable. + - The APOL1 G2 deletion is read through the VCF's `.tbi` when there is one, so only its window of chr22 is read. + +- Step 11's ROH summary table now comes from `og roh summary`, in the bash step and in the Nextflow ROH module, instead of an inline awk program. The table is the same, byte for byte, on 3 parity cases. The masked bands (centromeres and other blocks where ROH calls are not trusted) are one list in the Rust code, which `og roh summary` and the summary reader share. The bash step and the module no longer keep their own copy; the Python collector keeps its copy until it is retired, and a test checks that the two lists are the same. `og roh summary` writes the table to stdout without `--out`, so the bash step writes it on the host. + - The Rust `og` binary reads the clinical sections of summary.json: `og summary collect --only b`. That covers the ClinVar screen (stars, call floor, gnomAD frequencies and rsIDs from step 13's indexed VEP output, the cDNA check, scan and deep intronic tables), the carrier list, PharmCAT, CPIC, pypgx, Cyrius, CPSR, the clinical variant filter with its ACMG SF v3.3 tier and the two-hit rule, slivar with its phased compound-het pairs, and the CYP2D6 block. Nothing calls it yet: the reports still read the summary that `bin/collect_summary.py` writes. The sections are the same as the Python's on 13 parity cases, compared as JSON with key order; a fourteenth holds the deliberate fix below. Quoted headers in SMN, interval and dosage tables preserve deletion evidence. Non-ASCII consensus contigs leave the collector running. Each input is read once: the ClinVar hits feed four sections, and the clinical VCF is read line by line in one pass for its record count and its HIGH-impact records. On a synthetic folder with 300 ClinVar hits, a 200,000-record clinical VCF and a 200,000-row slivar table, it takes 0.3 s and 51 MB, where the Python collector took 1.3 s and 135 MB. With 50,000 ClinVar hits (far more than a genome has) it takes 0.6 s but 460 MB, where the Python took 3.5 s and 113 MB, because og builds the whole summary as a JSON tree before it writes it. What differs from the Python: - A clinical or slivar VCF named `.vcf.gz` that is plain text is read. The Python marked the section unreadable. - A PharmCAT report that is not UTF-8 is still unreadable, but the note names the fault in Rust's words. @@ -50,6 +58,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - The genotype lookup (`scripts/owngenome-lookup.sh`) now runs in the Rust `og` binary as `og lookup query`: one container and one process answer every query of the command line. It reads an indexed gVCF or annotated VCF through its `.tbi` or `.csi`, so it no longer starts bcftools containers or writes temporary slices, and it reads the site table, the pipeline tables and the gene lines once for all queries. An rsID that only the annotated VCF knows is looked for in one pass for every such rsID. It replaces the `resolve` and `report` commands of `bin/genome_lookup.py` (now `og lookup resolve` and `og lookup report`, with the same options); the file keeps only the query reading and the names rule that `bin/results_index.py` imports. The answers are the same, byte for byte, on the 56 parity cases, except for one change on purpose: - A gene's line `N of M bases have a genotype call` counts each base once. It summed the gVCF records, so a base that two records cover (a deletion and the reference block after it) counted twice, and the count could read higher than the truth. + - Step 38's shared DNA segments (the IBD report) now run in the Rust `og` binary: `og family ibd` replaces `bin/ibd_report.py`, in step 38 and in the Nextflow process `FAMILY_IBD_REPORT`. `ibd_pairs.tsv`, `ibd_segments.tsv` and `ibd_report.md` keep their names, columns and content. It reads one chromosome at a time and keeps one byte per member per marker, so on a synthetic family of five members over 22 chromosomes of 50,000 markers it takes 0.3 s and 6 MB instead of 6 s and 120 MB. What changes: - A missing or unreadable input (the chromosome list, a genetic map, a phased VCF, a hap-ibd file, the pedigree) stops the step with exit 2 and one `ERROR: cannot read : ` line. The Python stopped with exit 1 and a traceback. - A phased VCF record with fewer sample columns than its `#CHROM` line, a position that is not a number, or a record of another chromosome than the file's stops the step with exit 2 and names the file and line. The Python crashed later, or counted opposite homozygotes against the wrong markers. @@ -64,15 +73,18 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - `og pgx pypgx-summary` also looks up a VCF whose header has no `##contig` lines, by the contigs its index names. The Python found no defining variant in such a VCF. - Steps 08 and 36 treat a T1K allele quality that is not a finite number (`nan`, `inf`) like a quality of 0: the HLA gene is held back and not compared. The Python passed a `nan` quality to PharmCAT. - Errors read `ERROR: `; a broken JSON names the byte, not the line and column, and a file that is not a BAM says so instead of gzip's message. + - Step 05 and the Nextflow `ANNOTSV` now build AnnotSV's input and check its class 4 and 5 calls with the Rust `og` binary: `og sv add-callers` replaces `bin/cnvpytor_to_vcf.py` and `og sv postfilter` replaces `bin/annotsv_postfilter.py`. `_annotsv_input.vcf` and `_sv_class4_5_checked.tsv` keep their columns and rows; on synthetic inputs (393,157 output records; 20,000 class 4-5 rows against 150,000 transcripts) both write the same bytes as the Python, add-callers in 0.8 s and 156 MB where the Python took 3.6 s and 471 MB, the checks in 1.7 s and 144 MB where it took 44 s and 360 MB. Both tools apply one read-depth rule for long Delly calls. `cnvpytor_to_vcf.py` keeps only that rule for step 39, and `annotsv_postfilter.py` only the column names and the consensus join the reports import. - A Delly `RDCN` that is not a whole number of copies (`1.5`, `inf`) is no evidence in the class 4-5 table: `Depth_support` says `not checked` and the class stays. The Python read it as `no` and moved the call to class 3. Delly writes whole numbers, so a default run does not change. - Inputs are read as gzip by their first bytes, not by a `.gz` name. - A base record whose POS is not a whole number, a record with fewer than 8 columns, a missing input file or a rule-table row whose start or end is not a number stops the step with an `ERROR:` line and exit 1, not a Python traceback. + - Steps 06, 17 and 31 (and the CLINVAR_CDNA_CHECK and COMPHET_PHASE modules) now read the reads at a variant with the Rust `og` binary: `og cdna check` (the cDNA read check of each ClinVar or CPSR hit), `og cdna scan` (the genome-wide cDNA scan) and `og comphet phase` (the phase of each compound-het pair). They replace `bin/cdna_scan.py` and `bin/comphet_phase.py`, which are gone, and the read check of `bin/clip_check.py`, which keeps only the names the report imports. Step 32 and the PYPGX module read the BAM's contig names with `og bam contigs` instead of a pysam one-liner. These steps no longer need pysam; they still run in `PYPGX_IMAGE`, so CI can run the deleted Python beside `og`. The tables, file names and options stay the same; 45 parity cases on made-up BAMs hold them. What changes: - The cDNA scan lists its loci in the BAM's contig order, then by position. It sorted the contig names as text, so `chr10` came before `chr2`. Only the row order changes. - The scan reads each record once, even when it spans two exon regions, and reads the contigs on several threads. On a synthetic BAM of 2 million reads and 59,940 exons it takes 1.3 s and 22 MB where the Python took 2.2 s and 82 MB. - Each table is written to a temporary file and renamed, so a failed run leaves no half-written table. - A row of step 31's per-variant table with fewer columns than its header stops `og comphet phase` with exit 2 and the line. The Python crashed with a traceback. + - Step 41 (blood groups) now runs in the Rust `og` binary: `og bloodtype report` replaces `bin/bloodtype_report.py`, which is gone, and `og bloodtype check-markers` checks that the marker table and the rules agree. The report reads the 318 marker positions through the VCF's `.tbi` or `.csi` instead of the whole VCF; without a usable index it reads the VCF once, line by line, as before. On a synthetic 5,000,000-record bgzipped VCF it takes 0.04 s with the index and 0.9 s without, against 1.9 s for the Python. The Nextflow `BLOODTYPE` process now stages the VCF's index beside it. The table, the evidence table and the printed summary are the same, byte for byte, on the 34 parity cases. What changes: - The RhD call reads the copy-number catalog and assesses the RHD row once per run. The Python read the catalog up to five times. - Each table is written to a temporary file and renamed, so a failed run leaves no half-written table. @@ -114,20 +126,26 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - Step 38, step 39 with a families file, `run-all.sh` with `FAMILIES` and the family workflow now need `og` (`scripts/setup.sh --og`). - The family page now keeps a child's results to act on closed, as the child's own page does, and what it shows of a child no longer depends on which result is closed or where it lies. While a child has a result still closed, or the page cannot read the child's results, these close for that child, one box each: the painting's grandparents (every stretch drawn from its parent alone), the chromosome map image, all de novo and compound-het rows, the child's medicines and HLA cells in the side by side, and the carrier screen of a couple the child is part of. Each "Show the result" opens one result; the places come back once the last one is opened, and a reload closes them again. While the results cannot be read, each box opens only its own place. On a child's page, a copy-number dot and a card's fact that name a closed result's gene now close with it too, on a phone in the card's line as well. `results.json` adds `veil.spans`, `veil.facts` and `veil.widget`; nothing is renamed or removed. The veil is still a browser veil: `results.json` and `family.json` hold every row. + - Step 40, the 23andMe and AncestryDNA raw-data files, now runs on the Rust `og` binary: `og consumer export`, `og consumer build-sites` and `og consumer check` replace `bin/export_consumer.py`. On a synthetic gVCF of 552 MB (32 million records) with 617,044 sites it takes about 2 s and 43 MB in a Linux container on 4 cores, where the Python took 30 s and 660 MB; the files hold the same rows. `setup.sh --consumer-sites` reads dbSNP on 4 cores without the grep pass. The zips are now the same bytes for the same day (`--date`, else `SOURCE_DATE_EPOCH`, else today). Fixes: a call of three or more alleles stops the export with the site, instead of a file the strict reader refuses; two dbSNP records of one rsID at one position have their ALT alleles joined before the strand test; a bad site row or an unreadable zip is reported with the file and line instead of a traceback; a good export removes the `.txt` files a failed one left. The musl builds of `og` use the mimalloc allocator, which keeps a long multi-threaded read at its live memory. Step 40 needs `og`: `scripts/setup.sh ` builds it, and `scripts/setup.sh --og` builds only it. - Steps 21, 32 and 9b (and the CYRIUS, PYPGX and STRANGER_RFC1 modules) now run the CYP2D6 depth check and the RFC1 motif count with `og`, and `bin/cyp2d6_depth_check.py` and `bin/rfc1_motif.py` are gone. A run that reaches these steps needs the binary: run `./scripts/setup.sh --og` once after setup and after each pull; without it the step stops with `og is not built`. The output files, their names and formats do not change; in CI, every e2e case that runs these steps in bash checks `og` against the deleted Python, byte for byte. + - Step 33's estimate of mosaic loss of chrY or of one chrX now runs in the Rust `og` binary: `og qc sex-loss` replaces `bin/mosaic_sex_loss.py`, which is gone. The `mosaic_*` rows it adds to `qc/_sample_qc.tsv` are the same, byte for byte, down to the last digit of CPython's exact variance, and it reads a whole genome's mosdepth windows about three times faster. Step 33 and the Nextflow `SAMPLE_QC` process now need `og`: run `scripts/setup.sh --og` once. Three things change on purpose: - A depth of `nan` or `inf` in the mosdepth windows is now named in `mosaic_reason` as unreadable. Before, the median of a block with `nan` depended on where it sat. - A VCF site whose `AD` adds up past 4,294,967,295 reads is now skipped. Before, it stopped the tool with an `OverflowError`, and the table got no `mosaic_*` rows. - A regions file or VCF that is not UTF-8 is named in `mosaic_reason` with its line number, not Python's byte offset into its read buffer. A step 33 table that is not UTF-8 still stops the tool (exit 1), now with its line number instead of Python's `UnicodeDecodeError` text. + - The checks CI runs on the pipeline itself now run in the Rust `og` binary: `og dev contract` (the output contract, from `scripts/ci/check-output-contract.py`), `og dev schema` (a `summary.json` against its schema, from `tests/schema/validate.py`) and `og dev report-only` (the report-only error strategy, from `scripts/ci/check-report-only.py`). The two Python checks and their two test files are gone; their checks are cargo tests in Guard's `rust` job. `tests/schema/validate.py` stays only as an in-process helper of three Python test files (the unit-tests job has no `og`), with the same rules as `og dev schema`. Each now fails on what it let pass: - `og dev contract --static`: a `nextflow_schema.json` default of another JSON type than `nextflow.config`'s value (`0` for `false`, `1` for `true`) now fails. A results file the run check cannot read (a truncated gzip) is a `FAIL` line that names it, not a traceback. Commented-out Groovy no longer counts. - `og dev schema`: `additionalProperties: false` now refuses a key the schema does not list; `true` no longer matches `enum: [1]`; NaN fails `minimum`. A schema with an unknown keyword or type name, or a `$ref` to nothing or back to itself, anywhere in it, is now exit 2 (a broken schema), not a pass or a traceback. A document that is not JSON prints `ERROR:` and exits 1. - `og dev report-only`: a `//` inside a string is no longer read as a comment, and a nested brace in the strategy closure, or a setting before it, no longer hides a `withName` block. - `--repo` defaults to the current folder for `og dev contract` and `og dev report-only`. + - Step 06's ClinVar screen runs in the Rust `og` binary, with one table of ClinVar review stars: `og clinvar join-adjacent` (the combined-allele joiner; its `build` mode reads the reference itself, in place of the `bcftools +fill-from-fasta` pipe), `og clinvar hits` (the console TSV) and `og clinvar deep-intronic` (the always-report list) replace `bin/clinvar_join_adjacent.awk` and `bin/clinvar_hits.awk`, which are gone. On 19 cases og writes what the awk wrote, byte for byte, with one fix: an empty GENEINFO segment (between two `|`) is left out of the TSV's gene column, as the run summary already did. Step 06 now needs `bin/og` (`./scripts/setup.sh --og`) and an uncompressed reference with its `.fai`, the documented default; a gzip reference is refused. + - Step 25's scoring files are written by the Rust `og` binary: `og prs format` replaces `collect_summary.py prs-format` in `PRS_PREPARE` and `scripts/25-prs.sh`. It reads one score at a time, so `PRS_PREPARE` peaks at 0.7 GB instead of 11.5 GB on the default scores and asks for 2 GB instead of 16 GB, and it runs in seconds instead of minutes. The files it writes hold the same rows, labels and notes. Fixes: two scoring files with one PGS id (`PGS1.txt` and `PGS1.txt.gz`) now stop the step with `ERROR: PGS1 has two scoring files in DIR: PGS1.txt, PGS1.txt.gz`, where the second overwrote the first and both were counted; a score list that is not UTF-8 gives `ERROR: : not UTF-8 at byte N`, not a Python traceback; a scoring file whose name and `#pgs_id` differ takes its label from the list's row for the `#pgs_id`, as its sex and evaluations already did. With `--keep-ambiguous`, a malformed `.fai` line (`ERROR: cannot read the reference REF and its .fai: .fai line N: ...`) or a `.fai` that does not match the FASTA (`the .fai does not match the FASTA: ...`) now stops the step, where the Python skipped short `.fai` lines and read whatever byte sat at the offset. The score files are gzip level 1 instead of 9: the same text, a little larger. + - Step 49 (clonal haematopoiesis) now prepares the CHIP gene models and writes its two tables with the Rust `og` binary: `og chip prepare`, `og chip report` and `og chip check-table` replace the commands of `bin/chip_report.py`, which keeps only the detection limit and the column names the reports import. On Ensembl 116's full GFF3, prepare takes 1.7 s and 6 MB instead of 15 s and 20 MB. The two tables, `chip.gff3` and `chip.bed` are the same, byte for byte. What changes: a missing input file or a position that is not a whole number stops with an `ERROR:` line and exit 1, not a Python traceback, and each output is written whole or not at all. Step 49 now needs `og` (`scripts/setup.sh --og`). - VEP consequence ranking (steps 23 and 31) and the gnomAD constraint join now run in the Rust `og` binary, inside `BCFTOOLS_IMAGE`: `og vcf csq-rank` replaces `bin/csq_rank.awk` and `og table constraint-join` replaces `bin/constraint_join.awk`, which are gone. Neither awk program runs on the host anymore (step 23 still uses the host's `awk` for small table edits), and the ranker is no longer the slow stage of step 23's main pipe. Steps 23 and 31 now need `og`: `scripts/setup.sh --og` builds it. Their output is the same, byte for byte, except for these changes. @@ -135,17 +153,20 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - A ranking floor that is not a number, or a `--keep` other than `all` or `noncommon`, is a usage error (exit 2). awk read such a floor as 0. - An empty constraint file now stops with `ERROR: the constraint table is empty` (exit 3). awk read the summary table as the constraint file and blamed the wrong file. - An unreadable input is reported with its path and the reason: exit 2 from `og table constraint-join`, exit 1 from `og vcf csq-rank`. + - The STRchive catalog, Exomiser's zips and the genetic map are now set up by the Rust `og` binary: `og str catalog` (replaces `bin/strchive_convert.py`, which is gone), `og setup unzip` and `og setup genetic-map` (replace two Python programs inside `scripts/setup.sh`). The catalog and its ranges table are the same, byte for byte, and so are the unpacked Exomiser data and the maps. `setup.sh` builds `og` first when these need it. The STRchive catalog is converted once more after this update, because its stamp now names the converter's format instead of the Python file's hash. - Steps 26b and 51: each genetic map file is written whole or not at all. Before, a setup killed while it wrote a map could leave a truncated `.rfmix.map` that every later setup took as done. Step 26b's maps now carry a stamp, `maps/DONE`, written last; an install without it, or with a map file missing, writes every map again. `tests/test_setup_fine_maps.sh` checks this on a cut-short map. - `setup.sh --exomiser-data`: a zip member whose name leaves the folder (an absolute path or `..`) is refused, and nothing is unpacked. Python's `extractall` wrote it elsewhere, under a name with those parts dropped. - A genetic map row with fewer than four fields, a STRchive record with a missing coordinate, a locus structure with no repeat, and an extra STR locus whose `LocusId` or `VariantId` is not text now stop with exit 2 and a message, not a Python traceback. - A damaged zip whose sizes claim more than the file holds, or a member that inflates past its recorded size, now stops with exit 2 before memory or disk fills up. - `setup.sh --strchive-data` builds `og` before it removes an outdated catalog, so a failed build keeps the catalog that was there. + - Step 26b (fine-grained ancestry) now runs in the Rust `og` binary: `og ancestry-fine panel-map`, `check-map`, `order`, `sample-bed`, `chroms` and `report` replace the commands of `bin/ancestry_fine_report.py`, in the step script, the Nextflow module and `setup.sh --ancestry-fine-panel`. The module now keeps only the group names, colours and minimum that the report and the docs demo read. The outputs are the same, byte for byte, except as listed below. No option, output file or format changes. Steps 26b and the panel install now need `og`, and `setup.sh` builds it when it is missing. - A failed write no longer leaves a partial table. `sample-bed` used to write each level's ADMIXTURE input as it went, so a `.P` file with the wrong number of rows left the earlier level's `.bed`, `.bim`, `.fam` and `.P.in` behind for the next command. Every input is now read and checked first, and every output is written under a temporary name and renamed into place. `run_admixture.sh` comes last. - A missing or malformed input (a `.Q` or `.P` file that is not there, a position that is not a whole number, an RFMix file without its codes line) is now an `ERROR:` line and exit 1, not a Python traceback. - The genotypes VCF is read as a stream: only one byte per reference SNV is kept, whatever the VCF's size. On a synthetic panel of 500,000 SNVs, `sample-bed` takes 0.5 s and 67 MB, where the Python took 1.5 s and 370 MB. - The painting PNG has the same pixels. Its compressed bytes differ, because another deflate writes them. The comment at the top of `run_admixture.sh` names `og ancestry-fine sample-bed`. + - Step 27 (CPIC lookup) reads PharmCAT's report with `og pgx cpic-report`, in the Rust `og` binary, instead of `bin/pgx_parse.py`, which is gone. The Nextflow `CPIC_LOOKUP` module runs the same command. Step 27 needs `og`: `scripts/setup.sh ` builds it, and `scripts/setup.sh --og` builds only it. The phenotype table, the comparison table and the recommendations are the same, byte for byte, on every case we test, with these fixes: the tables are written under a temporary name and renamed into place, so a stopped step leaves no half-written file; a field of the report with an unexpected type reads as missing instead of stopping the step with a Python traceback; two or more guidance sources the parser does not know are listed by name, not in an order that changed from run to run; the `Date:` line is the date in UTC, and the hidden option `--date YYYY-MM-DD` sets it for tests. No output file, column or option changes. - Four VCF filters of steps 04, 05 and 07 now run in the Rust `og` binary: `og sv inv-filter` (step 04), `og sv svlen-fix` (step 05), `og pgx refblock-fill` and `og pgx mito-merge` (step 07). They replace `bin/manta_inv_filter.py`, `bin/sv_svlen_fix.py`, `bin/pgx_refblock_fill.py` and `bin/pgx_mito_merge.py`, which are gone. Their output is the same, byte for byte, except for these fixes. Steps 04, 05 and 07 now need `og`: `scripts/setup.sh ` builds it, and `scripts/setup.sh --og` builds only it. @@ -158,14 +179,20 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - Malformed input to any of the four (a record with too few columns, a `POS` that is not a whole number) now stops with exit 2 and a message that names the file and line, not a Python traceback. - The CYP2D6 depth check and the RFC1 motif count, ported to the Rust `og` binary: `og pgx cyp2d6-bed`, `og pgx cyp2d6-measure` and `og pgx cyp2d6-check` (from `bin/cyp2d6_depth_check.py`) and `og str motif` (from `bin/rfc1_motif.py`) read BAMs with htslib, without pysam. On 67 cases they write what the Python writes, byte for byte, with one fix: a mosdepth regions file that lists one region twice with two depths is now refused (exit 2); the Python averaged the two. Steps 21, 32 and 9b run them (see above); no option, output file or format changes. + - Genome map, first pieces, in the Rust `og` binary. `og map build` writes one person's map document: the chromosome bands, the runs of homozygosity, and every finding with a place on the genome (its own position, or its gene's span). A result the page keeps closed (a finding to act on, and APOE on every page) is tagged with its id on every row that could reveal it, so the page's own closed-result rule hides those rows too. `og map genes` builds the gene and exon files once per GENCODE release, offline. No pipeline output, option or page changes yet; the live page starts drawing the map in a later change. The format is in `rust/crates/og-map/README.md`. + - The run-log commands are now `og run state`, `og run eta`, `og run headroom` and `og run delta`, with the same options and the same output. The command lines of `bin/run_state.py`, `bin/run_eta.py`, `bin/run_headroom.py` and `bin/run_delta.py` are gone: run as commands they exit 1 and name the og command; the modules stay for the live site, its alerts, the database re-check and the summary, which import them. `scripts/generate-report.sh` writes the delta with `og run delta --newest-history `, so og picks the previous run by the one rule the summary uses too, and the report now needs og in `bin/.og/` (`scripts/setup.sh --og`), as steps 04, 05 and 07 do. On macOS, `bin/og` runs in a Linux container; docs/run-state.md says how to run the commands there. - `og run delta` copied every summary section once per section it compared and looked ClinVar hits up one by one. On a synthetic summary with 50,000 hits it took 24 s and 770 MB, against 0.3 s for the Python. It now borrows the sections and hashes the keys: 0.3 s and 310 MB, with the same three files byte for byte. + - Run logs in the Rust `og` binary. `og run state`, `og run eta`, `og run headroom` and `og run delta` read Nextflow traces, weblog messages, run-all.sh status and the per-box step history the way `bin/run_state.py`, `bin/run_eta.py`, `bin/run_headroom.py` and `bin/run_delta.py` do, and write the same files byte for byte; a parity test runs both on the same inputs. `og run tasks` gives the step of each trace row, and the Nextflow CI check of ClinVar reruns now reads the traces with it. Two fixes: a PRS percentile of 0 in the delta report now reads 0, not "none", in og and in `run_delta.py`; and `generate-report.sh` now compares a run with the newest `history/` folder that holds a summary.json, as the delta tool itself does: a newer folder without one, left by a copy that stopped half way, no longer stops the report with "the delta against the previous run was not written". og reads a trace memory cell such as "1.2.3 GB" as no value; the Python modules of the live alerts still stop on it. + - Rust building blocks for the ports that look things up by position. `og-io` reads the lines of a region from a bgzipped file through its `.tbi` or `.csi` index, as `tabix` prints them, reading only the blocks the index points at. `og-intervals` reads bases from an uncompressed FASTA through its `.fai`, as `samtools faidx` prints them, and refuses a compressed FASTA or a `.fai` that does not match the file. It also adds sorted interval sets (union, intersection, lookups by binary search, BED files such as the PAR regions) and reciprocal overlap. Nothing calls them yet, so no pipeline output, option or file changes. + - `og` runs natively on a Mac that hosts the live site, so the genome map works there too. `scripts/setup.sh --og` and each release now also make `og-darwin-arm64` and `og-darwin-x86_64`, and `bin/og` runs them on macOS. They are built without the new cargo feature `bam` (on by default, so the Linux builds are unchanged): the BAM tools say so and exit 2 there. `rust/og.sh check` refuses a stale macOS binary, and a release's `SHA256SUMS` lists all four. ## Releases - 1.0.0 (not released yet): [docs/releases/v1.0.0.md](docs/releases/v1.0.0.md) + - 0.2.0 to 0.8.2: https://github.com/GeiserX/OwnGenome/releases diff --git a/docs/rust.md b/docs/rust.md index e7f77e242..17ed2cecf 100644 --- a/docs/rust.md +++ b/docs/rust.md @@ -21,9 +21,7 @@ rust/ crates/og-qc/ og qc sex-loss: mosaic loss of Y or X for step 33 crates/og-setup/ what setup.sh runs once: the STRchive catalog, zips, the genetic map crates/og-steps/ the step helpers: constraint-join, prs format, step 40's consumer export; og family (step 38); og pgx outside-calls|hla-concordance|pypgx-summary (steps 08, 32, 36); og paraphase (step 35b) - crates/og-summary/ the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample; - og summary collect --only a, the first seven summary sections (no caller yet); - --only b, the clinical sections: ClinVar, carrier, PGx, CPSR, ACMG SF, two hits + crates/og-summary/ the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample; og summary collect --only a|b|c, QC, clinical and the remaining summary sections (no caller yet); og roh summary (step 11) crates/og-pedigree/ the pedigree rules: families files, members, kinship, PARs crates/og-chip/ step 49, clonal haematopoiesis: og chip prepare, report, check-table crates/og-cn/ copy number from depth (steps 32, 35, 41, 46): og cn bed, call, intervals, check-bam, marker-regions, marker-read diff --git a/modules/local/roh/main.nf b/modules/local/roh/main.nf index bf7b9c3ca..86df6a0e2 100644 --- a/modules/local/roh/main.nf +++ b/modules/local/roh/main.nf @@ -45,24 +45,11 @@ process ROH { | bcftools roh \${ROH_FLAGS} -o ${meta.id}_roh.txt - # Summary: autosomal segments with 5 Mb or more outside GRCh38's - # centromere and heterochromatin bands (the acen, gvar and stalk rows of - # UCSC's cytoBand.hg38.txt, merged; the line of scripts/11-roh-analysis.sh - # and bin/collect_summary.py), the threshold of docs/11-roh-analysis.md; - # length_bp and length_mb are of that part. The bash step writes the same - # file. - MASKED_BANDS="chr1:121700000-143200000 chr2:91800000-96000000 chr3:87800000-98600000 chr4:48200000-51800000 chr5:46100000-51400000 chr6:58500000-62600000 chr7:58100000-62100000 chr8:43200000-47200000 chr9:42200000-61500000 chr10:38000000-41600000 chr11:51000000-55800000 chr12:33200000-37800000 chr13:0-18900000 chr14:0-18200000 chr15:0-20500000 chr16:35300000-47000000 chr17:22700000-27400000 chr18:15400000-21500000 chr19:19900000-31900000 chr20:25700000-30400000 chr21:0-13000000 chr22:0-17400000 chrX:58100000-63800000 chrY:10300000-10600000 chrY:26600000-57227415" - echo "# ROH Summary for ${meta.id}" > ${meta.id}_roh_summary.txt - echo "# Segments >=5MB on autosomes (excludes chrX/chrY)" >> ${meta.id}_roh_summary.txt - printf 'chrom\\tstart\\tend\\tlength_bp\\tlength_mb\\n' >> ${meta.id}_roh_summary.txt - awk -v bands="\$MASKED_BANDS" ' - BEGIN { n = split(bands, b, " "); for (i = 1; i <= n; i++) { split(b[i], x, /[:-]/); c[i] = x[1]; s[i] = x[2]; e[i] = x[3] } } - \$1 == "RG" && \$3 ~ /^chr([1-9]|1[0-9]|2[0-2])\$/ { - m = 0 - for (i = 1; i <= n; i++) if (c[i] == \$3) { o = (\$5 < e[i] ? \$5 : e[i]) - (\$4 - 1 > s[i] ? \$4 - 1 : s[i]); if (o > 0) m += o } - if (m > \$6) m = \$6 - l = \$6 - m - if (l >= 5000000) printf "%s\\t%s\\t%s\\t%d\\t%.1f\\n", \$3, \$4, \$5, l, l / 1e6 - }' ${meta.id}_roh.txt >> ${meta.id}_roh_summary.txt + # centromere and heterochromatin bands (MASKED_BANDS in og's step 11 + # code, the one copy), the threshold of docs/11-roh-analysis.md; + # length_bp and length_mb are of that part. The bash step runs the same + # command. + og roh summary --sample ${meta.id} --roh ${meta.id}_roh.txt --out ${meta.id}_roh_summary.txt cat <<-END_VERSIONS > versions.yml "${task.process}": diff --git a/rust/Cargo.lock b/rust/Cargo.lock index 352e03131..c5d9ae54a 100644 --- a/rust/Cargo.lock +++ b/rust/Cargo.lock @@ -840,8 +840,11 @@ dependencies = [ "flate2", "indexmap", "memchr", + "og-ancestry", "og-bam", + "og-chip", "og-clinvar", + "og-cn", "og-io", "og-steps", "pycompat", diff --git a/rust/README.md b/rust/README.md index c42b40b64..39a8457c2 100644 --- a/rust/README.md +++ b/rust/README.md @@ -10,7 +10,7 @@ OwnGenome's own tools, ported from `bin/*.py` one at a time into one static bina | `crates/og-bam` | BAM reading as pysam does it (fetch, `reference_end`, `cigarstring`, `get_blocks`), on htslib linked statically, and the tools that read BAMs: the reads at a variant (`reads`: one CIGAR walk per read, SA pieces, filters per tool), `og cdna check`, `og cdna scan`, `og comphet phase`, `og bam contigs`, the CYP2D6 depth check and the RFC1 motif count | | `crates/og-pedigree` | the pedigree rules of step 38: families files, members, kinship, declared relationships, PARs | | `crates/og-qc` | `og qc sex-loss`: step 33's mosaic loss of Y or X, with CPython's medians, `fmean` and exact `variance` | -| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC) and `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar), beside the Python collector, which still writes it | +| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC), `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar) and `--only c` (the remaining summary sections), beside the Python collector, which still writes summary.json; `og roh summary` (step 11) | | `crates/og-chip` | step 49, clonal haematopoiesis: `og chip prepare`, `og chip report`, `og chip check-table` | | `crates/og-cn` | copy number from depth, steps 32, 35, 41 and 46: `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions`, `og cn marker-read` | | `crates/og-bloodtype` | step 41, blood groups: `og bloodtype report`, `og bloodtype check-markers`, `og bloodtype systems` | diff --git a/rust/crates/og-summary/Cargo.toml b/rust/crates/og-summary/Cargo.toml index d17584417..325b1f475 100644 --- a/rust/crates/og-summary/Cargo.toml +++ b/rust/crates/og-summary/Cargo.toml @@ -12,6 +12,9 @@ og-io.workspace = true og-bam.workspace = true og-clinvar.workspace = true og-steps.workspace = true +og-ancestry.workspace = true +og-chip.workspace = true +og-cn.workspace = true indexmap.workspace = true memchr.workspace = true diff --git a/rust/crates/og-summary/src/collect.rs b/rust/crates/og-summary/src/collect.rs index 15a29828f..e351fd308 100644 --- a/rust/crates/og-summary/src/collect.rs +++ b/rust/crates/og-summary/src/collect.rs @@ -10,6 +10,10 @@ //! section's state. Group `b` is the clinical sections ([`group_b`]): //! clinvar, carrier, pharmcat, cpic, pypgx, cyrius, cpsr, clinical (with //! the ACMG SF tier and the two-hit rule) and slivar, and the CYP2D6 block. +//! Group `c` is copy number, HLA, the polygenic scores, +//! ancestry, structural variants, repeats, telomeres, runs of +//! homozygosity, the haplogroups, the opt-in step tables, chrM, traits with +//! the APOL1 G2 deletion, and the couple screen ([`group_c`]). //! //! How it reads a folder: one walk of the folder ([`scan::FolderIndex`]) //! instead of a `glob` per candidate path; each input read once (the @@ -25,6 +29,7 @@ pub mod clinical; pub mod clinvar; pub mod cpsr; pub mod group_b; +pub mod group_c; pub mod model; pub mod pgx; pub mod scan; @@ -41,7 +46,8 @@ use sections::{Callability, Coverage, Data, Mosaic, ReadCount, SampleQc, SexChec use std::io::Write; use std::path::Path; -const USAGE: &str = "usage: og summary collect --sample SAMPLE --sample-dir SAMPLE_DIR --only {a,b} +const USAGE: &str = + "usage: og summary collect --sample SAMPLE --sample-dir SAMPLE_DIR --only {a,b,c} --out OUT [--declared-sex DECLARED_SEX] [--assets ASSETS] "; @@ -51,7 +57,7 @@ const OPTS: &[Opt] = &[ Opt("--declared-sex", Kind::Value, false), Opt("--out", Kind::Value, true), Opt("--assets", Kind::Value, false), - Opt("--only", Kind::Choice(&["a", "b"]), true), + Opt("--only", Kind::Choice(&["a", "b", "c"]), true), ]; /// What one collect reads. @@ -64,6 +70,18 @@ pub struct Input { pub declared_sex: Option, /// The repository's `assets/` folder. pub assets: String, + /// The group of sections to read: `a`, `b` or `c`. + pub group: Group, + /// The folder of the two tables group c reads from `bin/`. + pub tables: String, +} + +/// The group of sections a collect reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Group { + A, + B, + C, } /// `og summary collect`: exit 0 when summary.json is written (whatever @@ -83,6 +101,12 @@ pub fn cli(argv: &[String], out: &mut impl Write, err: &mut impl Write) -> u8 { sample_dir: a.text("--sample-dir").to_owned(), declared_sex: a.get("--declared-sex").map(str::to_owned), assets: a.get("--assets").map_or_else(default_assets, str::to_owned), + tables: group_c::util::default_tables(), + group: match a.text("--only") { + "b" => Group::B, + "c" => Group::C, + _ => Group::A, + }, }; // A part of the sections must never replace the sample's complete // summary.json, so there is no default path while `--only` exists. @@ -149,6 +173,34 @@ pub fn collect(input: &Input) -> Result { let s = pycompat::mtime::escape(&input.sample); let find = |pats: &[String]| index.first(pats).map(str::to_owned); let file = |rel: String| -> (String, String) { (crate::join(&input.sample_dir, &rel), rel) }; + if input.group == Group::C { + // The declared sex the chrY filter of CNVpytor reads comes from + // step 33's table when the run status and manifest have none. + let qc = find(&[format!("qc/{s}_sample_qc.tsv")]).map(file); + let qc_declared = qc.and_then(|(p, _)| { + let rows = sections::read_tsv(&p).ok()?; + SampleQc::read(&p, &rows) + .ok()? + .declared_sex() + .map(str::to_owned) + }); + let declared_sex = declared( + input, + run.as_ref(), + manifest.as_ref(), + qc_declared.as_deref(), + ); + let sections = read_group_c(input, &index, run.as_ref(), declared_sex.as_deref()); + return Ok(Summary { + sample: input.sample.clone(), + generated_utc: time::utc_minute(now()).unwrap_or_default(), + run, + manifest, + declared_sex, + sections, + cyp2d6: None, + }); + } let variants_at = find(&[format!("vcf/{s}.vcf.gz")]); let (variants, rest) = std::thread::scope(|scope| { @@ -194,25 +246,10 @@ pub fn collect(input: &Input) -> Result { Some(Data::SampleQc(q)) => q.declared_sex(), _ => None, }); - let declared_sex = [ - input.declared_sex.as_deref(), - run.as_ref().and_then(|r| r.declared_sex.as_deref()), - manifest - .as_ref() - .and_then(|m| m.value("run", "declared_sex")), - qc_declared, - ] - .into_iter() - .flatten() - .find(|v| !v.is_empty()) - .map(str::to_owned); - - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_or(0.0, |d| d.as_secs_f64()); + let declared_sex = declared(input, run.as_ref(), manifest.as_ref(), qc_declared); Ok(Summary { sample: input.sample.clone(), - generated_utc: time::utc_minute(now).unwrap_or_default(), + generated_utc: time::utc_minute(now()).unwrap_or_default(), run, manifest, declared_sex, @@ -221,6 +258,67 @@ pub fn collect(input: &Input) -> Result { }) } +fn now() -> f64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0.0, |d| d.as_secs_f64()) +} + +/// The declared sex: the command line's, the run status's, the +/// manifest's, then step 33's; the first that is not empty. +fn declared( + input: &Input, + run: Option<&RunStatus>, + manifest: Option<&Manifest>, + qc: Option<&str>, +) -> Option { + [ + input.declared_sex.as_deref(), + run.and_then(|r| r.declared_sex.as_deref()), + manifest.and_then(|m| m.value("run", "declared_sex")), + qc, + ] + .into_iter() + .flatten() + .find(|v| !v.is_empty()) + .map(str::to_owned) +} + +/// Reads the sections of group c, then runs the parts of the post pass +/// that change them. +fn read_group_c( + input: &Input, + index: &FolderIndex, + run: Option<&RunStatus>, + declared_sex: Option<&str>, +) -> Vec
{ + let c = group_c::Ctx { + sample: &input.sample, + dir: &input.sample_dir, + assets: &input.assets, + tables: &input.tables, + index, + }; + let reads = group_c::read_all(&c); + let mut sections: Vec
= group_c::GROUP_C + .iter() + .zip(reads) + .map(|(spec, r)| { + let r = r.and_then(|found| match found { + None => Ok(None), + Some((f, v)) => with_mtime(&f.path, f.rel, Data::Json(v)), + }); + Section::new(*spec, r, run) + }) + .collect(); + let mut view: Vec<(Spec, model::State, Option<&mut Data>)> = sections + .iter_mut() + .map(|s| (s.spec, s.state, s.data.as_mut())) + .collect(); + group_c::post(&c, &mut view, &group_c::Others::default(), declared_sex); + sections +} + /// What one reader gave: the file (relative), its mtime and its data. type Read = Result, String>; diff --git a/rust/crates/og-summary/src/collect/group_c.rs b/rust/crates/og-summary/src/collect/group_c.rs new file mode 100644 index 000000000..15ab27782 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c.rs @@ -0,0 +1,284 @@ +//! Group c of `og summary collect`: copy number, HLA, polygenic scores, +//! structural variants, repeats, telomeres, runs of homozygosity, +//! haplogroups, the step tables of the opt-in steps, chrM, traits with the +//! APOL1 G2 deletion, and the couple screen. +//! +//! Each section is read on its own, and an error stays in its section. The +//! three parts of the Python collector's post pass that change these +//! sections' data run here too, after every section is read: the repeat +//! readings with the traits and CPIC states, Locityper's primary calls, and +//! the chrY filter of CNVpytor's count for a declared female. What they +//! need from group b (the CPIC state, pypgx and Cyrius) comes in +//! [`Others`]; `--only c` reads none of it, so it is empty there. + +mod chrm; +mod cn; +mod hla; +mod repeats; +pub mod roh; +mod sv; +mod tables; +mod traits; +pub mod util; + +use super::model::{Spec, State}; +use super::scan::FolderIndex; +use super::sections::Data; +use pycompat::json::Value; + +/// The sections of group c, in summary.json's order. +pub const GROUP_C: [Spec; 32] = [ + spec("couple", "Couple carrier screen", "39"), + spec( + "parascopy", + "SMN1/SMN2 copy number (Parascopy, short reads)", + "35", + ), + spec("paralogs", "Paralog genes (Paraphase, PacBio HiFi)", "35b"), + spec("hla", "HLA typing (T1K)", "08"), + spec("prs", "Polygenic risk scores", "25"), + spec( + "ancestry_fine", + "Fine-grained ancestry (ADMIXTURE, RFMix)", + "26b", + ), + spec("manta", "Structural variants (Manta)", "04"), + spec("delly", "Structural variants (Delly)", "19"), + spec("cnvpytor", "Copy number (CNVpytor)", "18"), + spec("sv_consensus", "SV consensus merge", "22"), + spec( + "annotsv", + "Structural variants, class 4 and 5 (AnnotSV)", + "05", + ), + spec("expansions", "Repeat expansions (ExpansionHunter)", "09"), + spec("telomere", "Telomere content (TelomereHunter)", "10"), + spec("roh", "Runs of homozygosity", "11"), + spec("haplogroup", "Mitochondrial haplogroup", "12"), + spec("y_haplogroup", "Y-chromosome haplogroup (Yleaf)", "37"), + spec("y_tree", "Y placement on the current public tree", "37"), + spec("y_str", "Y-STR markers (HipSTR)", "45"), + spec( + "consumer_export", + "Raw-data export (23andMe v5 layout)", + "40", + ), + spec("bloodtype", "Blood groups (predicted)", "41"), + spec("lpa", "Lp(a): LPA KIV-2 repeat count", "46"), + spec("vntyper", "MUC1 VNTR frameshifts (VNtyper)", "52"), + spec("locityper", "Locityper second opinion at hard loci", "53"), + spec("mito", "Mitochondrial variants", "20"), + spec("traits", "Traits panel", "42"), + spec("archaic", "Archaic ancestry (IBDmix)", "43"), + spec("ancient", "Ancient populations (AADR PCA)", "44"), + spec("viral", "Viral DNA in the unmapped reads", "47"), + spec("exomiser", "Symptom-driven ranking (Exomiser)", "48"), + spec("chip", "Clonal haematopoiesis (CHIP)", "49"), + spec("mei", "Mobile element insertions (MEGAnE)", "50"), + spec("mosaic", "Mosaic chromosomal alterations (MoChA)", "51"), +]; + +const fn spec(key: &'static str, title: &'static str, step: &'static str) -> Spec { + Spec { key, title, step } +} + +/// A file a section found: its path and its path relative to the folder. +#[derive(Debug, Clone)] +pub struct Found { + pub path: String, + pub rel: String, +} + +/// What a reader sees: the sample, its folder, the assets and one walk of +/// the folder. +pub struct Ctx<'a> { + pub sample: &'a str, + pub dir: &'a str, + pub assets: &'a str, + /// The folder of `hla_alleles.tsv` and `str_intermediate_ranges.tsv`. + pub tables: &'a str, + pub index: &'a FolderIndex, +} + +impl Ctx<'_> { + /// The sample name as a file pattern matches it literally. + pub fn s(&self) -> String { + pycompat::mtime::escape(self.sample) + } + + /// `first_existing(d, patterns)`. + pub fn find(&self, pats: &[String]) -> Option { + self.index.first(pats).map(|rel| Found { + path: crate::join(self.dir, rel), + rel: rel.to_owned(), + }) + } + + /// `first_existing` of one pattern. + pub fn find1(&self, pat: String) -> Option { + self.find(&[pat]) + } + + /// `os.path.join(d, rel)`. + pub fn path(&self, rel: &str) -> String { + crate::join(self.dir, rel) + } + + /// `os.path.join(os.path.dirname(found), name)` when it is a file. + pub fn beside(&self, f: &Found, name: &str) -> Option { + let dir = |p: &str| p.rsplit_once('/').map_or(String::new(), |x| x.0.to_owned()); + let path = crate::join(&dir(&f.path), name); + crate::is_file(&path).then(|| Found { + rel: crate::join(&dir(&f.rel), name), + path, + }) + } + + /// A table of the `bin/` folder ([`util::default_tables`]). + pub fn table(&self, name: &str) -> String { + crate::join(self.tables, name) + } + + /// `assets/`. + pub fn asset(&self, name: &str) -> String { + crate::join(self.assets, name) + } +} + +/// What a section reader gives: the file, its data, and for the sections +/// the post pass reads, what it needs, typed. +pub type Out = Result, String>; + +/// The states and data of group b the post pass reads. Empty under +/// `--only c`, which reads no group b section. +#[derive(Debug, Default, Clone)] +pub struct Others { + pub cpic: Option, + pub traits: Option, + /// pypgx's CYP2D6 call and its section state, when that section is ok + /// or stale. + pub pypgx: Option<(State, String)>, + /// Cyrius's genotype and Filter (either may be absent), when that + /// section is ok or stale. + pub cyrius: Option<(State, Option, Option)>, +} + +/// Reads every section of group c, in [`GROUP_C`] order. +pub fn read_all(c: &Ctx<'_>) -> Vec { + let readers: [fn(&Ctx<'_>) -> Out; 32] = [ + tables::couple, + cn::parascopy, + cn::paralogs, + hla::hla, + tables::prs, + tables::ancestry_fine, + sv::manta, + sv::delly, + sv::cnvpytor, + sv::sv_consensus, + sv::annotsv, + repeats::expansions, + tables::telomere, + roh::section, + tables::haplogroup, + tables::y_haplogroup, + tables::y_tree, + tables::y_str, + tables::consumer_export, + tables::bloodtype, + tables::lpa, + tables::vntyper, + tables::locityper, + chrm::mito, + traits::traits, + tables::archaic, + tables::ancient, + tables::viral, + tables::exomiser, + tables::chip, + tables::mei, + tables::mosaic, + ]; + // The readers are independent: one thread each, over a small pool, so a + // big input (a VCF, a table of thousands of rows) never holds up the + // others. A panic stays in its section. + let jobs = std::thread::available_parallelism().map_or(4, |n| n.get().min(8)); + let next = std::sync::atomic::AtomicUsize::new(0); + let slots: Vec>> = (0..readers.len()) + .map(|_| std::sync::Mutex::new(None)) + .collect(); + std::thread::scope(|scope| { + for _ in 0..jobs { + scope.spawn(|| { + loop { + let i = next.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let Some(r) = readers.get(i) else { break }; + let out = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| r(c))) + .unwrap_or_else(|_| Err("internal error: the reader stopped".to_owned())); + if let Ok(mut slot) = slots[i].lock() { + *slot = Some(out); + } + } + }); + } + }); + slots + .into_iter() + .map(|m| { + m.into_inner() + .ok() + .flatten() + .unwrap_or_else(|| Err("internal error: the reader stopped".to_owned())) + }) + .collect() +} + +/// The post pass over the read sections (`sections` in [`GROUP_C`] +/// order, each with its state and data): the repeat readings with the +/// traits and CPIC states, each Locityper locus's primary call, and the +/// chrY filter of CNVpytor's count. +pub fn post( + c: &Ctx<'_>, + sections: &mut [(Spec, State, Option<&mut Data>)], + others: &Others, + declared_sex: Option<&str>, +) { + let state = |key: &str, secs: &[(Spec, State, Option<&mut Data>)]| { + secs.iter() + .find(|(s, ..)| s.key == key) + .map_or(State::Missing, |x| x.1) + }; + let traits_state = others.traits.unwrap_or_else(|| state("traits", sections)); + let cpic_state = others.cpic.unwrap_or(State::Ok); + let hla_state = state("hla", sections); + let hla_data = sections + .iter() + .find(|(s, ..)| s.key == "hla") + .and_then(|x| x.2.as_deref()) + .and_then(json) + .cloned(); + for (spec, st, data) in sections.iter_mut() { + let live = matches!(st, State::Ok | State::Stale); + let Some(Data::Json(v)) = data.as_deref_mut() else { + continue; + }; + match spec.key { + "expansions" if live => repeats::post(c, v, traits_state, cpic_state), + "locityper" if live => { + tables::locityper_post(v, hla_state, hla_data.as_ref(), others); + } + "cnvpytor" => sv::cnvpytor_sex_filter(v, declared_sex), + _ => {} + } + } +} + +fn json(d: &Data) -> Option<&Value> { + match d { + Data::Json(v) => Some(v), + _ => None, + } +} + +#[cfg(test)] +mod tests; diff --git a/rust/crates/og-summary/src/collect/group_c/chrm.rs b/rust/crates/og-summary/src/collect/group_c/chrm.rs new file mode 100644 index 000000000..e232bccf6 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/chrm.rs @@ -0,0 +1,477 @@ +//! Step 20's chrM section: the PASS calls and the heteroplasmic ones, the +//! poly-C tracts counted apart, the annotation table with its ClinVar +//! classes, the confirmed variants of `assets/mito_confirmed.tsv`, the large +//! deletions and the mtDNA copy number. + +use super::util::{Obj, asset_rows, cells, lines, opt_s, pairs, round, s, strs, vcf_lines}; +use super::{Ctx, Found, Out}; +use crate::collect::sections::py_error; +use indexmap::IndexMap; +use pycompat::csv::{DictReader, DictRow, Quoting}; +use pycompat::json::Value; + +/// The D-loop poly-C tracts, rCRS m.303-315 and m.16184-16193. +const POLYC_TRACTS: [(i128, i128); 2] = [(303, 315), (16184, 16193)]; +const MITO_ANNOT_COLS: [&str; 13] = [ + "chrom", + "pos", + "ref", + "alt", + "filter", + "af", + "clinvar_id", + "clinvar_significance", + "clinvar_review", + "clinvar_condition", + "gnomad_af_hom", + "gnomad_af_het", + "gnomad_max_hl", +]; +const CLINVAR_PLP: [&str; 3] = [ + "Pathogenic", + "Likely_pathogenic", + "Pathogenic/Likely_pathogenic", +]; +const CLINVAR_BENIGN: [&str; 4] = [ + "Benign", + "Likely_benign", + "Benign/Likely_benign", + "not_provided", +]; +const MITO_DEL_TOLERANCE: i128 = 50; +const MITO_DEL_LIST_PCT: f64 = 1.0; +const MITO_CN_COLS: [&str; 8] = [ + "status", + "reason", + "chrm_reads", + "chrm_length", + "autosomal_reads", + "autosomal_length", + "depth_ratio", + "mtdna_copies", +]; +const HET_LOW: f64 = 0.05; +const HET_HIGH: f64 = 0.95; + +fn n_chars(x: &str) -> i128 { + x.chars().count() as i128 +} + +/// The rCRS positions a chrM allele changes. +fn mito_span(pos: i128, r: &str, a: &str) -> (i128, i128) { + if n_chars(r) == n_chars(a) { + (pos, pos + n_chars(r) - 1) + } else { + (pos + 1, pos + (n_chars(r) - 1).max(1)) + } +} + +pub fn in_polyc(pos: i128, r: &str, a: &str) -> bool { + let (lo, hi) = mito_span(pos, r, a); + POLYC_TRACTS.iter().any(|(a, b)| lo <= *b && hi >= *a) +} + +/// `(pos, ref, alt)` without the bases both share at the end, then at the +/// start, one base always kept. +pub fn min_allele(mut pos: i128, r: &str, a: &str) -> (i128, String, String) { + let mut r: Vec = r.chars().collect(); + let mut a: Vec = a.chars().collect(); + while r.len() > 1 && a.len() > 1 && r.last() == a.last() { + r.pop(); + a.pop(); + } + let mut i = 0; + while r.len() - i > 1 && a.len() - i > 1 && r[i] == a[i] { + i += 1; + pos += 1; + } + (pos, r[i..].iter().collect(), a[i..].iter().collect()) +} + +fn py_int(x: Option<&str>) -> Result { + match x { + None => Err(py_error( + "TypeError", + "int() argument must be a string, a bytes-like object or a real number, not 'NoneType'", + )), + Some(x) => crate::int(x).map_err(|e| py_error("ValueError", &e)), + } +} + +fn py_float(x: Option<&str>) -> Result { + match x { + None => Err(py_error( + "TypeError", + "float() argument must be a string or a real number, not 'NoneType'", + )), + Some(x) => crate::float(x).map_err(|e| py_error("ValueError", &e)), + } +} + +/// `row[k]`, the column required. +fn col<'a>(r: &'a DictRow, k: &str) -> Result, String> { + super::util::at(r, k) +} + +fn confirmed_fields(k: &DictRow) -> Result { + let mut o = Obj::new(); + for (key, c) in [ + ("hgvs", "hgvs"), + ("gene", "gene"), + ("disease", "disease"), + ("clinvar_id", "clinvar_id"), + ("pmids", "pmids"), + ("note", "heteroplasmy_note"), + ] { + o.put(key, opt_s(col(k, c)?)); + } + Ok(o) +} + +/// Python's `max` of floats: the first, replaced by each later one that is +/// greater. +fn py_max(xs: &[f64]) -> Option { + let mut it = xs.iter().copied(); + let mut cur = it.next()?; + for x in it { + if x > cur { + cur = x; + } + } + Some(cur) +} + +type Confirmed = (IndexMap<(i128, String, String), DictRow>, Vec); + +fn mito_confirmed(assets: &str) -> Result, String> { + let rows = match asset_rows(assets, "mito_confirmed.tsv") { + Ok(r) => r, + // A missing assets folder is a note, not an unreadable section. + Err(e) if e.starts_with("mito_confirmed.tsv is not in ") => return Ok(Err(e)), + Err(e) => return Err(e), + }; + let mut snv = IndexMap::new(); + let mut dels = Vec::new(); + for r in rows { + match col(&r, "kind")? { + Some("snv") => { + let key = ( + py_int(col(&r, "pos")?)?, + col(&r, "ref")?.unwrap_or("").to_owned(), + col(&r, "alt")?.unwrap_or("").to_owned(), + ); + snv.insert(key, r); + } + Some("deletion") => dels.push(r), + _ => {} + } + } + Ok(Ok((snv, dels))) +} + +fn mito_extended(c: &Ctx<'_>, vcf: &Found) -> Result { + let mut out = Obj::new(); + let (snv, dels) = match mito_confirmed(c.assets)? { + Ok((snv, dels)) => (Some(snv), dels), + Err(note) => { + out.put("confirmed_note", s(¬e)); + (None, Vec::new()) + } + }; + if let Some(snv) = &snv { + let mut records = Vec::new(); + vcf_lines(&vcf.path, |l| records.push(cells(l)))?; + let mut confirmed = Vec::new(); + for r in records { + if r.len() < 8 { + continue; + } + let (fmt, val): (Vec<&str>, Vec<&str>) = if r.len() > 9 { + (r[8].split(':').collect(), r[9].split(':').collect()) + } else { + (Vec::new(), Vec::new()) + }; + let afs: Vec<&str> = match fmt.iter().position(|f| *f == "AF") { + Some(i) if i < val.len() => val[i].split(',').collect(), + _ => Vec::new(), + }; + let pos = py_int(Some(&r[1]))?; + for (i, alt) in r[4].split(',').enumerate() { + if let Some(k) = snv.get(&min_allele(pos, &r[3], alt)) { + confirmed.push( + confirmed_fields(k)? + .set("pos", Value::Int(py_int(col(k, "pos")?)?)) + .set("filter", s(&r[6])) + .set("af", s(afs.get(i).copied().unwrap_or("."))) + .val(), + ); + } + } + } + out.put("confirmed", Value::Array(confirmed)); + } + let s_ = c.s(); + match c.find1(format!("mito/{s_}_chrM_deletions.tsv")) { + None => out.put( + "deletions_note", + s("no deletion table (mito/_chrM_deletions.tsv): step 20 ran before it looked for large deletions, eKLIPse wrote no table (see the step's log), or the Nextflow MITO_DELETIONS task failed (see the Nextflow log)"), + ), + Some(p) => { + let rows = super::util::table(&p.path, false)?.rows; + let mut listed = Vec::new(); + for r in &rows { + let bkp5 = py_int(col(r, "bkp5")?)?; + let bkp3 = py_int(col(r, "bkp3")?)?; + let length = match col(r, "length")? { + Some(".") => Value::Null, + x => Value::Int(py_int(x)?), + }; + let level = py_float(col(r, "level_pct")?)?; + let mut known = None; + for k in &dels { + let kp = py_int(col(k, "pos")?)?; + if (bkp5 + 1 - kp).abs() <= MITO_DEL_TOLERANCE { + let ke = py_int(col(k, "end")?)?; + if (bkp3 - 1 - ke).abs() <= MITO_DEL_TOLERANCE { + known = Some(k); + break; + } + } + } + if known.is_some() || level >= MITO_DEL_LIST_PCT { + let mut row = Obj::new() + .set("bkp5", Value::Int(bkp5)) + .set("bkp3", Value::Int(bkp3)) + .set("length", length) + .set("level_pct", Value::Float(level)); + if let Some(k) = known { + row.put("known", confirmed_fields(k)?.val()); + } + listed.push(row.val()); + } + } + out.put( + "deletions", + Obj::new() + .set("source", s(&p.rel)) + .set("total", super::util::count(rows.len())) + .set("listed", Value::Array(listed)) + .set("list_floor_pct", Value::Float(MITO_DEL_LIST_PCT)) + .val(), + ); + } + } + match c.find1(format!("mito/{s_}_mtdna_copies.tsv")) { + Some(p) => { + let m: IndexMap = pairs(&p.path)?.into_iter().collect(); + out.put( + "copies", + MITO_CN_COLS + .iter() + .fold(Obj::new(), |o, k| { + o.set(k, opt_s(m.get(*k).map(String::as_str).filter(|v| *v != "."))) + }) + .val(), + ); + } + None => out.put( + "copies_note", + s("no copy-number table (mito/_mtdna_copies.tsv): step 20 ran before it counted mtDNA copies, or the Nextflow MITO_DELETIONS task failed (see the Nextflow log)"), + ), + } + Ok(out) +} + +fn mtime(p: &str) -> Result { + pycompat::mtime::mtime(p).map_err(|e| super::util::os_note(&crate::os_error(&e, p))) +} + +/// The annotation table's sources and its PASS rows, or a note. +fn mito_annotation(c: &Ctx<'_>, vcf: &Found) -> Result<(Obj, Vec), String> { + let Some(p) = c.find1(format!("mito/{}_chrM_annotated.tsv", c.s())) else { + return Ok(( + Obj::new().set("note", s("no annotation table (mito/_chrM_annotated.tsv): step 20 ran before it annotated, or the Nextflow MITO_VARIANTS_ANNOTATE task failed (see the Nextflow log)")), + Vec::new(), + )); + }; + if mtime(&p.path)? < mtime(&vcf.path)? { + return Ok(( + Obj::new().set( + "note", + s(&format!( + "{} is older than the chrM calls: rerun step 20", + p.rel + )), + ), + Vec::new(), + )); + } + let mut sources: IndexMap = IndexMap::new(); + let mut body = Vec::new(); + for line in lines(&p.path)? { + if let Some(rest) = line.strip_prefix("# ") { + let c: Vec<&str> = rest.split('\t').collect(); + if c.len() >= 3 { + let v = if c[1] == "not annotated" { + format!("not annotated: {}", c[2]) + } else if c[2] != "-" { + format!("{} ({})", c[1], c[2]) + } else { + c[1].to_owned() + }; + sources.insert(c[0].to_owned(), v); + } + } else if !pycompat::num::py_strip(&line).is_empty() { + body.push(Ok::<_, std::io::Error>(line + "\n")); + } + } + let rows = DictReader::new(body.into_iter(), '\t', Quoting::Minimal) + .collect::, _>>() + .map_err(|e| py_error(e.exception(), &e.to_string()))? + .into_iter() + .filter(|r| r.get("filter").flatten() == Some("PASS")) + .collect(); + let src = |k: &str| opt_s(sources.get(k).map(String::as_str)); + Ok(( + Obj::new() + .set("source", s(&p.rel)) + .set("clinvar", src("clinvar")) + .set("gnomad", src("gnomad")), + rows, + )) +} + +fn clinvar_terms(sig: Option<&str>) -> Vec<&str> { + sig.unwrap_or("") + .split(['|', ',']) + .filter(|t| !t.is_empty() && *t != ".") + .collect() +} + +pub fn mito(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("mito/{}_chrM_filtered.vcf.gz", c.s())) else { + return Ok(None); + }; + let mut records = Vec::new(); + vcf_lines(&p.path, |l| records.push(cells(l)))?; + let (mut passed, mut het) = (0usize, 0usize); + let (mut polyc, mut sites) = (Vec::new(), Vec::new()); + for r in &records { + if r.len() < 10 || r[6] != "PASS" { + continue; + } + passed += 1; + let fmt: Vec<&str> = r[8].split(':').collect(); + let val: Vec<&str> = r[9].split(':').collect(); + let Some(i) = fmt + .iter() + .position(|f| *f == "AF") + .filter(|i| *i < val.len()) + else { + continue; + }; + let afs: Result, _> = val[i] + .split(',') + .filter(|x| !matches!(*x, "" | ".")) + .map(pycompat::num::parse_float) + .collect(); + let Some(af) = afs.ok().and_then(|a| py_max(&a)) else { + continue; + }; + if !(HET_LOW..HET_HIGH).contains(&af) { + continue; + } + let pos = py_int(Some(&r[1]))?; + if r[4].split(',').any(|a| in_polyc(pos, &r[3], a)) { + polyc.push( + Obj::new() + .set( + "variant", + s(&format!("{}:{} {}>{}", r[0], r[1], r[3], r[4])), + ) + .set("af", round(af, 3)) + .val(), + ); + } else { + het += 1; + sites.push( + Obj::new() + .set("pos", Value::Int(pos)) + .set("ref", s(&r[3])) + .set("alt", s(&r[4])) + .set("af", round(af, 3)) + .val(), + ); + } + } + let (mut ann, rows) = mito_annotation(c, &p)?; + let mut listed = Vec::new(); + let mut plp = 0i128; + for r in &rows { + let terms = clinvar_terms(r.get("clinvar_significance").flatten()); + let is_plp = terms.iter().any(|t| CLINVAR_PLP.contains(t)); + plp += i128::from(is_plp); + let het_row = r + .get("af") + .flatten() + .and_then(|x| pycompat::num::parse_float(x).ok()) + .is_some_and(|af| (HET_LOW..HET_HIGH).contains(&af)); + if is_plp + || het_row + || (!terms.is_empty() && !terms.iter().all(|t| CLINVAR_BENIGN.contains(t))) + { + let mut o = Obj::new(); + for k in MITO_ANNOT_COLS { + if k != "chrom" && k != "filter" { + o.put(k, opt_s(r.get(k).flatten())); + } + } + let pos = py_int(col(r, "pos")?)?; + let rf = col(r, "ref")?.unwrap_or(""); + let al = col(r, "alt")?.unwrap_or(""); + o.put("polyc", Value::Bool(in_polyc(pos, rf, al))); + listed.push(o.val()); + } + } + if ann.0.contains_key("source") { + ann.put("variants", Value::Array(listed)); + ann.put("clinvar_plp", Value::Int(plp)); + } + let mut out = mito_extended(c, &p)?; + out.put("pass", super::util::count(passed)); + out.put("heteroplasmic", super::util::count(het)); + out.put("heteroplasmic_sites", Value::Array(sites)); + out.put("heteroplasmy_floor", Value::Float(HET_LOW)); + out.put("polyc_heteroplasmic", Value::Array(polyc)); + out.put( + "polyc_tracts", + strs(POLYC_TRACTS.iter().map(|(a, b)| format!("m.{a}-{b}"))), + ); + out.put("annotation", ann.val()); + Ok(Some((p, out.val()))) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn min_allele_trims_end_then_start() { + assert_eq!(min_allele(302, "AC", "ACC"), (302, "A".into(), "AC".into())); + assert_eq!(min_allele(10, "ACT", "AGT"), (11, "C".into(), "G".into())); + assert_eq!(min_allele(5, "A", "G"), (5, "A".into(), "G".into())); + } + + #[test] + fn polyc_span() { + assert!(in_polyc(302, "A", "AC")); + assert!(!in_polyc(300, "A", "G")); + assert!(in_polyc(16189, "T", "C")); + } + + #[test] + fn python_max_keeps_a_leading_nan() { + assert!(py_max(&[f64::NAN, 0.5]).is_some_and(f64::is_nan)); + assert_eq!(py_max(&[0.5, f64::NAN]), Some(0.5)); + } +} diff --git a/rust/crates/og-summary/src/collect/group_c/cn.rs b/rust/crates/og-summary/src/collect/group_c/cn.rs new file mode 100644 index 000000000..04e455967 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/cn.rs @@ -0,0 +1,778 @@ +//! Step 35 (Parascopy and the depth counts, short reads) and step 35b +//! (Paraphase, PacBio HiFi): SMN1 and SMN2 at exon 7 with the SMA reading, +//! the HBA, CYP21A2, interval and gene-dosage tables, every other catalog +//! locus, and the Paraphase genes with SMN1 from both tools. + +use super::util::{ + Obj, Table, asset_rows, at, base, columns, count, isdigit, lines, opt_s, or, or_dot, pairs, s, + strs, table, +}; +use super::{Ctx, Found, Out}; +use og_cn::markers::{MARKERS, Status}; +use og_cn::smn::SMN1_EXON7; +use pycompat::csv::DictRow; +use pycompat::json::Value; + +/// SMN2 exon 7, c.840 (the base that tells SMN2 from SMN1), GRCh38 1-based. +pub const SMN2_EXON7: (&str, i64) = ("chr5", 70_076_526); + +/// Tables in paralogs/ that other parts of steps 35 and 35b write, not +/// catalog rows. +const NOT_CATALOG_TABLES: [&str; 2] = ["hba", "paraphase"]; +/// The columns of a Parascopy copy-number table. +const PARASCOPY_COLUMNS: [&str; 9] = [ + "chrom", + "start", + "end", + "agCN", + "agCN_filter", + "agCN_qual", + "psCN", + "psCN_qual", + "psCN_filter", +]; +const NO_SMN_TABLE: &str = "no SMN1 table (step 35 wrote no SMN copy-number table)"; +const MARKER_FIELDS: [&str; 8] = [ + "marker", + "rsid", + "status", + "alt_reads", + "depth", + "fraction", + "copies", + "message", +]; +const DOSAGE_COLUMNS: [&str; 6] = ["locus", "status", "copies", "estimate", "reason", "reading"]; +const INTERVAL_COLUMNS: [&str; 9] = [ + "locus", "name", "chrom", "start", "end", "status", "copies", "expected", "change", +]; + +/// A copy number read at one position from a Parascopy table. +#[derive(Debug, Clone, PartialEq)] +pub struct CnAt { + pub cn: Option, + pub text: String, + pub agcn: String, + pub agcn_qual: String, + pub agcn_filter: String, +} + +impl CnAt { + fn none(text: &str) -> CnAt { + CnAt { + cn: None, + text: text.to_owned(), + agcn: ".".into(), + agcn_qual: ".".into(), + agcn_filter: ".".into(), + } + } + + fn json(&self) -> Value { + Obj::new() + .set("cn", cn(self.cn)) + .set("text", s(&self.text)) + .set("agcn", s(&self.agcn)) + .set("agcn_qual", s(&self.agcn_qual)) + .set("agcn_filter", s(&self.agcn_filter)) + .val() + } +} + +fn cn(x: Option) -> Value { + x.map_or(Value::Null, Value::Int) +} + +/// The region of a row and its homologous regions, 1-based closed; `None` +/// for a row without a whole start and end. +fn regions(get: &dyn Fn(&str) -> Option) -> Option> { + let chrom = get("chrom")?; + let start = crate::int(&get("start")?).ok()?; + let end = crate::int(&get("end")?).ok()?; + let mut out = vec![(chrom, start + 1, end)]; + let homologous = get("homologous_regions").filter(|h| !h.is_empty()); + for h in homologous.as_deref().unwrap_or("*").split(',') { + if let Some(r) = region(pycompat::num::py_strip(h)) { + out.push(r); + } + } + Some(out) +} + +/// `^([^:]+):(\d+)-(\d+)` at the start of a text. +fn region(h: &str) -> Option<(String, i128, i128)> { + let (c, rest) = h.split_once(':')?; + if c.is_empty() { + return None; + } + let digits = |x: &str| -> Option<(i128, usize)> { + let n = x.bytes().take_while(u8::is_ascii_digit).count(); + (n > 0) + .then(|| (x[..n].parse().ok(), n)) + .and_then(|(v, n)| Some((v?, n))) + }; + let (b, n) = digits(rest)?; + let rest = rest[n..].strip_prefix('-')?; + let (e, _) = digits(rest)?; + Some((c.to_owned(), b, e)) +} + +/// The paralog-specific copy number (psCN) of the row, or of its +/// homologous region, that holds a position, with the row's aggregate +/// value (agCN). `cn` is `None` unless the filter is PASS, the quality 20 +/// or more and the value a whole number (Parascopy's threshold). +fn cn_at_rows<'r>( + rows: impl Iterator Option>, + chrom: &str, + pos: i64, + nothing: &str, +) -> CnAt { + let pos = i128::from(pos); + for get in rows { + let Some(regs) = regions(get) else { continue }; + let Some(idx) = regs + .iter() + .position(|(c, b, e)| c == chrom && *b <= pos && pos <= *e) + else { + continue; + }; + let val = |k: &str| get(k).filter(|v| !v.is_empty()); + let cns = val("psCN").unwrap_or_default(); + let quals = val("psCN_qual").unwrap_or_default(); + let cns: Vec<&str> = cns.split(',').collect(); + let quals: Vec<&str> = quals.split(',').collect(); + let cn_txt = cns.get(idx).copied().unwrap_or("?"); + let q_txt = quals.get(idx).copied().unwrap_or("*"); + let filt = val("psCN_filter").unwrap_or_else(|| ".".into()); + let q = pycompat::num::parse_float(q_txt).ok(); + let ok = filt == "PASS" && q.is_some_and(|q| q >= 20.0) && isdigit(cn_txt); + return CnAt { + cn: if ok { crate::int(cn_txt).ok() } else { None }, + text: format!( + "{cn_txt} (quality {q_txt}, filter {filt}){}", + if ok { "" } else { ", not resolved" } + ), + agcn: val("agCN").unwrap_or_else(|| ".".into()), + agcn_qual: val("agCN_qual").unwrap_or_else(|| ".".into()), + agcn_filter: val("agCN_filter").unwrap_or_else(|| ".".into()), + }; + } + CnAt::none(nothing) +} + +/// `parascopy_cn_at`: over the rows of step 35's table. +/// A lookup of a row's cell by column. +type Getter<'a> = Box Option + 'a>; + +pub fn parascopy_cn_at(rows: &[DictRow], chrom: &str, pos: i64) -> CnAt { + let getters: Vec> = rows + .iter() + .map(|r| { + Box::new(move |k: &str| r.get(k).flatten().map(str::to_owned)) + as Box Option> + }) + .collect(); + cn_at_rows( + getters + .iter() + .map(|g| g.as_ref() as &dyn Fn(&str) -> Option), + chrom, + pos, + "no row covers exon 7", + ) +} + +/// `paraphase_report.parascopy_smn1`: SMN1 at exon 7 from step 35's SMN +/// table read line by line, each line zipped with the header. +pub fn parascopy_smn1(path: &str) -> Result<(Option, String), String> { + let ls = lines(path)?; + let mut it = ls.iter(); + let header: Vec<&str> = it.next().map_or_else(Vec::new, |h| h.split('\t').collect()); + let rows: Vec> = it + .map(|l| header.iter().copied().zip(l.split('\t')).collect()) + .collect(); + let getters: Vec> = rows + .iter() + .map(|r| { + Box::new(move |k: &str| r.get(k).map(|v| (*v).to_owned())) + as Box Option> + }) + .collect(); + let r = cn_at_rows( + getters + .iter() + .map(|g| g.as_ref() as &dyn Fn(&str) -> Option), + SMN1_EXON7.0, + SMN1_EXON7.1, + "no row covers SMN1 exon 7", + ); + Ok((r.cn, r.text)) +} + +/// What an SMN1 copy number means for SMA, in one line. +pub fn smn_reading(smn1: Option) -> String { + match smn1 { + None => { + "SMN1 not resolved (quality under 20 or filter not PASS): no carrier reading".into() + } + Some(0) => "no SMN1 copy: the pattern of spinal muscular atrophy (SMA) itself".into(), + Some(1) => "one SMN1 copy: the pattern of an SMA carrier".into(), + Some(2) => "two SMN1 copies: not a carrier by copy number; a 2+0 carrier (both copies on one chromosome) cannot be ruled out".into(), + Some(n) => format!("{n} SMN1 copies: not a carrier by copy number"), + } +} + +/// A copy-aware marker table (metric, value) beside a count; a table that +/// is not there or has no known status reads as not assessed, with the +/// reason in its message. +pub fn read_marker_table(path: &str, step: &str, key: &str) -> Value { + let m = MARKERS + .iter() + .find(|m| m.key == key) + .expect("a marker of og-cn"); + let statuses = [ + Status::Present, + Status::NotSeen, + Status::Uncertain, + Status::TooLittleDepth, + Status::NotAssessed, + ] + .map(Status::as_str); + let mut out: indexmap::IndexMap<&str, String> = + MARKER_FIELDS.iter().map(|k| (*k, ".".to_owned())).collect(); + out.insert("marker", m.name.to_owned()); + out.insert("rsid", m.dbsnp.0.to_owned()); + out.insert("status", "not assessed".to_owned()); + let name = base(path); + let done = |out: indexmap::IndexMap<&str, String>| { + Value::Object( + out.into_iter() + .map(|(k, v)| (k.to_owned(), s(&v))) + .collect(), + ) + }; + if !crate::is_file(path) { + out.insert( + "message", + format!( + "{}: not assessed, step {step} wrote no {name} (a run from before this reader, or the check did not run).", + m.name + ), + ); + return done(out); + } + let kv: std::collections::HashMap = + pairs(path).unwrap_or_default().into_iter().collect(); + if !kv + .get("status") + .is_some_and(|v| statuses.contains(&v.as_str())) + { + out.insert( + "message", + format!( + "{}: not assessed, {name} has no status row ({}).", + m.name, + statuses.join(", ") + ), + ); + return done(out); + } + for k in MARKER_FIELDS { + if let Some(v) = kv.get(k).filter(|v| !v.is_empty()) { + out.insert(k, v.clone()); + } + } + done(out) +} + +/// `{table: path}` of step 35's catalog tables (`__copy_number.tsv`). +fn parascopy_tables(c: &Ctx<'_>) -> Vec<(String, Found)> { + let pat = format!("paralogs/{}_*_copy_number.tsv", c.s()); + let skip = c.sample.chars().count() + 1; + c.index + .all(&pat) + .into_iter() + .filter_map(|rel| { + let b = base(rel); + let n = b.chars().count(); + let name: String = b + .chars() + .skip(skip) + .take(n.saturating_sub(skip + "_copy_number.tsv".len())) + .collect(); + (!NOT_CATALOG_TABLES.contains(&name.as_str())).then(|| { + ( + name, + Found { + path: c.path(rel), + rel: rel.to_owned(), + }, + ) + }) + }) + .collect() +} + +/// A table whose header must hold `need`. +fn has_all(cols: &[String], need: &[&str]) -> bool { + need.iter().all(|k| cols.iter().any(|c| c == k)) +} + +/// The first `agCN_filter` that starts with "not assessed". +fn not_assessed(rows: &[DictRow]) -> Result, String> { + for r in rows { + if r.get("agCN_filter") + .flatten() + .is_some_and(|f| f.starts_with("not assessed")) + { + return Ok(at(r, "agCN_filter")?.map(str::to_owned)); + } + } + Ok(None) +} + +/// Step 35 (Parascopy, short reads): SMN1 and SMN2 at exon 7, their +/// aggregate copy number and the SMA reading; the HBA, interval, CYP21A2 +/// and dosage tables; and every other catalog table, one locus each. +pub fn parascopy(c: &Ctx<'_>) -> Out { + let sm = c.sample; + let tables = parascopy_tables(c); + let direct = |name: &str| { + let rel = format!("paralogs/{sm}_{name}"); + Found { + path: c.path(&rel), + rel, + } + }; + let hp = direct("hba_copy_number.tsv"); + let ip = direct("cn_intervals.tsv"); + let cp = direct("cyp21a2_psv.tsv"); + let dp = direct("cn_dosage.tsv"); + let smn = tables.iter().find(|(n, _)| n == "smn").map(|x| x.1.clone()); + let Some(p) = smn + .clone() + .or_else(|| tables.first().map(|x| x.1.clone())) + .or_else(|| { + [&hp, &ip, &cp, &dp] + .into_iter() + .find(|f| crate::is_file(&f.path)) + .cloned() + }) + else { + return Ok(None); + }; + let (smn1, smn2, reading, nrows) = match &smn { + Some(f) => { + let t = table(&f.path, false)?; + if !has_all(&columns(&f.path)?, &PARASCOPY_COLUMNS) { + return Err(format!( + "{} lacks the chrom, start, end, agCN, agCN_filter, agCN_qual, psCN, psCN_qual or psCN_filter column", + base(&f.path) + )); + } + let smn1 = parascopy_cn_at(&t.rows, SMN1_EXON7.0, SMN1_EXON7.1); + let smn2 = parascopy_cn_at(&t.rows, SMN2_EXON7.0, SMN2_EXON7.1); + let reading = match not_assessed(&t.rows)? { + Some(na) => format!("SMN1 and SMN2 {na}: no carrier reading"), + None if smn1.text == "no row covers exon 7" => { + "no row of step 35's table covers SMN1 exon 7: no carrier reading".into() + } + None => smn_reading(smn1.cn), + }; + (smn1, smn2, reading, t.rows.len()) + } + None => ( + CnAt::none(NO_SMN_TABLE), + CnAt::none(NO_SMN_TABLE), + format!("{NO_SMN_TABLE}: no carrier reading"), + 0, + ), + }; + let others: Vec<&(String, Found)> = tables.iter().filter(|(n, _)| n != "smn").collect(); + let mut catalog: indexmap::IndexMap = indexmap::IndexMap::new(); + if !others.is_empty() { + for r in asset_rows(c.assets, "paralog_loci.tsv")? { + let k = at(&r, "table")?.unwrap_or("None").to_owned(); + catalog.insert(k, r); + } + } + let mut order: Vec<&(String, Found)> = others; + order.sort_by(|a, b| { + let key = |n: &str| catalog.get_index_of(n).unwrap_or(catalog.len()); + key(&a.0).cmp(&key(&b.0)).then_with(|| a.0.cmp(&b.0)) + }); + let mut loci = Vec::new(); + for (n, f) in order { + loci.push(parascopy_locus(n, &f.path, catalog.get(n))?); + } + let opt = |f: &Found, read: fn(&str) -> Result| -> Result { + if crate::is_file(&f.path) { + read(&f.path) + } else { + Ok(Value::Null) + } + }; + let data = Obj::new() + .set("smn1", smn1.json()) + .set( + "smn2", + Obj::new() + .set("cn", cn(smn2.cn)) + .set("text", s(&smn2.text)) + .val(), + ) + .set("agcn", s(&smn1.agcn)) + .set("agcn_qual", s(&smn1.agcn_qual)) + .set("agcn_filter", s(&smn1.agcn_filter)) + .set( + "smn1_marker", + read_marker_table( + &c.path(&format!("paralogs/{sm}_smn1_marker.tsv")), + "35", + "SMN1_27134", + ), + ) + .set("rows", count(nrows)) + .set("hba", opt(&hp, read_hba)?) + .set("cn_intervals", opt(&ip, read_cn_intervals)?) + .set("cyp21a2", opt(&cp, read_hba)?) + .set("dosage", opt(&dp, read_cn_dosage)?) + .set("reading", s(&reading)) + .set("loci", Value::Array(loci)); + Ok(Some((p, data.val()))) +} + +/// One catalog table that is not SMN1's: the locus's aggregate copy number +/// per region of its profile, the catalog's genes, tag and reading. +fn parascopy_locus(name: &str, path: &str, row: Option<&DictRow>) -> Result { + let cat = + |k: &str, default: &str| row.map_or_else(|| default.to_owned(), |r| or(r, k, default)); + let out = Obj::new() + .set("table", s(name)) + .set("genes", s(&cat("genes", name))) + .set("tag", s(&cat("tag", "."))) + .set("reading", s(&cat("reading", "."))); + let t = table(path, false)?; + let with = |o: Obj, status: &str, cnv: Value, text: &str| { + o.set("status", s(status)) + .set("cn", cnv) + .set("text", s(text)) + .val() + }; + if !has_all(&t.head, &PARASCOPY_COLUMNS) { + let text = format!("unreadable: {} lacks the copy-number columns", base(path)); + return Ok(with(out, "unreadable", Value::Null, &text)); + } + if let Some(na) = not_assessed(&t.rows)? { + return Ok(with(out, "not assessed", Value::Null, &na)); + } + struct Reg { + chrom: Option, + start: Option, + end: Option, + cn: String, + qual: String, + filter: String, + ok: bool, + } + let regs: Vec = t + .rows + .iter() + .map(|r| { + let get = |k: &str| r.get(k).flatten().map(str::to_owned); + let q = pycompat::num::parse_float(&or(r, "agCN_qual", "")).ok(); + let cnv = or_dot(r, "agCN"); + let ok = isdigit(&cnv) + && r.get("agCN_filter").flatten() == Some("PASS") + && q.is_some_and(|q| q >= 20.0); + Reg { + chrom: get("chrom"), + start: get("start"), + end: get("end"), + cn: cnv, + qual: or_dot(r, "agCN_qual"), + filter: or_dot(r, "agCN_filter"), + ok, + } + }) + .collect(); + if regs.is_empty() { + return Ok(with( + out, + "not assessed", + Value::Null, + "not assessed: the table has no region row", + )); + } + let values: std::collections::HashSet<&str> = regs + .iter() + .filter(|r| r.ok) + .map(|r| r.cn.as_str()) + .collect(); + if values.len() == 1 && regs.iter().all(|r| r.ok) { + let n = crate::int(values.into_iter().next().unwrap_or_default()) + .map_err(|e| super::super::sections::py_error("ValueError", &e))?; + let mut min = f64::INFINITY; + for r in ®s { + min = min.min( + crate::float(&r.qual) + .map_err(|e| super::super::sections::py_error("ValueError", &e))?, + ); + } + let text = format!( + "{n} ({} region(s), quality {} or more, filter PASS)", + regs.len(), + pycompat::fmt::general(min, 6) + ); + return Ok(with(out, "called", Value::Int(n), &text)); + } + let none = |x: &Option| x.clone().unwrap_or_else(|| "None".into()); + let text = regs + .iter() + .map(|r| { + let b = match &r.start { + Some(b) if isdigit(b) => { + crate::int(b).map_or_else(|_| b.clone(), |v| (v + 1).to_string()) + } + b => none(b), + }; + format!( + "{}:{b}-{} {} (quality {}, filter {})", + none(&r.chrom), + none(&r.end), + r.cn, + r.qual, + r.filter + ) + }) + .collect::>() + .join("; "); + let called = regs.iter().any(|r| r.ok); + let text = format!( + "{text}{}", + if called { + "" + } else { + ": no region with a whole copy number, filter PASS and quality 20 or more" + } + ); + let out = out.set("reading", s(".")); + Ok(with( + out, + if called { "called" } else { "not resolved" }, + Value::Null, + &text, + )) +} + +/// Step 35's interval table: the rows with a deletion or a duplication, and +/// a count of the rows by status. +fn read_cn_intervals(path: &str) -> Result { + let t = table(path, false)?; + if !has_all(&t.head, &INTERVAL_COLUMNS) { + return Err(format!( + "{} lacks the locus, name, status, copies, expected or change column", + base(path) + )); + } + short_row(path, &t, &INTERVAL_COLUMNS)?; + let mut by: indexmap::IndexMap = indexmap::IndexMap::new(); + let mut changes = Vec::new(); + for r in &t.rows { + let cell = |k: &str| r.get(k).flatten().unwrap_or_default().to_owned(); + *by.entry(cell("status")).or_default() += 1; + if matches!(cell("change").as_str(), "deletion" | "duplication") { + let keep = [ + "locus", "name", "chrom", "start", "end", "copies", "expected", "change", + ]; + changes.push( + keep.iter() + .fold(Obj::new(), |o, k| o.set(k, s(&cell(k)))) + .val(), + ); + } + } + Ok(Obj::new() + .set("rows", count(t.rows.len())) + .set( + "by_status", + Value::Object(by.into_iter().map(|(k, v)| (k, Value::Int(v))).collect()), + ) + .set("changes", Value::Array(changes)) + .val()) +} + +/// Step 35's gene-dosage table: one entry per locus, in the table's order. +fn read_cn_dosage(path: &str) -> Result { + if !has_all(&columns(path)?, &DOSAGE_COLUMNS) { + return Err(format!( + "{} lacks the {} or reading column", + base(path), + DOSAGE_COLUMNS[..5].join(", ") + )); + } + let t = table(path, false)?; + short_row(path, &t, &DOSAGE_COLUMNS)?; + Ok(Value::Array( + t.rows + .iter() + .map(|r| super::util::dots(r, &DOSAGE_COLUMNS)) + .collect(), + )) +} + +/// A row cut short of a column makes the table unreadable. +fn short_row(path: &str, t: &Table, cols: &[&str]) -> Result<(), String> { + match t + .rows + .iter() + .position(|r| cols.iter().any(|k| !matches!(r.get(k), Some(Some(_))))) + { + Some(i) => Err(format!("{} line {} lacks a field", base(path), i + 2)), + None => Ok(()), + } +} + +/// Step 35's HBA table (metric, value), and its CYP21A2 table the same way; +/// a table without a status row reads as not assessed. +fn read_hba(path: &str) -> Result { + let mut out: indexmap::IndexMap = indexmap::IndexMap::new(); + for l in lines(path)? { + if let Some((k, v)) = l.split_once('\t') + && k != "metric" + { + out.insert(k.to_owned(), s(v)); + } + } + let ok = + matches!(out.get("status"), Some(Value::Str(v)) if v == "called" || v == "not assessed"); + if !ok { + return Ok(Obj::new() + .set("status", s("not assessed")) + .set( + "message", + s(&format!("not assessed: {} has no status row", base(path))), + ) + .val()); + } + Ok(Value::Object(out)) +} + +/// The Paraphase regions shown gene by gene, in the order of +/// assets/paraphase_genes.tsv. +fn paralog_regions(assets: &str) -> Result, String> { + let mut out: Vec = Vec::new(); + for r in asset_rows(assets, "paraphase_genes.tsv")? { + let g = at(&r, "region")?.unwrap_or("None").to_owned(); + if !out.contains(&g) { + out.push(g); + } + } + Ok(out) +} + +/// Step 35b (Paraphase, PacBio HiFi): the genes of the catalog regions, a +/// count of the other regions, the variant rows per region, and SMN1 +/// beside step 35's Parascopy call. +pub fn paralogs(c: &Ctx<'_>) -> Out { + let s_ = c.sample; + let Some(p) = c.find1(format!("paralogs/{}_paraphase_copy_number.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + if !t.rows.is_empty() && !has_all(&t.head, &["region", "gene", "copy_number", "status"]) { + return Err(format!( + "{} lacks the region, gene, copy_number or status column", + base(&p.path) + )); + } + let regions = paralog_regions(c.assets)?; + let region_of = |r: &DictRow| r.get("region").flatten().map(str::to_owned); + let mut genes: Vec<(usize, Value, Option, Option)> = Vec::new(); + let mut others = 0; + let mut others_called = 0; + for r in &t.rows { + let reg = region_of(r); + match reg + .as_ref() + .and_then(|g| regions.iter().position(|x| x == g)) + { + Some(i) => { + let cell = |k: &str| opt_s(r.get(k).flatten()); + genes.push(( + i, + Obj::new() + .set("region", cell("region")) + .set("gene", cell("gene")) + .set("copy_number", cell("copy_number")) + .set("status", cell("status")) + .set("detail", s(&or_dot(r, "detail"))) + .val(), + r.get("gene").flatten().map(str::to_owned), + r.get("copy_number").flatten().map(str::to_owned), + )); + } + None => { + others += 1; + if r.get("status").flatten() == Some("called") { + others_called += 1; + } + } + } + } + genes.sort_by_key(|g| g.0); + let mut variants: indexmap::IndexMap = indexmap::IndexMap::new(); + if let Some(vp) = c.beside(&p, &format!("{s_}_paraphase_variants.tsv")) { + for r in table(&vp.path, false)?.rows { + let k = match r.get("region") { + None => String::new(), + Some(None) => "null".into(), + Some(Some(v)) => v.to_owned(), + }; + *variants.entry(k).or_default() += 1; + } + } + let smn1 = genes + .iter() + .find(|g| g.2.as_deref() == Some("SMN1")) + .and_then(|g| g.3.clone()) + .filter(|x| isdigit(x)) + .and_then(|x| crate::int(&x).ok()); + let parascopy = match c.beside(&p, &format!("{s_}_smn_copy_number.tsv")) { + Some(pp) => Some(parascopy_smn1(&pp.path)?), + None => None, + }; + let agree = match (smn1, ¶scopy) { + (Some(a), Some((Some(b), _))) => s(if a == *b { "agree" } else { "disagree" }), + _ => Value::Null, + }; + let parascopy_json = parascopy.map_or(Value::Null, |(n, text)| { + Obj::new().set("cn", cn(n)).set("text", s(&text)).val() + }); + let data = Obj::new() + .set( + "genes", + Value::Array(genes.into_iter().map(|g| g.1).collect()), + ) + .set("other_regions", count(others)) + .set("other_regions_called", count(others_called)) + .set( + "variants", + Value::Object( + variants + .into_iter() + .map(|(k, v)| (k, Value::Int(v))) + .collect(), + ), + ) + .set( + "smn1", + Obj::new() + .set("paraphase", cn(smn1)) + .set("parascopy", parascopy_json) + .set("agree", agree) + .val(), + ); + let _ = strs::<[&str; 0], &str>; + Ok(Some((p, data.val()))) +} diff --git a/rust/crates/og-summary/src/collect/group_c/hla.rs b/rust/crates/og-summary/src/collect/group_c/hla.rs new file mode 100644 index 000000000..f5429d12f --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/hla.rs @@ -0,0 +1,400 @@ +//! Step 08 (T1K): the HLA alleles per gene with their qualities, the +//! candidate cap, the HFE label, the coeliac DQ haplotypes and the table +//! alleles marked present or absent; OptiType's concordance and the KIR +//! pass when they ran. The rules are docs/08-hla-typing.md's "What the +//! report derives". + +use super::util::{Obj, at, bin_rows, count, key_error, lines, s, strs, table}; +use super::{Ctx, Out}; +use og_steps::pgx_rules::HLA_GENES; +use pycompat::json::Value; + +/// A quality-0 allele with more candidates than this is "no confident call". +pub const MAX_CANDIDATES: usize = 3; +const HFE_LABEL: &str = "T1K sequence allele, not the HFE C282Y/H63D genotype: that genotype comes from the variant calls (ClinVar section)"; + +/// One gene's row of T1K's genotype table. +#[derive(Debug, Clone)] +pub struct Locus { + pub gene: String, + pub alleles: Vec, + pub quality: Vec, + pub low_confidence: bool, + pub candidates: Vec, + pub one_allele_type: bool, +} + +impl Locus { + fn json(&self) -> Value { + let mut o = Obj::new() + .set("gene", s(&self.gene)) + .set("alleles", strs(&self.alleles)) + .set("quality", strs(&self.quality)) + .set("low_confidence", Value::Bool(self.low_confidence)) + .set( + "withheld_from_pharmcat", + Value::Bool(self.low_confidence && HLA_GENES.contains(&self.gene.as_str())), + ) + .set( + "candidates", + Value::Array(self.candidates.iter().map(|n| count(*n)).collect()), + ) + .set("one_allele_type", Value::Bool(self.one_allele_type)); + if name(&self.gene) == "HFE" { + o.put("label", s(HFE_LABEL)); + } + o.val() + } +} + +/// T1K's quality as a number; one that is not a number counts as -1, as in +/// step 36. +pub fn quality(q: &str) -> f64 { + pycompat::num::parse_float(q).unwrap_or(-1.0) +} + +/// `HLA-DQB1*02:01:01` or `DQB1*02:01:01` is `DQB1*02:01:01`. +fn name(x: &str) -> &str { + let x = pycompat::num::py_strip(x); + x.strip_prefix("HLA-").unwrap_or(x) +} + +/// T1K's allele field: one allele, or equally likely candidates separated +/// by commas. +fn candidates(allele: &str) -> Vec<&str> { + allele + .split(',') + .map(name) + .filter(|a| !a.is_empty()) + .collect() +} + +/// The alleles and their qualities of columns 3 and 6 (`.` is no allele). +fn alleles(c: &[&str], skip_empty: bool) -> (Vec, Vec) { + let (mut a, mut q) = (Vec::new(), Vec::new()); + for i in [2, 5] { + let x = c.get(i).copied().unwrap_or(""); + if (skip_empty && x.is_empty()) || x == "." || x.is_empty() { + continue; + } + a.push(x.to_owned()); + q.push(c.get(i + 2).copied().unwrap_or("").to_owned()); + } + (a, q) +} + +/// One locus with the candidate cap, the candidate counts and whether T1K +/// typed one allele. +fn locus(gene: &str, alleles: Vec, quals: Vec, n_diff: &str) -> Locus { + // Step 36's rule: an allele quality of 0 or below withholds the gene. + let low = quals.iter().any(|q| quality(q) <= 0.0); + let mut counts = Vec::new(); + let mut kept = Vec::new(); + for (i, a) in alleles.iter().enumerate() { + let q = quals.get(i).map_or("", String::as_str); + let cands = candidates(a); + counts.push(cands.len()); + kept.push(if quality(q) <= 0.0 && cands.len() > MAX_CANDIDATES { + cands[..MAX_CANDIDATES].join(",") + } else { + a.clone() + }); + } + let one = pycompat::num::py_strip(n_diff) == "1" && kept.len() == 1; + Locus { + gene: gene.to_owned(), + alleles: kept, + quality: quals, + low_confidence: low, + candidates: counts, + one_allele_type: one, + } +} + +/// `yes` when every candidate of the allele is in the group, `no` when +/// none is, `mixed` when only some are. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Match { + Yes, + No, + Mixed, +} + +fn matches(allele: &str, group: &str) -> Match { + let prefix = format!("{group}:"); + let hits: Vec = candidates(allele) + .iter() + .map(|c| *c == group || c.starts_with(&prefix)) + .collect(); + if !hits.is_empty() && hits.iter().all(|h| *h) { + Match::Yes + } else if hits.iter().any(|h| *h) { + Match::Mixed + } else { + Match::No + } +} + +/// Over both alleles of a locus. +fn carries(l: &Locus, group: &str) -> Match { + let m: Vec = l.alleles.iter().map(|a| matches(a, group)).collect(); + if m.contains(&Match::Yes) { + Match::Yes + } else if m.contains(&Match::Mixed) { + Match::Mixed + } else { + Match::No + } +} + +fn both(a: Match, b: Match) -> &'static str { + if a == Match::Yes && b == Match::Yes { + "present" + } else if a == Match::No || b == Match::No { + "absent" + } else { + "uncertain" + } +} + +/// The DQ haplotypes, the DQB1*02 dose and each table allele present or +/// absent. +fn derived(loci: &[Locus], table: &[indexmap::IndexMap]) -> Result { + let mut by: indexmap::IndexMap<&str, &Locus> = indexmap::IndexMap::new(); + for l in loci { + by.entry(name(&l.gene)).or_insert(l); + } + let usable = |gene: &str| -> (Option<&Locus>, String) { + match by.get(gene) { + Some(l) if !l.alleles.is_empty() => { + if l.low_confidence { + (None, format!("HLA-{gene} low confidence")) + } else { + (Some(*l), String::new()) + } + } + _ => (None, format!("HLA-{gene} not typed")), + } + }; + let (dqa, ra) = usable("DQA1"); + let (dqb, rb) = usable("DQB1"); + let reason = [ra, rb] + .into_iter() + .filter(|r| !r.is_empty()) + .collect::>() + .join("; "); + let mut co = Obj::new() + .set("status", s("not_determined")) + .set("reason", s(&reason)) + .set("dq2_5", Value::Null) + .set("dq2_5_phase", Value::Null) + .set("dq2_2", Value::Null) + .set("dq8", Value::Null) + .set("dqb1_02_dose", Value::Null); + if let Some(dqb) = dqb { + let m: Vec = dqb.alleles.iter().map(|a| matches(a, "DQB1*02")).collect(); + if !m.contains(&Match::Mixed) { + let n = m.iter().filter(|x| **x == Match::Yes).count(); + co.put( + "dqb1_02_dose", + count(if dqb.one_allele_type { 2 * n } else { n }), + ); + } + } + if let (Some(dqa), Some(dqb)) = (dqa, dqb) { + co.put("status", s("derived")); + co.put("reason", s("")); + let dq25 = both(carries(dqa, "DQA1*05"), carries(dqb, "DQB1*02")); + co.put("dq2_5", s(dq25)); + co.put( + "dq2_2", + s(both(carries(dqa, "DQA1*02:01"), carries(dqb, "DQB1*02:02"))), + ); + co.put( + "dq8", + s(both(carries(dqa, "DQA1*03"), carries(dqb, "DQB1*03:02"))), + ); + if dq25 == "present" { + let (drb1, why) = usable("DRB1"); + let phase = match drb1.map(|l| carries(l, "DRB1*03")) { + Some(Match::Yes) => "likely cis (DRB1*03 present)".to_owned(), + Some(Match::No) => "possible trans (no DRB1*03)".to_owned(), + _ => format!( + "phase unknown ({})", + if why.is_empty() { + "DRB1*03 uncertain" + } else { + &why + } + ), + }; + co.put("dq2_5_phase", s(&phase)); + } + } + let mut alleles = Vec::new(); + for r in table { + let cell = |k: &str| r.get(k).cloned().ok_or_else(|| key_error(k)); + let allele = cell("allele")?; + let gene = allele.split('*').next().unwrap_or_default(); + let (l, why) = usable(gene); + let state = match l.map(|l| carries(l, &allele)) { + Some(Match::Yes) => "present", + Some(Match::No) => "absent", + Some(Match::Mixed) => "uncertain", + None => "not_determined", + }; + alleles.push( + Obj::new() + .set("allele", s(&allele)) + .set("kind", s(&cell("kind")?)) + .set("association", s(&cell("association")?)) + .set("pmids", s(&cell("pmids")?)) + .set("citation", s(&cell("citation")?)) + .set("state", s(state)) + .set("reason", s(&why)) + .val(), + ); + } + Ok(Obj::new() + .set("coeliac", co.val()) + .set("alleles", Value::Array(alleles)) + .val()) +} + +/// The first `#` line of a file step 08 wrote instead of a result. +fn first_comment(path: &str) -> Result { + Ok(lines(path)? + .iter() + .find(|l| l.starts_with('#')) + .map(|l| pycompat::num::py_strip(l.trim_start_matches('#')).to_owned()) + .unwrap_or_default()) +} + +/// OptiType's concordance with T1K per class I gene, or why it did not run. +fn optitype(c: &Ctx<'_>) -> Result, String> { + let conc = c.find1(format!("hla_optitype/{}_hla_concordance.tsv", c.s())); + let res = c.find1(format!("hla_optitype/{}_optitype_result.tsv", c.s())); + let Some(conc) = conc else { + let Some(res) = res else { return Ok(None) }; + let why = first_comment(&res.path)?; + let why = if why.is_empty() { + "no concordance table: step 08 did not compare the callers".to_owned() + } else { + why + }; + return Ok(Some( + Obj::new() + .set("genes", Value::Array(Vec::new())) + .set("reason", s(&why)) + .val(), + )); + }; + let mut genes = Vec::new(); + for r in table(&conc.path, false)?.rows { + let cell = |k: &str| pycompat::num::py_strip(r.get(k).flatten().unwrap_or("")).to_owned(); + if !cell("Gene").is_empty() { + genes.push( + Obj::new() + .set("gene", s(&cell("Gene"))) + .set("t1k", s(&cell("T1K"))) + .set("optitype", s(&cell("OptiType"))) + .set("agree", s(&cell("Agree"))) + .set("note", s(&cell("Note"))) + .val(), + ); + } + } + let reason = if genes.is_empty() { + "the concordance table has no rows" + } else { + "" + }; + Ok(Some( + Obj::new() + .set("genes", Value::Array(genes)) + .set("reason", s(reason)) + .val(), + )) +} + +/// The KIR pass: the genes T1K gave an allele, with each quality, and the +/// genes it gave none. +fn kir(c: &Ctx<'_>) -> Result, String> { + let s_ = c.s(); + let Some(p) = c.find(&[ + format!("kir_t1k/{s_}_kir_genotype.tsv"), + format!("kir/{s_}_kir_genotype.tsv"), + ]) else { + return Ok(None); + }; + let (mut typed, mut not_called) = (Vec::new(), Vec::new()); + for l in lines(&p.path)? { + let c: Vec<&str> = l.split('\t').collect(); + if c.len() < 3 || c[0].starts_with('#') { + continue; + } + let (a, q) = alleles(&c, true); + if a.is_empty() { + not_called.push(c[0].to_owned()); + } else { + typed.push( + Obj::new() + .set("gene", s(c[0])) + .set("alleles", strs(&a)) + .set("quality", strs(&q)) + .val(), + ); + } + } + let reason = if !typed.is_empty() || !not_called.is_empty() { + String::new() + } else { + let why = first_comment(&p.path)?; + if why.is_empty() { + "the KIR genotype file has no gene rows".to_owned() + } else { + why + } + }; + Ok(Some( + Obj::new() + .set("typed", Value::Array(typed)) + .set("not_called", strs(¬_called)) + .set("reason", s(&reason)) + .val(), + )) +} + +/// Step 08's T1K genotype table, the derived calls, and OptiType and KIR +/// when they ran. +pub fn hla(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find(&[ + format!("hla_t1k/{s_}_hla_genotype.tsv"), + format!("hla/{s_}_hla_genotype.tsv"), + "hla/*_hla_genotype.tsv".to_owned(), + ]) else { + return Ok(None); + }; + let mut loci = Vec::new(); + for l in lines(&p.path)? { + let c: Vec<&str> = l.split('\t').collect(); + if c.len() < 3 || c[0].starts_with('#') { + continue; + } + let (a, q) = alleles(&c, false); + loci.push(locus(c[0], a, q, c.get(1).copied().unwrap_or(""))); + } + let table = bin_rows(&c.table("hla_alleles.tsv"))?; + let mut out = Obj::new() + .set("loci", Value::Array(loci.iter().map(Locus::json).collect())) + .set("derived", derived(&loci, &table)?); + if let Some(o) = optitype(c)? { + out.put("optitype", o); + } + if let Some(k) = kir(c)? { + out.put("kir", k); + } + let _ = at; + Ok(Some((p, out.val()))) +} diff --git a/rust/crates/og-summary/src/collect/group_c/repeats.rs b/rust/crates/og-summary/src/collect/group_c/repeats.rs new file mode 100644 index 000000000..9d9610fd0 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/repeats.rs @@ -0,0 +1,933 @@ +//! Step 09 (ExpansionHunter) and 9b (Stranger, the RFC1 and other motif +//! counts): every locus with its counts, how well each allele was sized, +//! Stranger's grade, the intermediate zone, and what the counts of CFTR's +//! TG and poly-T tracts and UGT1A1's TA repeat mean. + +use super::super::model::State; +use super::util::{Obj, bin_rows, cells, count, get_default, s, strs, table, vcf_lines}; +use super::{Ctx, Out}; +use indexmap::IndexMap; +use pycompat::json::Value; + +/// The five loci the report names first. +pub const EH_LOCI: [&str; 5] = ["HTT", "FMR1", "C9ORF72", "ATXN1", "DMPK"]; +/// Fewer spanning reads than this: a low-confidence size. +const MIN_SPANNING: i128 = 2; +const MOTIF_KEYS: [&str; 7] = [ + "verdict", + "note", + "dominant_motif", + "dominant_class", + "dominant_share", + "motif_shares", + "repeat_reads", +]; +/// The repeats STRchive leaves out, read when step 09 sized them. +const EXTRA_SETUP: &str = "STR_EXTRA_LOCI=yes ./scripts/setup.sh --strchive-data "; +const CFTR_R117H: &str = "rs78655421"; +const CFTR_SOURCE: &str = "Deignan et al. 2020, PMID 32404922"; +const UGT1A1_SOURCE: &str = "Gammal et al. 2016, PMID 26417955"; +const HTTLPR_READING: &str = "not tested by any step. CPIC's 2023 SSRI guideline (Bousman et al. 2023, PMID 37032427) found that the data on SLC6A4 do not support its use in antidepressant prescribing, so a result would carry no recommendation."; + +/// How well ExpansionHunter sized an allele, worst first; a locus takes +/// the worst of its alleles. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum Sizing { + NoCall, + FlankingOnly, + InRepeat, + LowConfidence, + Sized, + Unknown, +} + +impl Sizing { + pub fn word(self) -> &'static str { + match self { + Sizing::NoCall => "no_call", + Sizing::FlankingOnly => "flanking_only", + Sizing::InRepeat => "in_repeat", + Sizing::LowConfidence => "low_confidence", + Sizing::Sized => "sized", + Sizing::Unknown => "unknown", + } + } + + fn of(w: &str) -> Option { + [ + Sizing::NoCall, + Sizing::FlankingOnly, + Sizing::InRepeat, + Sizing::LowConfidence, + Sizing::Sized, + Sizing::Unknown, + ] + .into_iter() + .find(|z| z.word() == w) + } + + /// Sized well enough to read a count. + fn read(self) -> bool { + matches!(self, Sizing::Sized | Sizing::Unknown) + } +} + +fn str_int(x: &str) -> Option { + crate::int(x).ok() +} + +/// One allele's sizing from ExpansionHunter's SO, ADSP and REPCI fields. +fn allele_sizing(so: &str, adsp: &str, ci: &str, cn: &str, path_min: Option) -> Sizing { + let Some(n) = str_int(cn) else { + return Sizing::NoCall; + }; + match so { + "" | "." => return Sizing::Unknown, + "FLANKING" => return Sizing::FlankingOnly, + "INREPEAT" => return Sizing::InRepeat, + _ => {} + } + let hi = if ci.is_empty() { + None + } else { + str_int(ci.rsplit('-').next().unwrap_or(ci)) + }; + let few = str_int(adsp).is_some_and(|a| a < MIN_SPANNING); + let reaches = matches!((path_min, hi), (Some(m), Some(h)) if n < m && m <= h); + if few || reaches { + Sizing::LowConfidence + } else { + Sizing::Sized + } +} + +/// A per-allele FORMAT value as `n` parts; one value serves every allele. +fn split_n(x: &str, n: usize) -> Vec { + let parts: Vec<&str> = if x.is_empty() { + Vec::new() + } else { + x.split('/').collect() + }; + if parts.len() == n { + parts.into_iter().map(str::to_owned).collect() + } else if parts.len() == 1 { + vec![parts[0].to_owned(); n] + } else { + vec![String::new(); n] + } +} + +/// `bin/str_intermediate_ranges.tsv` by locus. +fn ranges(c: &Ctx<'_>) -> Result>, String> { + let mut out = IndexMap::new(); + for r in bin_rows(&c.table("str_intermediate_ranges.tsv"))? { + let k = r + .get("locus") + .cloned() + .ok_or_else(|| super::util::key_error("locus"))?; + out.insert(k, r); + } + Ok(out) +} + +fn mtime(p: &str) -> Result { + pycompat::mtime::mtime(p).map_err(|e| super::util::os_note(&crate::os_error(&e, p))) +} + +/// The motif table `name` unless it is older than Stranger's file. +fn motif_table(c: &Ctx<'_>, name: &str, after: &str) -> Result, String> { + let Some(p) = c.find1(format!("expansion_hunter/{}_{name}", c.s())) else { + return Ok(None); + }; + if mtime(&p.path)? < mtime(after)? { + return Ok(None); + } + table(&p.path, false).map(Some) +} + +fn motif_row(r: &pycompat::csv::DictRow) -> Value { + MOTIF_KEYS + .iter() + .fold(Obj::new(), |o, k| o.set(k, get_default(r, k, "."))) + .val() +} + +/// `{locus: reason}` of the loci the catalog step 09 ran on gives no +/// thresholds (its NoThresholds field), from the copy beside its VCF. +fn not_graded(c: &Ctx<'_>) -> IndexMap { + let mut out = IndexMap::new(); + let Some(p) = c.find1(format!("expansion_hunter/{}_eh_catalog.json", c.s())) else { + return out; + }; + let Ok(text) = std::fs::read(&p.path).map(String::from_utf8) else { + return out; + }; + let Ok(Ok(Value::Array(cat))) = text.map(|t| pycompat::json::loads(&t)) else { + return out; + }; + for e in cat { + let Value::Object(m) = e else { continue }; + let (Some(id), Some(why)) = (m.get("LocusId"), m.get("NoThresholds")) else { + continue; + }; + if pycompat::value::truthy(id) + && pycompat::value::truthy(why) + && let (Value::Str(id), Ok(why)) = (id, pycompat::value::py_str(why)) + { + out.insert(id.clone(), why); + } + } + out +} + +/// One locus as the reports read it. +struct Locus { + json: Obj, + locus: String, +} + +/// Step 09's ExpansionHunter VCF, or Stranger's copy of it with a grade per +/// locus when it is not older than the VCF. +pub fn expansions(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let eh = c.find(&[ + format!("expansion_hunter/{s_}_eh.vcf"), + format!("expansion_hunter/{s_}.vcf"), + "expansion_hunter/*_eh.vcf".to_owned(), + ]); + let mut st = c.find(&[ + format!("expansion_hunter/{s_}_eh_stranger.vcf"), + "expansion_hunter/*_eh_stranger.vcf".to_owned(), + ]); + let outdated = match (&st, &eh) { + (Some(a), Some(b)) => mtime(&a.path)? < mtime(&b.path)?, + _ => false, + }; + if outdated { + st = None; + } + let Some(p) = st.clone().or(eh) else { + return Ok(None); + }; + let ranges = ranges(c)?; + let stp = st.as_ref().map(|f| f.path.as_str()); + let rfc1 = match stp { + Some(after) => { + motif_table(c, "rfc1_motif.tsv", after)?.and_then(|t| t.rows.first().map(motif_row)) + } + None => None, + }; + let mut motifs: IndexMap = IndexMap::new(); + if let Some(after) = stp + && let Some(t) = motif_table(c, "str_motifs.tsv", after)? + { + for r in &t.rows { + if let Some(l) = r.get("locus").flatten().filter(|l| !l.is_empty()) { + motifs.insert(l.to_owned(), motif_row(r)); + } + } + } + let ungraded = if st.is_some() { + not_graded(c) + } else { + IndexMap::new() + }; + let mut records = Vec::new(); + vcf_lines(&p.path, |l| records.push(cells(l)))?; + let (mut loci, mut flagged, mut no_status, mut graded_none) = + (Vec::::new(), Vec::new(), 0, Vec::::new()); + let mut tested = 0; + for r in &records { + if r.len() < 10 { + continue; + } + tested += 1; + let info: IndexMap<&str, &str> = r[7] + .split(';') + .map(|kv| kv.split_once('=').unwrap_or((kv, ""))) + .collect(); + let get = |k: &str| info.get(k).copied().filter(|v| !v.is_empty()); + let lid = get("REPID") + .or(get("VARID")) + .map_or_else(|| format!("{}:{}", r[0], r[1]), str::to_owned); + let rep = get("VARID").map_or_else(|| lid.clone(), str::to_owned); + let fmt: Vec<&str> = r[8].split(':').collect(); + let val: Vec<&str> = r[9].split(':').collect(); + let field = |k: &str| { + fmt.iter() + .position(|f| *f == k) + .and_then(|i| val.get(i)) + .copied() + .unwrap_or("") + }; + let repcn = field("REPCN"); + let counts: Vec<&str> = if repcn.is_empty() || repcn == "." { + Vec::new() + } else { + repcn.split('/').collect() + }; + let n = counts.len().max(1); + let path_min = info.get("STR_PATHOLOGIC_MIN").and_then(|v| str_int(v)); + let cn_list: Vec<&str> = if counts.is_empty() { + vec!["."] + } else { + counts.clone() + }; + let sizing: Vec = split_n(field("SO"), n) + .iter() + .zip(split_n(field("ADSP"), n)) + .zip(split_n(field("REPCI"), n)) + .zip(cn_list) + .map(|(((so, ad), ci), cn)| allele_sizing(so, &ad, &ci, cn, path_min)) + .collect(); + let status: Vec<&str> = if st.is_some() { + info.get("STR_STATUS") + .copied() + .unwrap_or("") + .split(',') + .filter(|x| !x.is_empty() && *x != ".") + .collect() + } else { + Vec::new() + }; + let worst = sizing.iter().copied().min().unwrap_or(Sizing::NoCall); + let dots = |k: &str| { + info.get(k) + .copied() + .unwrap_or("") + .trim_matches('.') + .to_owned() + }; + let mut item = Obj::new() + .set("locus", s(&rep)) + .set( + "repeat_count", + s(if repcn.is_empty() { "." } else { repcn }), + ) + .set("status", s(&status.join(","))) + .set("normal_max", s(&dots("STR_NORMAL_MAX"))) + .set("pathologic_min", s(&dots("STR_PATHOLOGIC_MIN"))) + .set("support", s(field("SO"))) + .set("adsp", s(field("ADSP"))) + .set("ci", s(field("REPCI"))) + .set("sizing", s(worst.word())) + .set( + "allele_sizing", + s(&sizing + .iter() + .map(|z| z.word()) + .collect::>() + .join("/")), + ) + .set("zone", s("")) + .set("zone_label", s("")) + .set("zone_citation", s("")); + let sized: Vec = counts + .iter() + .zip(&sizing) + .filter(|(_, z)| z.read()) + .filter_map(|(c, _)| str_int(c)) + .collect(); + if let Some(rng) = ranges.get(&rep) { + let bound = |k: &str| -> Result { + let v = rng.get(k).ok_or_else(|| super::util::key_error(k))?; + crate::int(v).map_err(|e| super::super::sections::py_error("ValueError", &e)) + }; + if !sized.is_empty() { + let (lo, hi) = (bound("intermediate_min")?, bound("intermediate_max")?); + if sized.iter().any(|c| lo <= *c && *c <= hi) { + let cell = + |k: &str| rng.get(k).cloned().ok_or_else(|| super::util::key_error(k)); + item.put( + "zone", + s(&format!( + "{}-{}", + cell("intermediate_min")?, + cell("intermediate_max")? + )), + ); + item.put("zone_label", s(&cell("label")?)); + item.put("zone_citation", s(&cell("citation")?)); + } + } + } + let abnormal = status.iter().any(|x| *x != "normal"); + if rep == "RFC1" && st.is_some() && abnormal { + item.put( + "rfc1_motif", + rfc1.clone().unwrap_or_else(|| { + Obj::new() + .set("verdict", s("not_checked")) + .set("note", s("no motif count beside Stranger's output (step 9b runs it with the sample's BAM): confirm the flag with repeat-primed PCR")) + .val() + }), + ); + } + if rep != "RFC1" + && abnormal + && let Some(m) = motifs.get(&rep) + { + item.put("motif", m.clone()); + } + if st.is_some() + && status.is_empty() + && let Some(why) = ungraded.get(&lid) + { + item.put("not_graded", s(why)); + } + if st.is_some() { + if status.is_empty() && item.0.contains_key("not_graded") { + if !graded_none.contains(&lid) { + graded_none.push(lid.clone()); + } + } else if status.is_empty() { + no_status += 1; + } else if abnormal { + let mut f = Obj::new(); + for k in ["locus", "repeat_count", "status", "sizing"] { + f.put(k, item.0[k].clone()); + } + for k in ["rfc1_motif", "motif"] { + if let Some(v) = item.0.get(k) { + f.put(k, v.clone()); + } + } + flagged.push(f.val()); + } + } + loci.push(Locus { + json: item, + locus: rep, + }); + } + let by_unsorted = by_locus(loci.iter().map(|l| (&l.locus, &l.json))); + loci.sort_by(|a, b| { + let rank = |l: &str| { + EH_LOCI + .iter() + .position(|k| *k == l) + .unwrap_or(EH_LOCI.len()) + }; + rank(&a.locus) + .cmp(&rank(&b.locus)) + .then_with(|| a.locus.cmp(&b.locus)) + }); + let key_loci = EH_LOCI + .iter() + .map(|k| { + Obj::new() + .set("locus", s(k)) + .set( + "repeat_count", + by_unsorted + .get(*k) + .map_or_else(|| s("not in output"), |l| l.repeat_count.clone()), + ) + .val() + }) + .collect(); + let word = |l: &Locus, k: &str| match l.json.0.get(k) { + Some(Value::Str(x)) => x.clone(), + _ => String::new(), + }; + let unsized_: Vec = loci + .iter() + .filter(|l| !Sizing::of(&word(l, "sizing")).is_some_and(Sizing::read)) + .map(|l| l.locus.clone()) + .collect(); + let intermediate: Vec = loci + .iter() + .filter(|l| !word(l, "zone").is_empty()) + .map(|l| l.locus.clone()) + .collect(); + let readings = readings(c, &by_unsorted, State::Ok, State::Ok)?; + let mut out = Obj::new() + .set("records", count(tested)) + .set("key_loci", Value::Array(key_loci)) + .set( + "loci", + Value::Array(loci.into_iter().map(|l| l.json.val()).collect()), + ) + .set("stranger", Value::Bool(st.is_some())) + .set("stranger_outdated", Value::Bool(outdated)) + .set("unsized", strs(&unsized_)) + .set("intermediate", strs(&intermediate)) + .set("readings", readings); + if st.is_some() { + out.put("flagged", Value::Array(flagged)); + out.put("no_status", count(no_status)); + out.put( + "not_graded", + Value::Object( + graded_none + .iter() + .map(|k| (k.clone(), s(&ungraded[k]))) + .collect(), + ), + ); + } + + Ok(Some((p, out.val()))) +} + +/// What a reading needs of a locus: its count and how well it was sized. +#[derive(Debug, Clone)] +struct Sized { + repeat_count: Value, + sizing: Option, +} + +/// `{locus: l}` over loci in this order, the last of a name kept. +fn by_locus<'a>(loci: impl Iterator) -> IndexMap { + loci.map(|(k, o)| { + let get = |f: &str| o.0.get(f).cloned().unwrap_or(Value::Null); + let sizing = match get("sizing") { + Value::Str(w) => Sizing::of(&w), + _ => None, + }; + ( + k.clone(), + Sized { + repeat_count: get("repeat_count"), + sizing, + }, + ) + }) + .collect() +} + +/// The two counts of a sized record, sorted. +fn counts(item: Option<&Sized>) -> Option<[i128; 2]> { + let item = item?; + if !item.sizing.is_some_and(Sizing::read) { + return None; + } + let Value::Str(rc) = &item.repeat_count else { + return None; + }; + let mut c: Vec> = rc.split('/').map(str_int).collect(); + if c.len() == 1 && c[0].is_some() { + c.push(c[0]); + } + match c[..] { + [Some(a), Some(b)] => Some(if a <= b { [a, b] } else { [b, a] }), + _ => None, + } +} + +fn text_of(v: &Value) -> String { + match v { + Value::Str(x) => x.clone(), + other => pycompat::value::py_str(other).unwrap_or_default(), + } +} + +fn sizing_text(item: &Sized) -> String { + item.sizing.map_or("None", Sizing::word).to_owned() +} + +/// What a section state other than ok says about another step's table. +fn other_state(st: State) -> &'static str { + match st { + State::Stale => "from an earlier run", + State::Failed => "its step failed in the last run", + _ => "unreadable", + } +} + +/// R117H's genotype in step 42's traits table: (genotype, why not read, +/// a note on a table of an earlier run). +fn r117h(c: &Ctx<'_>, state: State) -> (Option, String, String) { + if matches!(state, State::Failed | State::Unreadable) { + return ( + None, + format!("step 42's traits table is {}", other_state(state)), + String::new(), + ); + } + let row = c + .find1(format!("traits/{}_traits.tsv", c.s())) + .and_then(|p| table(&p.path, false).ok()) + .and_then(|t| { + t.rows + .into_iter() + .find(|r| r.get("rsid").flatten() == Some(CFTR_R117H)) + }); + let called = row.as_ref().map(|r| { + ( + super::util::or(r, "status", ""), + super::util::or(r, "genotype", ""), + ) + }); + match called { + Some((st, gt)) if st == "called" => ( + Some(gt), + String::new(), + if state == State::Stale { + " (R117H from step 42's traits table of an earlier run)".into() + } else { + String::new() + }, + ), + _ => ( + None, + "no step 42 traits row for rs78655421 was called".into(), + String::new(), + ), + } +} + +fn a_copies(gt: &str) -> usize { + gt.split('/').filter(|x| *x == "A").count() +} + +fn cftr(c: &Ctx<'_>, by: &IndexMap, traits: State) -> Value { + let out = Obj::new() + .set("locus", s("CFTR")) + .set("repeat", s("intron 9 (legacy IVS8) TG and poly-T tract")) + .set("source", s(CFTR_SOURCE)); + let not_sized = |o: Obj, reading: String| { + o.set("status", s("not_sized")) + .set("genotype", s(".")) + .set("phase", s(".")) + .set("reading", s(&reading)) + .val() + }; + let (Some(tg), Some(t)) = (by.get("CFTR_TG"), by.get("CFTR_T")) else { + return not_sized( + out, + format!( + "not in step 09's output: {EXTRA_SETUP} adds the tract to the catalog (docs/09-str-expansions.md)." + ), + ); + }; + let (Some(a), Some(b)) = (counts(Some(tg)), counts(Some(t))) else { + return not_sized( + out, + format!( + "ExpansionHunter did not size both tracts (TG {}, {}; poly-T {}, {}): not read.", + text_of(&tg.repeat_count), + sizing_text(tg), + text_of(&t.repeat_count), + sizing_text(t) + ), + ); + }; + let pairs: Vec = if a[0] == a[1] || b[0] == b[1] { + vec![format!("TG{}-{}T/TG{}-{}T", a[0], b[0], a[1], b[1])] + } else { + vec![ + format!("TG{}-{}T/TG{}-{}T", a[0], b[0], a[1], b[1]), + format!("TG{}-{}T/TG{}-{}T", a[1], b[0], a[0], b[1]), + ] + }; + let genotype = if pairs.len() == 1 { + pairs[0].clone() + } else { + format!("TG{}/TG{} {}T/{}T", a[0], a[1], b[0], b[1]) + }; + let ints = |x: [i128; 2]| Value::Array(x.iter().map(|v| Value::Int(*v)).collect()); + let out = out + .set("status", s("read")) + .set("genotype", s(&genotype)) + .set("pairings", strs(&pairs)) + .set("tg", ints(a)) + .set("poly_t", ints(b)); + let five = b.iter().filter(|n| **n == 5).count(); + let mut other: Vec = b + .iter() + .copied() + .filter(|n| !matches!(n, 5 | 7 | 9)) + .collect(); + other.sort_unstable(); + other.dedup(); + let odd = if other.is_empty() { + String::new() + } else { + format!( + " A poly-T length of {} is none of the usual 5T, 7T and 9T.", + other + .iter() + .map(|n| format!("{n}T")) + .collect::>() + .join(", ") + ) + }; + let (r117, why, note) = r117h(c, traits); + let r117_text = s(r117.as_deref().filter(|x| !x.is_empty()).unwrap_or(".")); + if five == 0 { + let mut line = format!( + "No 5T copy ({}T/{}T): the tract adds no 5T allele of its own. The poly-T length still matters beside R117H, which is milder and of variable penetrance on a 7T or 9T copy and CF-causing with 5T in cis.", + b[0], b[1] + ); + match &r117 { + None => line += &format!(" R117H was not read ({why})."), + Some(g) if a_copies(g) == 2 => { + let copies = if b[0] == b[1] { + format!("both {}T copies", b[0]) + } else { + format!("both copies ({}T and {}T)", b[0], b[1]) + }; + line += &format!(" R117H ({g}) is read: it sits on {copies}.{note}"); + } + Some(g) if a_copies(g) > 0 => { + let copies = if b[0] == b[1] { + format!("{}T", b[0]) + } else { + format!("{}T or {}T", b[0], b[1]) + }; + line += &format!( + " R117H ({g}) is read: with no 5T copy it sits on a {copies} copy.{note}" + ); + } + Some(_) => line += &format!(" No R117H copy.{note}"), + } + return out + .set("five_t", Value::Bool(false)) + .set("phase", s("not_needed")) + .set("r117h", r117_text) + .set("reading", s(&(line + &odd))) + .val(); + } + let tg_text = if pairs.len() == 1 { + let tgs: Vec = if a[0] == a[1] { vec![a[0]] } else { a.to_vec() }; + format!( + "{}{}", + tgs.iter() + .map(|x| format!("TG{x}")) + .collect::>() + .join(" and "), + if five == 1 { + " on the 5T copy" + } else { + " on the 5T copies" + } + ) + } else { + format!( + "TG{} or TG{} on the 5T copy, not told by the counts ({})", + a[0], + a[1], + pairs.join(" or ") + ) + }; + let line = format!( + "5T: {five} copy{} with the 5T poly-T tract, {tg_text}. 5T lowers the splicing of exon 10 and is a variable-penetrance allele: with TG12 or TG13 on the same copy it more often causes CBAVD or another CFTR-related disorder when the other copy carries a CF-causing variant.", + if five > 1 { "s" } else { "" } + ); + let (phase, text) = match &r117 { + None => ( + "not_read", + format!(" R117H was not read ({why}), so whether it sits on the 5T copy is not known."), + ), + Some(g) => { + let n = a_copies(g); + let (p, t) = if n == 0 { + ( + "not_needed", + " No R117H copy: the 5T is read on its own.".to_owned(), + ) + } else if n == 2 || five == 2 { + ( + "in_cis", + format!( + " R117H ({g}) and 5T share a copy: R117H with 5T in cis is a CF-causing allele of variable penetrance." + ), + ) + } else { + ( + "unknown", + " One R117H copy and one 5T copy: phase unknown. R117H lies 17.6 kb from the tract, farther than a read pair spans, so whether they share a copy (in cis, a CF-causing allele of variable penetrance) or not (in trans) is not read; testing a parent tells them apart.".to_owned(), + ) + }; + (p, t + ¬e) + } + }; + out.set("five_t", Value::Bool(true)) + .set("phase", s(phase)) + .set("r117h", r117_text) + .set("reading", s(&(line + &text + &odd))) + .val() +} + +fn ugt1a1_name(n: i128) -> Option<&'static str> { + match n { + 6 => Some("*36"), + 7 => Some("*1"), + 8 => Some("*28"), + 9 => Some("*37"), + _ => None, + } +} + +fn ugt1a1(c: &Ctx<'_>, by: &IndexMap, cpic: State) -> Value { + let out = Obj::new() + .set("locus", s("UGT1A1")) + .set("repeat", s("promoter TA repeat")) + .set("source", s(UGT1A1_SOURCE)); + let not_sized = |o: Obj, reading: String| { + o.set("status", s("not_sized")) + .set("genotype", s(".")) + .set("pharmcat", s(".")) + .set("agreement", s("not_compared")) + .set("reading", s(&reading)) + .val() + }; + let Some(item) = by.get("UGT1A1") else { + return not_sized( + out, + format!( + "not in step 09's output: {EXTRA_SETUP} adds the repeat to the catalog (docs/09-str-expansions.md). PharmCAT (step 7) still reads UGT1A1 *28." + ), + ); + }; + let Some(cn) = counts(Some(item)) else { + return not_sized( + out, + format!( + "ExpansionHunter did not size the repeat ({}, {}): not read.", + text_of(&item.repeat_count), + sizing_text(item) + ), + ); + }; + let names: Vec = cn + .iter() + .map(|n| ugt1a1_name(*n).map_or_else(|| format!("TA{}", n - 1), str::to_owned)) + .collect(); + let genotype = names.join("/"); + let dec = names.iter().filter(|n| *n == "*28" || *n == "*37").count(); + let ta = cn + .iter() + .map(|n| format!("TA{}", n - 1)) + .collect::>() + .join("/"); + let out = out + .set("status", s("read")) + .set("genotype", s(&genotype)) + .set( + "ta_units", + Value::Array(cn.iter().map(|v| Value::Int(*v)).collect()), + ); + let mut reading = format!( + "{genotype} ({ta}, {}/{} TA units as ExpansionHunter counts them): ", + cn[0], cn[1] + ); + if !cn.iter().all(|n| ugt1a1_name(*n).is_some()) { + reading += "a TA length outside TA5 to TA8 is not a named allele here, so its function is not read and it is not compared with PharmCAT (step 7)."; + return out + .set("pharmcat", s(".")) + .set("agreement", s("not_compared")) + .set("reading", s(&reading)) + .val(); + } + reading += &if dec == 0 { + "no decreased-function TA allele.".to_owned() + } else { + format!( + "{dec} decreased-function allele{} (*28 and *37 are decreased function).", + if dec > 1 { "s" } else { "" } + ) + }; + let (mut pc, mut agreement) = (".".to_owned(), "not_compared"); + let readable = !matches!(cpic, State::Failed | State::Unreadable); + let row = c + .find1(format!("cpic/{}_phenotypes.tsv", c.s())) + .filter(|_| readable) + .and_then(|p| table(&p.path, false).ok()) + .and_then(|t| { + t.rows + .into_iter() + .find(|r| r.get("Gene").flatten() == Some("UGT1A1")) + }); + if let Some(row) = row { + let d = super::util::or(&row, "Diplotype", ""); + let d = pycompat::num::py_strip(&d); + pc = if d.is_empty() { + ".".into() + } else { + d.to_owned() + }; + let alleles: Vec<&str> = pc.split('/').collect(); + if alleles.len() == 2 + && !pc.contains("possible") + && super::util::or(&row, "Status", "") != "not called" + { + let n = alleles + .iter() + .filter(|a| { + a.split(['+', ' ', '[', ']']) + .any(|x| x == "*28" || x == "*37") + }) + .count(); + agreement = if n == dec { "agrees" } else { "differs" }; + } + } + let old = if cpic == State::Stale && agreement != "not_compared" { + " (step 27's table is from an earlier run)" + } else { + "" + }; + reading += &match agreement { + "agrees" => { + format!(" PharmCAT (step 7) called {pc}: the same number of *28 and *37 copies{old}.") + } + "differs" => format!( + " PharmCAT (step 7) called {pc}, a different number of *28 and *37 copies{old}: read the reads at chr2:233760234 before using either." + ), + _ if !readable => format!( + " Not compared with PharmCAT (step 27's table is {}).", + other_state(cpic) + ), + _ => " Not compared with PharmCAT (no unambiguous UGT1A1 call from step 27).".to_owned(), + }; + out.set("pharmcat", s(&pc)) + .set("agreement", s(agreement)) + .set("reading", s(&reading)) + .val() +} + +/// What the counts of the extra repeats mean, and the 5-HTTLPR line no +/// step can test. +fn readings( + c: &Ctx<'_>, + by: &IndexMap, + traits: State, + cpic: State, +) -> Result { + Ok(Value::Array(vec![ + cftr(c, by, traits), + ugt1a1(c, by, cpic), + Obj::new() + .set("locus", s("SLC6A4")) + .set("repeat", s("5-HTTLPR promoter length polymorphism")) + .set("status", s("not_tested")) + .set("genotype", s(".")) + .set("reading", s(HTTLPR_READING)) + .set("source", s("Bousman et al. 2023, PMID 37032427")) + .val(), + ])) +} + +/// The post pass: the readings again, with the traits and CPIC states, over +/// the loci as the section lists them. +pub fn post(c: &Ctx<'_>, v: &mut Value, traits: State, cpic: State) { + let Value::Object(m) = v else { return }; + let loci: Vec<(String, Obj)> = match m.get("loci") { + Some(Value::Array(xs)) => xs + .iter() + .filter_map(|x| match x { + Value::Object(o) => match o.get("locus") { + Some(Value::Str(l)) => Some((l.clone(), Obj(o.clone()))), + _ => None, + }, + _ => None, + }) + .collect(), + _ => return, + }; + let by = by_locus(loci.iter().map(|(k, o)| (k, o))); + if let Ok(r) = readings(c, &by, traits, cpic) { + m.insert("readings".into(), r); + } +} diff --git a/rust/crates/og-summary/src/collect/group_c/roh.rs b/rust/crates/og-summary/src/collect/group_c/roh.rs new file mode 100644 index 000000000..489aa1a7e --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/roh.rs @@ -0,0 +1,496 @@ +//! Step 11 (bcftools roh): the runs of homozygosity of chr1 to chr22 +//! outside GRCh38's centromere and heterochromatin bands, F_ROH, the +//! relationship band it is nearest to and the one-chromosome (UPD) flag; +//! and `og roh summary`, the step's own table of the long runs. +//! +//! [`MASKED_BANDS`] is the one copy of the bands: the summary section, +//! CNVpytor's count and `og roh summary` (step 11 and the ROH module) all +//! read it here. + +use super::util::{Obj, count, lines, s}; +use super::{Ctx, Out}; +use pycompat::json::Value; +use std::io::Write; + +/// GRCh38's centromeres and heterochromatin: the acen, gvar and stalk rows +/// of UCSC's cytoBand.hg38.txt (the file setup.sh installs as +/// reference/cytoBand.hg38.txt), neighbouring rows merged, as (chrom, +/// 0-based start, exclusive end). Reads there map badly, so a run of +/// homozygosity or a copy number call over them is mostly an artefact. +/// Regenerate with +/// gzip -cd cytoBand.txt.gz | awk '$1 ~ /^chr([0-9]+|X|Y)$/ && $5 ~ /^(acen|gvar|stalk)$/' \ +/// | sort -k1,1V -k2,2n | awk '{ if ($1 == c && $2 == e) e = $3; else { if (c != "") printf "%s:%d-%d ", c, s, e; +/// c = $1; s = $2; e = $3 } } END { print c ":" s "-" e }' +pub const MASKED_BANDS: [(&str, i128, i128); 25] = [ + ("chr1", 121_700_000, 143_200_000), + ("chr2", 91_800_000, 96_000_000), + ("chr3", 87_800_000, 98_600_000), + ("chr4", 48_200_000, 51_800_000), + ("chr5", 46_100_000, 51_400_000), + ("chr6", 58_500_000, 62_600_000), + ("chr7", 58_100_000, 62_100_000), + ("chr8", 43_200_000, 47_200_000), + ("chr9", 42_200_000, 61_500_000), + ("chr10", 38_000_000, 41_600_000), + ("chr11", 51_000_000, 55_800_000), + ("chr12", 33_200_000, 37_800_000), + ("chr13", 0, 18_900_000), + ("chr14", 0, 18_200_000), + ("chr15", 0, 20_500_000), + ("chr16", 35_300_000, 47_000_000), + ("chr17", 22_700_000, 27_400_000), + ("chr18", 15_400_000, 21_500_000), + ("chr19", 19_900_000, 31_900_000), + ("chr20", 25_700_000, 30_400_000), + ("chr21", 0, 13_000_000), + ("chr22", 0, 17_400_000), + ("chrX", 58_100_000, 63_800_000), + ("chrY", 10_300_000, 10_600_000), + ("chrY", 26_600_000, 57_227_415), +]; + +/// GRCh38's chr1 to chr22 (UCSC hg38.chrom.sizes). +pub const AUTOSOME_BP: i128 = 2_875_001_522; +/// The runs F_ROH and the one-chromosome flag count (docs/11-roh-analysis.md). +pub const ROH_MIN_BP: f64 = 5e6; +/// One chromosome carrying most of the long runs: at least this much of it +/// in runs of 5 Mb or more ... +pub const UPD_MIN_BP: f64 = 20e6; +/// ... and more than this share of all such autosomal runs. +pub const UPD_SHARE: f64 = 0.5; + +/// Bases of the 1-based, inclusive span `start..=end` inside the masked +/// bands of `chrom`. +pub fn masked_overlap(chrom: &str, start: i128, end: i128) -> i128 { + MASKED_BANDS + .iter() + .filter(|(c, ..)| *c == chrom) + .map(|(_, a, b)| overlap(*a, *b, start, end)) + .sum() +} + +fn overlap(a: i128, b: i128, start: i128, end: i128) -> i128 { + (end.min(b) - (start - 1).max(a)).max(0) +} + +/// Whether a chromosome name is one of chr1 to chr22. +pub fn is_autosome(chrom: &str) -> bool { + chrom + .strip_prefix("chr") + .and_then(|n| { + n.parse::() + .ok() + .filter(|v| (1..=22).contains(v)) + .map(|v| (v, n)) + }) + .is_some_and(|(v, n)| n == v.to_string()) +} + +/// [`AUTOSOME_BP`] without the autosomal bands: the length F_ROH is a share +/// of, since a run's part in a band is not counted either. +pub fn autosome_unmasked_bp() -> i128 { + AUTOSOME_BP + - MASKED_BANDS + .iter() + .filter(|(c, ..)| is_autosome(c)) + .map(|(_, a, b)| b - a) + .sum::() +} + +/// docs/11-roh-analysis.md's table: F of a child of the parents. A value +/// goes to the row whose F is nearest on a log scale (the boundaries are +/// the geometric means of neighbouring rows), and under 1/128 (half the +/// second-cousin F) to "no sign of related parents". +pub fn roh_band(f: f64) -> &'static str { + let gm = |a: f64, b: f64| (a * b).powf(0.5); + let bands = [ + (1.0 / 128.0, "no sign of related parents"), + (gm(1.0 / 64.0, 1.0 / 32.0), "second cousins (F 1/64)"), + ( + gm(1.0 / 32.0, 1.0 / 16.0), + "first cousins once removed (F 1/32)", + ), + (gm(1.0 / 16.0, 1.0 / 8.0), "first cousins (F 1/16)"), + ( + gm(1.0 / 8.0, 1.0 / 4.0), + "half siblings, uncle and niece, double first cousins (F 1/8)", + ), + (f64::INFINITY, "parent and child, full siblings (F 1/4)"), + ]; + bands + .iter() + .find(|(top, _)| f < *top) + .map_or("parent and child, full siblings (F 1/4)", |b| b.1) +} + +/// assets/imprinted_regions.tsv, the same rows (a test keeps the two +/// equal): chrom, the region's span (none for the whole chromosome), the +/// region, the maternal and paternal UPD syndromes and the PMIDs. +/// chrom, span, then four text fields. +type Imprinted = ( + &'static str, + Option<(i128, i128)>, + &'static str, + &'static str, + &'static str, + &'static str, +); + +pub const IMPRINTED_REGIONS: [Imprinted; 6] = [ + ( + "chr6", + None, + "6q24 (PLAGL1, HYMAI)", + "-", + "Transient neonatal diabetes mellitus (6q24-related)", + "20301706", + ), + ( + "chr7", + None, + "7p12 (GRB10), 7q32 (MEST)", + "Silver-Russell syndrome", + "-", + "27585961", + ), + ( + "chr11", + Some((0, 22_000_000)), + "11p15 (H19/IGF2, KCNQ1OT1)", + "Silver-Russell syndrome", + "Beckwith-Wiedemann syndrome", + "29377879;27585961", + ), + ( + "chr14", + None, + "14q32.2 (DLK1-MEG3)", + "Temple syndrome", + "Kagami-Ogata syndrome", + "24891339;26377239", + ), + ( + "chr15", + Some((20_500_000, 33_400_000)), + "15q11-q13 (SNRPN, UBE3A)", + "Prader-Willi syndrome", + "Angelman syndrome", + "22237428;27615419", + ), + ( + "chr20", + None, + "20q13.32 (GNAS)", + "Mulchandani-Bhoj-Conlin syndrome (growth failure)", + "Pseudohypoparathyroidism type 1B", + "26248010;29959430", + ), +]; + +/// The imprinted regions a flagged chromosome joins: every row of the +/// whole chromosome, and a row with a region only when a run overlaps it. +fn imprinted(chrom: &str, runs: &[(i128, i128)]) -> Value { + Value::Array( + IMPRINTED_REGIONS + .iter() + .filter(|r| r.0 == chrom) + .filter(|r| { + r.1.is_none_or(|(a, b)| runs.iter().any(|&(s, e)| overlap(a, b, s, e) > 0)) + }) + .map(|r| { + Obj::new() + .set("region", s(r.2)) + .set("maternal_upd", s(r.3)) + .set("paternal_upd", s(r.4)) + .set("pmid", s(r.5)) + .val() + }) + .collect(), + ) +} + +fn r1(x: f64, n: i32) -> Value { + super::util::round(x, n) +} + +/// Step 11's bcftools roh output: runs on chrX and chrY counted apart, runs +/// on any contig other than chr1 to chr22 left out, and the part of a run +/// in a masked band not counted. +pub fn section(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find(&[ + format!("vcf/{s_}_roh.txt"), + format!("roh/{s_}_roh.txt"), + "roh/*_roh.txt".to_owned(), + ]) else { + return Ok(None); + }; + let (mut total, mut largest, mut masked) = (0.0_f64, 0.0_f64, 0.0_f64); + let (mut segs, mut sex_segs) = (0usize, 0usize); + let mut big = Vec::new(); + // chrom -> the bases of its long runs, and the runs (start, end) + let mut per_chrom: indexmap::IndexMap)> = + indexmap::IndexMap::new(); + for line in lines(&p.path)? { + let c = pycompat::text::split(&line, None, None); + if c.len() < 6 || c[0] != "RG" { + continue; + } + let (Ok(start), Ok(end), Ok(mut length)) = ( + crate::int(c[3]), + crate::int(c[4]), + pycompat::num::parse_float(c[5]), + ) else { + continue; + }; + if !length.is_finite() { + return Err(format!( + "{} has a run of length {}", + crate::basename(&p.path), + c[5] + )); + } + if matches!(c[2], "chrX" | "chrY" | "X" | "Y") { + sex_segs += 1; + continue; + } + if !is_autosome(c[2]) { + continue; + } + let inside = length.min(masked_overlap(c[2], start, end) as f64); + masked += inside; + length -= inside; + if length <= 0.0 { + continue; + } + segs += 1; + total += length; + largest = largest.max(length); + if length >= ROH_MIN_BP { + let pc = per_chrom.entry(c[2].to_owned()).or_default(); + pc.0 += length; + pc.1.push((start, end)); + let mut row = Obj::new() + .set("region", s(&format!("{}:{}-{}", c[2], c[3], c[4]))) + .set("mb", r1(length / 1e6, 1)); + if inside != 0.0 { + row.put("masked_mb", r1(inside / 1e6, 1)); + } + big.push(row.val()); + } + } + let long_bp: f64 = per_chrom.values().fold(0.0, |a, v| a + v.0); + let f_roh = long_bp / autosome_unmasked_bp() as f64; + let mut upd = Value::Null; + // The first chromosome of the most bases, as max() picks. + let top = + per_chrom.iter().fold( + None::<(&String, &(f64, Vec<(i128, i128)>))>, + |best, kv| match best { + Some(b) if b.1.0 >= kv.1.0 => Some(b), + _ => Some(kv), + }, + ); + if let Some((chrom, (bp, runs))) = top + && *bp >= UPD_MIN_BP + && *bp > UPD_SHARE * long_bp + { + upd = Obj::new() + .set("chrom", s(chrom)) + .set("mb", r1(bp / 1e6, 1)) + .set("share", r1(bp / long_bp, 2)) + .set("imprinted", imprinted(chrom, runs)) + .val(); + } + let relationship = if upd == Value::Null { + s(roh_band(f_roh)) + } else { + Value::Null + }; + let data = Obj::new() + .set("segments", count(segs)) + .set("total_mb", r1(total / 1e6, 1)) + .set("largest_mb", r1(largest / 1e6, 1)) + .set("autosomal_over_5mb", Value::Array(big)) + .set("masked_mb", r1(masked / 1e6, 1)) + .set("sex_chromosome_segments", count(sex_segs)) + .set("long_mb", r1(long_bp / 1e6, 1)) + .set("f_roh", r1(f_roh, 4)) + .set("relationship", relationship) + .set("upd_flag", upd); + Ok(Some((p, data.val()))) +} + +// --- og roh summary ------------------------------------------------------------ + +const USAGE: &str = "usage: og roh summary --sample SAMPLE --roh ROH [--out OUT]\n"; + +/// A field as awk reads it as a number: the longest leading decimal +/// number, else 0. +fn awk_num(x: &str) -> f64 { + let b = x.as_bytes(); + let mut i = 0; + if matches!(b.first(), Some(b'+' | b'-')) { + i += 1; + } + let digits = |i: &mut usize| { + let s = *i; + while b.get(*i).is_some_and(u8::is_ascii_digit) { + *i += 1; + } + *i - s + }; + let mut n = digits(&mut i); + if b.get(i) == Some(&b'.') { + i += 1; + n += digits(&mut i); + } + if n == 0 { + return 0.0; + } + let mantissa_end = i; + if matches!(b.get(i), Some(b'e' | b'E')) { + let mut j = i + 1; + if matches!(b.get(j), Some(b'+' | b'-')) { + j += 1; + } + if digits(&mut j) > 0 { + i = j; + } else { + i = mantissa_end; + } + } + x[..i].parse().unwrap_or(0.0) +} + +/// The rows of step 11's summary: every run on chr1 to chr22 whose part +/// outside the masked bands is 5 Mb or more, with that part's length. +pub fn summary_rows(roh: &str) -> Result { + let mut out = String::new(); + for line in lines(roh)? { + // awk's fields: runs of blanks, leading ones dropped. + let f: Vec<&str> = line.split([' ', '\t']).filter(|x| !x.is_empty()).collect(); + if f.len() < 3 || f[0] != "RG" || !is_autosome(f[2]) { + continue; + } + let get = |i: usize| f.get(i).copied().unwrap_or(""); + let (start, end, len) = (awk_num(get(3)), awk_num(get(4)), awk_num(get(5))); + let mut m = 0.0; + for (c, a, b) in MASKED_BANDS { + if c == f[2] { + let o = end.min(b as f64) - (start - 1.0).max(a as f64); + if o > 0.0 { + m += o; + } + } + } + if m > len { + m = len; + } + let l = len - m; + if l >= 5_000_000.0 { + out.push_str(&format!( + "{}\t{}\t{}\t{}\t{}\n", + f[2], + get(3), + get(4), + l.trunc() as i64, + pycompat::fmt::fixed(l / 1e6, 1) + )); + } + } + Ok(out) +} + +/// `og roh summary`: step 11's summary table of the long runs, for the bash +/// step and the ROH module alike, into `--out` or on stdout. Exit 0 when +/// written, 1 when the roh file cannot be read or the table written, 2 for +/// a command line it cannot read. +pub fn cli(argv: &[String], out: &mut impl Write, err: &mut impl Write) -> u8 { + use crate::args::{Kind, Opt, parse}; + const OPTS: &[Opt] = &[ + Opt("--sample", Kind::Value, true), + Opt("--roh", Kind::Value, true), + Opt("--out", Kind::Value, false), + ]; + let a = match parse(argv, OPTS) { + Ok(a) => a, + Err(e) => { + let _ = write!(err, "{USAGE}"); + let _ = writeln!(err, "og roh summary: error: {e}"); + return 2; + } + }; + let mut run = || -> Result<(), String> { + let rows = summary_rows(a.text("--roh"))?; + let body = format!( + "# ROH Summary for {}\n# Segments >=5MB on autosomes (excludes chrX/chrY)\nchrom\tstart\tend\tlength_bp\tlength_mb\n{rows}", + a.text("--sample") + ); + match a.get("--out") { + Some(path) => crate::write_file(path, body.as_bytes()), + None => out + .write_all(body.as_bytes()) + .and_then(|()| out.flush()) + .map_err(|e| format!("stdout: {e}")), + } + }; + crate::finish(run(), err) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::testutil::Dir; + + #[test] + fn awk_reads_the_leading_number() { + assert_eq!(awk_num("7000000"), 7_000_000.0); + assert_eq!(awk_num("8000000.5"), 8_000_000.5); + assert_eq!(awk_num("1e3x"), 1000.0); + assert_eq!(awk_num("1e"), 1.0); + assert_eq!(awk_num("x"), 0.0); + assert_eq!(awk_num("-2"), -2.0); + } + + #[test] + fn the_summary_keeps_the_unmasked_part_of_long_autosomal_runs() { + let d = Dir::new("roh-summary"); + let roh = d.put( + "roh.txt", + "# RG\nRG\tS\tchrY\t1\t9000000\t9000000\t9\t9\n\ + RG\tS\tchr1\t120000001\t145000000\t25000000\t9\t9\n\ + RG\tS\tchr2\t10000001\t17000000\t7000000\t9\t9\n\ + RG\tS\tchr13\t1\t25000000\t25000000\t9\t9\n\ + RG\tS\tchr5\t1\t4999999\t4999999\t9\t9\n", + ); + let argv: Vec = ["--sample", "S", "--roh", &roh] + .iter() + .map(|s| (*s).to_owned()) + .collect(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + assert_eq!(cli(&argv, &mut out, &mut err), 0); + assert_eq!( + String::from_utf8(out).unwrap(), + "# ROH Summary for S\n# Segments >=5MB on autosomes (excludes chrX/chrY)\n\ + chrom\tstart\tend\tlength_bp\tlength_mb\n\ + chr2\t10000001\t17000000\t7000000\t7.0\n\ + chr13\t1\t25000000\t6100000\t6.1\n" + ); + } + + #[test] + fn a_missing_roh_file_exits_1_and_no_sample_exits_2() { + let argv = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::>(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + assert_eq!( + cli( + &argv(&["--sample", "S", "--roh", "/no/such"]), + &mut out, + &mut err + ), + 1 + ); + assert_eq!(cli(&argv(&["--roh", "x"]), &mut out, &mut err), 2); + } +} diff --git a/rust/crates/og-summary/src/collect/group_c/sv.rs b/rust/crates/og-summary/src/collect/group_c/sv.rs new file mode 100644 index 000000000..4bcf320d3 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/sv.rs @@ -0,0 +1,466 @@ +//! Structural variants: Manta's and Delly's record counts, CNVpytor's +//! calls with the significance and quality filters, step 22's consensus +//! count, and step 05's class 4 and 5 calls with their support from the +//! consensus. + +use super::roh::masked_overlap; +use super::util::{Obj, at, base, cells, count, int, lines, number, s, strs, table, vcf_lines}; +use super::{Ctx, Found, Out}; +use pycompat::csv::DictRow; +use pycompat::json::Value; +use std::collections::HashMap; + +/// A record count and the records that PASS. +fn count_vcf(path: &str) -> Result<(usize, usize), String> { + let (mut total, mut pass) = (0, 0); + vcf_lines(path, |l| { + total += 1; + if l.split(|&c| c == b'\t').nth(6) == Some(b"PASS") { + pass += 1; + } + })?; + Ok((total, pass)) +} + +fn totals(p: Found) -> Out { + let (t, ps) = count_vcf(&p.path)?; + Ok(Some(( + p, + Obj::new() + .set("total", count(t)) + .set("pass", count(ps)) + .val(), + ))) +} + +/// Step 04 (Manta). +pub fn manta(c: &Ctx<'_>) -> Out { + match c.find(&[ + "manta/results/variants/diploidSV.vcf.gz".to_owned(), + "manta/*diploidSV.vcf.gz".to_owned(), + "manta/**/diploidSV.vcf.gz".to_owned(), + ]) { + Some(p) => totals(p), + None => Ok(None), + } +} + +/// Step 19 (Delly). +pub fn delly(c: &Ctx<'_>) -> Out { + match c.find1(format!("delly/{}_sv.vcf.gz", c.s())) { + Some(p) => totals(p), + None => Ok(None), + } +} + +/// Step 22's consensus count. +pub fn sv_consensus(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("sv_merged/{}_sv_consensus.vcf.gz", c.s())) else { + return Ok(None); + }; + let (t, _) = count_vcf(&p.path)?; + Ok(Some((p, Obj::new().set("consensus", count(t)).val()))) +} + +/// A CNVpytor call is kept in the filtered count when fewer than half of +/// its reads have mapping quality 0 and fewer than half of its bases are N, +/// and it touches no masked band. +const CNV_MAX_Q0: f64 = 0.5; +const CNV_MAX_PN: f64 = 0.5; + +/// `(\S+):(\d+)-(\d+)` over a whole cell. +fn span(x: &str) -> Option<(&str, i128, i128)> { + let (c, rest) = x.rsplit_once(':')?; + let (a, b) = rest.split_once('-')?; + let digits = |v: &str| !v.is_empty() && v.bytes().all(|d| d.is_ascii_digit()); + if c.is_empty() || c.chars().any(char::is_whitespace) || !digits(a) || !digits(b) { + return None; + } + Some((c, a.parse().ok()?, b.parse().ok()?)) +} + +/// Step 18 (CNVpytor): the calls, the significant ones (e-value under +/// 0.01), and those kept after the quality and band filters. +pub fn cnvpytor(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("cnvpytor/{}_cnvs.txt", c.s())) else { + return Ok(None); + }; + let (mut total, mut dels, mut dups, mut sig, mut kept, mut kept_y) = (0, 0, 0, 0, 0, 0); + let (mut q0s, mut gaps, mut bands) = (0, 0, 0); + for line in lines(&p.path)? { + let c = pycompat::text::split(&line, None, None); + if c.is_empty() { + continue; + } + total += 1; + dels += usize::from(c[0] == "deletion"); + dups += usize::from(c[0] == "duplication"); + let m = if c.len() > 4 { span(c[1]) } else { None }; + match c.get(4).map(|v| pycompat::num::parse_float(v)) { + Some(Ok(e)) if e.partial_cmp(&0.01).is_none_or(|o| o.is_lt()) => {} + _ => continue, + } + sig += 1; + let (q0, pn) = if c.len() > 9 { + (number(c[8]), number(c[9])) + } else { + (None, None) + }; + if q0.is_some_and(|q| q >= CNV_MAX_Q0) { + q0s += 1; + } else if pn.is_some_and(|q| q >= CNV_MAX_PN) { + gaps += 1; + } else if let Some((ch, a, b)) = m + && masked_overlap(ch, a, b) > 0 + { + bands += 1; + } else { + kept += 1; + kept_y += usize::from(m.is_some_and(|(ch, ..)| ch == "chrY" || ch == "Y")); + } + } + let data = Obj::new() + .set("total", count(total)) + .set("deletions", count(dels)) + .set("duplications", count(dups)) + .set("significant", count(sig)) + .set("significant_filtered", count(kept)) + .set("filtered_chry", count(kept_y)) + .set( + "dropped", + Obj::new() + .set("q0", count(q0s)) + .set("gap", count(gaps)) + .set("band", count(bands)) + .val(), + ); + Ok(Some((p, data.val()))) +} + +/// Takes the chrY calls out of the filtered count of a declared female; +/// with no declared sex they stay, and the data says so. +pub fn cnvpytor_sex_filter(v: &mut Value, declared_sex: Option<&str>) { + let Value::Object(m) = v else { return }; + let (Some(Value::Int(kept)), Some(Value::Int(y))) = ( + m.get("significant_filtered").cloned(), + m.get("filtered_chry").cloned(), + ) else { + return; + }; + let declared = declared_sex.filter(|x| !x.is_empty()); + m.insert("declared_sex".into(), declared.map_or(Value::Null, s)); + if declared == Some("female") { + if let Some(Value::Object(d)) = m.get_mut("dropped") { + d.insert("chry_female".into(), Value::Int(y)); + } + m.insert("significant_filtered".into(), Value::Int(kept - y)); + } +} + +// --- step 05, AnnotSV class 4 and 5 --------------------------------------------- + +/// Two breakpoints within this many bases of a consensus record's are the +/// same SV. +const CONSENSUS_BP: i128 = 1000; + +/// AnnotSV writes chromosomes without `chr`; step 22 writes them with it. +fn chrom_key(c: &str) -> &str { + if c.get(..3).is_some_and(|p| p.eq_ignore_ascii_case("chr")) { + &c[3..] + } else { + c + } +} + +fn info_value<'a>(info: &'a str, key: &str) -> &'a str { + info.split(';') + .find_map(|kv| { + let (k, v) = kv.split_once('=').unwrap_or((kv, "")); + (k == key).then_some(v) + }) + .unwrap_or("") +} + +fn kind(t: &str) -> String { + let t = pycompat::text::upper(t); + if t.starts_with("DUP") { + "DUP".into() + } else { + t + } +} + +struct Rec { + pos: i128, + end: i128, + supp: i128, + vec: String, +} + +/// Step 22's consensus: caller names, and the records by (chrom, type). +/// The header and the records of the consensus VCF by (chrom, id). +type Consensus = (Vec, HashMap<(String, String), Vec>); + +fn read_consensus(path: &str) -> Result { + let mut names = Vec::new(); + let mut by: HashMap<(String, String), Vec> = HashMap::new(); + for line in lines(path)? { + if line.starts_with("##") { + continue; + } + let c: Vec<&str> = line.split('\t').collect(); + if line.starts_with("#CHROM") { + names = c.iter().skip(9).map(|x| (*x).to_owned()).collect(); + continue; + } + if c.len() < 8 { + continue; + } + let Ok(pos) = crate::int(c[1]) else { continue }; + let end = match info_value(c[7], "END") { + "" => Ok(pos), + v => crate::int(v), + }; + let supp = match info_value(c[7], "SUPP") { + "" => Ok(0), + v => crate::int(v), + }; + let (Ok(end), Ok(supp)) = (end, supp) else { + continue; + }; + by.entry((chrom_key(c[0]).to_owned(), kind(info_value(c[7], "SVTYPE")))) + .or_default() + .push(Rec { + pos, + end, + supp, + vec: info_value(c[7], "SUPP_VEC").to_owned(), + }); + } + Ok((names, by)) +} + +/// Support and the supporting callers of each row from the consensus: the +/// record of the same type whose two breakpoints lie within 1,000 bp (the +/// nearest, the first of a tie), else 1 and the row's own caller; `None` +/// for both without a consensus file. +fn consensus_support(rows: &[DictRow], path: Option<&str>) -> Result, String> { + let Some(path) = path.filter(|p| crate::is_file(p)) else { + return Ok(rows.iter().map(|_| (Value::Null, Value::Null)).collect()); + }; + let (names, by) = read_consensus(path)?; + let mut out = Vec::new(); + for r in rows { + let cell = |k: &str| r.get(k).flatten().unwrap_or_default(); + let mut best: Option<(i128, &Rec)> = None; + if let (Ok(start), Ok(end)) = (crate::int(cell("SV_start")), crate::int(cell("SV_end"))) { + let key = ( + chrom_key(cell("SV_chrom")).to_owned(), + kind(cell("SV_type")), + ); + for rec in by.get(&key).map_or(&[][..], Vec::as_slice) { + let (dp, de) = ((rec.pos - start).abs(), (rec.end - end).abs()); + if dp <= CONSENSUS_BP && de <= CONSENSUS_BP && best.is_none_or(|(d, _)| dp + de < d) + { + best = Some((dp + de, rec)); + } + } + } + out.push(match best { + Some((_, rec)) => { + let callers: Vec = rec + .vec + .chars() + .enumerate() + .filter(|&(_, x)| x == '1') + .map(|(i, _)| { + names + .get(i) + .cloned() + .unwrap_or_else(|| format!("caller {}", i + 1)) + }) + .collect(); + (int(rec.supp), s(&callers.join(","))) + } + None => (int(1), s(&super::util::or(r, "Caller", "."))), + }); + } + Ok(out) +} + +/// `{FORMAT key: value}` of the sample column (the column after FORMAT, +/// unless it is `Annotation_mode`). +fn sample_format(r: &DictRow, head: &[String]) -> HashMap { + let keys: Vec<&str> = r.iter().map(|(k, _)| k).collect(); + let _ = head; + let Some(i) = keys.iter().position(|k| *k == "FORMAT").map(|i| i + 1) else { + return HashMap::new(); + }; + let Some(col) = keys.get(i).filter(|k| **k != "Annotation_mode") else { + return HashMap::new(); + }; + let fmt = r.get("FORMAT").flatten().unwrap_or_default(); + let val = r.get(col).flatten().unwrap_or_default(); + fmt.split(':') + .zip(val.split(':')) + .map(|(k, v)| (k.to_owned(), v.to_owned())) + .collect() +} + +/// Step 05's class 4 and 5 SVs, each with the callers that support it from +/// step 22's consensus; a call step 05's rules moved to class 3 listed +/// apart. +pub fn annotsv(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("annotsv/{}_sv_class4_5_checked.tsv", c.s())) else { + return Ok(None); + }; + let size = std::fs::metadata(&p.path).map_or(0, |m| m.len()); + let t = if size > 0 { + table(&p.path, false)? + } else { + super::util::Table { + head: Vec::new(), + rows: Vec::new(), + } + }; + let need = [ + "SV_chrom", + "SV_start", + "SV_end", + "SV_type", + "ACMG_class", + "Caller", + "Exon_edge", + "Population_AF", + "Common", + ]; + if !t.rows.is_empty() && !need.iter().all(|k| t.has(k)) { + let mut sorted = need.to_vec(); + sorted.sort_unstable(); + return Err(format!( + "{} lacks a column of [{}]", + base(&p.path), + sorted + .iter() + .map(|k| crate::repr(k)) + .collect::>() + .join(", ") + )); + } + let mut cons = c.find1(format!("sv_merged/{}_sv_consensus.vcf.gz", c.s())); + let mut cons_note = Value::Null; + let support = match consensus_support(&t.rows, cons.as_ref().map(|f| f.path.as_str())) { + Ok(x) => x, + Err(e) => { + let f = cons.take().map(|f| f.path).unwrap_or_default(); + let why = e.split_once(": ").map_or(e.as_str(), |x| x.1); + cons_note = s(&format!("{} could not be read: {why}", base(&f))); + consensus_support(&t.rows, None)? + } + }; + let mut keep = Vec::new(); + let mut moved = Vec::new(); + for (r, (sp, callers)) in t.rows.iter().zip(support) { + // A row cut short reads None there, as Python's DictReader gives it. + let cell = + |k: &str| -> Result { Ok(at(r, k)?.unwrap_or("None").to_owned()) }; + let cellv = |k: &str| -> Result { Ok(super::util::opt_s(at(r, k)?)) }; + let af = match at(r, "Population_AF")? { + None | Some("") => Value::Null, + Some(v) => pycompat::num::parse_float(v).map_or(Value::Null, Value::Float), + }; + let genes: Vec = super::util::or(r, "Gene_name", "") + .split(';') + .filter(|g| !g.is_empty()) + .map(str::to_owned) + .collect(); + let fmt = sample_format(r, &t.head); + let fmt_get = |k: &str| { + fmt.get(k) + .filter(|v| !v.is_empty()) + .map_or(Value::Null, |v| s(v)) + }; + let acmg = cell("ACMG_class")?; + let support_callers = match &callers { + Value::Str(x) if !x.is_empty() => strs(x.split(',')), + _ => Value::Array(Vec::new()), + }; + let row = Obj::new() + .set( + "sv", + s(&format!( + "{}:{}-{}", + cell("SV_chrom")?, + cell("SV_start")?, + cell("SV_end")? + )), + ) + .set("type", cellv("SV_type")?) + .set("gt", fmt_get("GT")) + .set("ps", fmt_get("PS")) + .set("length", s(&super::util::or(r, "SV_length", ""))) + .set("acmg_class", cellv("ACMG_class")?) + .set("genes", strs(&genes)) + .set("caller", cellv("Caller")?) + .set("support", sp) + .set("support_callers", support_callers) + .set("exon_edge", cellv("Exon_edge")?) + .set( + "exon_edge_detail", + s(&super::util::or(r, "Exon_edge_detail", "")), + ) + .set("population_af", af) + .set( + "population_source", + s(&super::util::or(r, "Population_source", "")), + ) + .set("common", Value::Bool(at(r, "Common")? == Some("yes"))) + .set( + "annotsv_class", + match r.get("AnnotSV_class").flatten().filter(|v| !v.is_empty()) { + Some(v) => s(v), + None => cellv("ACMG_class")?, + }, + ) + .set("rule", s(&super::util::or(r, "Rule", ""))) + .set("rule_detail", s(&super::util::or(r, "Rule_detail", ""))); + if matches!(acmg.as_str(), "4" | "5") { + keep.push(row.val()); + } else { + moved.push(row.val()); + } + } + let field = |v: &Value, k: &str| -> Value { + match v { + Value::Object(m) => m.get(k).cloned().unwrap_or(Value::Null), + _ => Value::Null, + } + }; + let single = keep + .iter() + .filter(|x| field(x, "support") == Value::Int(1)) + .count(); + let edge = keep + .iter() + .filter(|x| field(x, "exon_edge") == s("yes")) + .count(); + let edge_checked = !keep + .iter() + .any(|x| field(x, "exon_edge") == s("not checked")); + let common = keep + .iter() + .filter(|x| field(x, "common") == Value::Bool(true)) + .count(); + let data = Obj::new() + .set("count", count(keep.len())) + .set("rows", Value::Array(keep)) + .set("downgraded", Value::Array(moved)) + .set("consensus", cons.map_or(Value::Null, |f| s(&f.rel))) + .set("consensus_note", cons_note) + .set("single_caller", count(single)) + .set("exon_edge", count(edge)) + .set("exon_edge_checked", Value::Bool(edge_checked)) + .set("common", count(common)); + let _ = cells; + Ok(Some((p, data.val()))) +} diff --git a/rust/crates/og-summary/src/collect/group_c/tables.rs b/rust/crates/og-summary/src/collect/group_c/tables.rs new file mode 100644 index 000000000..d13e7d843 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/tables.rs @@ -0,0 +1,1560 @@ +//! The sections that read one step table each: the couple screen, the +//! polygenic scores, fine ancestry, telomeres, the haplogroups, the Y-STR +//! panel, the raw-data export, blood groups, archaic and ancient ancestry, +//! Lp(a), VNtyper, Locityper, viral DNA, CHIP, Exomiser, mobile elements +//! and MoChA. + +use super::super::model::State; +use super::util::{ + Obj, Table, at, base, count, dots, dots_strict, get_default, isdigit, kv_dot, kv_get, + metric_map, opt_s, or, or_dot, ragged, round, s, strs, table, +}; +use super::{Ctx, Found, Others, Out}; +use crate::collect::sections::{num, py_error}; +use indexmap::IndexMap; +use pycompat::json::Value; + +fn unreadable(msg: String) -> Result { + Err(msg) +} + +/// Python's `sorted(need)` of a set, as its list repr. +fn sorted_repr(need: &[&str]) -> String { + let mut v: Vec<&str> = need.to_vec(); + v.sort_unstable(); + format!( + "[{}]", + v.iter() + .map(|k| crate::repr(k)) + .collect::>() + .join(", ") + ) +} + +fn float_err(x: &str) -> String { + py_error( + "ValueError", + &format!("could not convert string to float: {}", crate::repr(x)), + ) +} + +fn int_of(x: &str) -> Result { + crate::int(x).map_err(|e| py_error("ValueError", &e)) +} + +fn opt_num(x: Option<&str>) -> Value { + num(x).map_or(Value::Null, Value::Float) +} + +/// `{k: r[k] or "."}` over the header's own first line, read as text. +fn first_line(path: &str) -> Result, String> { + let l = super::util::lines(path)?; + let h = l.first().map_or("", String::as_str); + Ok(h.trim_end_matches('\r') + .split('\t') + .map(str::to_owned) + .collect()) +} + +/// Base64 of a file's bytes (`base64.b64encode`). +fn b64(bytes: &[u8]) -> String { + const T: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let mut out = String::with_capacity(bytes.len().div_ceil(3) * 4); + for ch in bytes.chunks(3) { + let b = [ch[0], *ch.get(1).unwrap_or(&0), *ch.get(2).unwrap_or(&0)]; + let n = (u32::from(b[0]) << 16) | (u32::from(b[1]) << 8) | u32::from(b[2]); + for i in 0..4 { + if i <= ch.len() { + out.push(T[((n >> (18 - 6 * i)) & 63) as usize] as char); + } else { + out.push('='); + } + } + } + out +} + +/// The base64 of a picture under `limit` bytes, else `null`. +fn picture(f: Option, limit: u64) -> Result { + let Some(f) = f else { return Ok(Value::Null) }; + let size = std::fs::metadata(&f.path) + .map_err(|e| super::util::os_note(&crate::os_error(&e, &f.path)))? + .len(); + if size >= limit { + return Ok(Value::Null); + } + let bytes = + std::fs::read(&f.path).map_err(|e| super::util::os_note(&crate::os_error(&e, &f.path)))?; + Ok(s(&b64(&bytes))) +} + +// --- step 39 -------------------------------------------------------------------- + +pub fn couple(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("couple/{}_couple_carrier.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, true)?; + let need = [ + "gene", + "condition", + "result", + "offspring_risk", + "partner_a", + "findings_a", + "partner_b", + "findings_b", + "explanation", + ]; + if need.iter().any(|k| !t.has(k)) { + return unreadable(format!( + "{} lacks a column of {}", + base(&p.path), + need.join(", ") + )); + } + let sample = c.sample; + let cell = |r: &pycompat::csv::DictRow, k: &str| r.get(k).flatten().unwrap_or("").to_owned(); + for (i, r) in t.rows.iter().enumerate() { + let bad = ragged(r, &need) + || (cell(r, "partner_a") != sample && cell(r, "partner_b") != sample) + || !matches!( + cell(r, "result").as_str(), + "shared" | "x_linked" | "one_partner" + ); + if bad { + return unreadable(format!( + "{} line {} is not a row of {sample}'s couple table", + base(&p.path), + i + 2 + )); + } + } + let side = |r: &pycompat::csv::DictRow| { + let (own, other) = if cell(r, "partner_a") == sample { + ("a", "b") + } else { + ("b", "a") + }; + Obj::new() + .set("gene", s(&cell(r, "gene"))) + .set("condition", s(&cell(r, "condition"))) + .set("risk", s(&cell(r, "offspring_risk"))) + .set("own", s(&cell(r, &format!("findings_{own}")))) + .set( + "partner_findings", + s(&cell(r, &format!("findings_{other}"))), + ) + .set("explanation", s(&cell(r, "explanation"))) + .val() + }; + let partner = t.rows.first().map(|r| { + if cell(r, "partner_a") == sample { + cell(r, "partner_b") + } else { + cell(r, "partner_a") + } + }); + let of = |kind: &str| { + Value::Array( + t.rows + .iter() + .filter(|r| cell(r, "result") == kind) + .map(side) + .collect(), + ) + }; + let v = Obj::new() + .set("partner", opt_s(partner.as_deref())) + .set("shared", of("shared")) + .set("x_linked", of("x_linked")) + .set( + "one_partner", + count( + t.rows + .iter() + .filter(|r| cell(r, "result") == "one_partner") + .count(), + ), + ) + .val(); + Ok(Some((p, v))) +} + +// --- step 25 -------------------------------------------------------------------- + +pub fn prs(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("prs/{s_}_prs_summary.tsv")) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let mut rows = Vec::new(); + let mut adjusted = false; + for r in &t.rows { + let pct = or(r, "Percentile", ""); + let pct = if pct == "NA" { String::new() } else { pct }; + adjusted |= !pct.is_empty(); + let group = match r.get("Ancestry_Group") { + None => s(""), + Some(Some("NA")) => s(""), + Some(v) => opt_s(v), + }; + let evaluated = match r.get("Evaluated_In") { + Some(Some(x)) => strs(x.split(';').filter(|x| !x.is_empty())), + _ => Value::Null, + }; + rows.push( + Obj::new() + .set("condition", get_default(r, "Condition", "")) + .set("pgs_id", get_default(r, "PGS_ID", "")) + .set("score", get_default(r, "Score_SUM", "")) + .set("matched", get_default(r, "Variants_Matched", "")) + .set("total", get_default(r, "Variants_Total", "")) + .set("chrx", get_default(r, "ChrX", "")) + .set("ambiguous", get_default(r, "Ambiguous", "")) + .set("applies_to", get_default(r, "Applies_To", "")) + .set("sex_check", get_default(r, "Sex_Check", "")) + .set("percentile", s(&pct)) + .set("group", group) + .set("evaluated_in", evaluated) + .set("effect", s(&or(r, "Effect", ""))) + .val(), + ); + } + let mut out = Obj::new() + .set("scores", Value::Array(rows)) + .set("adjusted", Value::Bool(adjusted)); + if let Some(a) = c.find(&[ + format!("ancestry/{s_}_ancestry.tsv"), + format!("prs/{s_}_ancestry.tsv"), + ]) { + let mut kv: IndexMap = IndexMap::new(); + for r in table(&a.path, false)?.rows { + if let Some(Some(k)) = r.get("key") + && !k.is_empty() + { + kv.insert(k.to_owned(), get_default(&r, "value", "")); + } + } + let g = |k: &str| kv.get(k).cloned().unwrap_or_else(|| s("")); + out.put( + "ancestry", + Obj::new() + .set("population", g("population")) + .set("panel", g("reference_panel")) + .set("low_confidence", g("population_low_confidence")) + .val(), + ); + } + Ok(Some((p, out.val()))) +} + +// --- step 26b ------------------------------------------------------------------- + +pub fn ancestry_fine(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("ancestry_fine/{s_}_ancestry_fine.tsv")) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let need = ["level", "code", "group", "fraction", "se", "low", "high"]; + if !t.first_has(&need) { + return unreadable(format!( + "{} has no rows or lacks a column of {}", + base(&p.path), + sorted_repr(&need) + )); + } + let levels = ["continent", "region", "painting", "left_out", "not_painted"]; + let mut by: IndexMap<&str, Vec> = levels.iter().map(|l| (*l, Vec::new())).collect(); + for r in &t.rows { + let Some(Some(level)) = r.get("level") else { + continue; + }; + if let Some(list) = by.get_mut(level) { + list.push( + Obj::new() + .set("code", opt_s(at(r, "code")?)) + .set("group", opt_s(at(r, "group")?)) + .set("fraction", opt_num(at(r, "fraction")?)) + .set("low", opt_num(at(r, "low")?)) + .set("high", opt_num(at(r, "high")?)) + .val(), + ); + } + } + let mut out = Obj::new(); + for (k, v) in by { + out.put(k, Value::Array(v)); + } + out.put("min_references", count(og_ancestry::MIN_REFERENCES)); + let png = c.find1(format!("ancestry_fine/{s_}_painting.png")); + out.put("painting_png", Value::Null); + out.put( + "legend", + Value::Array( + og_ancestry::COLOURS + .iter() + .map(|(code, rgb)| { + Obj::new() + .set("code", s(code)) + .set( + "group", + s(og_ancestry::label(og_ancestry::Level::Continent, code)), + ) + .set( + "colour", + s(&format!("#{:02x}{:02x}{:02x}", rgb[0], rgb[1], rgb[2])), + ) + .val() + }) + .collect(), + ), + ); + out.put("painting_png", picture(png, 2_000_000)?); + Ok(Some((p, out.val()))) +} + +// --- step 44 -------------------------------------------------------------------- + +pub fn ancient(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("ancient/{s_}_ancient.tsv")) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let need = [ + "rank", "group", "n", "distance", "dates", "country", "locality", + ]; + if !t.first_has(&need) { + return unreadable(format!( + "{} has no rows or lacks a column of {}", + base(&p.path), + sorted_repr(&need) + )); + } + let info = match c.find1(format!("ancient/{s_}_ancient_info.tsv")) { + Some(ip) => metric_map(&table(&ip.path, false)?, "key", "value"), + None => IndexMap::new(), + }; + let nearest = t + .rows + .iter() + .map(|r| { + need.iter() + .fold(Obj::new(), |o, k| o.set(k, opt_s(r.get(k).flatten()))) + .val() + }) + .collect(); + let mut out = Obj::new().set("nearest", Value::Array(nearest)); + for k in [ + "release", + "sites_used", + "panel_sites", + "ancient_people", + "ancient_groups", + "present_day_people", + ] { + out.put(k, s(&kv_dot(&info, k))); + } + let svg = c.find1(format!("ancient/{s_}_ancient_pca.svg")); + out.put("plot_svg", picture(svg, 3_000_000)?); + Ok(Some((p, out.val()))) +} + +// --- step 10 -------------------------------------------------------------------- + +const GENOME_BASES: f64 = 3.1e9; +const TELOMERE_FLOOR_PER_30X: f64 = 2000.0; + +pub fn telomere(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find(&[ + format!("telomere/{s_}/{s_}/{s_}_summary.tsv"), + format!("telomere/{s_}/{s_}_summary.tsv"), + "telomere/**/*_summary.tsv".to_owned(), + ]) else { + return Ok(None); + }; + let lines: Vec> = super::util::lines(&p.path)? + .into_iter() + .filter(|l| !pycompat::num::py_strip(l).is_empty()) + .map(|l| l.split('\t').map(str::to_owned).collect()) + .collect(); + if lines.len() < 2 { + return Ok(Some((p, Obj::new().set("tel_content", s(".")).val()))); + } + let (head, row) = (&lines[0], &lines[1]); + let i = head.iter().position(|h| h == "tel_content").unwrap_or(10); + let mut out = Obj::new().set("tel_content", s(row.get(i).map_or(".", String::as_str))); + let col = |names: &[&str]| -> Option<&str> { + names.iter().find_map(|n| { + head.iter() + .position(|h| h == n) + .and_then(|i| row.get(i)) + .map(String::as_str) + }) + }; + let intratel = num(col(&["intratel_reads", "intratelomeric_reads"])); + let total = num(col(&["total_reads"])); + let lengths: Vec = col(&["read_lengths", "read_length"]) + .unwrap_or("") + .split([',', ';', '|', ' ']) + .filter_map(|x| num(Some(x))) + .filter(|x| *x != 0.0) + .collect(); + let Some(intratel) = intratel else { + out.put( + "read_check", + s("no intratelomeric read count in the summary"), + ); + return Ok(Some((p, out.val()))); + }; + out.put( + "intratelomeric_reads", + Value::Int(crate::trunc(intratel).map_err(|e| py_error("OverflowError", &e))?), + ); + let total = total.filter(|t| *t != 0.0); + let (Some(total), false) = (total, lengths.is_empty()) else { + out.put( + "read_check", + s("depth unknown (no total_reads or read_lengths in the summary): not checked"), + ); + return Ok(Some((p, out.val()))); + }; + let longest = lengths.iter().copied().fold(f64::NEG_INFINITY, f64::max); + let depth = total * longest / GENOME_BASES; + let floor = TELOMERE_FLOOR_PER_30X * depth / 30.0; + out.put("depth", round(depth, 1)); + let fl = pycompat::fmt::round_int(floor) + .map_err(|e| py_error(e.exception(), "cannot round")) + .and_then(|x| crate::trunc(x).map_err(|e| py_error("OverflowError", &e)))?; + out.put("read_floor", Value::Int(fl)); + out.put("low_reads", Value::Bool(intratel < floor)); + Ok(Some((p, out.val()))) +} + +// --- steps 12 and 37 ------------------------------------------------------------ + +const EXPLORE_DOCS: &str = "https://github.com/GeiserX/OwnGenome/blob/main/docs/"; + +/// `urllib.parse.quote(x, safe="")`. +fn quote(x: &str) -> String { + let mut out = String::new(); + for b in x.bytes() { + if b.is_ascii_alphanumeric() || matches!(b, b'_' | b'.' | b'-' | b'~') { + out.push(b as char); + } else { + out.push_str(&format!("%{b:02X}")); + } + } + out +} + +fn explore(kind: &str, hg: &str) -> Value { + let mut hg = pycompat::num::py_strip(hg); + if kind == "y" { + hg = hg.split('*').next().unwrap_or(""); + } + if matches!(hg, "" | "." | "NA" | "insufficient markers") { + return Value::Array(Vec::new()); + } + let q = quote(hg); + let link = |name: &str, url: String| Obj::new().set("name", s(name)).set("url", s(&url)).val(); + let (ftdna, yfull, doc) = if kind == "y" { + ( + format!("https://discover.familytreedna.com/y-dna/{q}"), + ("YFull YTree", format!("https://www.yfull.com/tree/{q}/")), + "37-y-haplogroup.md#explore-your-branch-further", + ) + } else { + ( + format!("https://discover.familytreedna.com/mtdna/{q}"), + ("YFull MTree", format!("https://www.yfull.com/mtree/{q}/")), + "12-mito-haplogroup.md#explore-your-branch-further", + ) + }; + Value::Array(vec![ + link("FamilyTreeDNA Discover (free, no upload)", ftdna), + link(yfull.0, yfull.1), + link( + "More routes, and the files to share", + format!("{EXPLORE_DOCS}{doc}"), + ), + ]) +} + +pub fn haplogroup(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("mito/{s_}_haplogroup.txt")) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let hg = match t.rows.first() { + None => String::new(), + Some(r) => { + let named = r.get("Haplogroup").flatten().filter(|x| !x.is_empty()); + let second = || { + r.iter() + .nth(1) + .and_then(|(_, v)| v) + .filter(|x| !x.is_empty()) + }; + named + .or_else(second) + .unwrap_or("") + .trim_matches('"') + .to_owned() + } + }; + let mut out = Obj::new() + .set("haplogroup", s(if hg.is_empty() { "." } else { &hg })) + .set("explore", explore("mt", &hg)); + if let Some(hc) = c.find1(format!("mito/{s_}_haplocheck.txt")) { + let t = table(&hc.path, false)?; + let first: Option> = t.rows.first().map(|r| { + r.iter() + .filter(|(k, _)| !k.is_empty()) + .map(|(k, v)| { + ( + k.trim_matches('"').to_owned(), + v.unwrap_or("").trim_matches('"').to_owned(), + ) + }) + .collect() + }); + let Some(first) = first.filter(|m| m.contains_key("Contamination Status")) else { + return unreadable(format!( + "{} has no Contamination Status column", + base(&hc.path) + )); + }; + let dot = |k: &str| { + first + .get(k) + .filter(|v| !v.is_empty()) + .map_or(".", String::as_str) + .to_owned() + }; + out.put("contamination_status", s(&dot("Contamination Status"))); + out.put("contamination_level", s(&dot("Contamination Level"))); + } + Ok(Some((p, out.val()))) +} + +pub fn y_haplogroup(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("y_haplogroup/{}_y_haplogroup.txt", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let Some(r) = t.rows.first().filter(|_| t.has("Hg")) else { + return unreadable(format!( + "{} has no Hg column or no sample row", + base(&p.path) + )); + }; + let hg = pycompat::num::py_strip(r.get("Hg").flatten().unwrap_or("")).to_owned(); + let v = Obj::new() + .set( + "haplogroup", + s(if hg.is_empty() || hg == "NA" { + "insufficient markers" + } else { + &hg + }), + ) + .set("valid_markers", s(&or_dot(r, "Valid_markers"))) + .set("qc_score", s(&or_dot(r, "QC-score"))) + .set("explore", explore("y", &hg)) + .val(); + Ok(Some((p, v))) +} + +pub fn y_tree(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("y_haplogroup/{s_}_y_tree.txt")) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let need = ["Hg", "Level", "Tree"]; + let Some(r) = t.rows.first().filter(|_| need.iter().all(|k| t.has(k))) else { + return unreadable(format!( + "{} lacks a column of {}", + base(&p.path), + need.join(", ") + )); + }; + let hg = pycompat::num::py_strip(r.get("Hg").flatten().unwrap_or("")).to_owned(); + let mut out = Obj::new().set( + "haplogroup", + s(if hg.is_empty() || hg == "NA" { + "no branch has a derived call" + } else { + &hg + }), + ); + for (k, col) in [ + ("level", "Level"), + ("tree", "Tree"), + ("path_derived", "Path_derived"), + ("path_ancestral", "Path_ancestral"), + ("path_no_call", "Path_no_call"), + ("ancestral_in_terminal", "Ancestral_in_terminal"), + ("outvoted_ancestral", "Outvoted_ancestral"), + ("snps_called", "SNPs_called"), + ("min_depth", "Min_depth"), + ] { + out.put(k, s(&or_dot(r, col))); + } + out.put("explore", explore("y", &hg)); + if let Some(lvl) = c.find1(format!("y_haplogroup/{s_}_y_tree_path.tsv")) { + let pt = table(&lvl.path, false)?; + if !pt.has("Branch") { + return unreadable(format!("{} has no Branch column", base(&lvl.path))); + } + let keys = [ + "Level", + "Branch", + "SNPs", + "On_tables", + "Derived", + "Ancestral", + "No_call", + "Ancestral_SNPs", + ]; + out.put( + "path", + Value::Array( + pt.rows + .iter() + .map(|row| { + keys.iter() + .fold(Obj::new(), |o, k| o.set(k, s(&or(row, k, "")))) + .val() + }) + .collect(), + ), + ); + } + Ok(Some((p, out.val()))) +} + +// --- step 45 -------------------------------------------------------------------- + +pub fn y_str(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("y_str/{}_y_str.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let need = ["order", "panel", "marker", "count", "confidence"]; + if !t.first_has(&need) { + return unreadable(format!( + "{} has no rows or lacks a column of {}", + base(&p.path), + need.join(", ") + )); + } + let keep: Vec> = t + .rows + .iter() + .map(|r| need.iter().map(|k| or(r, k, "")).collect()) + .collect(); + let kinds = ["high", "low", "no call", "not callable"]; + let counts: Vec = kinds + .iter() + .map(|c| keep.iter().filter(|r| r[4] == *c).count()) + .collect(); + if counts.iter().sum::() != keep.len() { + return unreadable(format!( + "{} has a confidence other than high, low, no call or not callable", + base(&p.path) + )); + } + let mut out = Obj::new().set( + "markers", + Value::Array( + keep.iter() + .map(|r| { + need.iter() + .zip(r) + .fold(Obj::new(), |o, (k, v)| o.set(k, s(v))) + .val() + }) + .collect(), + ), + ); + for (k, n) in kinds.iter().zip(counts) { + out.put(&k.replace(' ', "_"), count(n)); + } + Ok(Some((p, out.val()))) +} + +// --- step 40 -------------------------------------------------------------------- + +pub fn consumer_export(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("consumer_export/{}_consumer_export.tsv", c.s())) else { + return Ok(None); + }; + let mut data: IndexMap> = IndexMap::new(); + for r in table(&p.path, false)?.rows { + if let Some(Some(k)) = r.get("key") + && !k.is_empty() + { + let v = match r.get("value") { + None => Some(String::new()), + Some(v) => v.map(str::to_owned), + }; + data.insert(k.to_owned(), v); + } + } + let text = |k: &str| data.get(k).cloned().flatten().unwrap_or_default(); + for k in ["sites", "called", "no_call"] { + if !isdigit(&text(k)) { + return unreadable(format!("{} has no whole-number '{k}' row", base(&p.path))); + } + } + let mut reasons = Obj::new(); + for (k, v) in &data { + if let Some(why) = k.strip_prefix("no_call: ") { + let v = v.clone().unwrap_or_default(); + if !isdigit(&v) { + return unreadable(format!( + "{} has a '{k}' row without a whole-number count", + base(&p.path) + )); + } + reasons.put(why, Value::Int(int_of(&v)?)); + } + } + let blocks = text("from_reference_blocks"); + let blocks = if blocks.is_empty() { + 0 + } else { + int_of(&blocks)? + }; + let dot = |k: &str| { + let v = text(k); + s(if v.is_empty() { "." } else { &v }) + }; + let v = Obj::new() + .set("sites", Value::Int(int_of(&text("sites"))?)) + .set("called", Value::Int(int_of(&text("called"))?)) + .set("no_call", Value::Int(int_of(&text("no_call"))?)) + .set("from_reference_blocks", Value::Int(blocks)) + .set("template_sites", dot("template_sites")) + .set("file_23andme", dot("file_23andme")) + .set("file_ancestrydna", dot("file_ancestrydna")) + .set("no_call_reasons", reasons.val()) + .val(); + Ok(Some((p, v))) +} + +// --- step 41 -------------------------------------------------------------------- + +pub fn bloodtype(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("bloodtype/{}_bloodtype.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + if !t.first_has(&["system", "phenotype", "confidence"]) { + return unreadable(format!( + "{} has no system, phenotype and confidence columns, or no row", + base(&p.path) + )); + } + let mut by: IndexMap, String> = IndexMap::new(); + let mut systems = Vec::new(); + for r in &t.rows { + let system = r.get("system").flatten(); + let pheno = or_dot(r, "phenotype"); + by.insert(system.map(str::to_owned), pheno.clone()); + systems.push( + Obj::new() + .set("system", opt_s(system)) + .set("phenotype", s(&pheno)) + .set("genotype", s(&or_dot(r, "genotype"))) + .set("confidence", s(&or_dot(r, "confidence"))) + .val(), + ); + } + let get = |k: &str| s(by.get(&Some(k.to_owned())).map_or(".", String::as_str)); + let v = Obj::new() + .set("systems", Value::Array(systems)) + .set("abo", get("ABO")) + .set("rhd", get("RhD")) + .val(); + Ok(Some((p, v))) +} + +// --- step 43 -------------------------------------------------------------------- + +fn all_digits(x: &str) -> bool { + !x.is_empty() && x.bytes().all(|b| b.is_ascii_digit()) +} + +pub fn archaic(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("archaic/{}_archaic.tsv", c.s())) else { + return Ok(None); + }; + let mut data: IndexMap = IndexMap::new(); + for r in table(&p.path, false)?.rows { + if let Some(Some(k)) = r.get("key") + && !k.is_empty() + { + data.insert(k.to_owned(), or(&r, "value", "")); + } + } + let mut longest: Vec<(i128, &String)> = data + .iter() + .filter_map(|(k, v)| { + let n = k.strip_prefix("longest_").filter(|n| all_digits(n))?; + Some((crate::int(n).ok()?, v)) + }) + .collect(); + longest.sort_by_key(|x| x.0); + let mut out = Obj::new() + .set( + "population", + s(data.get("population").map_or(".", String::as_str)), + ) + .set("longest", strs(longest.iter().map(|x| x.1.as_str()))); + for k in [ + "neanderthal_segments", + "neanderthal_bp", + "denisovan_segments", + "denisovan_bp", + "analysed_bp", + ] { + let v = data.get(k).map_or("", String::as_str); + if !all_digits(v) { + return unreadable(format!("{} has no whole number for {k}", base(&p.path))); + } + out.put(k, Value::Int(int_of(v)?)); + } + for k in ["neanderthal_percent", "denisovan_percent"] { + let v = data.get(k).map_or("", String::as_str); + let ok = v + .split_once('.') + .is_some_and(|(a, b)| all_digits(a) && all_digits(b)); + if !ok { + return unreadable(format!("{} has no percentage for {k}", base(&p.path))); + } + out.put(k, s(v)); + } + Ok(Some((p, out.val()))) +} + +// --- step 46 -------------------------------------------------------------------- + +const LPA_RISK_ALLELES: [(&str, &str); 2] = [("rs10455872", "G"), ("rs3798220", "C")]; + +pub fn lpa(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("lpa/{s_}_lpa_kiv2.tsv")) else { + return Ok(None); + }; + let kv = metric_map(&table(&p.path, false)?, "metric", "value"); + if !matches!( + kv_get(&kv, "status"), + Some("estimated" | "too little depth" | "not assessed") + ) { + return unreadable(format!( + "{} has no status row (estimated, too little depth or not assessed)", + base(&p.path) + )); + } + let mut data = Obj::new(); + for k in [ + "status", + "kiv2_copies", + "ci95_low", + "ci95_high", + "class", + "kiv2_depth", + "flank_depth", + "mapq0_share", + "message", + "advice", + ] { + data.put(k, s(&kv_dot(&kv, k))); + } + let dir = p.path.rsplit_once('/').map_or("", |x| x.0); + data.put( + "null_allele", + super::cn::read_marker_table( + &crate::join(dir, &format!("{}_lpa_kiv2_4925.tsv", c.sample)), + "46", + "LPA_KIV2_4925", + ), + ); + let mut variants = Vec::new(); + let mut risk = 0i128; + let mut any_read = false; + if let Some(t) = c.find1(format!("traits/{s_}_traits.tsv")) { + for r in table(&t.path, false)?.rows { + let rsid = r.get("rsid").flatten(); + let Some(alt) = LPA_RISK_ALLELES + .iter() + .find(|(id, _)| Some(*id) == rsid) + .map(|x| x.1) + else { + continue; + }; + let called = r.get("status").flatten() == Some("called"); + let gt = r.get("genotype").flatten().unwrap_or(""); + let n = called.then(|| gt.split(['/', '|']).filter(|a| *a == alt).count() as i128); + if let Some(n) = n { + risk += n; + any_read = true; + } + variants.push( + Obj::new() + .set("rsid", opt_s(rsid)) + .set("genotype", s(&or_dot(&r, "genotype"))) + .set("status", s(&or_dot(&r, "status"))) + .set("risk_alleles", n.map_or(Value::Null, Value::Int)) + .val(), + ); + } + } + let class = kv_dot(&kv, "class"); + let mut reasons = Vec::new(); + if risk != 0 { + reasons.push(format!( + "{risk} risk allele{} at rs10455872 and rs3798220", + if risk != 1 { "s" } else { "" } + )); + } + if class == "low" { + reasons.push("a low KIV-2 repeat count".to_owned()); + } + let read = kv_dot(&kv, "status") == "estimated" || any_read; + let flag_text = if !reasons.is_empty() { + format!( + "Genetic signs of a high Lp(a): {}. Measure Lp(a) in blood.", + reasons.join(" and ") + ) + } else if read { + "No genetic sign of a high Lp(a) from these markers. They explain only part of it: the KIV-2 count explained 21% to 27% of the variation in a Danish population study.".to_owned() + } else { + "Not determined: neither the KIV-2 count nor the two risk variants could be read. Measure Lp(a) in blood.".to_owned() + }; + data.put("variants", Value::Array(variants)); + data.put("risk_alleles", Value::Int(risk)); + data.put("flag", s(if reasons.is_empty() { "." } else { "high" })); + data.put("flag_text", s(&flag_text)); + Ok(Some((p, data.val()))) +} + +// --- step 52 -------------------------------------------------------------------- + +const VNTYPER_STATUSES: [&str; 4] = ["positive", "possible", "negative", "not assessed"]; + +pub fn vntyper(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("vntyper/{}_vntyper.tsv", c.s())) else { + return Ok(None); + }; + let kv = metric_map(&table(&p.path, false)?, "metric", "value"); + if !kv_get(&kv, "status").is_some_and(|st| VNTYPER_STATUSES.contains(&st)) { + return unreadable(format!( + "{} has no status row ({})", + base(&p.path), + VNTYPER_STATUSES.join(", ") + )); + } + let v = [ + "status", + "confidence", + "variant", + "flag", + "alt_depth", + "vntr_mean_depth", + "coverage_qc", + "sensitivity", + "message", + "advice", + ] + .iter() + .fold(Obj::new(), |o, k| o.set(k, s(&kv_dot(&kv, k)))) + .val(); + Ok(Some((p, v))) +} + +// --- step 53 -------------------------------------------------------------------- + +const LOCITYPER_COLUMNS: [&str; 13] = [ + "locus", + "region", + "status", + "genotype", + "haplotype_1", + "haplotype_2", + "quality", + "total_reads", + "unexplained_reads", + "weighted_distance", + "warnings", + "filter", + "reason", +]; + +pub fn locityper(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("locityper/{}_locityper.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + if !t.first_has(&LOCITYPER_COLUMNS) { + return unreadable(format!( + "{} lacks a row or one of the columns {}", + base(&p.path), + LOCITYPER_COLUMNS.join(", ") + )); + } + let loci: Vec = t.rows.iter().map(|r| dots(r, &LOCITYPER_COLUMNS)).collect(); + let n = |k: &str, want: &str| t.rows.iter().filter(|r| or_dot(r, k) == want).count(); + let v = Obj::new() + .set("called", count(n("status", "called"))) + .set("passed", count(n("filter", "pass"))) + .val(); + let mut out = Obj::new().set("loci", Value::Array(loci)); + if let Value::Object(m) = v { + out.0.extend(m); + } + Ok(Some((p, out.val()))) +} + +fn text_at<'a>(v: &'a Value, k: &str) -> Option<&'a Value> { + match v { + Value::Object(m) => m.get(k), + _ => None, + } +} + +/// Each Locityper locus's primary call (`locityper_primary`), as text. +pub fn locityper_post(v: &mut Value, hla_state: State, hla: Option<&Value>, others: &Others) { + let old = |st: State| { + if st == State::Stale { + " (stale: from an earlier run)" + } else { + "" + } + }; + let hla = hla.filter(|_| matches!(hla_state, State::Ok | State::Stale)); + let mut by_gene: IndexMap = IndexMap::new(); + if let Some(Value::Array(loci)) = hla.and_then(|h| text_at(h, "loci")) { + for x in loci { + if let Some(Value::Str(g)) = text_at(x, "gene") { + by_gene.insert(g.clone(), x); + } + } + } + let mut primary: IndexMap<&str, String> = IndexMap::new(); + for g in ["HLA-DRB1", "HLA-DQB1"] { + let x = by_gene + .get(g) + .or_else(|| by_gene.get(&g.replace("HLA-", ""))); + primary.insert( + g, + match x { + None => "T1K (step 08) did not run".to_owned(), + Some(x) => { + let alleles = match text_at(x, "alleles") { + Some(Value::Array(a)) => a + .iter() + .map(|a| pycompat::value::py_str(a).unwrap_or_default()) + .collect::>() + .join("/"), + _ => String::new(), + }; + let low = text_at(x, "low_confidence").is_some_and(pycompat::value::truthy); + format!( + "T1K (step 08): {}{}{}", + if alleles.is_empty() { + "no allele" + } else { + &alleles + }, + if low { " (low confidence)" } else { "" }, + old(hla_state) + ) + } + }, + ); + } + let typed = hla + .and_then(|h| text_at(h, "kir")) + .and_then(|k| text_at(k, "typed")) + .filter(|t| pycompat::value::truthy(t)); + primary.insert( + "KIR", + match typed { + Some(Value::Array(t)) => format!( + "T1K KIR pass (step 08): {} genes typed{}", + t.len(), + old(hla_state) + ), + Some(Value::Object(t)) => format!( + "T1K KIR pass (step 08): {} genes typed{}", + t.len(), + old(hla_state) + ), + _ => "T1K's KIR pass (step 08, KIR=true) gave no call".to_owned(), + }, + ); + let mut cyp = Vec::new(); + if let Some((st, call)) = &others.pypgx { + cyp.push(format!("pypgx (step 32): {call}{}", old(*st))); + } + if let Some((st, genotype, filter)) = &others.cyrius { + let usable = if filter.as_deref() == Some("PASS") { + String::new() + } else { + format!( + " (not usable: Filter {})", + filter + .as_deref() + .filter(|f| !f.is_empty()) + .unwrap_or("missing") + ) + }; + cyp.push(format!( + "Cyrius (step 21): {}{usable}{}", + genotype.as_deref().unwrap_or("none"), + old(*st) + )); + } + primary.insert( + "CYP2D6", + if cyp.is_empty() { + "pypgx (step 32) and Cyrius (step 21) did not run".to_owned() + } else { + cyp.join("; ") + }, + ); + primary.insert("FCGR", "no other step calls FCGR".to_owned()); + let Value::Object(m) = v else { return }; + let Some(Value::Array(loci)) = m.get_mut("loci") else { + return; + }; + for x in loci { + let Value::Object(x) = x else { continue }; + let locus = match x.get("locus") { + Some(Value::Str(l)) => l.clone(), + _ => String::new(), + }; + let p = primary + .get(locus.as_str()) + .cloned() + .unwrap_or_else(|| "no other step calls this locus".to_owned()); + x.insert("primary".into(), s(&p)); + } +} + +// --- step 47 -------------------------------------------------------------------- + +pub fn viral(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("viral/{}_viral.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, false)?; + let need = [ + "virus", + "accession", + "reads", + "breadth", + "relative_depth", + "copies_per_cell", + "call", + ]; + if !t.first_has(&need) { + return unreadable(format!( + "{} has no rows or lacks a column of {}", + base(&p.path), + need.join(", ") + )); + } + let all: Vec<&str> = need + .iter() + .copied() + .chain(["human_depth", "note"]) + .collect(); + let keep = t + .rows + .iter() + .map(|r| dots_strict(r, &all)) + .collect::, _>>()?; + let get = |v: &Value, k: &str| match text_at(v, k) { + Some(Value::Str(x)) => x.clone(), + _ => String::new(), + }; + let ici = keep + .iter() + .find(|r| get(r, "call") == "inherited integrated HHV-6 likely") + .map(|r| get(r, "virus")); + let human = get(&keep[0], "human_depth"); + let hhv6 = match ici { + Some(v) => format!("inherited integrated {v} likely"), + None if human == "." => "not assessed (no human depth)".to_owned(), + None => "no integrated HHV-6".to_owned(), + }; + let found: Vec = keep + .iter() + .filter(|r| { + let n = get(r, "reads"); + isdigit(&n) && crate::int(&n).is_ok_and(|n| n > 0) + }) + .cloned() + .collect(); + let v = Obj::new() + .set("hhv6", s(&hhv6)) + .set("human_depth", s(&human)) + .set("found", Value::Array(found)) + .set("genomes", count(keep.len())) + .val(); + Ok(Some((p, v))) +} + +// --- step 49 -------------------------------------------------------------------- + +pub fn chip(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let (Some(p), Some(dp)) = ( + c.find1(format!("chip/{s_}_chip.tsv")), + c.find1(format!("chip/{s_}_chip_depth.tsv")), + ) else { + return Ok(None); + }; + let calls = table(&p.path, false)?; + let depth = table(&dp.path, false)?; + let need = [ + "gene", + "protein_change", + "vaf", + "alt_reads", + "depth", + "status", + "note", + ]; + let head = first_line(&p.path)?; + if need.iter().any(|k| !head.iter().any(|h| h == k)) { + return unreadable(format!( + "{} lacks a column of {}", + base(&p.path), + need.join(", ") + )); + } + let dneed = ["gene", "mean_depth", "detection_limit_vaf", "status"]; + if !depth.first_has(&dneed) { + return unreadable(format!( + "{} has no gene, mean_depth, detection_limit_vaf and status columns, or no row", + base(&dp.path) + )); + } + let chip: Vec = calls + .rows + .iter() + .filter(|r| r.get("status").flatten() == Some("chip")) + .map(|r| dots(r, &need)) + .collect(); + let mut covered = Vec::new(); + for r in &depth.rows { + if r.get("status").flatten() == Some("covered") { + let x = r.get("mean_depth").flatten().ok_or_else(|| { + py_error( + "TypeError", + "float() argument must be a string or a real number, not 'NoneType'", + ) + })?; + covered.push(pycompat::num::parse_float(x).map_err(|_| float_err(x))?); + } + } + let med = if covered.is_empty() { + None + } else { + // NaN has no ordering. Never let sorting move an invalid depth + // away from the median and produce a usable sensitivity estimate. + Some( + pycompat::stats::median(covered) + .map_err(|e| py_error(e.exception(), "cannot convert float NaN to integer"))?, + ) + }; + let limit = match med { + None => Value::Null, + Some(m) => og_chip::binom::detection_limit(m) + .map_err(|e| py_error("ValueError", &e.to_string()))? + .map_or(Value::Null, Value::Float), + }; + let v = Obj::new() + .set("calls", Value::Array(chip.clone())) + .set("other", count(calls.rows.len() - chip.len())) + .set( + "genes", + Value::Array(depth.rows.iter().map(|r| dots(r, &dneed)).collect()), + ) + .set("median_depth", med.map_or(Value::Null, Value::Float)) + .set("median_limit", limit) + .set( + "not_covered", + Value::Array( + depth + .rows + .iter() + .filter(|r| r.get("status").flatten() != Some("covered")) + .map(|r| opt_s(r.get("gene").flatten())) + .collect(), + ), + ) + .val(); + Ok(Some((p, v))) +} + +// --- step 50 -------------------------------------------------------------------- + +pub fn mei(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("mei/{}_mei_sites.tsv", c.s())) else { + return Ok(None); + }; + let sites = table(&p.path, false)?; + let need = [ + "gene", + "element", + "condition", + "inheritance", + "status", + "mean_depth", + "call", + ]; + if !sites.first_has(&need) { + return unreadable(format!( + "{} has no rows or lacks a column of {}", + base(&p.path), + need.join(", ") + )); + } + let dir = p.path.rsplit_once('/').map_or("", |x| x.0); + let cp = crate::join(dir, &format!("{}_mei.tsv", c.sample)); + if !crate::is_file(&cp) { + return unreadable(format!("{} is missing beside {}", base(&cp), base(&p.path))); + } + let calls = table(&cp, false)?; + if ["chrom", "pos", "element", "copies"] + .iter() + .any(|k| !calls.has(k)) + { + return unreadable(format!( + "{} lacks a column of chrom, pos, element, copies", + base(&cp) + )); + } + let mut counts: std::collections::BTreeMap = Default::default(); + for r in &calls.rows { + let e = r.get("element").flatten().unwrap_or("null").to_owned(); + *counts.entry(e).or_default() += 1; + } + let v = Obj::new() + .set("calls", count(calls.rows.len())) + .set( + "by_element", + Value::Object(counts.into_iter().map(|(k, n)| (k, count(n))).collect()), + ) + .set( + "sites", + Value::Array(sites.rows.iter().map(|r| dots(r, &need)).collect()), + ) + .val(); + Ok(Some((p, v))) +} + +// --- step 51 -------------------------------------------------------------------- + +const MOCHA_SUMMARY_KEYS: [&str; 12] = [ + "status", + "message", + "depth", + "het_sites", + "het_per_10mb", + "chromosomes", + "floor_read", + "floor_min", + "floor", + "calls_mocha", + "calls_reported", + "phasing", +]; +const MOCHA_CALL_KEYS: [&str; 11] = [ + "chrom", + "start", + "end", + "length_mb", + "arms", + "type", + "cell_fraction", + "n_hets", + "lod_baf_phase", + "reported", + "reason", +]; + +pub fn mosaic(c: &Ctx<'_>) -> Out { + let s_ = c.s(); + let Some(p) = c.find1(format!("mosaic/{s_}_mocha_summary.tsv")) else { + return Ok(None); + }; + let kv = metric_map(&table(&p.path, false)?, "metric", "value"); + if !matches!(kv_get(&kv, "status"), Some("assessed" | "not assessed")) { + return unreadable(format!( + "{} has no status row (assessed or not assessed)", + base(&p.path) + )); + } + let mut data = MOCHA_SUMMARY_KEYS + .iter() + .fold(Obj::new(), |o, k| o.set(k, s(&kv_dot(&kv, k)))); + let Some(cf) = c.find1(format!("mosaic/{s_}_mocha.tsv")) else { + return unreadable(format!( + "{} has no {}_mocha.tsv beside it", + base(&p.path), + c.sample + )); + }; + let head = first_line(&cf.path)?; + if MOCHA_CALL_KEYS.iter().any(|k| !head.iter().any(|h| h == k)) { + return unreadable(format!( + "{} lacks a column of {}", + base(&cf.path), + MOCHA_CALL_KEYS.join(", ") + )); + } + let rows: Table = table(&cf.path, false)?; + let reported = |r: &pycompat::csv::DictRow| r.get("reported").flatten() == Some("yes"); + let calls: Vec = rows + .rows + .iter() + .filter(|r| reported(r)) + .map(|r| dots(r, &MOCHA_CALL_KEYS)) + .collect(); + let n = calls.len(); + data.put("calls", Value::Array(calls)); + data.put( + "not_reported", + count(rows.rows.iter().filter(|r| !reported(r)).count()), + ); + let said = kv_dot(&kv, "calls_reported"); + if said != "." && said != n.to_string() { + return unreadable(format!( + "{} has {n} reported calls, {} says {said}", + base(&cf.path), + base(&p.path) + )); + } + Ok(Some((p, data.val()))) +} + +// --- step 48 -------------------------------------------------------------------- + +pub fn exomiser(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("exomiser/{}_exomiser_top.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, true)?; + let header = first_line(&p.path)?; + let need = [ + "rank", + "gene", + "moi", + "exomiser_score", + "phenotype_score", + "variant", + "genotype", + "consequence", + "hgvs", + "acmg", + "disease", + ]; + if need.iter().any(|k| !header.iter().any(|h| h == k)) { + return unreadable(format!( + "{} lacks a column of {}", + base(&p.path), + need.join(", ") + )); + } + if t.rows.iter().any(|r| ragged(r, &need)) { + return unreadable(format!( + "{} has a line that does not have one field per column", + base(&p.path) + )); + } + let dir = p.path.rsplit_once('/').map_or("", |x| x.0); + let tp = crate::join(dir, &format!("{}_exomiser_terms.tsv", c.sample)); + let mut terms = Vec::new(); + if crate::is_file(&tp) { + for r in table(&tp, true)?.rows { + terms.push( + Obj::new() + .set("id", get_default(&r, "hpo_id", "")) + .set("label", get_default(&r, "label", "")) + .val(), + ); + } + } + let cell = |r: &pycompat::csv::DictRow, k: &str| s(r.get(k).flatten().unwrap_or("")); + let mut genes: Vec<(String, Obj, Vec)> = Vec::new(); + for r in &t.rows { + let rank = r.get("rank").flatten().unwrap_or("").to_owned(); + if genes.last().is_none_or(|g| g.0 != rank) { + let o = ["rank", "gene", "moi", "exomiser_score", "phenotype_score"] + .iter() + .fold(Obj::new(), |o, k| o.set(k, cell(r, k))); + genes.push((rank, o, Vec::new())); + } + let v = [ + "variant", + "genotype", + "consequence", + "hgvs", + "acmg", + "disease", + ] + .iter() + .fold(Obj::new(), |o, k| o.set(k, cell(r, k))) + .val(); + if let Some(g) = genes.last_mut() { + g.2.push(v); + } + } + let v = Obj::new() + .set("terms", Value::Array(terms)) + .set( + "genes", + Value::Array( + genes + .into_iter() + .map(|(_, o, vs)| o.set("variants", Value::Array(vs)).val()) + .collect(), + ), + ) + .val(); + Ok(Some((p, v))) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn base64_matches_python() { + assert_eq!(b64(b""), ""); + assert_eq!(b64(b"f"), "Zg=="); + assert_eq!(b64(b"fo"), "Zm8="); + assert_eq!(b64(b"foo"), "Zm9v"); + assert_eq!(b64(b"foob"), "Zm9vYg=="); + } + + #[test] + fn quote_keeps_unreserved_only() { + assert_eq!(quote("R-M269"), "R-M269"); + assert_eq!(quote("H5a7 x/y"), "H5a7%20x%2Fy"); + assert_eq!(quote("é"), "%C3%A9"); + } +} diff --git a/rust/crates/og-summary/src/collect/group_c/tests.rs b/rust/crates/og-summary/src/collect/group_c/tests.rs new file mode 100644 index 000000000..4a93b5111 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/tests.rs @@ -0,0 +1,204 @@ +//! Group c through `collect`: each section's state, and the post pass +//! (CNVpytor's chrY filter, Locityper's primary calls, the repeat readings). +//! The section readers are checked against the Python collector by the +//! summary-collect parity cases (tests/parity/cases.tsv). + +use super::super::{Group, Input, collect}; +use super::{GROUP_C, Others}; +use crate::testutil::Dir; +use pycompat::json::Value; + +fn summary(d: &Dir, declared_sex: Option<&str>) -> Value { + let input = Input { + sample: "S".into(), + sample_dir: d.path("S"), + declared_sex: declared_sex.map(str::to_owned), + assets: d.path("assets"), + group: Group::C, + tables: concat!(env!("CARGO_MANIFEST_DIR"), "/../../../bin").into(), + }; + collect(&input).unwrap().to_json() +} + +fn at<'a>(v: &'a Value, path: &[&str]) -> &'a Value { + path.iter().fold(v, |v, k| match v { + Value::Object(m) => m.get(*k).unwrap_or_else(|| panic!("no {k} in {v:?}")), + Value::Array(a) => &a[k.parse::().unwrap()], + _ => panic!("{k} of {v:?}"), + }) +} + +fn text(v: &Value) -> &str { + match v { + Value::Str(s) => s, + _ => panic!("not a text: {v:?}"), + } +} + +const CNVS: &str = "deletion\tchrY:2800001-2900000\t100000\t0.0\t1e-15\t1\t1\t1\t0.05\t0\t9\n\ +deletion\tchr2:10000001-10100000\t100000\t0.4\t1e-9\t1\t1\t1\t0.01\t0\t9\n\ +deletion\tchr1:122000001-123000000\t1000000\t0.1\t1e-12\t1\t1\t1\t0.2\t0\t9\n"; + +#[test] +fn chip_nan_depth_is_unreadable() { + let d = Dir::new("c-chip-nan"); + d.put( + "S/chip/S_chip.tsv", + "gene\tprotein_change\tvaf\talt_reads\tdepth\tstatus\tnote\n", + ); + d.put("S/chip/S_chip_depth.tsv", "gene\tmean_depth\tdetection_limit_vaf\tstatus\nA\t30\t.\tcovered\nB\tnan\t.\tcovered\nC\t40\t.\tcovered\n"); + let v = summary(&d, None); + let chip = at(&v, &["sections", "chip"]); + assert_eq!(text(at(chip, &["state"])), "unreadable"); + assert_eq!(at(chip, &["data"]), &Value::Object(Default::default())); +} + +#[test] +fn chip_infinity_only_fails_when_the_median_is_nonfinite() { + let d = Dir::new("c-chip-inf"); + d.put( + "S/chip/S_chip.tsv", + "gene\tprotein_change\tvaf\talt_reads\tdepth\tstatus\tnote\n", + ); + for (depths, median) in [ + (vec!["30", "inf", "40"], Some(40.0)), + (vec!["30", "-inf", "40"], Some(30.0)), + (vec!["inf"], None), + (vec!["-inf"], None), + (vec!["-inf", "inf"], None), + ] { + let mut body = "gene\tmean_depth\tdetection_limit_vaf\tstatus\n".to_owned(); + for (i, value) in depths.iter().enumerate() { + body.push_str(&format!("G{i}\t{value}\t.\tcovered\n")); + } + d.put("S/chip/S_chip_depth.tsv", body); + let v = summary(&d, None); + let chip = at(&v, &["sections", "chip"]); + if let Some(median) = median { + assert_eq!(text(at(chip, &["state"])), "ok", "{depths:?}"); + assert_eq!(at(chip, &["data", "median_depth"]), &Value::Float(median)); + } else { + assert_eq!(text(at(chip, &["state"])), "unreadable", "{depths:?}"); + assert_eq!(at(chip, &["data"]), &Value::Object(Default::default())); + } + } +} + +#[test] +fn every_section_of_group_c_is_written_in_order() { + let d = Dir::new("c-order"); + let v = summary(&d, None); + let Value::Object(secs) = at(&v, &["sections"]) else { + panic!() + }; + let keys: Vec<&str> = secs.keys().map(String::as_str).collect(); + let want: Vec<&str> = GROUP_C.iter().map(|s| s.key).collect(); + assert_eq!(keys, want); + assert!(secs.values().all(|s| text(at(s, &["state"])) == "missing")); +} + +#[test] +fn cnvpytor_drops_chry_for_a_declared_female_only() { + let d = Dir::new("c-cnv"); + d.put("S/cnvpytor/S_cnvs.txt", CNVS); + let female = summary(&d, Some("female")); + let data = at(&female, &["sections", "cnvpytor", "data"]); + assert_eq!(at(data, &["significant_filtered"]), &Value::Int(1)); + assert_eq!(at(data, &["dropped", "chry_female"]), &Value::Int(1)); + assert_eq!(at(data, &["dropped", "band"]), &Value::Int(1)); + let male = summary(&d, Some("male")); + let data = at(&male, &["sections", "cnvpytor", "data"]); + assert_eq!(at(data, &["significant_filtered"]), &Value::Int(2)); + assert_eq!(at(data, &["declared_sex"]), &Value::Str("male".into())); +} + +#[test] +fn locityper_names_the_primary_call_of_each_locus() { + let d = Dir::new("c-loci"); + d.put( + "S/hla_t1k/S_hla_genotype.tsv", + "HLA-DRB1\t2\tDRB1*03:01\t1\t40\tDRB1*07:01\t1\t0\n", + ); + let head = "locus\tregion\tstatus\tgenotype\thaplotype_1\thaplotype_2\tquality\ttotal_reads\tunexplained_reads\tweighted_distance\twarnings\tfilter\treason\n"; + d.put( + "S/locityper/S_locityper.tsv", + format!( + "{head}HLA-DRB1\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t\n\ + HLA-DQB1\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t\n\ + CYP2D6\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t\n\ + OTHER\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t\n" + ), + ); + let v = summary(&d, None); + let loci = at(&v, &["sections", "locityper", "data", "loci"]); + let primary = |i: &str| text(at(loci, &[i, "primary"])).to_owned(); + assert_eq!( + primary("0"), + "T1K (step 08): DRB1*03:01/DRB1*07:01 (low confidence)" + ); + assert_eq!(primary("1"), "T1K (step 08) did not run"); + assert_eq!( + primary("2"), + "pypgx (step 32) and Cyrius (step 21) did not run" + ); + assert_eq!(primary("3"), "no other step calls this locus"); +} + +#[test] +fn locityper_reads_pypgx_and_cyrius_from_group_b() { + let mut v = pycompat::json::loads(r#"{"loci": [{"locus": "CYP2D6"}]}"#).unwrap(); + let others = Others { + pypgx: Some((super::State::Stale, "*1/*4".into())), + cyrius: Some((super::State::Ok, Some("*1/*2".into()), None)), + ..Others::default() + }; + super::tables::locityper_post(&mut v, super::State::Missing, None, &others); + assert_eq!( + text(at(&v, &["loci", "0", "primary"])), + "pypgx (step 32): *1/*4 (stale: from an earlier run); Cyrius (step 21): *1/*2 (not usable: Filter missing)" + ); +} + +#[test] +fn a_cftr_5t_with_r117h_reads_the_phase() { + let d = Dir::new("c-cftr"); + // Invented calls. Keep FORMAT separate from the values, as in the parity generator. + let vcf = concat!( + "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n", + "chr7\t1\t.\tT\t\t.\tPASS\tVARID=CFTR_TG;REPID=CFTR_TG\t", + "GT:SO:REPCN:REPCI:ADSP\t", + "0/1:SPANNING/SPANNING:11/12:11-11/12-12:8/9\n", + "chr7\t2\t.\tT\t\t.\tPASS\tVARID=CFTR_T;REPID=CFTR_T\t", + "GT:SO:REPCN:REPCI:ADSP\t", + "0/1:SPANNING/SPANNING:5/7:5-5/7-7:8/9\n", + ); + d.put("S/expansion_hunter/S_eh.vcf", vcf); + d.put( + "S/traits/S_traits.tsv", + "rsid\tgene\ttrait\tgenotype\tstatus\treading\tancestry\tnote\tpmids\n\ +rs78655421\tCFTR\tR117H\tG/A\tcalled\tr\ta\tn\tp\n", + ); + let v = summary(&d, None); + let cftr = at(&v, &["sections", "expansions", "data", "readings", "0"]); + assert_eq!(text(at(cftr, &["phase"])), "unknown"); + assert_eq!(text(at(cftr, &["r117h"])), "G/A"); + assert!(text(at(cftr, &["reading"])).contains("phase unknown")); +} + +#[test] +fn a_stale_section_says_from_which_run() { + let d = Dir::new("c-stale"); + d.put( + "S/logs/run_status.tsv", + "meta\tstarted_epoch\t4102444800\nstep\t11\tskipped\n", + ); + d.put( + "S/vcf/S_roh.txt", + "RG\tS\tchr2\t10000001\t17000000\t7000000\t9\t9\n", + ); + let v = summary(&d, None); + let roh = at(&v, &["sections", "roh"]); + assert_eq!(text(at(roh, &["state"])), "stale"); + assert!(text(at(roh, &["note"])).contains("step 11 was skipped")); + assert_eq!(at(roh, &["data", "largest_mb"]), &Value::Float(7.0)); +} diff --git a/rust/crates/og-summary/src/collect/group_c/traits.rs b/rust/crates/og-summary/src/collect/group_c/traits.rs new file mode 100644 index 000000000..6a553b36f --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/traits.rs @@ -0,0 +1,667 @@ +//! Step 42's traits section: the traits table, its genotype-rules lines, +//! and the APOL1 G2 deletion read from the sample's VCF beside the APOL1 +//! rule, whose single-base sites read G1 only. + +use super::util::{Obj, base, count, ragged, s, table}; +use super::{Ctx, Out}; +use pycompat::json::Value; + +/// The columns of step 42's genotype-rules table. +pub const RULE_OUT_COLUMNS: [&str; 15] = [ + "rule", + "gene", + "trait", + "rsids", + "chrom", + "pos", + "genotypes", + "gq", + "phase", + "result", + "status", + "reading", + "ancestry", + "note", + "pmids", +]; +/// The rules whose line the traits table holds already. +const TRAITS_TABLE_RULES: [&str; 1] = ["APOE"]; +const TRAIT_COLUMNS: [&str; 9] = [ + "rsid", "gene", "trait", "genotype", "status", "reading", "ancestry", "note", "pmids", +]; + +pub fn traits(c: &Ctx<'_>) -> Out { + let Some(p) = c.find1(format!("traits/{}_traits.tsv", c.s())) else { + return Ok(None); + }; + let t = table(&p.path, true)?; + if !t.first_has(&TRAIT_COLUMNS) { + return Err(format!( + "{} has no rows or lacks a column of {}", + base(&p.path), + TRAIT_COLUMNS.join(", ") + )); + } + if let Some(i) = t.rows.iter().position(|r| ragged(r, &TRAIT_COLUMNS)) { + return Err(format!( + "{} line {} does not have one field per column", + base(&p.path), + i + 2 + )); + } + let cell = |r: &pycompat::csv::DictRow, k: &str| r.get(k).flatten().unwrap_or("").to_owned(); + let row = |r: &pycompat::csv::DictRow| { + TRAIT_COLUMNS + .iter() + .fold(Obj::new(), |o, k| o.set(k, s(&cell(r, k)))) + .val() + }; + let apoe = |r: &&pycompat::csv::DictRow| cell(r, "gene") == "APOE"; + let mut out = Obj::new() + .set( + "rows", + Value::Array(t.rows.iter().filter(|r| !apoe(r)).map(row).collect()), + ) + .set( + "apoe", + Value::Array(t.rows.iter().filter(apoe).map(row).collect()), + ); + for st in ["called", "not_covered", "other_allele"] { + out.put( + st, + count( + t.rows + .iter() + .filter(|r| !apoe(r) && cell(r, "status") == st) + .count(), + ), + ); + } + out.put("rules", genotype_rules(c)?); + Ok(Some((p, out.val()))) +} + +/// Step 42's genotype-rules lines, `null` without the table; the APOL1 line +/// gets the G2 reading. +fn genotype_rules(c: &Ctx<'_>) -> Result { + let Some(p) = c.find1(format!("traits/{}_genotype_rules.tsv", c.s())) else { + return Ok(Value::Null); + }; + let t = table(&p.path, true)?; + if !t.rows.is_empty() && RULE_OUT_COLUMNS.iter().any(|k| !t.has(k)) { + return Err(format!( + "{} lacks a column of {}", + base(&p.path), + RULE_OUT_COLUMNS.join(", ") + )); + } + if let Some(i) = t.rows.iter().position(|r| ragged(r, &RULE_OUT_COLUMNS)) { + return Err(format!( + "{} line {} does not have one field per column", + base(&p.path), + i + 2 + )); + } + let mut out = Vec::new(); + for r in &t.rows { + let cell = |k: &str| r.get(k).flatten().unwrap_or("").to_owned(); + if TRAITS_TABLE_RULES.contains(&cell("rule").as_str()) { + continue; + } + let mut o = RULE_OUT_COLUMNS + .iter() + .fold(Obj::new(), |o, k| o.set(k, s(&cell(k)))); + if cell("rule") == apol1::G2_RULE { + let g2 = apol1::read(c); + let both = apol1::combine(&cell("result"), &cell("status"), &g2); + let mut v = g2.json(); + v.put("result", s(&both.0)); + v.put("reading", s(&both.1)); + o.put("g2", v.val()); + } + out.push(o.val()); + } + Ok(Value::Array(out)) +} + +/// APOL1 G2 (rs71785313), a 6-base deletion: read from the VCF records over +/// its span, and put together with the G1 result of step 42's APOL1 rule. +pub mod apol1 { + use super::*; + + pub const G2_RULE: &str = "APOL1"; + const CHROM: &str = "chr22"; + const POS: i64 = 36_265_995; + const REF: &str = "AATAATT"; + const ALT: &str = "A"; + /// GRCh38 chr22:36265975-36266025, the plus strand. + const CONTEXT_START: i64 = 36_265_975; + const CONTEXT: &str = "AGAAGCTAAACATTCTCAACAATAATTATAAGATTCTGCAGGCGGACCAAG"; + const CONTEXT_END: i64 = CONTEXT_START + CONTEXT.len() as i64 - 1; + /// The anchor base of the left-aligned record to the end of the span a + /// 6-base deletion there can be written over. + pub const SPAN: (i64, i64) = (36_265_995, 36_266_005); + const TWO_COPIES: &str = "the high-risk genotype of Genovese 2010, which went with focal segmental glomerulosclerosis and hypertension-attributed kidney failure in African Americans. A risk factor, not a diagnosis."; + + /// What `read_g2` gave, with the file it read. + #[derive(Debug, Clone, PartialEq)] + pub struct G2 { + pub status: &'static str, + pub copies: Option, + pub gq: String, + pub text: String, + /// The VCF read, relative to the folder; `None` with no VCF. A + /// read error writes it before the text. + pub source: Option, + pub source_first: bool, + } + + impl G2 { + pub fn json(&self) -> Obj { + let o = Obj::new() + .set("status", s(self.status)) + .set( + "copies", + self.copies.map_or(Value::Null, |c| Value::Int(c.into())), + ) + .set("gq", s(&self.gq)); + let src = super::super::util::opt_s(self.source.as_deref()); + if self.source_first { + o.set("source", src).set("text", s(&self.text)) + } else { + o.set("text", s(&self.text)).set("source", src) + } + } + } + + /// The context with `r` at `pos` replaced by `a`, or `None` when `r` is + /// not the GRCh38 bases there. + pub fn apply(pos: i64, r: &str, a: &str) -> Option { + let i = pos - CONTEXT_START; + if i < 0 || pos + r.len() as i64 - 1 > CONTEXT_END { + return None; + } + let i = i as usize; + if CONTEXT.get(i..i + r.len()) != Some(r) { + return None; + } + Some(format!("{}{a}{}", &CONTEXT[..i], &CONTEXT[i + r.len()..])) + } + + fn haplotype() -> Option { + apply(POS, REF, ALT) + } + + fn digits(x: &str) -> bool { + pycompat::text::isdigit(x) + } + + fn record_end(r: &[String]) -> Option { + if let Some(info) = r.get(7) { + for (i, _) in info.match_indices("END=") { + if i == 0 || info.as_bytes()[i - 1] == b';' { + let d: String = info[i + 4..] + .chars() + .take_while(char::is_ascii_digit) + .collect(); + if !d.is_empty() { + return d.parse().ok(); + } + } + } + } + Some(r[1].parse::().ok()? + r[3].chars().count() as i64 - 1) + } + + fn overlaps(r: &[String]) -> bool { + r.len() > 4 + && r[0] == CHROM + && digits(&r[1]) + && r[1].parse::().is_ok_and(|p| p <= SPAN.1) + && record_end(r).is_some_and(|e| e >= SPAN.0) + } + + fn symbolic(a: &str) -> bool { + a == "*" || a == "." || (a.len() >= 2 && a.starts_with('<') && a.ends_with('>')) + } + + /// G2 from the records over [`SPAN`]; `gvcf` says a base with no + /// record has no reads. + pub fn read_g2(records: &[Vec], gvcf: bool) -> G2 { + let g2_hap = haplotype(); + let (mut copies, mut covered, mut gqs, mut problems) = ( + 0i64, + Vec::<(i64, i64)>::new(), + Vec::::new(), + Vec::::new(), + ); + let (mut seen, mut g2_records) = (false, 0); + 'records: for r in records { + if !overlaps(r) { + continue; + } + if r.len() < 10 { + problems.push("a record over G2 has no sample column".into()); + continue; + } + seen = true; + let pos: i64 = r[1].parse().unwrap_or(0); + let end = record_end(r).unwrap_or(pos); + let rf = r[3].as_str(); + let alleles: Vec<&str> = std::iter::once(rf).chain(r[4].split(',')).collect(); + let mut fmt = indexmap::IndexMap::new(); + for (k, v) in r[8].split(':').zip(r[9].split(':')) { + fmt.insert(k, v); + } + let field = |k: &str| fmt.get(k).copied().unwrap_or(".").to_owned(); + let gt = field("GT"); + let gt: Vec<&str> = gt.split(['/', '|']).collect(); + if gt.iter().any(|g| !digits(g)) { + problems.push(format!("a no-call at {}:{pos}", r[0])); + continue; + } + let idx: Vec = gt.iter().map(|g| g.parse().unwrap_or(usize::MAX)).collect(); + if idx.iter().any(|i| *i >= alleles.len()) { + problems.push(format!( + "a genotype at {}:{pos} names an allele the record lacks", + r[0] + )); + continue; + } + if idx.iter().all(|i| *i == 0) { + covered.push((pos, end)); + gqs.push(field("GQ")); + continue; + } + if r[6] != "PASS" && r[6] != "." { + problems.push(format!( + "a call at {}:{pos} that failed the caller's filter ({})", + r[0], r[6] + )); + continue; + } + let mut n = 0; + let mut stopped = false; + for &i in &idx { + if i == 0 { + continue; + } + let a = alleles[i]; + let inside = pos >= CONTEXT_START && end <= CONTEXT_END; + if inside && apply(pos, rf, rf).is_none() { + problems.push(format!( + "REF {rf} at {}:{pos} is not the GRCh38 base (is the VCF on GRCh38?)", + r[0] + )); + stopped = true; + break; + } + let hap = if symbolic(a) || !inside { + None + } else { + apply(pos, rf, a) + }; + if hap.is_some() && hap == g2_hap { + n += 1; + } else { + problems.push(format!( + "an allele other than G2 inside its span ({}:{pos} {rf}>{a})", + r[0] + )); + stopped = true; + break; + } + } + if stopped { + continue; + } + if n > 0 { + g2_records += 1; + if g2_records > 1 { + problems.push("more than one record calls the G2 deletion".into()); + break 'records; + } + } + copies += n; + covered.push((pos, end)); + gqs.push(field("GQ")); + } + let gq = if gqs.is_empty() { + ".".to_owned() + } else { + gqs.join(",") + }; + let g2 = |status, copies, gq: String, text: String| G2 { + status, + copies, + gq, + text, + source: None, + source_first: false, + }; + if let Some(p) = problems.first() { + return g2( + "not_read", + None, + gq, + format!("G2 (rs71785313) not read: {p}."), + ); + } + if copies > 0 { + return g2( + "called", + Some(copies), + gq, + format!( + "G2 (rs71785313): {copies} cop{} of the deletion called.", + if copies > 1 { "ies" } else { "y" } + ), + ); + } + if (SPAN.0..=SPAN.1).all(|p| covered.iter().any(|(a, b)| *a <= p && p <= *b)) { + return g2( + "called", + Some(0), + gq, + "G2 (rs71785313): the reference over the whole deletion span, no G2 copy.".into(), + ); + } + if !gvcf && !seen { + return g2( + "no_call_record", + Some(0), + ".".into(), + "G2 (rs71785313): no G2 deletion among the VCF's calls.".into(), + ); + } + g2( + "not_covered", + None, + gq, + "G2 (rs71785313) not read: no genotype over part of the deletion span.".into(), + ) + } + + /// Only records overlapping the span. Like the Python window reader, + /// stop past the window or after leaving chr22 on sorted input. + fn window_records( + lines: impl IntoIterator>, + ) -> Result>, String> { + let mut out = Vec::new(); + let mut seen = false; + for line in lines { + let line = line?; + if line.starts_with('#') { + continue; + } + let c = super::super::util::cells(line.strip_suffix('\n').unwrap_or(&line).as_bytes()); + if c.len() < 2 || !digits(&c[1]) { + continue; + } + if c[0] != CHROM { + if seen { + break; + } + continue; + } + seen = true; + if c[1].parse::().is_ok_and(|p| p > i128::from(SPAN.1)) { + break; + } + if overlaps(&c) { + out.push(c); + } + } + Ok(out) + } + + /// The records over the G2 span, using tabix when a `.tbi` is there, + /// otherwise streaming up to the end of the window. + fn window(path: &str) -> Result>, String> { + let tbi = format!("{path}.tbi"); + let mut out = Vec::new(); + if std::path::Path::new(&tbi).exists() { + let mut rd = og_io::tabix::Reader::with_index(path, &tbi).map_err(|e| e.to_string())?; + for line in rd.query(CHROM, (SPAN.0 - 1) as u64, SPAN.1 as u64) { + let line = line.map_err(|e| e.to_string())?; + out.push(super::super::util::cells(&line)); + } + return Ok(out); + } + let mut rd = crate::text_lines(path)?; + window_records(std::iter::from_fn(|| { + let mut line = String::new(); + match rd.read_line(&mut line) { + Ok(true) => Some(Ok(line)), + Ok(false) => None, + Err(e) => Some(Err(format!( + "{}: {}", + base(path), + crate::collect::sections::gzip_message(&e) + ))), + } + })) + } + + /// G2 from the sample's gVCF when it is there, else its VCF. + pub fn read(c: &Ctx<'_>) -> G2 { + for (name, gvcf) in [ + (format!("vcf/{}.g.vcf.gz", c.sample), true), + (format!("vcf/{}.vcf.gz", c.sample), false), + ] { + let p = c.path(&name); + if !crate::is_file(&p) { + continue; + } + return match window(&p) { + Err(e) => G2 { + status: "not_read", + copies: None, + gq: ".".into(), + text: format!("G2 (rs71785313) not read: {name} could not be read ({e})."), + source: Some(name), + source_first: true, + }, + Ok(recs) => G2 { + source: Some(name), + ..read_g2(&recs, gvcf) + }, + }; + } + G2 { + status: "not_read", + copies: None, + gq: ".".into(), + text: "G2 (rs71785313) not read: no VCF of the sample in this folder.".into(), + source: None, + source_first: true, + } + } + + /// The G1 + G2 result and its reading, from step 42's APOL1 line + /// (`result`, `status`) and the G2 reading. + pub fn combine(result: &str, status: &str, g2: &G2) -> (String, String) { + let g1: Option = if status == "called" { + match result { + "G0/G0" => Some(0), + "G0/G1" => Some(1), + "G1/G1" => Some(2), + _ => None, + } + } else { + None + }; + let two = |label: &str| format!("Two APOL1 risk copies ({label}): {TWO_COPIES}"); + let Some(g1) = g1 else { + if g2.status == "called" && g2.copies == Some(2) { + return ( + "G2/G2".into(), + two("G2/G2") + + " The G1 sites were not read, but two G2 copies leave no copy for G1.", + ); + } + return ( + ".".into(), + "Not read: the G1 sites of the APOL1 rule were not read.".into(), + ); + }; + let Some(c) = g2.copies else { + return (".".into(), "The G1 result above stands on its own.".into()); + }; + if g1 + c > 2 { + return ( + ".".into(), + format!( + "{g1} G1 and {c} G2 copies do not fit two copies of the gene (G1 and G2 never share a copy): not read. Check the calls." + ), + ); + } + let mut label = std::iter::repeat_n("G0", (2 - g1 - c) as usize) + .chain(std::iter::repeat_n("G1", g1 as usize)) + .chain(std::iter::repeat_n("G2", c as usize)) + .collect::>() + .join("/"); + let mut reading = match g1 + c { + 2 => two(&label), + 1 => format!( + "One APOL1 risk copy ({label}): one copy did not raise the risk in Genovese 2010 (a recessive pattern)." + ), + _ => "No G1 and no G2 copy: no APOL1 kidney risk copy.".into(), + }; + if g2.status == "no_call_record" { + label += " (no G2 call)"; + reading += " No G2 deletion was called, but this VCF has no reference blocks, so a G2 copy at a position with no reads would not show."; + } + (label, reading) + } + + #[cfg(test)] + mod tests { + use super::*; + + #[test] + fn unindexed_window_stops_at_the_end_or_next_chromosome() { + let call = rec(POS, REF, ALT, "0/1", "."); + for stop in [ + format!("chr22\t{}\t.\tA\tT", SPAN.1 + 1), + "chr23\t1\t.\tA\tT".into(), + ] { + let lines = [ + Ok(call.join("\t")), + Ok(stop), + Err("must stop before reading this line".into()), + ]; + assert_eq!(window_records(lines).unwrap(), vec![call.clone()]); + } + } + + #[test] + fn unindexed_window_retains_only_overlaps() { + use std::io::Write; + let d = crate::testutil::Dir::new("g2-window-bounded"); + let path = d.put("calls.vcf", "##fileformat=VCFv4.2\n"); + let mut f = std::io::BufWriter::new( + std::fs::OpenOptions::new() + .append(true) + .open(&path) + .unwrap(), + ); + writeln!(f, "chr21\t1\t.\tA\tT\t.\tPASS\t.").unwrap(); + writeln!(f, "chr22\t1\t.\tA\tT\t.\tPASS\t.").unwrap(); + let block = rec( + SPAN.0 - 20, + "A", + "", + "0/0", + &format!("END={}", SPAN.1), + ); + let call = rec(POS, REF, ALT, "0/1", "."); + writeln!(f, "{}", block.join("\t")).unwrap(); + writeln!(f, "{}", call.join("\t")).unwrap(); + for i in 1..=250_000 { + writeln!(f, "chr22\t{}\t.\tA\tT\t.\tPASS\t.", SPAN.1 + i).unwrap(); + } + f.flush().unwrap(); + let records = window(&path).unwrap(); + assert_eq!(records.len(), 2); + assert_eq!(records, vec![block, call]); + } + + fn rec(pos: i64, r: &str, a: &str, gt: &str, info: &str) -> Vec { + [ + CHROM, + &pos.to_string(), + ".", + r, + a, + "50", + "PASS", + info, + "GT:GQ", + &format!("{gt}:40"), + ] + .iter() + .map(|x| (*x).to_owned()) + .collect() + } + + #[test] + fn the_marker_is_a_six_base_deletion_of_the_context() { + let h = haplotype().expect("REF is the GRCh38 bases"); + assert_eq!(CONTEXT.len() - h.len(), 6); + } + + #[test] + fn g2_left_aligned_is_one_copy() { + let g = read_g2(&[rec(POS, REF, ALT, "0/1", ".")], false); + assert_eq!((g.status, g.copies), ("called", Some(1))); + } + + #[test] + fn g2_shifted_is_matched() { + // The same deletion written three bases to the right: TAATTA>T + // after AATAATTATAA's repeat gives the same sequence. + let g = read_g2(&[rec(36_265_998, "ATTATA", "A", "1/1", ".")], false); + assert_eq!(g.status, "not_read", "{g:?}"); + let shifted = rec(36_265_996, "ATAATTA", "A", "1/1", "."); + let g = read_g2(&[shifted], false); + assert_eq!((g.status, g.copies), ("called", Some(2)), "{g:?}"); + } + + #[test] + fn reference_block_over_span_reads_zero() { + let g = read_g2( + &[rec(36_265_990, "A", "", "0/0", "END=36266010")], + true, + ); + assert_eq!((g.status, g.copies), ("called", Some(0))); + } + + #[test] + fn vcf_without_record_is_no_call_record() { + let g = read_g2(&[], false); + assert_eq!(g.status, "no_call_record"); + let g = read_g2(&[], true); + assert_eq!(g.status, "not_covered"); + } + + #[test] + fn two_g2_records_are_not_read() { + let a = rec(POS, REF, ALT, "0/1", "."); + let b = rec(36_265_996, "ATAATTA", "A", "0/1", "."); + let g = read_g2(&[a, b], false); + assert_eq!(g.status, "not_read"); + assert!(g.text.contains("more than one record")); + } + + #[test] + fn combine_counts_both() { + let g2 = read_g2(&[rec(POS, REF, ALT, "0/1", ".")], false); + assert_eq!(combine("G0/G1", "called", &g2).0, "G1/G2"); + let none = read_g2(&[], false); + let (label, reading) = combine("G0/G0", "called", &none); + assert_eq!(label, "G0/G0 (no G2 call)"); + assert!(reading.contains("no reference blocks")); + } + } +} diff --git a/rust/crates/og-summary/src/collect/group_c/util.rs b/rust/crates/og-summary/src/collect/group_c/util.rs new file mode 100644 index 000000000..4aba9f470 --- /dev/null +++ b/rust/crates/og-summary/src/collect/group_c/util.rs @@ -0,0 +1,385 @@ +//! What every reader of group c shares: the table reader with its header, +//! the cell lookups with the Python collector's three answers (no such +//! column, a row cut short, a cell), and an ordered JSON object builder. + +use crate::collect::sections::{num, py_error}; +use crate::{basename, repr, text_lines}; +use indexmap::IndexMap; +use pycompat::csv::{DictReader, DictRow, Quoting}; +use pycompat::json::Value; + +/// A table read as `csv.DictReader` reads it, with its header cells. +pub struct Table { + /// The header cells as written; empty for an empty file. + pub head: Vec, + pub rows: Vec, +} + +impl Table { + /// Whether the header names `col` (`col in reader.fieldnames`). + pub fn has(&self, col: &str) -> bool { + self.head.iter().any(|h| h == col) + } + + /// Whether the first row has every column (`need <= set(rows[0])`): + /// false for a table without rows. + pub fn first_has(&self, need: &[&str]) -> bool { + !self.rows.is_empty() && need.iter().all(|k| self.has(k)) + } +} + +/// Reads a tab-separated table with a header. `quote_none` reads `"` as a +/// plain character (`quoting=csv.QUOTE_NONE`). +pub fn table(path: &str, quote_none: bool) -> Result { + let mut lines = text_lines(path).map_err(|e| os_note(&e))?; + let iter = std::iter::from_fn(move || { + let mut s = String::new(); + match lines.read_line(&mut s) { + Ok(true) => Some(Ok(s)), + Ok(false) => None, + Err(e) => Some(Err(e)), + } + }); + let q = if quote_none { + Quoting::None + } else { + Quoting::Minimal + }; + let mut rd = DictReader::new(iter, '\t', q); + let csv_err = |e: pycompat::csv::Error| py_error(e.exception(), &e.to_string()); + let head = rd + .header() + .map_err(csv_err)? + .map(|h| h.names().to_vec()) + .unwrap_or_default(); + let rows = rd.collect::, _>>().map_err(csv_err)?; + Ok(Table { head, rows }) +} + +/// The lines of a text file, line ends removed (`open(p, errors="replace")`). +pub fn lines(path: &str) -> Result, String> { + let mut rd = text_lines(path).map_err(|e| os_note(&e))?; + let mut out = Vec::new(); + let mut s = String::new(); + while rd + .read_line(&mut s) + .map_err(|e| py_error("OSError", &crate::collect::sections::gzip_message(&e)))? + { + out.push(s.strip_suffix('\n').unwrap_or(&s).to_owned()); + s.clear(); + } + Ok(out) +} + +/// An `OSError` from opening a file, as the collector's catch-all wrote it. +pub fn os_note(e: &str) -> String { + let kind = if e.contains("No such file") { + "FileNotFoundError" + } else if e.contains("Permission denied") { + "PermissionError" + } else if e.contains("Is a directory") { + "IsADirectoryError" + } else { + "OSError" + }; + py_error(kind, e) +} + +/// `KeyError: 'k'`, a column a row lacks. +pub fn key_error(k: &str) -> String { + py_error("KeyError", &repr(k)) +} + +/// `row[k]`: an error when the header has no such column, `None` for a +/// row cut short. +pub fn at<'a>(r: &'a DictRow, k: &str) -> Result, String> { + r.get(k).ok_or_else(|| key_error(k)) +} + +/// `row[k]` that must be a text: a short row is an error too. +pub fn text<'a>(r: &'a DictRow, k: &str) -> Result<&'a str, String> { + at(r, k)?.ok_or_else(|| { + py_error( + "TypeError", + &format!("the cell {} of a short row is None", repr(k)), + ) + }) +} + +/// `row.get(k) or "."`. +pub fn or_dot(r: &DictRow, k: &str) -> String { + or(r, k, ".") +} + +/// `row.get(k) or default`. +pub fn or(r: &DictRow, k: &str, default: &str) -> String { + match r.get(k).flatten() { + Some(v) if !v.is_empty() => v.to_owned(), + _ => default.to_owned(), + } +} + +/// `row.get(k, default)`: the default without the column, `null` for a +/// row cut short. +pub fn get_default(r: &DictRow, k: &str, default: &str) -> Value { + match r.get(k) { + None => s(default), + Some(None) => Value::Null, + Some(Some(v)) => s(v), + } +} + +/// Whether a row has a cell past the header (Python's `None` key) or is +/// cut short of a column of `need`. +pub fn ragged(r: &DictRow, need: &[&str]) -> bool { + r.extra().is_some() || need.iter().any(|k| !matches!(r.get(k), Some(Some(_)))) +} + +/// `str.isdigit()` of an ASCII cell. +pub fn isdigit(x: &str) -> bool { + pycompat::text::isdigit(x) +} + +/// `float(x)` that is a number and not NaN. +pub fn number(x: &str) -> Option { + num(Some(x)) +} + +/// `os.path.basename`. +pub fn base(p: &str) -> &str { + basename(p) +} + +pub fn s(x: &str) -> Value { + Value::Str(x.to_owned()) +} + +pub fn opt_s(x: Option<&str>) -> Value { + x.map_or(Value::Null, s) +} + +pub fn strs(xs: I) -> Value +where + I: IntoIterator, + S: AsRef, +{ + Value::Array(xs.into_iter().map(|x| s(x.as_ref())).collect()) +} + +pub fn int(n: impl Into) -> Value { + Value::Int(n.into()) +} + +pub fn count(n: usize) -> Value { + Value::Int(n as i128) +} + +/// An object built in insertion order. +#[derive(Debug, Default, Clone)] +pub struct Obj(pub IndexMap); + +impl Obj { + pub fn new() -> Obj { + Obj::default() + } + + pub fn set(mut self, k: &str, v: Value) -> Obj { + self.0.insert(k.to_owned(), v); + self + } + + pub fn put(&mut self, k: &str, v: Value) { + self.0.insert(k.to_owned(), v); + } + + pub fn val(self) -> Value { + Value::Object(self.0) + } +} + +/// `{k: row.get(k) or "." for k in keys}`. +pub fn dots(r: &DictRow, keys: &[&str]) -> Value { + keys.iter() + .fold(Obj::new(), |o, k| o.set(k, s(&or_dot(r, k)))) + .val() +} + +/// `{k: row[k] or "." for k in keys}`: a column the row lacks is an error. +pub fn dots_strict(r: &DictRow, keys: &[&str]) -> Result { + let mut o = Obj::new(); + for k in keys { + let v = at(r, k)?.filter(|v| !v.is_empty()).unwrap_or("."); + o.put(k, s(v)); + } + Ok(o.val()) +} + +/// `{r[key]: r.get(value) for r in rows}` of a `metric\tvalue` table: the +/// last row of a key wins, in the place of its first. +pub fn metric_map(t: &Table, key: &str, value: &str) -> IndexMap, Option> { + let mut out = IndexMap::new(); + for r in &t.rows { + let k = r.get(key).flatten().map(str::to_owned); + let v = r.get(value).flatten().map(str::to_owned); + out.insert(k, v); + } + out +} + +/// `kv.get(k) or "."` over a [`metric_map`]. +pub fn kv_dot(kv: &IndexMap, Option>, k: &str) -> String { + match kv.get(&Some(k.to_owned())) { + Some(Some(v)) if !v.is_empty() => v.clone(), + _ => ".".to_owned(), + } +} + +/// `kv.get(k)` over a [`metric_map`]. +pub fn kv_get<'a>(kv: &'a IndexMap, Option>, k: &str) -> Option<&'a str> { + kv.get(&Some(k.to_owned())).and_then(|v| v.as_deref()) +} + +/// `[(metric, value)]` of a two-column table with a header line +/// (`read_tsv_pairs`): blank lines skipped, short lines left out. +pub fn pairs(path: &str) -> Result, String> { + let ls = lines(path)?; + let ls: Vec<&String> = ls + .iter() + .filter(|l| !pycompat::num::py_strip(l).is_empty()) + .collect(); + Ok(ls + .iter() + .skip(1) + .filter_map(|l| { + let c: Vec<&str> = l.split('\t').collect(); + (c.len() >= 2).then(|| (c[0].to_owned(), c[1].to_owned())) + }) + .collect()) +} + +/// `round(x, n)` as a JSON float. +pub fn round(x: f64, n: i32) -> Value { + Value::Float(pycompat::fmt::round(x, n).unwrap_or(x)) +} + +/// The rows of `assets/`, comment lines left out (`asset_rows`). +pub fn asset_rows(assets: &str, name: &str) -> Result, String> { + let p = crate::join(assets, name); + if !crate::is_file(&p) { + return Err(format!( + "{name} is not in {assets}: the report needs the repository's assets/ folder (render_report.py --assets)" + )); + } + let text = std::fs::read(&p).map_err(|e| py_error("OSError", &e.to_string()))?; + let text = + String::from_utf8(text).map_err(|e| py_error("UnicodeDecodeError", &e.to_string()))?; + let lines = text + .split_inclusive('\n') + .filter(|l| !l.starts_with('#')) + .map(|l| Ok::<_, std::io::Error>(l.to_owned())); + DictReader::new(lines, '\t', Quoting::None) + .collect::, _>>() + .map_err(|e| py_error(e.exception(), &e.to_string())) +} + +/// The header of a table as `csv.reader` reads its first row (empty for an +/// empty file). +pub fn columns(path: &str) -> Result, String> { + Ok(table(path, false)?.head) +} + +/// The folder of the tables the Python collector read beside itself in +/// `bin/` (`hla_alleles.tsv`, `str_intermediate_ranges.tsv`): the `bin/` of +/// the checkout the binary sits in (`bin/.og/og-*` and `rust/target/*/og` +/// both do). `OG_BIN_TABLES` names another folder. +pub fn default_tables() -> String { + if let Some(d) = std::env::var_os("OG_BIN_TABLES") { + return std::path::Path::new(&d).display().to_string(); + } + let exe = std::env::current_exe().unwrap_or_default(); + exe.ancestors() + .skip(1) + .flat_map(|a| [a.to_path_buf(), a.join("bin")]) + .find(|p| p.join("hla_alleles.tsv").is_file()) + .unwrap_or_else(|| std::path::PathBuf::from("bin")) + .display() + .to_string() +} + +/// The data rows of a `bin/` table: `#` and blank lines left out, the first +/// row the header, each row zipped with it (a short row lacks the keys). +pub fn bin_rows(path: &str) -> Result>, String> { + let text = std::fs::read(path).map_err(|e| { + py_error( + if e.kind() == std::io::ErrorKind::NotFound { + "FileNotFoundError" + } else { + "OSError" + }, + &crate::os_error(&e, path), + ) + })?; + let text = + String::from_utf8(text).map_err(|e| py_error("UnicodeDecodeError", &e.to_string()))?; + let mut head: Option> = None; + let mut out = Vec::new(); + for line in text.lines() { + if line.starts_with('#') || pycompat::num::py_strip(line).is_empty() { + continue; + } + let c: Vec<&str> = line.split('\t').collect(); + match &head { + None => head = Some(c), + Some(h) => out.push( + h.iter() + .zip(c) + .map(|(k, v)| ((*k).to_owned(), v.to_owned())) + .collect(), + ), + } + } + Ok(out) +} + +/// Each data line of a VCF, plain or gzip (BGZF inflated on many cores), +/// without its line end; `#` lines are left out. Universal newlines, as +/// Python reads text. An error is `: `. +pub fn vcf_lines(path: &str, mut each: impl FnMut(&[u8])) -> Result<(), String> { + use std::io::BufRead; + let fail = |e: std::io::Error| { + format!( + "{}: {}", + base(path), + crate::collect::sections::gzip_message(&e) + ) + }; + let mut r = og_io::bgzf::open(path, og_io::bgzf::workers()).map_err(fail)?; + let mut buf = Vec::with_capacity(1 << 16); + let mut one = |l: &[u8]| { + if l.first() != Some(&b'#') { + each(l); + } + }; + loop { + buf.clear(); + if r.read_until(b'\n', &mut buf).map_err(fail)? == 0 { + break; + } + let line = buf.strip_suffix(b"\n").unwrap_or(&buf); + if memchr::memchr(b'\r', line).is_none() { + one(line); + } else { + let line = line.strip_suffix(b"\r").unwrap_or(line); + line.split(|&c| c == b'\r').for_each(&mut one); + } + } + Ok(()) +} + +/// A VCF line's cells, as text (`errors="replace"`). +pub fn cells(line: &[u8]) -> Vec { + String::from_utf8_lossy(line) + .split('\t') + .map(str::to_owned) + .collect() +} diff --git a/rust/crates/og-summary/src/collect/scan.rs b/rust/crates/og-summary/src/collect/scan.rs index 2792c67d6..d50bd251a 100644 --- a/rust/crates/og-summary/src/collect/scan.rs +++ b/rust/crates/og-summary/src/collect/scan.rs @@ -88,21 +88,23 @@ impl FolderIndex { None } - /// Every file one pattern names, in sorted order (`glob` with the - /// order made stable), for the readers that take the newest of them. + /// Every file of one pattern, sorted (`sorted(glob(pattern))`, files + /// only). pub fn all(&self, pattern: &str) -> Vec<&str> { if !has_magic(pattern) { return self .files .get(pattern) - .map(String::as_str) - .into_iter() - .collect(); + .map(|f| vec![f.as_str()]) + .unwrap_or_default(); } let pat: Vec<&str> = pattern.split('/').collect(); self.sorted .iter() - .filter(|f| matches(&f.split('/').collect::>(), &pat)) + .filter(|f| { + let parts: Vec<&str> = f.split('/').collect(); + matches(&parts, &pat) + }) .map(String::as_str) .collect() } diff --git a/rust/crates/og-summary/src/collect/sections.rs b/rust/crates/og-summary/src/collect/sections.rs index a719a593a..a354c126f 100644 --- a/rust/crates/og-summary/src/collect/sections.rs +++ b/rust/crates/og-summary/src/collect/sections.rs @@ -20,6 +20,8 @@ pub enum Data { MosaicSexLoss(Mosaic), ReadCount(ReadCount), Variants(Variants), + /// A section of group c, kept as the ordered JSON it writes. + Json(Value), ClinVar(super::clinvar::ClinVar), Carrier(super::carrier::Carrier), PharmCat(super::pgx::PharmCat), @@ -41,6 +43,7 @@ impl Data { Data::MosaicSexLoss(d) => d.to_json(), Data::ReadCount(d) => d.to_json(), Data::Variants(d) => d.to_json(), + Data::Json(v) => v.clone(), Data::ClinVar(d) => d.to_json(), Data::Carrier(d) => d.to_json(), Data::PharmCat(d) => d.to_json(), @@ -768,7 +771,7 @@ fn chars(b: &[u8]) -> usize { /// What Python's gzip says for the same fault, where the two are the same /// fault; otherwise the reader's own words. -fn gzip_message(e: &std::io::Error) -> String { +pub(crate) fn gzip_message(e: &std::io::Error) -> String { if e.kind() == std::io::ErrorKind::UnexpectedEof { "Compressed file ended before the end-of-stream marker was reached".to_owned() } else { diff --git a/rust/crates/og-summary/src/collect/tests.rs b/rust/crates/og-summary/src/collect/tests.rs index 36e41683e..75031f745 100644 --- a/rust/crates/og-summary/src/collect/tests.rs +++ b/rust/crates/og-summary/src/collect/tests.rs @@ -21,6 +21,8 @@ fn summary(d: &Dir, sample_dir: &str) -> Value { sample_dir: d.path(sample_dir), declared_sex: None, assets: d.path("assets"), + group: super::Group::A, + tables: String::new(), }; collect(&input).unwrap().to_json() } diff --git a/rust/og/src/main.rs b/rust/og/src/main.rs index c32cb0e67..fa17bed77 100644 --- a/rust/og/src/main.rs +++ b/rust/og/src/main.rs @@ -50,7 +50,8 @@ const USAGE: &str = "usage: og [args] og prs format [args] og prs table [args] og mito annotate|deletions|copies [args] - og summary collect --only a|b [args] + og summary collect --only {a,b,c} [args] + og roh summary --sample SAMPLE --roh ROH [--out OUT] og chip prepare|report|check-table [args] og cn bed|call|intervals|check-bam|marker-regions|marker-read [args] og bloodtype report|check-markers|systems [args] @@ -305,6 +306,14 @@ fn main() -> ExitCode { )), _ => usage_error("og summary takes collect"), }, + "roh" => match rest.first().map(String::as_str) { + Some("summary") => ExitCode::from(og_summary::collect::group_c::roh::cli( + &rest[1..], + &mut std::io::stdout().lock(), + &mut std::io::stderr().lock(), + )), + _ => usage_error("og roh takes summary"), + }, "mito" => ExitCode::from(og_summary::mito::cli( rest.first().map(String::as_str), rest.get(1..).unwrap_or_default(), diff --git a/scripts/11-roh-analysis.sh b/scripts/11-roh-analysis.sh index af2dfa241..9b29031ec 100755 --- a/scripts/11-roh-analysis.sh +++ b/scripts/11-roh-analysis.sh @@ -2,7 +2,7 @@ # ROH Analysis — Runs of homozygosity (consanguinity screening) # Detects long stretches of homozygous DNA that indicate shared ancestry. # The summary keeps chr1 to chr22 only and leaves out the part of each segment in a centromere -# or heterochromatin band (MASKED_BANDS below), where few markers map. +# or heterochromatin band (og roh summary's MASKED_BANDS), where few markers map. set -euo pipefail SAMPLE=${1:?Usage: $0 } @@ -44,30 +44,18 @@ run_in \ "${BCFTOOLS_IMAGE}" \ bash -euo pipefail -c "bcftools view -f PASS,. -Ou /data/${SAMPLE}.vcf.gz | bcftools roh ${ROH_FLAGS[*]} -o /data/${SAMPLE}_roh.txt -" -# GRCh38's centromere and heterochromatin bands: the acen, gvar and stalk -# rows of UCSC's cytoBand.hg38.txt, neighbouring rows merged (0-based start, -# exclusive end). bin/collect_summary.py (MASKED_BANDS, which says how to -# regenerate it) and modules/local/roh/main.nf hold the same line. -MASKED_BANDS="chr1:121700000-143200000 chr2:91800000-96000000 chr3:87800000-98600000 chr4:48200000-51800000 chr5:46100000-51400000 chr6:58500000-62600000 chr7:58100000-62100000 chr8:43200000-47200000 chr9:42200000-61500000 chr10:38000000-41600000 chr11:51000000-55800000 chr12:33200000-37800000 chr13:0-18900000 chr14:0-18200000 chr15:0-20500000 chr16:35300000-47000000 chr17:22700000-27400000 chr18:15400000-21500000 chr19:19900000-31900000 chr20:25700000-30400000 chr21:0-13000000 chr22:0-17400000 chrX:58100000-63800000 chrY:10300000-10600000 chrY:26600000-57227415" - -# Summary: autosomal segments with 5 Mb or more outside those bands, the -# threshold of docs/11-roh-analysis.md and of the reports; length_bp and -# length_mb are of that part. The Nextflow module writes the same file. +# Summary: autosomal segments with 5 Mb or more outside GRCh38's centromere +# and heterochromatin bands (the acen, gvar and stalk rows of UCSC's +# cytoBand.hg38.txt, merged; MASKED_BANDS in og's step 11 code, the one copy), +# the threshold of docs/11-roh-analysis.md and of the reports; length_bp and +# length_mb are of that part. The Nextflow module runs the same command. SUMMARY="${GENOME_DIR}/${SAMPLE}/vcf/${SAMPLE}_roh_summary.txt" -{ - echo "# ROH Summary for ${SAMPLE}" - echo "# Segments >=5MB on autosomes (excludes chrX/chrY)" - printf 'chrom\tstart\tend\tlength_bp\tlength_mb\n' - awk -v bands="$MASKED_BANDS" ' - BEGIN { n = split(bands, b, " "); for (i = 1; i <= n; i++) { split(b[i], x, /[:-]/); c[i] = x[1]; s[i] = x[2]; e[i] = x[3] } } - $1 == "RG" && $3 ~ /^chr([1-9]|1[0-9]|2[0-2])$/ { - m = 0 - for (i = 1; i <= n; i++) if (c[i] == $3) { o = ($5 < e[i] ? $5 : e[i]) - ($4 - 1 > s[i] ? $4 - 1 : s[i]); if (o > 0) m += o } - if (m > $6) m = $6 - l = $6 - m - if (l >= 5000000) printf "%s\t%s\t%s\t%d\t%.1f\n", $3, $4, $5, l, l / 1e6 - }' "$OUTPUT" -} > "${SUMMARY}.tmp" +run_in \ + --cpus 1 --memory 1g \ + -v "${GENOME_DIR}/${SAMPLE}/vcf:/data" \ + -v "${PGP_ROOT}/bin:/pgp-bin:ro" \ + "${BCFTOOLS_IMAGE}" \ + /pgp-bin/og roh summary --sample "$SAMPLE" --roh "/data/${SAMPLE}_roh.txt" > "${SUMMARY}.tmp" mv -f "${SUMMARY}.tmp" "$SUMMARY" echo "=== ROH complete ===" diff --git a/tests/parity/cases.tsv b/tests/parity/cases.tsv index 191f8e4dd..ab0f2f172 100644 --- a/tests/parity/cases.tsv +++ b/tests/parity/cases.tsv @@ -838,6 +838,24 @@ summary-collect empty gen:summary-collect.sh,summary-collect/oracle.sh --sample summary-collect declared-arg gen:summary-collect.sh,summary-collect/oracle.sh --sample S --sample-dir declared-status --assets assets --declared-sex female --out declared-arg.json --only a declared-arg.json json:paths=schema_version,sample,run,databases,manifest,sections.coverage,sections.callability,sections.sex_check,sections.sample_qc,sections.mosaic_sex_loss,sections.read_count,sections.variants PYTHON_IMAGE summary-collect no-assets gen:summary-collect.sh,summary-collect/oracle.sh --sample S --sample-dir full --assets nowhere --out no-assets.json --only a no-assets.json json:paths=schema_version,sample,run,databases,manifest,sections.coverage,sections.callability,sections.sex_check,sections.sample_qc,sections.mosaic_sex_loss,sections.read_count,sections.variants PYTHON_IMAGE summary-collect no-folder gen:summary-collect.sh,summary-collect/oracle.sh --sample S --sample-dir no-such-folder --assets assets --out no-folder.json --only a no-folder.json json:paths=schema_version,sample,run,databases,manifest,sections.coverage,sections.callability,sections.sex_check,sections.sample_qc,sections.mosaic_sex_loss,sections.read_count,sections.variants PYTHON_IMAGE +summary-collect c-full gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir full --assets assets --out full.json --only c full.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-float-nan gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir float-nan --assets assets --out float-nan.json --only c float-nan.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-float-inf gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir float-inf --assets assets --out float-inf.json --only c float-inf.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-float--inf gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir float--inf --assets assets --out float--inf.json --only c float--inf.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-nf gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir nf --assets assets --out nf.json --only c nf.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-bad1 gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir bad1 --assets assets --out bad1.json --only c bad1.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-bad2 gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir bad2 --assets assets --out bad2.json --only c bad2.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-left gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-left --assets assets --out g2-left.json --only c g2-left.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-shifted gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-shifted --assets assets --out g2-shifted.json --only c g2-shifted.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-two gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-two --assets assets --out g2-two.json --only c g2-two.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-gvcf gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-gvcf --assets assets --out g2-gvcf.json --only c g2-gvcf.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-none gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-none --assets assets --out g2-none.json --only c g2-none.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-other gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-other --assets assets --out g2-other.json --only c g2-other.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-nocall gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-nocall --assets assets --out g2-nocall.json --only c g2-nocall.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +summary-collect c-g2-novcf gen:summary-collect-c.sh,summary-collect/oracle.sh --sample S --sample-dir g2-novcf --assets assets --out g2-novcf.json --only c g2-novcf.json json:paths=schema_version,sample,run,databases,manifest,sections.couple,sections.parascopy,sections.paralogs,sections.hla,sections.prs,sections.ancestry_fine,sections.manta,sections.delly,sections.cnvpytor,sections.sv_consensus,sections.annotsv,sections.expansions,sections.telomere,sections.roh,sections.haplogroup,sections.y_haplogroup,sections.y_tree,sections.y_str,sections.consumer_export,sections.bloodtype,sections.lpa,sections.vntyper,sections.locityper,sections.mito,sections.traits,sections.archaic,sections.ancient,sections.viral,sections.exomiser,sections.chip,sections.mei,sections.mosaic PYTHON_IMAGE +roh-summary runs gen:roh-summary.sh,roh-summary/oracle.sh --sample S --roh roh.txt --out S_roh_summary.txt S_roh_summary.txt bytes PYTHON_IMAGE +roh-summary empty gen:roh-summary.sh,roh-summary/oracle.sh --sample sample.1_A-b --roh empty.txt --out out.txt out.txt bytes PYTHON_IMAGE +roh-summary stdout gen:roh-summary.sh,roh-summary/oracle.sh --sample S --roh roh.txt - bytes PYTHON_IMAGE summary-collect cb-full gen:summary-clinical.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir cb-full --assets assets --out cb-full.json --only b cb-full.json json:paths=schema_version,sample,run,databases,manifest,sections.clinvar,sections.carrier,sections.pharmcat,sections.cpic,sections.pypgx,sections.cyrius,sections.cpsr,sections.clinical,sections.slivar,cyp2d6 PYTHON_IMAGE summary-collect cb-quoted-smn gen:summary-clinical.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir cb-quoted-smn --assets assets --out cb-quoted-smn.json --only b cb-quoted-smn.json json:paths=schema_version,sample,run,databases,manifest,sections.clinvar,sections.carrier,sections.pharmcat,sections.cpic,sections.pypgx,sections.cyrius,sections.cpsr,sections.clinical,sections.slivar,cyp2d6 PYTHON_IMAGE summary-collect cb-quoted-interval gen:summary-clinical.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir cb-quoted-interval --assets assets --out cb-quoted-interval.json --only b cb-quoted-interval.json json:paths=schema_version,sample,run,databases,manifest,sections.clinvar,sections.carrier,sections.pharmcat,sections.cpic,sections.pypgx,sections.cyrius,sections.cpsr,sections.clinical,sections.slivar,cyp2d6 PYTHON_IMAGE diff --git a/tests/parity/gen/roh-summary.sh b/tests/parity/gen/roh-summary.sh new file mode 100755 index 000000000..ecfd467c1 --- /dev/null +++ b/tests/parity/gen/roh-summary.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# roh-summary.sh: writes the bcftools roh outputs of the roh-summary parity +# cases in the current directory, the same every time: runs on chr1 to +# chr22 inside, across and outside the masked bands, runs on chrX, chrY and +# other contigs, the 5 Mb line from both sides, lengths whose megabases +# round on a half, blank-separated fields and lines that are not runs. +# scripts/ci/oracle-diff.sh runs it. +set -euo pipefail +export LC_ALL=C + +{ + printf '# This file was produced by: bcftools roh\n# RG\t[2]Sample\t[3]Chromosome\t[4]Start\t[5]End\t[6]Length (bp)\t[7]Number of markers\t[8]Quality (average fwd-bwd phred score)\n' + printf 'ST\tS\tchr1\t100\t0\t1.0\n' + printf 'RG\tS\tchrY\t2781480\t47781479\t45000000\t900\t50\n' + printf 'RG\tS\tchrX\t1\t9000000\t9000000\t900\t50\n' + printf 'RG\tS\tchrEBV\t1\t6000000\t6000000\t90\t50\n' + printf 'RG\tS\tchr1_KI270706v1_random\t1\t6000000\t6000000\t90\t50\n' + printf 'RG\tS\tchr1\t125200001\t143200000\t18000000\t300\t50\n' + printf 'RG\tS\tchr9\t42300001\t49300000\t7000000\t200\t50\n' + printf 'RG\tS\tchr1\t120000001\t145000000\t25000000\t500\t50\n' + printf 'RG\tS\tchr2\t10000001\t17000000\t7000000\t900\t50\n' + printf 'RG\tS\tchr13\t1\t25000000\t25000000\t900\t50\n' + printf 'RG\tS\tchr22\t17000001\t40000000\t23000000\t900\t50\n' + printf 'RG\tS\tchr3\t1\t5000000\t5000000\t900\t50\n' + printf 'RG\tS\tchr4\t1\t4999999\t4999999\t900\t50\n' + printf 'RG\tS\tchr5\t1\t5050000\t5050000\t900\t50\n' + printf 'RG\tS\tchr6\t1\t5150000\t5150000\t900\t50\n' + printf 'RG\tS\tchr7\t1\t5250000\t5250000\t900\t50\n' + printf 'RG S chr8 1 6000000 6000000 900 50\n' + printf ' RG\tS\tchr10\t100\t6000099\t6000000\t9\t9\n' + printf 'RG\tS\tchr23\t1\t9000000\t9000000\t900\t50\n' + printf 'RG\tS\tchr21\t1\t9000000\n' + printf 'RG\tS\n' + printf '\n' + printf 'RG\tS\tchr11\t50000001\t60000000\t10000000\t9\t9\n' + printf 'RG\tS\tchr12\t1\t8000000\t8000000.5\t9\t9\n' +} > roh.txt +printf '# empty\n' > empty.txt diff --git a/tests/parity/gen/summary-collect-c.sh b/tests/parity/gen/summary-collect-c.sh new file mode 100644 index 000000000..9c853a9e4 --- /dev/null +++ b/tests/parity/gen/summary-collect-c.sh @@ -0,0 +1,414 @@ +#!/usr/bin/env bash +# summary-collect-c.sh: writes the sample folders of the summary-collect +# parity cases of group c in the current directory, the same every time. +# Each folder is one case of og summary collect --only c against +# bin/collect_summary.py: every section of the group with a typical table, +# the Nextflow and other file names, files from an earlier run, unreadable +# tables of each kind, and the APOL1 G2 deletion written left-aligned, +# shifted, twice, under a reference block and not at all. +# py/ holds the modules the oracle imports, as they were at the oracle's +# commit, and the two bin/ tables it reads beside itself; assets/ is a copy +# of the repository's catalogs these sections read. +# scripts/ci/oracle-diff.sh runs it. +set -euo pipefail +export LC_ALL=C TZ=UTC +here=$(cd "$(dirname "$0")" && pwd) +root=$(cd "${here}/../../.." && pwd) +ORACLE_SHA=98aa9252ff77825a8a797d388b449931fa9a234e + +T0=1577836800 # 2020-01-01 00:00:00 UTC, the start of the run +OLD=201906011230.00 # before the run +NEW=202103040506.07 # after it, the time of every file not set otherwise + +at() { touch -t "$1" "${@:2}"; } # [[CC]YY]MMDDhhmm[.SS], in UTC +put() { mkdir -p "$(dirname "$1")"; printf '%b' "$2" > "$1"; at "$NEW" "$1"; } +putgz() { mkdir -p "$(dirname "$1")"; printf '%b' "$2" | gzip -n -c > "$1"; at "$NEW" "$1"; } + +# --- the oracle's modules ------------------------------------------------------- +git -C "$root" cat-file -e "${ORACLE_SHA}^{commit}" 2> /dev/null \ + || git -C "$root" fetch -q --depth=1 origin "$ORACLE_SHA" +mkdir -p py +for m in paraphase_report ancestry_fine_report annotsv_postfilter cnvpytor_to_vcf family_report chip_report \ + traits_report genotype_rules_indels pgx_outside_calls; do + git -C "$root" show "${ORACLE_SHA}:bin/${m}.py" > "py/${m}.py" +done +for t in hla_alleles.tsv str_intermediate_ranges.tsv; do + git -C "$root" show "${ORACLE_SHA}:bin/${t}" > "py/${t}" +done + +mkdir -p assets +for a in paraphase_genes.tsv paralog_loci.tsv mito_confirmed.tsv callability_genes.tsv; do + cp "${root}/assets/${a}" assets/ +done + +# All calls below are invented. Share the FORMAT header across repeat fixtures. +EH_FORMAT='GT:SO:REPCN:REPCI:ADSP' +VCF_HEAD='##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n' + +# full: every section, the bash names ------------------------------------------- +f=full +put $f/couple/S_couple_carrier.tsv "gene\tcondition\tresult\toffspring_risk\tpartner_a\tfindings_a\tpartner_b\tfindings_b\texplanation +CFTR\tCystic fibrosis\tshared\t1 in 4\tS\tc.1521_1523del (P)\tP2\tc.350G>A (LP)\tBoth carry a change. +DMD\tDuchenne\tx_linked\t1 in 4 sons\tP2\t.\tS\tc.10G>T (P)\tOn X. +GJB2\tDeafness\tone_partner\t.\tS\tc.35del (P)\tP2\t.\tOne side. +HBB\tBeta thalassaemia\tone_partner\t.\tP2\tc.20A>T\tS\t.\tOne side.\n" +put $f/paralogs/S_smn_copy_number.tsv "chrom\tstart\tend\tlocus\tagCN_filter\tagCN\tagCN_qual\tpsCN_filter\tpsCN\tpsCN_qual\thomologous_regions +chr5\t70950000\t70953000\tSMN1\tPASS\t4\t45.5\tPASS\t1,3\t31,25\tchr5:70075000-70078000 +chr5\t70960000\t70970000\tSMN1\tPASS\t4\t40\tLowQual\t2,2\t10,12\t*\n" +put $f/paralogs/S_smn1_marker.tsv "metric\tvalue\nmarker\tSMN1 g.27134T>G\nstatus\tnot seen\nalt_reads\t0\ndepth\t31\nfraction\t0.000\ncopies\t0\nmessage\tno read carries the G\n" +put $f/paralogs/S_gba1_copy_number.tsv "chrom\tstart\tend\tagCN_filter\tagCN\tagCN_qual\tpsCN_filter\tpsCN\tpsCN_qual +chr1\t155231478\t155240000\tPASS\t4\t60\tPASS\t2,2\t30,30 +chr1\t155240000\t155244699\tPASS\t4\t35.25\tPASS\t2,2\t30,30\n" +put $f/paralogs/S_pms2_copy_number.tsv "chrom\tstart\tend\tagCN_filter\tagCN\tagCN_qual\tpsCN_filter\tpsCN\tpsCN_qual +chr7\t5970000\t5980000\tPASS\t4\t50\tPASS\t2,2\t30,30 +chr7\t5980000\t5990000\tLowQual\t3\t12\tPASS\t2,1\t30,30\n" +put $f/paralogs/S_mytable_copy_number.tsv "chrom\tstart\tend\tagCN_filter\tagCN\tagCN_qual\tpsCN_filter\tpsCN\tpsCN_qual +chr9\t1\t100\tnot assessed: no read over the locus\t.\t.\t.\t.\t.\n" +put $f/paralogs/S_strc_copy_number.tsv "chrom\tstart\tend\tagCN\n" +put $f/paralogs/S_hba_copy_number.tsv "metric\tvalue\nstatus\tcalled\ncopies\t4\nreading\ttwo copies of each gene\n" +put $f/paralogs/S_cn_intervals.tsv "locus\tname\tchrom\tstart\tend\tstatus\tcopies\texpected\tchange +DGCR\t22q11.2\tchr22\t18900000\t21500000\tcalled\t2\t2\tnone +AZFc\tAZFc\tchrY\t23000000\t26000000\tcalled\t0\t1\tdeletion +X1\tdup\tchr1\t1\t2\tcalled\t3\t2\tduplication +X2\tlow\tchr2\t1\t2\tnot assessed\t.\t2\t.\n" +put $f/paralogs/S_cyp21a2_psv.tsv "metric\tvalue\nstatus\tcalled\ncopies\t2\nmodule_total\t4\nfinding\tnone\n" +put $f/paralogs/S_cn_dosage.tsv "locus\tstatus\tcopies\testimate\treason\treading +HP\tcalled\t2\t2.1\t\tusual +LCE3BC\tcalled\t1\t1.0\t.\tone deleted +CCL3L1\tnot assessed\t\t\tlow depth\t\n" +put $f/paralogs/S_paraphase_copy_number.tsv "region\tgene\tcopy_number\tstatus\tdetail +pms2\tPMS2\t2\tcalled\t +smn1\tSMN1\t2\tcalled\tok +smn1\tSMN2\t2\tcalled\t +zz_other\tX\t2\tcalled\t. +zz_other2\tY\t.\tno call\t. +hba\tHBA1\t1\tcalled\t +hba\tHBA2\t2\tcalled\t\n" +put $f/paralogs/S_paraphase_variants.tsv "region\tpos\nsmn1\t1\nsmn1\t2\npms2\t3\n" +put $f/hla_t1k/S_hla_genotype.tsv "#comment\tline +HLA-A\t2\tA*02:01:01\t10\t60\tA*01:01:01\t9\t50 +HLA-B\t2\tB*57:01:01:02\t10\t0\tB*08:01:01,B*08:01:02,B*08:01:03,B*08:01:04,B*08:01:05\t9\t0 +HLA-C\t1\tC*07:01:01\t10\t40\t.\t0\t-1 +HLA-DQA1\t2\tDQA1*05:01:01\t10\t40\tDQA1*02:01\t9\t30 +HLA-DQB1\t2\tDQB1*02:01:01\t10\t40\tDQB1*02:02:01\t9\t30 +HLA-DRB1\t2\tDRB1*03:01:01\t10\t40\tDRB1*07:01:01\t9\tnan +HFE\t1\tHFE*001\t3\t20\t.\t0\t-1 +short\t1\n" +put $f/hla_optitype/S_hla_concordance.tsv "Gene\tT1K\tOptiType\tAgree\tNote\nHLA-A\t A*02:01 \tA*02:01\tyes\t\n\t\t\t\t\nHLA-B\tB*57:01\tB*57:01\tyes\tok\n" +put $f/kir_t1k/S_kir_genotype.tsv "KIR2DL1\t2\tKIR2DL1*001\t5\t30\tKIR2DL1*002\t4\t20\nKIR3DL3\t0\t.\t0\t-1\t.\t0\t-1\n" +put $f/prs/S_prs_summary.tsv "Condition\tPGS_ID\tScore_SUM\tVariants_Matched\tVariants_Total\tChrX\tAmbiguous\tApplies_To\tSex_Check\tPercentile\tAncestry_Group\tEvaluated_In\tEffect +CAD\tPGS000018\t0.5\t100\t120\t0\t12 dropped\tall\tok\t75\tEUR\tEUR;;AFR\tHR 1.5 (EUR) +T2D\tPGS000014\t0.1\t10\t20\t0\t0\tall\tok\tNA\tNA\t\t +Short\tPGS1\n" +put $f/ancestry/S_ancestry.tsv "key\tvalue\npopulation\tEUR\nreference_panel\tHGDP+1kGP\n\tskipped\npopulation_low_confidence\n" +put $f/ancestry_fine/S_ancestry_fine.tsv "level\tcode\tgroup\tfraction\tse\tlow\thigh +continent\tWEA\tWest Eurasia\t0.95\t0.01\t0.93\t0.97 +continent\tAFR\tAfrica\t0.05\t0.01\tnan\tx +region\tIBE\tIberia\t0.60\t0.05\t0.5\t0.7 +left_out\tOCE\tOceania\t.\t.\t.\t. +not_painted\tchr21\t.\t.\t.\t.\t. +other\tZ\tZ\t1\t1\t1\t1\n" +put $f/ancestry_fine/S_painting.png "\x89PNG\r\n\x1a\nnot a real picture\x00\xff" +putgz $f/manta/results/variants/diploidSV.vcf.gz "${VCF_HEAD}chr1\t100\tM1\tA\t\t50\tPASS\tSVTYPE=DEL\tGT\t0/1\nchr1\t200\tM2\tA\t\t50\tMinQUAL\tSVTYPE=DUP\tGT\t0/1\nchr2\t5\n" +putgz $f/delly/S_sv.vcf.gz "${VCF_HEAD}chr1\t100\tD1\tA\t\t50\tPASS\tSVTYPE=DEL\tGT\t0/1\n" +put $f/cnvpytor/S_cnvs.txt "deletion\tchr2:10000001-10100000\t100000\t0.4\t1e-9\t1\t1\t1\t0.01\t0\t9 +deletion\tchr1:122000001-123000000\t1000000\t0.1\t1e-12\t1\t1\t1\t0.2\t0\t9 +duplication\tchr3:1000001-1200000\t200000\t1.8\t1e-6\t1\t1\t1\t0.8\t0\t9 +deletion\tchr5:1-100000\t100000\t0.0\t1e-20\t1\t1\t1\t0\t0.9\t0 +deletion\tchrY:2800001-2900000\t100000\t0.0\t1e-15\t1\t1\t1\t0.05\t0\t9 +deletion\tchr4:1000001-1050000\t50000\t0.6\t0.2\t1\t1\t1\t0\t0\t9 +duplication\tchr6:1-2\t1\t2\tnan\t1\t1\t1\t0\t0\t9 +duplication\tweird\t1\t2\t0.001 + +\n" +putgz $f/sv_merged/S_sv_consensus.vcf.gz "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tmanta\tdelly\ttiddit +chr1\t1000\tc1\tN\t\t.\tPASS\tSVTYPE=DEL;END=5000;SUPP=2;SUPP_VEC=110\tGT\t0/1\t0/1\t./. +chr2\t100\tc2\tN\t\t.\tPASS\tSVTYPE=DUP:TANDEM;END=900;SUPP=3;SUPP_VEC=1111\tGT\t0/1\t0/1\t0/1 +chr3\tx\tc3\tN\t\t.\tPASS\tSVTYPE=INV\tGT\t0/1\t0/1\t0/1\n" +put $f/annotsv/S_sv_class4_5_checked.tsv "AnnotSV_ID\tSV_chrom\tSV_start\tSV_end\tSV_length\tSV_type\tGene_name\tACMG_class\tFORMAT\tS\tAnnotation_mode\tCaller\tExon_edge\tExon_edge_detail\tPopulation_AF\tPopulation_source\tCommon\tAnnotSV_class\tRule\tRule_detail +a1\t1\t1100\t5100\t-4000\tDEL\tGENE1;GENE2;\t5\tGT:PS\t0/1:77\tsplit\tmanta\tyes\texon 2\t0.001\tgnomAD\tno\t5\t\t +a2\t2\t150\t880\t730\tDUP\tGENE3\t4\tGT\t1/1\tsplit\tdelly\tno\t\t\t\tno\t\t\t +a3\t7\t1\t100\t99\tINV\t\t3\tGT\t0/1\tsplit\tmanta\tnot checked\t\tx\tgnomAD\tyes\t4\tmhc\tthe MHC +a4\t9\t1\t100\t99\tDEL\tG9\t4\t\t\tfull\ttiddit\tno\t\t0.2\tgnomAD\tyes\t\t\t\n" +put $f/expansion_hunter/S_eh.vcf "${VCF_HEAD}chr4\t3074877\t.\tC\t\t.\tPASS\tEND=3074940;REF=21;RL=63;RU=CAG;VARID=HTT;REPID=HTT\t${EH_FORMAT}:ADFL:ADIR:LC\t\ +0/1:SPANNING/SPANNING:17/20:17-17/20-20:10/12:5/5:0/0:30\n" +EH_RECS="chr4\t3074877\t.\tC\t\t.\tPASS\tEND=3074940;REF=21;RL=63;RU=CAG;VARID=HTT;REPID=HTT;STR_STATUS=normal,normal;STR_NORMAL_MAX=26;STR_PATHOLOGIC_MIN=40\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:17/20:17-17/20-20:10/12 +chrX\t147912051\t.\tC\t\t.\tPASS\tEND=147912110;VARID=FMR1;REPID=FMR1;STR_STATUS=premutation;STR_NORMAL_MAX=44;STR_PATHOLOGIC_MIN=200\t${EH_FORMAT}\t\ +1:SPANNING:58:55-60:1 +chr4\t39348425\t.\tA\t\t.\tPASS\tEND=39348480;VARID=RFC1;REPID=RFC1;STR_STATUS=full_mutation;STR_NORMAL_MAX=.;STR_PATHOLOGIC_MIN=400\t${EH_FORMAT}\t\ +1/1:INREPEAT/INREPEAT:500/600:400-700/500-800:0/0 +chr7\t117548628\t.\tT\t\t.\tPASS\tEND=117548650;VARID=CFTR_TG;REPID=CFTR_TG\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:11/12:11-11/12-12:8/9 +chr7\t117548651\t.\tT\t\t.\tPASS\tEND=117548660;VARID=CFTR_T;REPID=CFTR_T\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:5/7:5-5/7-7:8/9 +chr2\t233760233\t.\tA\t\t.\tPASS\tEND=233760250;VARID=UGT1A1;REPID=UGT1A1;STR_STATUS=normal\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:7/8:7-7/8-8:5/6 +chr6\t16327636\t.\tT\t\t.\tPASS\tEND=16327700;VARID=ATXN1;REPID=ATXN1;STR_STATUS=normal;STR_NORMAL_MAX=35;STR_PATHOLOGIC_MIN=39\t${EH_FORMAT}\t\ +0/1:SPANNING/FLANKING:30/37:30-30/35-40:4/0 +chr9\t27573528\t.\tC\t\t.\tPASS\tEND=27573540;VARID=C9ORF72;REPID=C9ORF72\t${EH_FORMAT}\t\ +./.:.:.:.:. +chr1\t10\t.\tC\t\t.\tPASS\tEND=20;VARID=EXTRA1\t${EH_FORMAT}\t\ +0/1:SPANNING:8/9:8-8/9-9:1 +short\t1\t.\tC" +put $f/expansion_hunter/S_eh_stranger.vcf "${VCF_HEAD}${EH_RECS}\n" +at "$OLD" $f/expansion_hunter/S_eh.vcf +put $f/expansion_hunter/S_rfc1_motif.tsv "verdict\tnote\tdominant_motif\tdominant_class\tdominant_share\tmotif_shares\trepeat_reads\npathogenic\tAAGGG dominant\tAAGGG\tpathogenic\t0.9\tAAGGG:0.9\t40\n" +put $f/expansion_hunter/S_str_motifs.tsv "locus\tverdict\tnote\tdominant_motif\nFMR1\tinterrupted\tAGG\tCGG\n\tx\tx\tx\n" +put $f/expansion_hunter/S_eh_catalog.json '[{"LocusId": "EXTRA1", "NoThresholds": "no published range"}, {"LocusId": "X"}, 3]' +put $f/telomere/S/S/S_summary.tsv "PID\tsample\ttotal_reads\tread_lengths\tintratel_reads\ttel_content\nS\ttumor\t620000000\t100,150\t40\t1.5\n" +put $f/vcf/S_roh.txt "# RG\nRG\tS\tchrY\t2781480\t47781479\t45000000\t900\t50 +RG\tS\tchrEBV\t1\t6000000\t6000000\t90\t50 +RG\tS\tchr1\t125200001\t143200000\t18000000\t300\t50 +RG\tS\tchr9\t42300001\t49300000\t7000000\t200\t50 +RG\tS\tchr1\t120000001\t145000000\t25000000\t500\t50 +RG\tS\tchr2\t10000001\t17000000\t7000000\t900\t50 +RG\tS\tchr15\t23000001\t50000000\t27000000\t900\t50 +RG\tS\tchr3\t1\t2\tx\t1\t1\n" +put $f/mito/S_haplogroup.txt "\"SampleID\"\t\"Haplogroup\"\t\"Rank\"\n\"S\"\t\"H5a1\"\t\"0.9\"\n" +put $f/mito/S_haplocheck.txt "\"Sample\"\t\"Contamination Status\"\t\"Contamination Level\"\n\"S\"\t\"NO\"\t\"\"\n" +put $f/y_haplogroup/S_y_haplogroup.txt "Sample_name\tHg\tHg_marker\tTotal_reads\tValid_markers\tQC-score\nS\tR1b1a1b1a1a2*(xR-L21)\tR-M269\t100\t40\t0.98\n" +put $f/y_haplogroup/S_y_tree.txt "Hg\tLevel\tTree\tPath_derived\tSNPs_called\nR-Z2105\t12\tISOGG 2026\t30\t400\n" +put $f/y_haplogroup/S_y_tree_path.tsv "Level\tBranch\tSNPs\tDerived\n1\tA0-T\tM42\t1\n2\tR\n" +put $f/y_str/S_y_str.tsv "order\tpanel\tmarker\tcount\tconfidence\n1\tY-12\tDYS393\t13\thigh\n2\tY-12\tDYS390\t24\tlow\n3\tY-12\tDYS19\t\tno call\n4\tY-25\tDYS464\t.\tnot callable\n" +put $f/consumer_export/S_consumer_export.tsv "key\tvalue\nsites\t600000\ncalled\t590000\nno_call\t10000\nfrom_reference_blocks\t\ntemplate_sites\t631000\nno_call: low depth\t9000\nno_call: filtered\t1000\nfile_23andme\tS_23andme.txt\n" +put $f/bloodtype/S_bloodtype.tsv "system\tphenotype\tgenotype\tconfidence\nABO\tA\tA1/O1\thigh\nRhD\tRhD positive\t\tmedium\nKell\t\tK-/k+\t\n" +put $f/lpa/S_lpa_kiv2.tsv "metric\tvalue\nstatus\testimated\nkiv2_copies\t35\nci95_low\t30\nci95_high\t40\nclass\tlow\nmessage\tok\n" +put $f/lpa/S_lpa_kiv2_4925.tsv "metric\tvalue\nstatus\tpresent\nalt_reads\t4\ndepth\t200\n" +put $f/traits/S_traits.tsv "rsid\tgene\tchrom\tpos\ttrait\tgenotype\tstatus\treading\tancestry\tnote\tpmids +rs10455872\tLPA\tchr6\t1\tLp(a)\tA/G\tcalled\tone copy\tall\t.\t1 +rs3798220\tLPA\tchr6\t2\tLp(a)\tT|C\tcalled\tone copy\tall\t.\t1 +rs78655421\tCFTR\tchr7\t3\tR117H\tG/A\tcalled\tone copy\tall\t.\t1 +rs429358\tAPOE\tchr19\t4\tAPOE\tT/C\tcalled\te3/e4\tall\t.\t1 +rs1\tX\tchr1\t5\tx\t.\tnot_covered\tno reads\tall\t.\t1 +rs2\tX\tchr1\t6\tx\tC/C\tother_allele\tother\tall\t.\t1\n" +put $f/traits/S_genotype_rules.tsv "rule\tgene\ttrait\trsids\tchrom\tpos\tgenotypes\tgq\tphase\tresult\tstatus\treading\tancestry\tnote\tpmids +APOE\tAPOE\tAPOE\trs429358,rs7412\tchr19\t1,2\tT/C,C/C\t40\t.\te3/e4\tcalled\tr\tall\t.\t1 +TAS2R38\tTAS2R38\tbitter\trs1\tchr7\t1\tC/G\t40\tunphased\tPAV/AVI\tcalled\ttaster\tall\t.\t1 +APOL1\tAPOL1\tkidney\trs73885319,rs60910145\tchr22\t1,2\tA/G,T/T\t40\t.\tG0/G1\tcalled\tone G1\tAfrican\t.\t20647424\n" +putgz $f/vcf/S.vcf.gz "${VCF_HEAD}chr22\t36265995\trs71785313\tAATAATT\tA\t50\tPASS\t.\tGT:GQ\t0/1:42\n" +put $f/cpic/S_phenotypes.tsv "Gene\tDiplotype\tPhenotype\tStatus\nUGT1A1\t*1/*28\tIntermediate\tnon-normal\n" +put $f/vntyper/S_vntyper.tsv "metric\tvalue\nstatus\tnegative\nconfidence\t\nvariant\t.\nflag\tnone\nadvice\tnone\n" +put $f/locityper/S_locityper.tsv "locus\tregion\tstatus\tgenotype\thaplotype_1\thaplotype_2\tquality\ttotal_reads\tunexplained_reads\tweighted_distance\twarnings\tfilter\treason +HLA-DRB1\tchr6:1-2\tcalled\th1,h2\th1\th2\t30\t100\t1\t0.1\t\tpass\t +KIR\tchr19:1-2\tcalled\tk1,k2\tk1\tk2\t20\t100\t1\t0.1\t\tpass\t +CYP2D6\tchr22:1-2\tno call\t.\t.\t.\t.\t.\t.\t.\tlow\tfail\tlow depth +FCGR\tchr1:1-2\tcalled\tf\tf\tf\t9\t1\t1\t1\t\tfail\tq +ZZZ\tchr1:1-2\tcalled\tz\tz\tz\t9\t1\t1\t1\t\tpass\t\n" +putgz $f/mito/S_chrM_filtered.vcf.gz "${VCF_HEAD}chrM\t302\t.\tA\tAC,ACC\t50\tPASS\t.\tGT:AF\t0/1:0.3,0.2 +chrM\t3243\t.\tA\tG\t50\tPASS\t.\tGT:AF\t0/1:0.40 +chrM\t8344\t.\tA\tG\t50\tweak_evidence\t.\tGT:AF\t0/1:0.30 +chrM\t11778\t.\tG\tA\t50\tPASS\t.\tGT:AF\t1:0.99 +chrM\t16189\t.\tT\tC\t50\tPASS\t.\tGT:AF\t0/1:0.5 +chrM\t100\t.\tG\tA\t50\tPASS\t.\tGT:AF\t0/1:. +chrM\t150\t.\tC\tT\t50\tPASS\t.\tGT\t0/1 +chrM\t200\t.\tC\tT\t50\tPASS\t.\tGT:AF\t0/1:nan,0.5\n" +put $f/mito/S_chrM_annotated.tsv "# clinvar\t2026-10-01\t- +# gnomad\tnot annotated\tno file +chrom\tpos\tref\talt\tfilter\taf\tclinvar_id\tclinvar_significance\tclinvar_review\tclinvar_condition\tgnomad_af_hom\tgnomad_af_het\tgnomad_max_hl +chrM\t3243\tA\tG\tPASS\t0.40\t9589\tPathogenic\tcriteria\tMELAS\t0\t0\t0 +chrM\t302\tA\tAC\tPASS\t0.3\t.\t.\t.\t.\t.\t.\t. +chrM\t16189\tT\tC\tPASS\t0.99\t1\tBenign|not_provided\t.\t.\t.\t.\t. +chrM\t500\tC\tT\tPASS\t0.99\t2\tUncertain_significance\t.\t.\t.\t.\t. +chrM\t8344\tA\tG\tweak_evidence\t0.3\t9579\tPathogenic\t.\t.\t.\t.\t.\n" +put $f/mito/S_chrM_deletions.tsv "bkp5\tbkp3\tlength\tlevel_pct\n8469\t13447\t4977\t0.5\n100\t200\t.\t2.5\n300\t400\t99\t0.1\n" +put $f/mito/S_mtdna_copies.tsv "metric\tvalue\nstatus\testimated\nreason\t.\nchrm_reads\t100000\nmtdna_copies\t250\n" +put $f/archaic/S_archaic.tsv "key\tvalue\npopulation\tCEU\nneanderthal_segments\t120\nneanderthal_bp\t50000000\ndenisovan_segments\t3\ndenisovan_bp\t200000\nanalysed_bp\t2500000000\nneanderthal_percent\t2.00\ndenisovan_percent\t0.01\nlongest_10\tj\nlongest_2\tb\nlongest_1\ta\n" +put $f/ancient/S_ancient.tsv "rank\tgroup\tn\tdistance\tdates\tcountry\tlocality\n1\tSpain_BA\t10\t0.01\t2000 BCE\tSpain\tx\n2\tFrance_N\t5\t0.02\t\tFrance\t\n" +put $f/ancient/S_ancient_info.tsv "key\tvalue\nrelease\tv62\nsites_used\t500000\n" +put $f/ancient/S_ancient_pca.svg '\n' +put $f/viral/S_viral.tsv "virus\taccession\treads\tbreadth\trelative_depth\tcopies_per_cell\tcall\thuman_depth\tnote +phiX\tNC_001422\t500\t1.0\t.\t.\tspike-in\t30\t +HHV-6B\tNC_000898\t2000\t0.99\t1.0\t1.0\tinherited integrated HHV-6 likely\t30\t +EBV\tNC_007605\t0\t0\t0\t0\tnot found\t30\t\n" +put $f/exomiser/S_exomiser_top.tsv "rank\tgene\tmoi\texomiser_score\tphenotype_score\tvariant\tgenotype\tconsequence\thgvs\tacmg\tdisease +1\tGENE1\tAD\t0.9\t0.8\tchr1-1-A-G\t0/1\tmissense\tp.X\tLP\tD1 +1\tGENE1\tAD\t0.9\t0.8\tchr1-2-A-G\t0/1\tstop\tp.Y\tP\tD1 +2\tGENE2\tAR\t0.5\t0.4\tchr2-1-A-G\t1/1\tsplice\tc.1\tVUS\tD2\n" +put $f/exomiser/S_exomiser_terms.tsv "hpo_id\tlabel\nHP:0001250\tSeizure\nHP:0000001\n" +put $f/chip/S_chip.tsv "gene\tprotein_change\tvaf\talt_reads\tdepth\tstatus\tnote\tchrom +DNMT3A\tR882H\t0.05\t3\t60\tchip\t\tchr2 +TET2\tQ1\t0.5\t30\t60\tgermline\tlikely inherited\tchr4\n" +put $f/chip/S_chip_depth.tsv "gene\tmean_depth\tdetection_limit_vaf\tstatus\nDNMT3A\t30.5\t0.1\tcovered\nTET2\t40\t0.08\tcovered\nASXL1\t2\t.\tlow depth\nJAK2\t35\t0.1\tcovered\nTP53\t25\t0.1\tcovered\n" +put $f/mei/S_mei_sites.tsv "gene\telement\tcondition\tinheritance\tstatus\tmean_depth\tcall +FKTN\tSVA\tFukuyama\tAR\tcallable\t30\tnot seen +TAF1\tSVA\tXDP\tXL\tnot callable\t2\t\n" +put $f/mei/S_mei.tsv "chrom\tpos\telement\tcopies\nchr1\t1\tALU\t1\nchr2\t2\tL1\t1\nchr3\t3\tALU\t2\n" +put $f/mosaic/S_mocha_summary.tsv "metric\tvalue\nstatus\tassessed\ndepth\t30\ncalls_reported\t1\nfloor\t0.05\n" +put $f/mosaic/S_mocha.tsv "chrom\tstart\tend\tlength_mb\tarms\ttype\tcell_fraction\tn_hets\tlod_baf_phase\treported\treason +chr20\t1\t2\t10\tq\tLoss\t0.1\t50\t20\tyes\t +chr1\t1\t2\t0.5\tp\tCNN-LOH\t0.02\t5\t3\tno\tbelow floor\n" + +# nf: the Nextflow and other names, a run with stale files and a female -------- +f=nf +put $f/logs/run_status.tsv "meta\tstarted_epoch\t${T0}\nmeta\tdeclared_sex\tfemale\nstep\t08\tok\nstep\t18\tskipped (opt-in)\nstep\t10\tfailed (exit 1)\nstep\t11\tok\nstep\t12\tskipped\nstep\t22\tfailed (exit 1)\nstep\t25\tfailed (exit 2)\nstep\t37\tok\nstep\t53\tskipped\n" +put $f/hla/S_hla_genotype.tsv "HLA-DQA1\t2\tDQA1*03:01\t10\t0\tDQA1*01:01\t9\t30\nHLA-DQB1\t1\tDQB1*02:01\t10\t40\t.\t0\t-1\nHLA-DRB1\t2\tDRB1*04:01\t10\t40\tDRB1*07:01\t9\t40\n" +put $f/hla_optitype/S_optitype_result.tsv "# OptiType did not run: no class I reads\n" +put $f/kir/S_kir_genotype.tsv "# too few KIR reads\n" +put $f/cnvpytor/S_cnvs.txt "deletion\tchrY:2800001-2900000\t100000\t0.0\t1e-15\t1\t1\t1\t0.05\t0\t9\ndeletion\tchr2:10000001-10100000\t100000\t0.4\t1e-9\n" +at "$OLD" $f/cnvpytor/S_cnvs.txt +put $f/telomere/x/deep/T_summary.tsv "PID\tsample\tintratelomeric_reads\ttel_content\nS\ttumor\t61873\t334\n" +put $f/roh/S_roh.txt "RG\tS\tchr2\t10000001\t17000000\t7000000\t900\t50\n" +at "$OLD" $f/roh/S_roh.txt +put $f/mito/S_haplogroup.txt "SampleID\tRank\nS\tU5a\n" +at "$OLD" $f/mito/S_haplogroup.txt +put $f/y_haplogroup/S_y_haplogroup.txt "Sample_name\tHg\nS\tNA\n" +put $f/y_haplogroup/S_y_tree.txt "Hg\tLevel\tTree\n\t\t\n" +put $f/y_haplogroup/S_y_tree_path.tsv "Level\tBranch\tSNPs\n" +put $f/manta/x/S.diploidSV.vcf.gz "" +putgz $f/manta/x/diploidSV.vcf.gz "${VCF_HEAD}" +putgz $f/sv_merged/S_sv_consensus.vcf.gz "${VCF_HEAD}chr1\t1\t.\tN\t\t.\tPASS\tSVTYPE=DEL;END=10;SUPP=1;SUPP_VEC=1\tGT\t0/1\n" +at "$OLD" $f/sv_merged/S_sv_consensus.vcf.gz +put $f/annotsv/S_sv_class4_5_checked.tsv "" +put $f/prs/S_prs_summary.tsv "Condition\tPGS_ID\tPercentile\nX\tPGS1\tNA\n" +put $f/ancestry/S_ancestry.tsv "key\tvalue\n" +at "$OLD" $f/prs/S_prs_summary.tsv +put $f/expansion_hunter/S_eh_stranger.vcf "${VCF_HEAD}chr4\t3074877\t.\tC\t\t.\tPASS\tVARID=HTT;REPID=HTT;STR_STATUS=normal\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:17/20:17-17/20-20:10/12\n" +at "$OLD" $f/expansion_hunter/S_eh_stranger.vcf +put $f/expansion_hunter/S_eh.vcf "${VCF_HEAD}chr7\t117548628\t.\tT\t\t.\tPASS\tVARID=CFTR_TG;REPID=CFTR_TG\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:12/12:12-12/12-12:8/9 +chr7\t117548651\t.\tT\t\t.\tPASS\tVARID=CFTR_T;REPID=CFTR_T\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:5/5:5-5/5-5:8/9 +chr2\t233760233\t.\tA\t\t.\tPASS\tVARID=UGT1A1;REPID=UGT1A1\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:8/10:8-8/10-10:5/6\n" +put $f/locityper/S_locityper.tsv "locus\tregion\tstatus\tgenotype\thaplotype_1\thaplotype_2\tquality\ttotal_reads\tunexplained_reads\tweighted_distance\twarnings\tfilter\treason +HLA-DRB1\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t +HLA-DQB1\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t +CYP2D6\tx\tcalled\th\th\th\t1\t1\t1\t1\t\tpass\t\n" +at "$OLD" $f/locityper/S_locityper.tsv +put $f/lpa/S_lpa_kiv2.tsv "metric\tvalue\nstatus\ttoo little depth\n" +put $f/traits/S_traits.tsv "rsid\tgene\ttrait\tgenotype\tstatus\treading\tancestry\tnote\tpmids +rs10455872\tLPA\tLp(a)\t.\tnot_covered\tno reads\tall\t.\t1\n" +putgz $f/mito/S_chrM_filtered.vcf.gz "${VCF_HEAD}chrM\t3243\t.\tA\tG\t50\tPASS\t.\tGT:AF\t0/1:0.40\n" +put $f/mito/S_chrM_annotated.tsv "chrom\tpos\tref\talt\tfilter\taf\n" +at "$OLD" $f/mito/S_chrM_annotated.tsv +put $f/mosaic/S_mocha_summary.tsv "metric\tvalue\nstatus\tnot assessed\nmessage\ttoo shallow\ncalls_reported\t.\n" +put $f/mosaic/S_mocha.tsv "chrom\tstart\tend\tlength_mb\tarms\ttype\tcell_fraction\tn_hets\tlod_baf_phase\treported\treason\n" +put $f/vntyper/S_vntyper.tsv "metric\tvalue\nstatus\tnot assessed\n" +put $f/exomiser/S_exomiser_top.tsv "rank\tgene\tmoi\texomiser_score\tphenotype_score\tvariant\tgenotype\tconsequence\thgvs\tacmg\tdisease\n" +put $f/chip/S_chip.tsv "gene\tprotein_change\tvaf\talt_reads\tdepth\tstatus\tnote\n" +put $f/chip/S_chip_depth.tsv "gene\tmean_depth\tdetection_limit_vaf\tstatus\nDNMT3A\t2\t.\tlow depth\n" +put $f/viral/S_viral.tsv "virus\taccession\treads\tbreadth\trelative_depth\tcopies_per_cell\tcall\thuman_depth\tnote\nphiX\tNC\t0\t0\t.\t.\tnot found\t\t\n" +put $f/mei/S_mei_sites.tsv "gene\telement\tcondition\tinheritance\tstatus\tmean_depth\tcall\nFKTN\tSVA\tF\tAR\tcallable\t30\tseen\n" +put $f/mei/S_mei.tsv "chrom\tpos\telement\tcopies\n" +put $f/consumer_export/S_consumer_export.tsv "key\tvalue\nsites\t10\ncalled\t10\nno_call\t0\n" +put $f/paralogs/S_hba_copy_number.tsv "metric\tvalue\nnote\tno status\n" +put $f/paralogs/S_paraphase_copy_number.tsv "region\tgene\tcopy_number\tstatus\nsmn1\tSMN1\tx\tcalled\n" +put $f/paralogs/S_smn_copy_number.tsv "chrom\tstart\tend\tagCN_filter\tagCN\tagCN_qual\tpsCN_filter\tpsCN\tpsCN_qual +chr5\t1\t2\tnot assessed: no model\t.\t.\t.\t.\t.\n" + +# bad1, bad2: an unreadable table of each kind ------------------------------------ +f=bad1 +put $f/couple/S_couple_carrier.tsv "gene\tcondition\tresult\toffspring_risk\tpartner_a\tfindings_a\tpartner_b\tfindings_b\texplanation\nG\tc\tshared\t1\tA\tx\tB\ty\tz\n" +put $f/ancestry_fine/S_ancestry_fine.tsv "level\tcode\n" +put $f/hla_t1k/S_hla_genotype.tsv "HLA-A\t2\tA*02:01\t1\tx\n" +put $f/mito/S_haplogroup.txt "" +put $f/mito/S_haplocheck.txt "Sample\tStatus\nS\tNO\n" +put $f/y_haplogroup/S_y_haplogroup.txt "Sample\tGroup\nS\tR\n" +put $f/y_haplogroup/S_y_tree.txt "Hg\tLevel\nR\t1\n" +put $f/y_str/S_y_str.tsv "order\tpanel\tmarker\tcount\tconfidence\n1\tY\tD\t1\tmaybe\n" +put $f/consumer_export/S_consumer_export.tsv "key\tvalue\nsites\tmany\n" +put $f/bloodtype/S_bloodtype.tsv "system\tphenotype\n" +put $f/archaic/S_archaic.tsv "key\tvalue\nneanderthal_segments\t1\nneanderthal_bp\t1\ndenisovan_segments\t1\ndenisovan_bp\t1\nanalysed_bp\t1\nneanderthal_percent\t2\n" +put $f/lpa/S_lpa_kiv2.tsv "metric\tvalue\nstatus\tmaybe\n" +put $f/vntyper/S_vntyper.tsv "metric\tvalue\n" +put $f/locityper/S_locityper.tsv "locus\tstatus\nX\tcalled\n" +put $f/viral/S_viral.tsv "virus\treads\nX\t1\n" +put $f/chip/S_chip.tsv "gene\tvaf\n" +put $f/chip/S_chip_depth.tsv "gene\tmean_depth\tdetection_limit_vaf\tstatus\nX\t1\t1\tcovered\n" +put $f/mei/S_mei_sites.tsv "gene\telement\tcondition\tinheritance\tstatus\tmean_depth\tcall\nX\tSVA\tc\tAR\tcallable\t1\tseen\n" +put $f/mosaic/S_mocha_summary.tsv "metric\tvalue\nstatus\tassessed\ncalls_reported\t2\n" +put $f/mosaic/S_mocha.tsv "chrom\tstart\tend\tlength_mb\tarms\ttype\tcell_fraction\tn_hets\tlod_baf_phase\treported\treason\nchr1\t1\t2\t1\tp\tLoss\t0.1\t1\t1\tyes\t\n" +put $f/exomiser/S_exomiser_top.tsv "rank\tgene\n" +put $f/traits/S_traits.tsv "rsid\tgene\ttrait\tgenotype\tstatus\treading\tancestry\tnote\tpmids\nrs1\tX\tx\tA/A\tcalled\tr\n" +put $f/ancient/S_ancient.tsv "rank\tgroup\n1\tx\n" +put $f/annotsv/S_sv_class4_5_checked.tsv "SV_chrom\tSV_start\n1\t2\n" +put $f/paralogs/S_smn_copy_number.tsv "chrom\tstart\tend\n" +put $f/paralogs/S_paraphase_copy_number.tsv "region\tgene\nsmn1\tSMN1\n" +put $f/paralogs/S_cn_dosage.tsv "locus\tstatus\n" +put $f/telomere/S/S_summary.tsv "PID\tsample\tintratel_reads\tread_lengths\ttel_content\nS\ttumor\t40\t150\t1.5\n" +put $f/expansion_hunter/S_eh.vcf "${VCF_HEAD}chr7\t1\t.\tT\t\t.\tPASS\tVARID=CFTR_TG\t${EH_FORMAT}\t\ +0/1:SPANNING/FLANKING:11/12:11-11/12-12:8/0\nchr7\t2\t.\tT\t\t.\tPASS\tVARID=CFTR_T\t${EH_FORMAT}\t\ +0/1:SPANNING/SPANNING:7/9:7-7/9-9:8/9\n" +put $f/mito/S_chrM_deletions.tsv "bkp5\tbkp3\n1\t2\n" +putgz $f/mito/S_chrM_filtered.vcf.gz "${VCF_HEAD}" +put $f/vcf/S_roh.txt "RG\tS\tchr1\t1\t2\tx\n" + +f=bad2 +put $f/couple/S_couple_carrier.tsv "gene\tcondition\n" +put $f/mei/S_mei_sites.tsv "gene\telement\tcondition\tinheritance\tstatus\tmean_depth\tcall\nX\tSVA\tc\tAR\tcallable\t1\tseen\n" +put $f/mei/S_mei.tsv "chrom\tpos\n" +put $f/mosaic/S_mocha_summary.tsv "metric\tvalue\nstatus\tassessed\n" +put $f/traits/S_traits.tsv "rsid\tgene\ttrait\tgenotype\tstatus\treading\tancestry\tnote\tpmids\nrs1\tX\tx\tA/A\tcalled\tr\ta\tn\tp\n" +put $f/traits/S_genotype_rules.tsv "rule\tgene\nAPOL1\tAPOL1\n" +put $f/archaic/S_archaic.tsv "key\tvalue\nneanderthal_segments\tmany\n" +put $f/viral/S_viral.tsv "virus\taccession\treads\tbreadth\trelative_depth\tcopies_per_cell\tcall\nX\tA\t1\t1\t1\t1\tc\n" +put $f/consumer_export/S_consumer_export.tsv "key\tvalue\nsites\t1\ncalled\t1\nno_call\t0\nno_call: other\tsome\n" +put $f/exomiser/S_exomiser_top.tsv "rank\tgene\tmoi\texomiser_score\tphenotype_score\tvariant\tgenotype\tconsequence\thgvs\tacmg\tdisease\n1\tG\n" +put $f/y_haplogroup/S_y_tree.txt "Hg\tLevel\tTree\nR\t1\tT\n" +put $f/y_haplogroup/S_y_tree_path.tsv "Level\tSNPs\n1\tM\n" +put $f/paralogs/S_cn_intervals.tsv "locus\tname\tchrom\tstart\tend\tstatus\tcopies\texpected\tchange\nX\tx\tchr1\t1\t2\tcalled\n" +put $f/mito/S_chrM_filtered.vcf.gz "not gzip\n" +put $f/delly/S_sv.vcf.gz "\x1f\x8b\x08\x00broken" +put $f/expansion_hunter/S_eh_stranger.vcf "${VCF_HEAD}chr4\t39348425\t.\tA\t\t.\tPASS\tVARID=RFC1;REPID=RFC1;STR_STATUS=full_mutation\t${EH_FORMAT}\t\ +1/1:INREPEAT/INREPEAT:500/600:400-700/500-800:0/0\n" + +# Non-finite cells follow each Python reader's rules, not a global filter. +for special in nan inf -inf; do + f="float-${special}" + cp -R full "$f" + put "$f/chip/S_chip_depth.tsv" "gene\tmean_depth\tdetection_limit_vaf\tstatus\nA\t30\t.\tcovered\nB\t${special}\t.\tcovered\nC\t40\t.\tcovered\n" + for spec in \ + 'paralogs/S_smn_copy_number.tsv:agCN_qual,psCN_qual' \ + 'paralogs/S_pms2_copy_number.tsv:agCN_qual,psCN_qual' \ + 'ancestry_fine/S_ancestry_fine.tsv:fraction,low,high' \ + 'telomere/S/S/S_summary.tsv:read_lengths' \ + 'mito/S_chrM_deletions.tsv:level_pct' \ + 'mito/S_chrM_annotated.tsv:af'; do + file=${spec%%:*} + cols=${spec#*:} + [ -f "$f/$file" ] || { echo "missing float fixture: $file" >&2; exit 1; } + awk -F '\t' -v OFS='\t' -v cols="$cols" -v value="$special" ' + BEGIN { split(cols, names, ","); for (i in names) want[names[i]] = 1 } + NR == 1 { for (i = 1; i <= NF; i++) if ($i in want) change[i] = 1 } + NR > 1 { for (i in change) $i = value } + { print } + ' "$f/$file" > "$f/cells.tmp" + mv -f "$f/cells.tmp" "$f/$file" + done + put "$f/hla_t1k/S_hla_genotype.tsv" "HLA-A\t2\tA*02:01\t1\t${special}\tA*01:01\t1\t${special}\n" + # Python raises an uncaught OverflowError if an infinite read length + # reaches its floor calculation. Keep this full-collector comparison + # on its defined no-depth path; Rust isolates that error to the section. + if [ "$special" != nan ]; then + put "$f/telomere/S/S/S_summary.tsv" "PID\tsample\ttotal_reads\tread_lengths\tintratel_reads\ttel_content\nS\ttumor\t0\t${special}\t40\t1.5\n" + fi + put "$f/vcf/S_roh.txt" "RG\tS\tchr2\t1\t7000000\t${special}\n" + put "$f/cnvpytor/S_cnvs.txt" "deletion\tchr2:1-100000\t100000\t0.4\t${special}\t1\t1\t1\t${special}\t${special}\t9\n" + putgz "$f/mito/S_chrM_filtered.vcf.gz" "${VCF_HEAD}chrM\t3243\t.\tA\tG\t50\tPASS\t.\tGT:AF\t0/1:${special}\n" +done + +# g2-*: the APOL1 G2 deletion beside step 42's APOL1 line -------------------------- +g2() { + local d=$1 result=$2 status=$3 + put "$d/traits/S_traits.tsv" "rsid\tgene\ttrait\tgenotype\tstatus\treading\tancestry\tnote\tpmids\nrs1\tX\tx\tA/A\tcalled\tr\ta\tn\tp\n" + put "$d/traits/S_genotype_rules.tsv" "rule\tgene\ttrait\trsids\tchrom\tpos\tgenotypes\tgq\tphase\tresult\tstatus\treading\tancestry\tnote\tpmids\nAPOL1\tAPOL1\tkidney\trs1\tchr22\t1\tA/G\t40\t.\t${result}\t${status}\tr\tAfrican\t.\t1\n" +} +g2 g2-left G0/G1 called +putgz g2-left/vcf/S.vcf.gz "${VCF_HEAD}chr21\t1\t.\tA\tG\t50\tPASS\t.\tGT\t0/1\nchr22\t36265995\trs71785313\tAATAATT\tA\t50\tPASS\t.\tGT:GQ\t0/1:42\nchr22\t36266100\t.\tA\tG\t50\tPASS\t.\tGT\t0/1\n" +# The same deletion written one base to the right: the same sequence, so G2. +g2 g2-shifted G0/G0 called +putgz g2-shifted/vcf/S.vcf.gz "${VCF_HEAD}chr22\t36265996\t.\tATAATTA\tA\t50\tPASS\t.\tGT:GQ\t1/1:30\n" +g2 g2-two G1/G1 called +putgz g2-two/vcf/S.vcf.gz "${VCF_HEAD}chr22\t36265995\t.\tAATAATT\tA\t50\tPASS\t.\tGT:GQ\t0/1:42\nchr22\t36265996\t.\tATAATTA\tA\t50\tPASS\t.\tGT:GQ\t0/1:40\n" +g2 g2-gvcf . . +putgz g2-gvcf/vcf/S.g.vcf.gz "${VCF_HEAD}chr22\t36265990\t.\tA\t\t.\t.\tEND=36266010\tGT:GQ\t0/0:50\n" +putgz g2-gvcf/vcf/S.vcf.gz "${VCF_HEAD}chr22\t36265995\t.\tAATAATT\tA\t50\tPASS\t.\tGT:GQ\t1/1:42\n" +g2 g2-none G0/G0 called +putgz g2-none/vcf/S.vcf.gz "${VCF_HEAD}chr1\t1\t.\tA\tG\t50\tPASS\t.\tGT\t0/1\n" +g2 g2-other G1/G1 called +putgz g2-other/vcf/S.vcf.gz "${VCF_HEAD}chr22\t36265998\t.\tATTATA\tA\t50\tPASS\t.\tGT:GQ\t0/1:42\nchr22\t36266000\t.\tT\tC\t50\tLowQual\t.\tGT\t0/1\n" +g2 g2-nocall G0/G2 called +putgz g2-nocall/vcf/S.g.vcf.gz "${VCF_HEAD}chr22\t36265995\t.\tAATAATT\tA,\t50\tPASS\t.\tGT:GQ\t./.:0\n" +g2 g2-novcf G1/G1 called diff --git a/tests/parity/golden/summary-collect/c-bad2/bad2.json b/tests/parity/golden/summary-collect/c-bad2/bad2.json new file mode 100644 index 000000000..4b626ef33 --- /dev/null +++ b/tests/parity/golden/summary-collect/c-bad2/bad2.json @@ -0,0 +1,416 @@ +{ + "schema_version": 2, + "sample": "S", + "generated_utc": "2026-11-13 11:50 UTC", + "run": { + "status_file": false, + "started_utc": null, + "declared_sex": null + }, + "databases": { + "clinvar_release": null, + "cpsr_clinvar_release": null, + "hla_database": null + }, + "manifest": [], + "sections": { + "couple": { + "title": "Couple carrier screen", + "step": "39", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_couple_carrier.tsv lacks a column of gene, condition, result, offspring_risk, partner_a, findings_a, partner_b, findings_b, explanation", + "data": {} + }, + "parascopy": { + "title": "SMN1/SMN2 copy number (Parascopy, short reads)", + "step": "35", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_cn_intervals.tsv line 2 lacks a field", + "data": {} + }, + "paralogs": { + "title": "Paralog genes (Paraphase, PacBio HiFi)", + "step": "35b", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "hla": { + "title": "HLA typing (T1K)", + "step": "08", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "prs": { + "title": "Polygenic risk scores", + "step": "25", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "ancestry_fine": { + "title": "Fine-grained ancestry (ADMIXTURE, RFMix)", + "step": "26b", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "manta": { + "title": "Structural variants (Manta)", + "step": "04", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "delly": { + "title": "Structural variants (Delly)", + "step": "19", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_sv.vcf.gz: Compressed file ended before the end-of-stream marker was reached", + "data": {} + }, + "cnvpytor": { + "title": "Copy number (CNVpytor)", + "step": "18", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "sv_consensus": { + "title": "SV consensus merge", + "step": "22", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "annotsv": { + "title": "Structural variants, class 4 and 5 (AnnotSV)", + "step": "05", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "expansions": { + "title": "Repeat expansions (ExpansionHunter)", + "step": "09", + "state": "ok", + "source": "expansion_hunter/S_eh_stranger.vcf", + "file_date": "2021-03-04 05:06 UTC", + "note": null, + "data": { + "records": 1, + "key_loci": [ + { + "locus": "HTT", + "repeat_count": "not in output" + }, + { + "locus": "FMR1", + "repeat_count": "not in output" + }, + { + "locus": "C9ORF72", + "repeat_count": "not in output" + }, + { + "locus": "ATXN1", + "repeat_count": "not in output" + }, + { + "locus": "DMPK", + "repeat_count": "not in output" + } + ], + "loci": [ + { + "locus": "RFC1", + "repeat_count": "500/600", + "status": "full_mutation", + "normal_max": "", + "pathologic_min": "", + "support": "INREPEAT/INREPEAT", + "adsp": "0/0", + "ci": "400-700/500-800", + "sizing": "in_repeat", + "allele_sizing": "in_repeat/in_repeat", + "zone": "", + "zone_label": "", + "zone_citation": "", + "rfc1_motif": { + "verdict": "not_checked", + "note": "no motif count beside Stranger's output (step 9b runs it with the sample's BAM): confirm the flag with repeat-primed PCR" + } + } + ], + "stranger": true, + "stranger_outdated": false, + "unsized": [ + "RFC1" + ], + "intermediate": [], + "readings": [ + { + "locus": "CFTR", + "repeat": "intron 9 (legacy IVS8) TG and poly-T tract", + "source": "Deignan et al. 2020, PMID 32404922", + "status": "not_sized", + "genotype": ".", + "phase": ".", + "reading": "not in step 09's output: STR_EXTRA_LOCI=yes ./scripts/setup.sh --strchive-data adds the tract to the catalog (docs/09-str-expansions.md)." + }, + { + "locus": "UGT1A1", + "repeat": "promoter TA repeat", + "source": "Gammal et al. 2016, PMID 26417955", + "status": "not_sized", + "genotype": ".", + "pharmcat": ".", + "agreement": "not_compared", + "reading": "not in step 09's output: STR_EXTRA_LOCI=yes ./scripts/setup.sh --strchive-data adds the repeat to the catalog (docs/09-str-expansions.md). PharmCAT (step 7) still reads UGT1A1 *28." + }, + { + "locus": "SLC6A4", + "repeat": "5-HTTLPR promoter length polymorphism", + "status": "not_tested", + "genotype": ".", + "reading": "not tested by any step. CPIC's 2023 SSRI guideline (Bousman et al. 2023, PMID 37032427) found that the data on SLC6A4 do not support its use in antidepressant prescribing, so a result would carry no recommendation.", + "source": "Bousman et al. 2023, PMID 37032427" + } + ], + "flagged": [ + { + "locus": "RFC1", + "repeat_count": "500/600", + "status": "full_mutation", + "sizing": "in_repeat", + "rfc1_motif": { + "verdict": "not_checked", + "note": "no motif count beside Stranger's output (step 9b runs it with the sample's BAM): confirm the flag with repeat-primed PCR" + } + } + ], + "no_status": 0, + "not_graded": {} + } + }, + "telomere": { + "title": "Telomere content (TelomereHunter)", + "step": "10", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "roh": { + "title": "Runs of homozygosity", + "step": "11", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "haplogroup": { + "title": "Mitochondrial haplogroup", + "step": "12", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "y_haplogroup": { + "title": "Y-chromosome haplogroup (Yleaf)", + "step": "37", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "y_tree": { + "title": "Y placement on the current public tree", + "step": "37", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_y_tree_path.tsv has no Branch column", + "data": {} + }, + "y_str": { + "title": "Y-STR markers (HipSTR)", + "step": "45", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "consumer_export": { + "title": "Raw-data export (23andMe v5 layout)", + "step": "40", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_consumer_export.tsv has a 'no_call: other' row without a whole-number count", + "data": {} + }, + "bloodtype": { + "title": "Blood groups (predicted)", + "step": "41", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "lpa": { + "title": "Lp(a): LPA KIV-2 repeat count", + "step": "46", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "vntyper": { + "title": "MUC1 VNTR frameshifts (VNtyper)", + "step": "52", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "locityper": { + "title": "Locityper second opinion at hard loci", + "step": "53", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "mito": { + "title": "Mitochondrial variants", + "step": "20", + "state": "ok", + "source": "mito/S_chrM_filtered.vcf.gz", + "file_date": "2021-03-04 05:06 UTC", + "note": null, + "data": { + "confirmed": [], + "deletions_note": "no deletion table (mito/_chrM_deletions.tsv): step 20 ran before it looked for large deletions, eKLIPse wrote no table (see the step's log), or the Nextflow MITO_DELETIONS task failed (see the Nextflow log)", + "copies_note": "no copy-number table (mito/_mtdna_copies.tsv): step 20 ran before it counted mtDNA copies, or the Nextflow MITO_DELETIONS task failed (see the Nextflow log)", + "pass": 0, + "heteroplasmic": 0, + "heteroplasmic_sites": [], + "heteroplasmy_floor": 0.05, + "polyc_heteroplasmic": [], + "polyc_tracts": [ + "m.303-315", + "m.16184-16193" + ], + "annotation": { + "note": "no annotation table (mito/_chrM_annotated.tsv): step 20 ran before it annotated, or the Nextflow MITO_VARIANTS_ANNOTATE task failed (see the Nextflow log)" + } + } + }, + "traits": { + "title": "Traits panel", + "step": "42", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_genotype_rules.tsv lacks a column of rule, gene, trait, rsids, chrom, pos, genotypes, gq, phase, result, status, reading, ancestry, note, pmids", + "data": {} + }, + "archaic": { + "title": "Archaic ancestry (IBDmix)", + "step": "43", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_archaic.tsv has no whole number for neanderthal_segments", + "data": {} + }, + "ancient": { + "title": "Ancient populations (AADR PCA)", + "step": "44", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "viral": { + "title": "Viral DNA in the unmapped reads", + "step": "47", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "KeyError: 'human_depth'", + "data": {} + }, + "exomiser": { + "title": "Symptom-driven ranking (Exomiser)", + "step": "48", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_exomiser_top.tsv has a line that does not have one field per column", + "data": {} + }, + "chip": { + "title": "Clonal haematopoiesis (CHIP)", + "step": "49", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "mei": { + "title": "Mobile element insertions (MEGAnE)", + "step": "50", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_mei.tsv lacks a column of chrom, pos, element, copies", + "data": {} + }, + "mosaic": { + "title": "Mosaic chromosomal alterations (MoChA)", + "step": "51", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "S_mocha_summary.tsv has no S_mocha.tsv beside it", + "data": {} + } + } +} diff --git a/tests/parity/inputs/roh-summary/oracle.sh b/tests/parity/inputs/roh-summary/oracle.sh new file mode 100644 index 000000000..29bf8847a --- /dev/null +++ b/tests/parity/inputs/roh-summary/oracle.sh @@ -0,0 +1,21 @@ +#!/bin/sh +# The oracle of og roh summary (tests/parity/oracle.ref): the summary block +# of scripts/11-roh-analysis.sh as it was before og took it over, its +# MASKED_BANDS line and its awk, given og's options. +# sh ./oracle.sh PROG --sample SAMPLE --roh ROH [--out OUT] +prog=$1 +out=/dev/stdout +shift +while [ $# -gt 1 ]; do + case $1 in + --sample) SAMPLE=$2 ;; + --roh) OUTPUT=$2 ;; + --out) out=$2 ;; + esac + shift 2 +done +MASKED_BANDS=$(sed -n 's/^MASKED_BANDS="\(.*\)"$/\1/p' "$prog") +block=$(awk '/^SUMMARY=/ { on = 1; next } on && /^} > / { print "}"; exit } on { print }' "$prog") +[ -n "$MASKED_BANDS" ] && [ -n "$block" ] || { echo "oracle.sh: no summary block in $prog" >&2; exit 2; } +export SAMPLE OUTPUT MASKED_BANDS +eval "$block" > "$out" diff --git a/tests/parity/inputs/summary-collect/oracle.sh b/tests/parity/inputs/summary-collect/oracle.sh index 1a3e15a7c..3e33b6992 100644 --- a/tests/parity/inputs/summary-collect/oracle.sh +++ b/tests/parity/inputs/summary-collect/oracle.sh @@ -5,10 +5,9 @@ # --assets becomes ASSETS_DIR, the global render_report.py --assets sets; # --only is og's alone (the Python writes every section, and the case # compares the group's subtrees). The modules collect_summary imports are -# stubbed, except those a generator staged in pymods/ (the clinical cases -# need the real helpers); no other one is reached for a sample folder that -# holds only the sections the cases cover, except run_delta's two readers -# of history/ and recheck/, which find nothing here. +# stubbed, except those the group b and c generators stage in pymods/ +# and py/. No other module is reached for these sample folders, except +# run_delta's two readers of history/ and recheck/, which find nothing here. prog=$1 shift d=$(mktemp -d) @@ -16,6 +15,10 @@ sed -n 's/^import \([a-z_]*\) # noqa.*/\1/p' "$prog" | while read -r m; do if [ -f "pymods/$m.py" ]; then cp "pymods/$m.py" "$d/"; else : > "$d/$m.py"; fi done printf 'def read_recheck(path):\n return None\n\n\ndef newest_history(sample_dir):\n return None\n' > "$d/run_delta.py" +# The group c cases bring the modules the oracle imports, as they were at +# its commit, and the two tables it reads beside itself (py/, written by +# tests/parity/gen/summary-collect-c.sh). +if [ -d py ]; then cp py/* "$d"/; fi cp "$prog" "$d/collect_summary.py" PYTHONPATH=$d exec python3 -c ' import sys diff --git a/tests/parity/oracle.ref b/tests/parity/oracle.ref index 57c27da2f..f6798c54c 100644 --- a/tests/parity/oracle.ref +++ b/tests/parity/oracle.ref @@ -63,3 +63,4 @@ y-yleaf-positions 0baddb45aa3871a57469cf20295bdd832f72f5c0 file:bin/yleaf_run.py lookup 0baddb45aa3871a57469cf20295bdd832f72f5c0 file:bin/genome_lookup.py python3 lookup lookup-query 0baddb45aa3871a57469cf20295bdd832f72f5c0 file:bin/genome_lookup.py bash ./oracle.sh lookup query family-ibd 0baddb45aa3871a57469cf20295bdd832f72f5c0 file:bin/ibd_report.py sh ./ibd_oracle.sh family ibd +roh-summary 7c1470bc50933f48a1a0dde3c0e0b2b87bc48718 file:scripts/11-roh-analysis.sh sh ./oracle.sh roh summary diff --git a/tests/test_collect_summary.py b/tests/test_collect_summary.py index 62d7d1b93..150e5ee17 100644 --- a/tests/test_collect_summary.py +++ b/tests/test_collect_summary.py @@ -1905,11 +1905,19 @@ def report_sections_mito_roh_telomere_cnv(work, schema): """ROH on autosomes outside the bands, the telomere read floor, CNVpytor's filtered count, the D-loop poly-C tracts and the chrM annotation.""" import re as _re - # The masked bands: one line in the summary, step 11 and the ROH module. + # The masked bands: og's step 11 code holds the one copy that step 11 + # and the ROH module run (og roh summary); the summary's line agrees. + rs = open(os.path.join(REPO, "rust/crates/og-summary/src/collect/group_c/roh.rs")).read() + body = rs[rs.index("pub const MASKED_BANDS"):] + body = body[:body.index("];")] + rust_bands = " ".join(f"{c}:{int(a.replace('_', ''))}-{int(b.replace('_', ''))}" + for c, a, b in _re.findall(r'\("(chr\w+)", ([0-9_]+), ([0-9_]+)\)', body)) + check("bands: og's MASKED_BANDS is the line of bin/collect_summary.py", + rust_bands == collect_summary.MASKED_BANDS, rust_bands[:80]) for f in ("scripts/11-roh-analysis.sh", "modules/local/roh/main.nf"): - m = _re.search(r'^\s*MASKED_BANDS="([^"]*)"', open(os.path.join(REPO, f)).read(), _re.M) - check(f"bands: {f} holds the line of bin/collect_summary.py", - m is not None and m.group(1) == collect_summary.MASKED_BANDS, m and m.group(1)[:80]) + text = open(os.path.join(REPO, f)).read() + check(f"bands: {f} holds no copy of the bands and runs og roh summary", + "MASKED_BANDS=" not in text and "og roh summary" in text, f) # ROH: a 45 Mb chrY segment, an 18 Mb segment in chr1's 1q12 gvar block, # one over chr9's centromere, a 25 Mb chr1 segment with 21.5 Mb in the @@ -1943,13 +1951,17 @@ def report_sections_mito_roh_telomere_cnv(work, schema): collect_summary.masked_bands = orig check("ROH red control: without the band mask the gvar segment is the largest again", r0["largest_mb"] == 25.0 and len(r0["autosomal_over_5mb"]) == 4, r0) - # Step 11's awk on the same file lists the same segments. - script = open(os.path.join(REPO, "scripts/11-roh-analysis.sh")).read() - awk = script[script.index(" awk -v bands="):script.index('"$OUTPUT"', script.index(" awk -v bands=")) + len('"$OUTPUT"')] - out = subprocess.run(["bash", "-c", f'MASKED_BANDS="{collect_summary.MASKED_BANDS}"; OUTPUT="$1"; {awk}', "_", - f"{d}/vcf/S_roh.txt"], capture_output=True, text=True) - check("ROH: step 11's summary awk lists the segments the report lists", - out.returncode == 0 and out.stdout == "chr2\t10000001\t17000000\t7000000\t7.0\n", (out.stdout, out.stderr)) + # Step 11's summary (og roh summary) on the same file lists the same segments. + if OG: + out = subprocess.run([OG, "roh", "summary", "--sample", "S", "--roh", f"{d}/vcf/S_roh.txt", + "--out", f"{d}/vcf/S_roh_summary.txt"], capture_output=True, text=True) + rows = open(f"{d}/vcf/S_roh_summary.txt").read().splitlines()[3:] if out.returncode == 0 else [] + check("ROH: step 11's summary (og roh summary) lists the segments the report lists", + rows == ["chr2\t10000001\t17000000\t7000000\t7.0"], (rows, out.stderr)) + elif os.environ.get("OG_REQUIRED"): + sys.exit("FAIL: OG_REQUIRED is set but OG names no og binary") + else: + print("[SKIP] og roh summary: OG is not set (the rust job of guard.yml runs it)") # Telomere: 40 and 4,000 intratelomeric reads at 30x (620 M reads of 150 bp). head = ("PID\tsample\ttotal_reads\tread_lengths\trepeat_threshold_set\trepeat_threshold_used\tintratel_reads\t"