[semenko]

Summary

Version 3.0. Default output changes, so this is a major version: bam2tensor now keeps a methylation call only when the read gives good evidence for it, checks that the BAM matches the reference, and fixes the seven findings from the methylation review.

Merging this publishes 3.0 to PyPI (the release workflow tags a version change on main).

Which reads and calls are used

Rule	Default	Option
Proper pairs only: a paired read without flag 0x2 is skipped	on	--keep-improper-pairs
Intact CpG: the read must show the CpG's other base present, adjacent and equal to the reference, otherwise -1	always	none
Base quality: the base carrying the call needs Phred >= 20, otherwise -1	20	--min-base-quality N (0 = off)
3' end mask: calls within N bases of the 3' end of the fragment's original strand become -1	off	--mask-3prime N

• Intact CpG removes the false "unmethylated" call that a CpA/TpG variant molecule produces. It is applied identically to the Bismark (XM) and base-decoding paths. A variant on the base that carries the call looks exactly like conversion in a single read and is not caught.

• Base quality is justified by measurement on real EM-seq, at bases conversion cannot touch:

  Base quality	Error rate
  below 20	14.7%
  20 to 29	0.88%
  30 and above	0.14%

• 3' end mask removes end-repair fill-in. It is off by default because pipelines often trim for this elsewhere. The end is taken from the mates' own coordinates, not TLEN: Bismark and bwa define TLEN differently when mates extend past each other (70 of the 557 fixture pairs).

• EM over-conversion filter now ignores -1 entries, so a discarded call can no longer shield a fully unmethylated fragment.

• Guard warnings: a RuntimeWarning fires when the proper-pair or base-quality filter discards more than half of at least 1,000 reads / calls, so an aligner that never sets 0x2, or placeholder qualities, cannot silently empty the output.

Reference check

• A BAM whose header chromosome lengths differ from the FASTA is refused (--allow-length-mismatch turns the check off).
• A RuntimeWarning fires when more than 10% of at least 1,000 decoded CpGs mismatch the reference. Real data sits at 2.1%; a wrong reference of the same length at 78%.

Fixes from the methylation review

1. scipy >= 1.15 is required (scipy.sparse.get_index_dtype).
2. A CpG cache is no longer reused for a different FASTA with the same size and modification time.
3. A BAM aligned to a different reference no longer passes silently (the reference check above).
4. Two BAMs that would write the same output file are refused up front.
5. Mates are matched by name and position, so unrelated pairs sharing a query name are not merged.
6. A chromosome named twice in --expected-chromosomes is an error.
7. bam2tensor-inspect exits 1 when a file cannot be read.

Also: rows follow the --expected-chromosomes order, like the columns, whatever the FASTA's order.

Recorded with every output

metadata.json gains input_bam, aligner, every filter setting (min_mapping_quality, min_base_quality, mask_3prime, keep_improper_pairs) and extraction_stats: reads used and dropped, CpG observations, and how many were discarded for each reason. bam2tensor-inspect prints the filters.

Breaking changes

• Output: fewer rows (improper pairs dropped) and more -1 entries (not-intact CpGs, low base quality) than 2.8 for the same BAM.
• API: ExtractionResult has a third field, stats. Code that unpacks matrix, tlen = ... must change.
• Wrong reference: a BAM/FASTA pair with differing chromosome lengths is now an error.
• Format number stays 2. No release has written format 2 (2.8 writes format 1), so 3.0 is the first.

Verification

Real EM-seq cfDNA (tests/test.bam against hg38 chr1):

Setting	Fragments	Methylation
Defaults	11,062	0.664
Improper pairs kept, no quality filter	11,638	0.660
3' mask 10	11,062	0.674
3' mask 20	11,062	0.679

• The output equals an independent decoder of the same rules in five configurations: defaults; improper pairs kept with no quality filter; mask 10; mask 20; mask 20 with two worker processes.
• A first-principles chemistry simulation (strand, conversion, PCR, both read orientations) matches truth in all 16 aligner / library / pairing configurations.
• The Bismark and bwa-meth fixtures still extract to identical matrices, including with the mask at 10 and 40.
• Speed on the test BAM: 0.146 s before, 0.153 to 0.160 s after.
• nox: all 12 sessions pass locally (pre-commit, mypy, tests on 3.11 / 3.12 / 3.13, typeguard, xdoctest, docs-build); 492 tests.

Known limits

• gem3 writes no MD tag, so --filter-non-converted cannot evaluate its reads. Now documented.
• The stored fragment length (tlen) still follows each aligner's own TLEN convention.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/bam2tensor
__init__.py
__main__.py
embedding.py
functions.py					781, 1180, 1245, 1281, 1337
inspect.py
Project Total

This report was generated by python-coverage-comment-action