Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,13 @@ jobs:
steps:
- uses: actions/download-artifact@v4

- uses: actions/checkout@v4
with:
path: source

- name: Package example data
run: tar -czf madcats-demo.tar.gz --exclude=results -C source/examples madcats-demo

- name: Create Release
uses: softprops/action-gh-release@v2
with:
Expand All @@ -133,4 +140,5 @@ jobs:
madcats-linux-x86_64/*.tar.gz
madcats-macos-x86_64/*.tar.gz
madcats-macos-aarch64/*.tar.gz
madcats-demo.tar.gz
dist/*
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -184,3 +184,4 @@ docs/book/

# macOS
.DS_Store
examples/madcats-demo/results/
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# MADCATS
# MADCATS <img src="https://raw.githubusercontent.com/Breeding-Insight/MADCATS/main/docs/src/images/madcats-logo.png" align="right" width="160" alt="MADCATS logo" />

[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![PyPI](https://img.shields.io/pypi/v/madcats.svg)](https://pypi.org/project/madcats/)
Expand Down Expand Up @@ -81,6 +81,8 @@ pip install . # or: python3 madcats.py --help (no install needed)

## Quick Start

To try MADCATS on a small simulated project first, see [Try the Example](https://madcats.readthedocs.io/en/latest/getting-started/example.html).

```bash
# 1) Reconstruct each marker's Ref/Alt tags from the DArTag design files
madcats design \
Expand Down Expand Up @@ -131,7 +133,7 @@ Full documentation is available at [madcats.readthedocs.io](https://madcats.read
- [Evidence Model](https://madcats.readthedocs.io/en/latest/reference/evidence-model.html): how alleles are counted, discovered and denoised

The design rationale and version-by-version measurements behind every default are recorded in the
[development reference](docs/DEVELOPMENT.md).
[development record](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md).

## Commands

Expand Down
Binary file added assets/madcats-logo-full.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 1 addition & 2 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ and VCF-ready output. `madcats.py` uses only Python 3's standard library,
exposes all analysis thresholds on the command line, and writes a QC and
provenance JSON beside every result. (The name also reflects the tool's
origins as a FASTQ-to-MADC workflow, where MADC denotes the Missing
Allele Discovery Counts format; `madc_pipeline.py` remains as a
compatibility shim for existing scripts.)
Allele Discovery Counts format.)

**A prior MADC is not an input to the workflow.** Generation uses the FASTQs
plus the assay design: either the DArTag primer/probe files or a
Expand Down
5 changes: 4 additions & 1 deletion docs/src/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<img src="images/madcats-logo.png" align="right" width="180" alt="MADCATS logo">

# Welcome to MADCATS!

[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
Expand Down Expand Up @@ -69,7 +71,8 @@ New to MADCATS? Start here:

1. [Installation](getting-started/installation.md): install with pip or download a binary
2. [Quick Start](getting-started/quickstart.md): your first MADC in two commands
3. [Choosing a Workflow](getting-started/workflows.md): which commands your platform and inputs need
3. [Try the Example](getting-started/example.md): run a small simulated project and check it against the truth
4. [Choosing a Workflow](getting-started/workflows.md): which commands your platform and inputs need

## Scope

Expand Down
2 changes: 1 addition & 1 deletion docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

- [Installation](getting-started/installation.md)
- [Quick Start](getting-started/quickstart.md)
- [Try the Example](getting-started/example.md)
- [Choosing a Workflow](getting-started/workflows.md)
- [Input Formats](getting-started/input-formats.md)

Expand Down Expand Up @@ -39,7 +40,6 @@
- [Architecture](development/architecture.md)
- [Contributing](development/contributing.md)
- [Changelog](development/changelog.md)
- [Development Reference](development/reference.md)

-----------

Expand Down
2 changes: 1 addition & 1 deletion docs/src/citation.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,5 +47,5 @@ polyploids from messy sequencing data. Genetics 210(3):789-807.
In your methods section:

> Microhaplotype read counts were generated from the demultiplexed FASTQ files with MADCATS
> v0.12.1 (Sandercock, 2026), using panel tags reconstructed from the vendor design files and
> v0.13.0 (Sandercock, 2026), using panel tags reconstructed from the vendor design files and
> default evidence and denoising settings.
10 changes: 5 additions & 5 deletions docs/src/commands/generate.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# madcats generate

Generate a MADC-compatible microhaplotype count table from per-sample FASTQs and a panel.
Generate a MADC microhaplotype count table from per-sample FASTQs and a panel.

## Synopsis

Expand Down Expand Up @@ -55,7 +55,7 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de
| `--sample-lookup <FILE>` | none | Two-column CSV/TSV mapping FASTQ-derived ids to sample names. Every run sample must be mapped |
| `--tag-length <INT>` | 81 | Tag length |

## Read Assignment (stage 2; these settings key the cache)
## Read Assignment (stage 2)

| Option | Default | Description |
|--------|---------|-------------|
Expand All @@ -64,7 +64,7 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de
| `--primer-mismatches {0,1}` | 1 | Mismatches allowed while assigning a primer/anchor. With `0`, reads from Match haplotypes whose distinguishing SNP falls inside the anchor are lost |
| `--gapped-rescue {on,off}` | `on` | Give reads that fail substitution-only identity one gapped alignment, so undesigned indels are counted |
| `--max-indel-length <INT>` | 10 | Widest gap (alignment band) a gapped alignment may use |
| `--partial-read-mode {off,assign}` | `assign` | Add a read shorter than the tag to the one full-length allele (already passing the evidence rules) whose start it matches, only when it is longer than every prefix shared between that locus's alleles. `off` drops such reads |
| `--partial-read-mode {on,off}` | `on` | Add a read shorter than the tag to the one full-length allele (already passing the evidence rules) whose start it matches, only when it is longer than every prefix shared between that locus's alleles. `off` drops such reads |
| `--min-partial-length <INT>` | tag length − 9 | Minimum length of a credited partial read |
| `--cache-dir <DIR>` | none | Reusable per-sample count cache |
| `--jobs <INT>` | up to 4 | Samples processed in parallel (also parallelizes denoising) |
Expand All @@ -85,8 +85,8 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de

| Option | Default | Description |
|--------|---------|-------------|
| `--denoise-distance {0,1,2}` | 1 | Per-sample abundance-skew denoising distance |
| `--global-denoise-distance {0,1,2}` | 1 | Edit distance for the pooled same-allele merge; `0` disables |
| `--denoise-distance {0,1,2}` | 1 | Per-sample denoising: the most differing bases allowed between a rare sequence and its parent; `0` disables |
| `--global-denoise-distance {0,1,2}` | 1 | Most differing bases allowed in the pooled same-allele merge; `0` disables |
| `--denoise-alpha <FLOAT>` | 2.0 | UNOISE-style abundance-skew exponent |

## Output (stages 6–7)
Expand Down
3 changes: 2 additions & 1 deletion docs/src/development/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,11 @@ madcats.py`), as the installed `madcats` command, or frozen into a standalone bi
| Path | Contents |
|------|----------|
| `madcats.py` | The whole tool: library functions and the command line |
| `madc_pipeline.py` | Compatibility shim re-exporting `madcats` under the tool's former name |
| `tests/` | Unit and end-to-end tests (`unittest`) |
| `scripts/` | Diagnostic and conversion scripts (e.g. `madc_to_reference_tags.py`, `validate_inferred_design.py`) |
| `docs/` | This documentation (mdBook), plus `DEVELOPMENT.md`, the long-form development record |
| `examples/madcats-demo/` | Simulated example project and the script that generates it |
| `assets/` | Full-size logo (2028 × 2028 PNG); the docs use a 600 px copy in `docs/src/images/` |
| `packaging/` | PyInstaller entry point for the standalone binaries |
| `pyproject.toml` | Package metadata for PyPI |
| `.github/workflows/` | Tests on every push; PyPI and binary releases on version tags |
Expand Down
29 changes: 20 additions & 9 deletions docs/src/development/changelog.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,28 @@
# Changelog

Notable changes by version. Measurements behind each change are in the
[Development Reference](reference.md).
[development record](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md) on GitHub.

## 0.13.0

- Removed the `madc_pipeline` compatibility module. Import `madcats` instead; the bundled
scripts already do.
- `--partial-read-mode` takes `on`/`off` like the other switches; the old value `assign` still
works as `on`, with a warning.
- Clearer command-line help: every command has a description, every option has help text, and
the program is named `madcats` in usage lines.
- The development record (`docs/DEVELOPMENT.md`) is no longer published on the docs site; it
stays in the repository.
- A small simulated example project, with its true haplotypes and genotypes, is included in
`examples/madcats-demo` and attached to each release as `madcats-demo.tar.gz`.
- The test suite checks that the command reference pages match each command's `--help`.

## Unreleased
## 0.12.1

- Packaged for PyPI: `pip install madcats` installs a `madcats` command.
- Standalone binaries for Linux and macOS are built for each release.
- New documentation site; the former README is now the development reference.
- New documentation site; the former README is now the development record, `docs/DEVELOPMENT.md`.
- Licensed under GPL-3.0-or-later.

## 0.12.1

- `generate`'s finalisation (per-sample denoising and the pooled pass) runs across the `--jobs`
workers. The selection is identical to the sequential passes. On a 44-run rhAmpSeq family the
pooled pass alone had taken 1,123 s in one process.
Expand All @@ -28,7 +39,7 @@ Notable changes by version. Measurements behind each change are in the

## 0.11.0

- Partial reads (`--partial-read-mode assign`, default): reads shorter than the tag are
- Partial reads (`--partial-read-mode assign`, now `on`; default): reads shorter than the tag are
added to an allele that already passed the evidence rules and whose start they match, only when
this cannot change an allele ratio.
`--min-partial-length` defaults to tag length − 9. The experimental `extend` mode was removed.
Expand Down Expand Up @@ -97,5 +108,5 @@ Notable changes by version. Measurements behind each change are in the

- `infer-design` for panels without a primer file.
- 0.3.x–0.4.x: quality-gated discovery and the exploratory audits that motivated 0.5.0.
- The tool was renamed from `madc_pipeline` to MADCATS. `madc_pipeline.py` remains as a
compatibility shim.
- The tool was renamed from `madc_pipeline` to MADCATS, with `madc_pipeline.py` kept as a
compatibility shim until 0.13.0.
10 changes: 9 additions & 1 deletion docs/src/development/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,18 @@ indel support, normalization and window freezing, output framing, paired-end mer
window-scan, gapped alignment, denoising and partial reads. Tests run on every push and pull
request across Python 3.9–3.13 (`.github/workflows/tests.yml`).

Two tests guard the documentation. `tests/test_docs.py` checks that each page in
`docs/src/commands/` lists exactly the options that command's `--help` shows, with matching
choices and defaults, so update the page whenever you add or change an option.
`tests/test_example.py` runs the example in `examples/madcats-demo` and checks the output against
its simulated truth. If a change alters the example's numbers, update
[Try the Example](../getting-started/example.md) to match. To change the example data itself,
edit and rerun `examples/madcats-demo/make_demo_data.py`.

Changes that touch counting should also be checked against real data. The standard is byte-for-byte:
a refactor regenerates existing tables identically from cache, and a behaviour change is measured
against a comparator before it becomes a default. Record those measurements in
`docs/DEVELOPMENT.md`.
[`docs/DEVELOPMENT.md`](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md).

## Code Style

Expand Down
1 change: 0 additions & 1 deletion docs/src/development/reference.md

This file was deleted.

110 changes: 110 additions & 0 deletions docs/src/getting-started/example.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Try the Example

MADCATS comes with a small simulated DArTag-style project that runs in seconds. The true
haplotypes and genotypes behind its reads are included, so you can check every row of the output
against what was simulated.

## Get the Data

Download it from the latest release (MADCATS 0.13.0 and later):

```bash
curl -LO https://github.com/Breeding-Insight/MADCATS/releases/latest/download/madcats-demo.tar.gz
tar -xzf madcats-demo.tar.gz
cd madcats-demo
```

The same files are in the repository under `examples/madcats-demo`.

| File | Contents |
|------|----------|
| `primers.csv` | DArT-style primer CSV for the 8 markers |
| `probe_design.tsv` | Probe design with one bracketed SNP per marker |
| `fastq/` | One FASTQ per sample, `S01.fastq.gz` … `S12.fastq.gz` |
| `sample_names.csv` | Renames the samples from `S01` to `Line_01`, and so on |
| `truth_haplotypes.csv` | Every simulated haplotype's 81-bp tag sequence |
| `truth_genotypes.csv` | Each sample's simulated diploid genotype at each marker |
| `make_demo_data.py` | The script that made all of the above; rerunning it reproduces the files exactly |

## What Was Simulated

Twelve diploid samples, eight markers with 81-bp tags, and about 550 reads per sample. Beyond each
marker's designed Ref and Alt alleles:

- **Chr02** and **Chr04** carry an extra microhaplotype: a second SNP on the Ref or Alt background.
- **Chr06** carries a haplotype with an undesigned 1-bp insertion.
- **Chr08** has no Alt allele in any sample.
- **Chr03** and **Chr07** have their primer on the reverse strand of the probe-design target.
- Every read has a 0.2% chance per base of a sequencing error, about 8% of reads are shorter than
the tag, and each sample has a few off-target reads.

## Run It

Reconstruct the panel from the design files, then count:

```bash
madcats design \
--primers primers.csv \
--probe-design probe_design.tsv \
--output results/reference_tags.csv

madcats generate \
--fastq-dir fastq \
--reference-tags results/reference_tags.csv \
--sample-lookup sample_names.csv \
--output results/demo_MADC.csv
```

`generate` writes the MADC table, `results/demo_MADC.csv.qc.json` and
`results/demo_MADC.csv.variants.csv`.

## Check the Results

The table has 19 rows: a Ref and an Alt row for each of the 8 markers, plus the 3 microhaplotypes
MADCATS discovered. With MADCATS 0.13.0 the discovered rows are:

| AlleleID | Variants | MarkerType | Reads (all samples) |
|----------|----------|------------|--------------------:|
| `Chr02_001257730\|RefMatch` | `67:A>T` | `snp` | 150 |
| `Chr04_002318006\|AltMatch` | `25:G>T;57:A>T` | `snp` | 122 |
| `Chr06_001644492\|RefMatch` | `29:G>GC;clip=1` | `indel` | 128 |

Things to look for:

- **Every simulated haplotype is a row, and nothing else is.** Compare the `AlleleSequence` column
with `truth_haplotypes.csv`. The sequencing errors were merged back into their true haplotypes by
denoising, so none of them became rows.
- **The insertion was recovered.** Its reads failed the substitution-only identity check and were
placed by gapped alignment. `Variants` reports the insertion left-aligned (`29:G>GC`), and
`clip=1` notes that it pushed the last Ref base out of the 81-bp window.
- **Chr08 still has an Alt row.** Canonical Ref and Alt rows are always written. Its single read
comes from a sequencing error that happened to hit the SNP base.
- **Counts follow the genotypes.** In each `Line_` column, a heterozygous sample splits its reads
between two haplotypes and a homozygous sample puts nearly all of them on one. Compare with
`truth_genotypes.csv`.
- **Every read is accounted for.** The `read_accounting` section of the QC JSON shows 6,989 reads in
total, 129 assigned by gapped alignment, 466 partial reads added to counts, 851 merged away by
per-sample denoising, and 64 in error sequences that failed the evidence rules. Each sample's
entry in `sample_qc` also counts its off-target reads (`reads_no_primer`).

## Try Changing Something

Each of these reruns `generate` with one setting changed:

| Change | What happens |
|--------|--------------|
| `--reproducible-min-samples 5 --depth-min-reads 0` | The 3 discovered rows disappear: each extra haplotype is carried by only 4 samples, and the depth route is off |
| `--gapped-rescue off` | The Chr06 insertion row disappears; its reads are no longer assigned |
| `--partial-read-mode off` | Same 19 rows, but reads in rows fall from 6,661 to 6,195 |

To compare two runs, use `validate`:

```bash
madcats validate --generated results/other_MADC.csv --expected results/demo_MADC.csv
```

## Next Steps

- [Choosing a Workflow](workflows.md): which commands your own data needs
- [Evidence Model](../reference/evidence-model.md): how reads are assigned, discovered and denoised
- [Output Formats](../reference/output-formats.md): every column of the MADC table
3 changes: 2 additions & 1 deletion docs/src/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ madcats --version
```

```
madcats 0.12.1
madcats 0.13.0
```

To run the test suite from a source checkout:
Expand All @@ -123,4 +123,5 @@ python3 -m unittest discover -s tests -v
## Next Steps

- [Quick Start](quickstart.md): generate your first MADC
- [Try the Example](example.md): run a small simulated project in seconds
- [Choosing a Workflow](workflows.md): which commands your data needs
1 change: 1 addition & 0 deletions docs/src/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ See [Output Formats](../reference/output-formats.md) for details.

## Next Steps

- [Try the Example](example.md): run these steps on a small simulated project and check the output
- [Choosing a Workflow](workflows.md): routes for other platforms and inputs
- [madcats generate](../commands/generate.md): every option, grouped by pipeline stage
- [Evidence Model](../reference/evidence-model.md): when a new sequence is reported as a row
Binary file added docs/src/images/madcats-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions docs/src/reference/evidence-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This page explains how `madcats generate` decides which reads count, which sequences become rows,
and how sequencing errors are removed. The design rationale and the measurements behind each default
are in the [Development Reference](../development/reference.md).
are in the [development record](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md) on GitHub.

## Read Assignment

Expand Down Expand Up @@ -61,8 +61,8 @@ depth route.
Canonical sequences and sequences that pass the evidence rules are protected: denoising never merges
them into anything else.

1. Per-sample denoising (UNOISE-style abundance skew). Within each sample, a rare sequence within
`--denoise-distance` (1) edits of a more abundant parent is merged into the parent when the
1. Per-sample denoising (UNOISE-style abundance skew). Within each sample, a rare sequence that differs at
no more than `--denoise-distance` (1) bases from a more abundant parent is merged into the parent when the
parent is at least `2^(alpha·d + 1)` times more abundant (`--denoise-alpha` 2.0). The closest
parent is chosen first, then the most abundant.
2. Pooled same-allele merge (`--global-denoise-distance`, 1). A sequence that failed the evidence
Expand Down
2 changes: 1 addition & 1 deletion docs/src/reference/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

MADCATS has been validated on five platforms, each against the best independent comparison available
for that dataset. This page summarizes the results. The full record, including every
version-to-version comparison, is in the [Development Reference](../development/reference.md).
version-to-version comparison, is in the [development record](https://github.com/Breeding-Insight/MADCATS/blob/main/docs/DEVELOPMENT.md) on GitHub.

## Performance at a Glance

Expand Down
Binary file added docs/theme/favicon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading