Add bwa-mem3 (TAPS and bisulfite/EM-seq) and rastair version 2 by eduard-watchmaker · Pull Request #638 · nf-core/methylseq

eduard-watchmaker · 2026-11-09T20:20:53Z

This PR adds bwa-mem3 as a methylation-aware aligner for both TAPS and bisulfite/EM-seq data, and rastair version 2 as an optional TAPS caller. Both TAPS callers now estimate read-end trims from each sample's M-bias by default.

What's new

--aligner bwamem3. bwa-mem3 aligns directional libraries against a conversion-aware index (bwa-mem3 index --meth); --pbat, --non_directional, --zymo and --single_cell stop with an error. The data type picks the mode:
- with --taps: TAPS mode (mem --meth=taps), then rastair calls methylation (--taps_caller).
- without --taps: bisulfite/EM-seq mode (mem --meth), then MethylDackel extracts methylation, as with --aligner bwameth.
Both modes use the same index. Alignment is followed by Picard MarkDuplicates and samtools stats/flagstat/idxstats.
--bwamem3_index. Pass a pre-built index as a directory, a .tar.gz or an index prefix (S3 works). Otherwise it's built from --fasta (keep it with --save_reference). docs/usage.md explains building it once for human-sized genomes, and the pipeline warns at launch when it would build one from a FASTA over 300 MB. iGenomes has no bwa-mem3 index; hosting one is proposed in Add bwa-mem3 index generation (plain and methylation --meth) references#171.
--taps_caller rastair2. rastair version 2 replaces the rastair 0.8 steps (rastair 0.8 stays the default). It jointly calls SNVs, indels and CpG methylation, writing a bgzipped VCF and a CpG BED. Single-end samples get --unpaired.
- rastair version 2 is free for non-commercial use only (University of Oxford), so it requires --accept_rastair2_license and prints the license notice.
- MultiQC gets bcftools stats of the VCF, General Statistics columns (CpG methylation, CpGs, depth), a methylation distribution, and a per-contig table that includes any spike-in controls. Spike-ins are not required.
M-bias read-end trims by default, for both callers.
- rastair 0.8: rastair mbias + rastair/mbiasparser, as before.
- rastair version 2: rastair2 mbias on --rastair2_mbias_region (default: the first 10 Mb of the longest contig; a methylated spike-in such as lambda is better, and needed for shallow libraries).
- When the trims can't be estimated (for example single-end samples), nothing is trimmed. --trim_OT/--trim_OB (now unset by default; previously 0,0,10,0 for rastair version 2 only) replace the estimate for both callers.
rastair 0.8 single-end warning. rastair 0.8 only calls properly paired reads, so single-end samples get no calls with --taps_caller rastair. The pipeline warns at the start of the TAPS step and the docs point to rastair version 2.
-profile test_taps. Mouse TAPS reads from the TAPS paper (GEO GSE112520, via rastair's test data) against mm10 chr19 plus methylated and unmodified spike-ins (methylseq: add bwa-mem3 --meth index and TAPS test data test-datasets#2370).
New tests.
- tests/bwamem3_taps.nf.test: rastair 0.8 (with the single-end warning), rastair version 2 with M-bias trims, rastair version 2 with --trim_OT/--trim_OB, and the licence-not-accepted error.
- tests/bwamem3_bisulfite.nf.test: default, skip_deduplication, save_reference + save_align_intermeds.
- A bwa-mem3 scenario in tests/index_downloads.nf.test.

Fixes picked up along the way

Declare nf-amazon without the 2.15.0 pin that came with the template sync (Important! Template update for nf-core/tools v4.2.1 #634), exactly as Drop version pin on nf-amazon plugin #642 does (which carries the CHANGELOG entry). That release doesn't work with Nextflow 25.10 (NoSuchMethodError on Session.getExecConfigProp), so every run reading S3 paths or using AWS Batch failed at startup, on dev too.
bwameth/align no longer writes into the shared staged index (bwameth/align: stop writing the fasta link into the shared staged index (race, read-only index) modules#13642). This replaces the local patch from Update bwameth to 0.2.10 #637 with the same code.
Preseq runs on the pre-deduplication Bismark BAM (bam_aligned from fastq_align_dedup_bismark: emit raw aligned BAM (bam_aligned) modules#13643).

Fixes from review and a pipeline-wide audit

After @FelixKrueger's review I audited the whole pipeline (channel pairing, every parameter against every aligner, and a stub run of about 480 parameter combinations). Fixed here:

rastair 0.8 trims: bam_taps_conversion paired BAMs, BAIs and M-bias trims by arrival order, so a sample could be called with another sample's trims. They are now joined by sample (local patch; the upstream fix has a two-sample test that fails on every run without it).
--collecthsmetrics never ran CollectHsMetrics (an empty GZI channel emptied the input); it does now.
--aligner bwamem without GPU emitted no BAM index or samtools stats, so --skip_deduplication left TAPS calling without input (local patch of fastq_align_dedup_bwamem until the upstream fix merges).
--rrbs now skips deduplication for bwamem and bwamem3 too, as for bismark and bwameth.
--taps with bismark stops with an error instead of producing no calls; --methyl_kit with targeted sequencing (MethylDackel) errors, since it removes the bedGraphs the targeted step reads.
Strandedness: bwa-meth and rastair 0.8 warn that they ignore --pbat/--non_directional/--zymo/--single_cell; rastair version 2 guesses read orientation for all of them.
Warnings for options that have no effect in the run (TAPS trims outside TAPS, --collecthsmetrics without targeted sequencing, --bamqc_regions_file without Qualimap, --slamseq outside Bismark or with --bismark_index).
Publishing: bismark_hisat coverage2cytosine and targeted outputs go under bismark_hisat/; GPU bwamem no longer publishes raw Parabricks BAMs to parabricks/ or stray indexes to deduplicated/.
Smaller: --bwamem3_index genome.fa.gz prefixes work (only .tar.gz is untarred); BWAMEM3_INDEX gets 12 CPUs and 24 h; the --ignore_3prime_r1/r2 help texts were swapped; the HISAT2 index test never used a pre-built index; --trim_OT/--trim_OB now take effect for rastair 0.8, so old params files that set them change results.

Most of these were already on dev; they are fixed here so they land together.

nf-core components

Everything is installed from nf-core/modules master, with no local subworkflows or branch installs:

fastq_align_dedup_bwamem3 and bwamem3/mem index_format (bwamem3/mem: index_format (BAI) input; new subworkflow fastq_align_dedup_bwamem3 modules#13640)
fasta_index_methylseq with bwamem3 (fasta_index_methylseq: add bwamem3 aligner modules#13622; the local patch only keeps the methurator input, as before)
bam_taps_conversion with use_rastair2 and a trims input (bam_taps_conversion: optional rastair version 2 caller (use_rastair2) modules#13641)
rastair2/call and the new rastair2/mbias (rastair2/mbias: new module; rastair2/call: take M-bias cutoffs modules#13678)
fastq_align_dedup_bismark bam_aligned (fastq_align_dedup_bismark: emit raw aligned BAM (bam_aligned) modules#13643) and bwameth/align (bwameth/align: stop writing the fasta link into the shared staged index (race, read-only index) modules#13642)

nf-core/modules#13699 (drop the plain .bwt.2bit.64 from --meth indexes) isn't needed here. With --bwamem3_index <prefix> the pipeline already stages only the files alignment reads.

Testing

nf-test: all 54 tests pass (everything except GPU) on native x86_64 with Docker and Nextflow 25.10.4. The bwa-mem3 snapshots were generated on x86_64. On Apple Silicon, MultiQC skips plot export under Rosetta, so snapshots made there lack multiqc_plots.
nf-core pipelines lint: 480 passed, 0 failed.

Human TAPS data on AWS Batch: two in-house NA12878 TAPS libraries (1 ng and 200 ng input, 250k read pairs each) on hg38 + lambda + an unmodified 2 kb control, with the bwa-mem3 index given as an S3 prefix. Pooled CpG methylation:

caller, trims	trims (1 ng / 200 ng)	NA12878	lambda	unmodified
rastair 0.8, M-bias (default)	OT `2,1,5,0` / `2,2,5,0`; OB `2,1,11,0` / `2,1,4,0`	0.496-0.498	0.980-0.982	0.004-0.005
rastair 2, M-bias on lambda (`--rastair2_mbias_region J02459.1`)	OT `2,0,5,0` / `2,0,6,0`; OB `4,0,4,0`	0.497	0.980-0.982	0.004-0.005
rastair 2, M-bias on the default region	none (too few reads in chr1:1-10 Mb at this depth)	0.492-0.493	0.961-0.964	0.004-0.005
rastair 2, fixed `0,0,10,0` (previous default)		0.498-0.499	0.973-0.975	0.004-0.005

Mapping 99.7% and duplication 5.9% / 8.9% match our production TAPS pipeline, as do the controls (lambda 0.98, unmodified 0.004-0.005).
With M-bias trims from lambda, rastair 0.8 and rastair version 2 agree on all three. At this depth the default region holds about 3,000 CpG observations, too few for an estimate; at 30x WGS it holds roughly a thousand times more. The docs recommend a spike-in region for shallow libraries.
Aligning the same reads in bisulfite mode inverts the controls, as expected. On hg38, BWAMEM3_MEM peaked at 23 GB of RAM.

PR checklist

Generated by Claude Code

🤖 Generated with Claude Code

- --aligner bwamem3: bwa-mem3 mem --meth=taps against a bwa-mem3 index --meth index (built from --fasta or given with --bwamem3_index), then Picard MarkDuplicates and samtools stats (local subworkflow FASTQ_ALIGN_DEDUP_BWAMEM3; FASTA_INDEX_METHYLSEQ patched for the index). - --taps_caller rastair2: rastair version 2 call (VCF + CpG BED), opt-in via --accept_rastair2_licence because of its non-commercial licence. meta.single_end adds --unpaired; --non_directional adds --guess-read-orientation; --trim_OT/--trim_OB set clipping. - Install bwamem3/index, bwamem3/mem and rastair2/call from nf-core/modules. - Schema, docs, CITATIONS and CHANGELOG. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- --bwamem3_index accepts an index prefix (e.g. s3://bucket/hg38.fa) as well as a directory or .tar.gz, and can come from a genome config (`bwamem3` attribute). - MultiQC for --taps_caller rastair2: bcftools stats of the VCF, and a local RASTAIR2_SUMMARY module (bin/rastair2_summary.sh) for genome-wide and per-contig CpG methylation and a methylation distribution. Works with or without spike-ins. - test_taps profile, assets/samplesheet_taps.csv and tests/bwamem3_taps.nf.test (rastair, rastair2, licence check) on TAPS test data pending in nf-core/test-datasets#2370. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Without --taps, --aligner bwamem3 aligns with `bwa-mem3 mem --meth` (unmethylated C->T) and MethylDackel extracts methylation, as for bwameth; with --taps it stays in TAPS mode with rastair. Both modes use the same `bwa-mem3 index --meth` index. Adds tests/bwamem3_bisulfite.nf.test. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rence nf-core/test-datasets#2370 now ships rastair's full mm10 chr19 (plus spike-ins) as reference/genome_taps.fa.gz instead of a 41 kb slice, so alignments are at real chr19 coordinates. Methylation and mapping figures are unchanged. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Uses reference/Bwamem3_Index.tar.gz from nf-core/test-datasets (methylseq branch, added in nf-core/test-datasets#2370). Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- docs/usage.md + schema: build the bwa-mem3 --meth index once for human-sized genomes and pass it with --bwamem3_index (prefix incl. S3, directory or .tar.gz); measured hg38 figures (23 GB peak for BWAMEM3_MEM; the plain .bwt.2bit.64 and .meth.fa aren't needed for alignment). - Warn at launch when --aligner bwamem3 has no --bwamem3_index and the FASTA is over 300 MB. - fasta_index_methylseq: sync the local patch with nf-core/modules#13622 (bwamem/bwamem3 share one branch; supplied tarballs unpack via UNTAR_BWAMEM). - rastair2_summary.sh: per-contig MultiQC columns in natural order (chr1..22, X, Y, M, then spike-ins). - CHANGELOG and test_taps profile description for the new TAPS test data. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Snapshots made on Apple Silicon lack MultiQC's exported plots (MultiQC skips plot export under Rosetta), so regenerate them on x86_64 to match CI. Also picks up the rastair2 per-contig column order and the UNTAR_BWAMEM alias. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-11-09T20:21:41Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 4.2.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the Synchronisation documentation.

github-actions · 2026-11-09T20:24:02Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 7f1866b

+| ✅ 258 tests passed       |+
#| ❔   8 tests were ignored |#
!| ❗   2 tests had warnings |!

Details

❗ Test warnings:

pipeline_todos - TODO string in CONTRIBUTING.md: Add any pipeline specific contribution guidelines here, such as coding styles, procedures, checklists etc.
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

files_exist - File is ignored: lib/nfcore_external_java_deps.jar
files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: .github/workflows/ci.yml
files_unchanged - File ignored due to lint config: assets/nf-core-methylseq_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-methylseq_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-methylseq_logo_dark.png
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/nf-test.yml
files_exist - File found: .github/actions/get-shards/action.yml
files_exist - File found: .github/actions/nf-test/action.yml
files_exist - File found: .github/workflows/pr-comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-methylseq_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/CONTRIBUTING.md
files_exist - File found: docs/images/nf-core-methylseq_logo_light.png
files_exist - File found: docs/images/nf-core-methylseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: nf-test.config
files_exist - File found: tests/default.nf.test
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: ro-crate-metadata.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-methylseq_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowMethylseq.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: manifest.diagram
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config variable (correctly) not found: validation.failUnrecognisedParams
nextflow_config - Config variable (correctly) not found: validation.failUnrecognisedHeaders
nextflow_config - Config timeline.enabled had correct value: True
nextflow_config - Config report.enabled had correct value: True
nextflow_config - Config trace.enabled had correct value: True
nextflow_config - Config dag.enabled had correct value: True
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config manifest.diagram file found: docs/images/4.2.0_metromap.svg
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 4.3.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.aligner= bismark
nextflow_config - Config default value correct: params.clip_r1= 0
nextflow_config - Config default value correct: params.clip_r2= 0
nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
nextflow_config - Config default value correct: params.nextseq_trim= 0
nextflow_config - Config default value correct: params.num_mismatches= 0.6
nextflow_config - Config default value correct: params.no_overlap= true
nextflow_config - Config default value correct: params.ignore_r1= 0
nextflow_config - Config default value correct: params.ignore_r2= 2
nextflow_config - Config default value correct: params.ignore_3prime_r1= 0
nextflow_config - Config default value correct: params.ignore_3prime_r2= 2
nextflow_config - Config default value correct: params.min_depth= 0
nextflow_config - Config default value correct: params.taps_caller= rastair
nextflow_config - Config default value correct: params.methurator_minimum_coverage= 1,10,15
nextflow_config - Config default value correct: params.methurator_compute_ci= false
nextflow_config - Config default value correct: params.methurator_t_max= 10
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/methylseq/
nf_test_content - 'tests/targeted_sequencing_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/targeted_sequencing_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/combined_index.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/combined_index.nf.test' contains outdir parameter
nf_test_content - 'tests/combined_index.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem_taps_use_gpu.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem_taps_use_gpu.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwamem3_taps.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem3_taps.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwamem3_taps.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem3_taps.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwamem3_taps.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem3_taps.nf.test' contains outdir parameter
nf_test_content - 'tests/default.nf.test' contains outdir parameter
nf_test_content - 'tests/default.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/default.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/default.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem_taps.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem_taps.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwamem3_bisulfite.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwamem3_bisulfite.nf.test' contains outdir parameter
nf_test_content - 'tests/bwamem3_bisulfite.nf.test' contains outdir parameter
nf_test_content - 'tests/bismark_hisat_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bismark_hisat_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/index_downloads.nf.test' contains outdir parameter
nf_test_content - 'tests/index_downloads.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwameth_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwameth_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/bwameth_use_gpu.nf.test' contains outdir parameter
nf_test_content - 'tests/bwameth_use_gpu.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bismark_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bismark_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/nextflow.config' contains modules_testdata_base_path
nf_test_content - 'tests/nextflow.config' contains pipelines_testdata_base_path
nf_test_content - 'tests/nextflow.config' contains pipelines_testdata_base_path
nf_test_content - 'tests/nextflow.config' contains pipelines_testdata_base_path
nf_test_content - 'nf-test.config' sets a testsDir
nf_test_content - 'nf-test.config' sets a workDir
nf_test_content - 'nf-test.config' sets a configFile
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/pr-comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
actions_nf_test - '.github/workflows/nf-test.yml' is triggered on expected events
actions_nf_test - '.github/workflows/nf-test.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 25.10.4, Config: 25.10.4
readme - README nf-core template version badge found.
readme - README Zenodo placeholder was replaced with DOI.
pipeline_if_empty_null - No ifEmpty(null) strings found
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: template-version-comment.yml
actions_schema_validation - Workflow validation passed: nf-test.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: update-gpu-snapshot.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: pr-comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: nf-test-gpu.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
local_component_structure - local subworkflows directory structure is correct 'subworkflows/local/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - PRESEQ_LCEXTRAP found in conf/base.config and Nextflow scripts.
base_config - METHURATOR_GTESTIMATOR found in conf/base.config and Nextflow scripts.
base_config - METHURATOR_PLOT found in conf/base.config and Nextflow scripts.
base_config - BISMARK_ALIGN found in conf/base.config and Nextflow scripts.
base_config - BISMARK_DEDUPLICATE found in conf/base.config and Nextflow scripts.
base_config - BISMARK_METHYLATIONEXTRACTOR found in conf/base.config and Nextflow scripts.
base_config - BWAMETH_ALIGN found in conf/base.config and Nextflow scripts.
base_config - PARABRICKS_FQ2BAMMETH found in conf/base.config and Nextflow scripts.
base_config - PARABRICKS_FQ2BAM found in conf/base.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 4.2.2
rocrate_readme_sync - RO-Crate descriptions are in sync with README.md.
container_configs - conf/containers_conda_lock_files_amd64.config is up to date
container_configs - conf/containers_conda_lock_files_arm64.config is up to date
container_configs - conf/containers_docker_amd64.config is up to date
container_configs - conf/containers_docker_arm64.config is up to date
container_configs - conf/containers_singularity_https_amd64.config is up to date
container_configs - conf/containers_singularity_https_arm64.config is up to date
container_configs - conf/containers_singularity_oras_amd64.config is up to date
container_configs - conf/containers_singularity_oras_arm64.config is up to date

Run details

nf-core/tools version 4.2.2
Run at 2026-11-12 18:08:35

The pipeline is on template 4.0.3, whose container_configs lint expects 'container =' entries; tools 4.2.1 had rewritten them as 'conda ='. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Regenerate conf/containers_conda_lock_files_*.config with nf-core tools 4.2.1 (conda = ...) - Apply dev's MultiQC static-plot ignore to the bwamem3 pipeline tests and drop plot entries from their snapshots - Sync ro-crate-metadata.json description with the README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nft-utils 1.2.1 (from the 4.2.1 template) returns removeNextflowVersion() as sorted TreeMaps, so the bwamem3 and bwamem3-index snapshots need the same order. Content unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Regenerate conf/containers_*.config with nf-core tools 4.2.1 (bwameth 0.2.10 + bwamem3/rastair2/bcftools) - README: keep the bwa-mem3 column and take #633's methurator correction for the bwa-mem + TAPS column - Sync ro-crate-metadata.json with the README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Install the nf-core subworkflow fastq_align_dedup_bwamem3 (nf-core/modules#13640) and drop the local copy. bwa-mem3 now writes the alignment BAI itself (bwamem3/mem updated for its index_format input). - Update fasta_index_methylseq to nf-core/modules#13622. The patch now only adds methylseq's methurator input. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…verride - Run both TAPS callers through BAM_TAPS_CONVERSION (nf-core/modules#13641, installed from its branch until it is merged). rastair version 2 now estimates trims with rastair2/mbias (#13678), like rastair 0.8 does with mbias + mbiasparser. When they can't be estimated, nothing is trimmed. - --trim_OT/--trim_OB (default unset) replace the estimate for both callers. - --rastair2_mbias_region: region for the rastair2 M-bias estimate, by default the first 10 Mb of the longest contig. - Warn when rastair 0.8 gets single-end samples: it only calls properly paired reads, so they get no methylation calls. - rastair 0.8 modules updated to nf-core master (version topics); the mbiasparser patch is no longer needed. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- bwamem3 TAPS: new test with --trim_OT/--trim_OB 0,0,10,0 (no rastair2 M-bias step; reproduces the previous default BED exactly); the default rastair2 test now checks the M-bias outputs (paired-end only); the rastair 0.8 test checks the single-end warning. - Snapshots regenerated on x86_64 (full suite, 53 tests): the bwa-mem3 BAI now comes from BWAMEM3_MEM (no SAMTOOLS_INDEX_ALIGN), rastair 0.8 reports version "0.8.2" through topics, and rastair2 outputs change with the estimated trims. All other snapshots are unchanged. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…he patch nf-core/modules#13643 added the bam_aligned emit upstream, so the local patch is no longer needed. main.nf only gains the licence header; outputs are unchanged. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts: # CHANGELOG.md

Update parabricks/fq2bam and parabricks/fq2bammeth (4.8.0-1), the samtools and Picard modules (1.25, 3.5.0), methyldackel and bwa/mem, and the fastq_align_dedup_bwameth, fastq_align_dedup_bwamem, fastq_align_bwa, bam_sort_stats_samtools, bam_stats_samtools and bam_methyldackel subworkflows to nf-core/modules master. fq2bam now takes known sites with their index. Add a GPU test for --aligner bwamem --taps (parabricks/fq2bam). Snapshots for the CPU tests were regenerated on x86_64; the GPU snapshots were built from AWS Batch GPU runs whose Parabricks 4.6 outputs reproduce the existing GPU snapshot exactly. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts: # ro-crate-metadata.json

Ignore the Parabricks fq2bam BAMs (parabricks/*.bam, *.bai) in snapshots, as for the other alignment BAMs: their headers hold run-specific paths, so the md5 changes between runs. Their reads are still checked through getReadsMD5, which drops leading zeros. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

heuermh · 2026-11-11T21:51:19Z

A note that I'll be testing this pull request (on rastair 0.8) on internal datasets over the next couple days, sorry for taking a little while. For sequencing, should this be merged at bwa-mem3 version 0.16.0, or should we push version 0.17.0 up through modules first?

Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eduard-watchmaker · 2026-11-11T22:22:16Z

A note that I'll be testing this pull request (on rastair 0.8) on internal datasets over the next couple days, sorry for taking a little while. For sequencing, should this be merged at bwa-mem3 version 0.16.0, or should we push version 0.17.0 up through modules first?

Thanks so much, much appreciated! Let me know how it goes and I will be very happy to add any suggestions.

I would love to upgrade to bwa mem3 0.17.0 but the builds fail to upload so I can't use these containers yet, I asked @nh13 to re-build/push and I can update this PR as soon as these images are out.

Actually they were just pushed but 0.17.0 do not carry any changes for TAPS, I think it's specific for EM/BS data.

nh13 · 2026-11-11T23:04:24Z

@eduard-watchmaker I think the CI runs completed for bioconda for 0.17.0. Let me know if you don't see them

nh13 · 2026-11-11T23:05:02Z

And I'll say, if you see any issues with bwa-mem3, I'd be glad to look into them and fix them!

eduard-watchmaker · 2026-11-11T23:06:58Z

Thanks so much @nh13 , I actually edited my message when I saw them published. For now the new version 0.17.0 doesnt really change for TAPS, but I will reach out if we need anything. So far, bwamem3 has been a great addition in every single aspect, so kudos to you (and the team involved) on this work!

Update Parabricks to 4.8.0-1 (with samtools 1.25 and Picard 3.5.0)

FelixKrueger

Main blocker is in the rastair 0.8 path: trims and BAMs reach RASTAIR_CALL unpaired, so samples can be called with each other's trims (inline). Also inline: RRBS dedup, non-directional libraries with bwamem3, .gz index prefixes and the CHANGELOG. Not in the diff: BWAMEM3_INDEX has no process label, so it gets 1 CPU and 4 h from conf/base.config, which won't build a human --meth index. The Parabricks/Picard bump and the nf-amazon fix would be easier to review as separate PRs.

This review was generated with Claude-assisted review and checked by me.

- bam_taps_conversion (local patch): join BAM, BAI and M-bias trims by sample before RASTAIR_MBIAS and RASTAIR_CALL, which took them as separate channels and so paired them by arrival order - --aligner bwamem3 skips deduplication with --rrbs, as bismark and bwameth do - --aligner bwamem3 errors on --pbat and --non_directional: bwa-mem3 --meth seeds read 1 as C->T and read 2 as G->A - --bwamem3_index: only .tar.gz goes to UNTAR, so a genome.fa.gz index prefix works - BWAMEM3_INDEX gets 12 CPUs and 24 h (the module has no resource label) - tests: rastair 0.8 call and methylKit output back in the snapshot, spike-in methylation checks for both callers, and a bwamem3 + --pbat error test - CHANGELOG: one sentence per new entry; --trim_OT/--trim_OB now take effect Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…dedness, publishing) - CollectHsMetrics: pass an empty GZI tuple instead of an empty channel, which made the targeted subworkflow's combine() emit nothing - fastq_align_dedup_bwamem (local patch): the CPU path now emits the BAM index and samtools stats, so --skip_deduplication gives TAPS calling its input and MultiQC its samtools sections - bam_taps_conversion (local patch): rastair 0.8 uses the reference as a value channel, as the rastair2 path does - --rrbs skips deduplication for bwamem too; bwamem publish selectors cover the GPU path, and PARABRICKS_FQ2BAM publishes only with --save_align_intermeds - --taps with bismark errors; bwamem3 errors on --zymo/--single_cell as well; bwameth and rastair 0.8 warn that they ignore non-directional options; rastair2 guesses read orientation for all of them - --methyl_kit with targeted sequencing (MethylDackel) errors; warn about options that have no effect in the run - bismark_hisat coverage2cytosine and targeted outputs go under bismark_hisat/ - docs: output.md bwa-mem/bwa-mem3 alignments, usage.md TAPS advice, schema help for --ignore_3prime_r1/r2; index_downloads uses the HISAT2 index Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use the American spelling, as rastair's own `rastair license` subcommand does, in the parameter name and the pipeline's text (review by heuermh). The nf-core meta.yml `licence` key is left as the schema requires. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep the plugin declared so S3 paths (the iGenomes default) resolve, without the 2.15.0 pin. This matches #642 line for line, which carries the CHANGELOG entry, so the two merge cleanly in either order. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… publishing - targeted sequencing: CollectHsMetrics now runs (coverage metrics and the MultiQC HsMetrics tables) - bwamem (CPU): samtools stats reach MultiQC - index_downloads: the HISAT2 scenario uses the pre-built index, so no genome preparation - bwamem TAPS GPU: no raw Parabricks BAMs in parabricks/ and no stray indexes in deduplicated/ (checked against a stub run with -profile gpu) Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ore lint nf-core lint fails any plugin without a pinned version (nextflow_config: "Plugin 'nf-amazon' does not have a pinned version"), and pinning a newer release would break other Nextflow versions and Seqera Platform. Like the nf-core template, declare no nf-amazon: Nextflow loads the matching release for S3 paths, as the AWS Batch validation runs with Nextflow 25.10.4 showed. Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nh13 · 2026-11-12T19:49:39Z

I’d suggest updating to 0.17.0 before merging: TAPS scoring is unchanged, but it includes performance and memory improvements. BS/EM-seq tests need revalidation because the default scoring changed to genomic; --meth-scoring collapsed retains the previous scoring.

eduard-watchmaker · 2026-11-12T20:30:05Z

@nh13 thanks, agreed. I've opened nf-core/modules#13715 to bump bwamem3/index and bwamem3/mem to 0.17.0. Once it merges, I'll update both modules here and regenerate the bwa-mem3 snapshots before #638 is merged.

For bisulfite and EM-seq, methylseq will use the new genomic default rather than pinning --meth-scoring collapsed. No released methylseq version has bwa-mem3 yet, so there are no existing results to keep compatible with. I've already run the methylseq bwa-mem3 tests against the 0.17.0 modules:

TAPS output is unchanged.
On the small E. coli bisulfite test, mapped reads drop from 284 to 217 of 20,000, as the stricter mismatch penalty drops marginal hits.

🤖 Addressed by Claude Code

eduard-watchmaker and others added 8 commits November 9, 2026 13:19

tests: add bwamem3 supplied-index download scenario

7c68c36

Uses reference/Bwamem3_Index.tar.gz from nf-core/test-datasets (methylseq branch, added in nf-core/test-datasets#2370). Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README: add the bwa-mem3 workflow and rastair version 2

bc31225

Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eduard-watchmaker and others added 4 commits November 9, 2026 13:41

eduard-watchmaker marked this pull request as ready for review November 10, 2026 02:25

eduard-watchmaker requested a review from a team as a code owner November 10, 2026 02:25

eduard-watchmaker requested review from FelixKrueger, ewels and sateeshperi November 10, 2026 02:25

eduard-watchmaker self-assigned this Nov 10, 2026

eduard-watchmaker and others added 3 commits November 10, 2026 12:08

Container configs for RASTAIR2_MBIAS; docs and CHANGELOG for TAPS trims

2002c9f

Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eduard-watchmaker requested a review from heuermh November 10, 2026 19:37

This was referenced Nov 10, 2026

rastair2/mbias: new module; rastair2/call: take M-bias cutoffs nf-core/modules#13678

Merged

bam_taps_conversion: optional rastair version 2 caller (use_rastair2) nf-core/modules#13641

Merged

eduard-watchmaker mentioned this pull request Nov 10, 2026

Add bwa-mem3 index generation (plain and methylation --meth) nf-core/references#171

Open

Merge branch 'dev' of github.com:nf-core/methylseq into taps_v2

88348b9

# Conflicts: # CHANGELOG.md

eduard-watchmaker mentioned this pull request Nov 11, 2026

Update Parabricks to 4.8.0-1 (with samtools 1.25 and Picard 3.5.0) #641

Merged

5 tasks done

eduard-watchmaker and others added 3 commits November 11, 2026 14:28

Merge nf-core/tools 4.2.2 template update (#640) into taps_v2

bf1443a

# Conflicts: # ro-crate-metadata.json

Merge taps_v2 (nf-core/tools 4.2.2 template update) into parabricks-4.8

50139bd

test: order the bwamem TAPS GPU snapshot versions as nf-test does

4cb0ac8

Generated by Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

eduard-watchmaker requested a review from nh13 November 11, 2026 23:07

eduard-watchmaker added 2 commits November 11, 2026 16:21

Merge pull request #641 from nf-core/parabricks-4.8

7fdbdd4

Update Parabricks to 4.8.0-1 (with samtools 1.25 and Picard 3.5.0)

Merge branch 'dev' into taps_v2

3af96c2

FelixKrueger reviewed Nov 12, 2026

View reviewed changes

eduard-watchmaker and others added 2 commits November 12, 2026 08:25

heuermh requested changes Nov 12, 2026

View reviewed changes

Comment thread nextflow_schema.json Resolved

heuermh reviewed Nov 12, 2026

View reviewed changes

Comment thread nextflow.config Resolved

eduard-watchmaker and others added 3 commits November 12, 2026 10:10

FelixKrueger mentioned this pull request Nov 12, 2026

aligner: --bwamem3 5-Base engine and shared 5-Base fixes FelixKrueger/Bismark#1209

Merged

This was referenced Nov 12, 2026

feat(index): support an output prefix and a minimal methylation index fg-labs/bwa-mem3#791

Closed

docs(meth): make directional-library requirements explicit in help and quick start fg-labs/bwa-mem3#792

Closed

nh13 mentioned this pull request Nov 12, 2026

Interest in optional tools for faster processing and methylation QC? #643

Open

eduard-watchmaker mentioned this pull request Nov 12, 2026

bwamem3/index, bwamem3/mem: bump bwa-mem3 to 0.17.0 nf-core/modules#13715

Merged

17 tasks done

Conversation

eduard-watchmaker commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's new

Fixes picked up along the way

Fixes from review and a pipeline-wide audit

nf-core components

Testing

PR checklist

Uh oh!

github-actions Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

heuermh commented Nov 11, 2026

Uh oh!

eduard-watchmaker commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nh13 commented Nov 11, 2026

Uh oh!

nh13 commented Nov 11, 2026

Uh oh!

eduard-watchmaker commented Nov 11, 2026

Uh oh!

FelixKrueger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nh13 commented Nov 12, 2026

Uh oh!

eduard-watchmaker commented Nov 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eduard-watchmaker commented Nov 9, 2026 •

edited

Loading

github-actions Bot commented Nov 9, 2026 •

edited

Loading

github-actions Bot commented Nov 9, 2026 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

eduard-watchmaker commented Nov 11, 2026 •

edited

Loading