Skip to content

Add Padfoot annotation and ReConPlot figures for the SV/CNA callers - #202

Merged
ljwharbers merged 32 commits into
IntGenomicsLab:devfrom
Tim-Yu:feat/padfoot-reconplot-on-savana-pr189
Oct 7, 2026
Merged

ljwharbers merged 32 commits into
IntGenomicsLab:devfrom
Tim-Yu:feat/padfoot-reconplot-on-savana-pr189

Conversation

@Tim-Yu

@Tim-Yu Tim-Yu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds two downstream steps for the SV and copy-number callers already in the pipeline, as discussed in #192:

  • Padfoot annotates somatic SVs and CNAs (gene and exon overlap per breakend, repeat context, complex-SV grouping, per-gene impact) once per caller pair with output for a sample: Severus + the top-ranked Wakhan solution, and SAVANA SVs + SAVANA absolute copy number. Paired and tumour-only. Output under <sample>/padfoot/{severus_wakhan,savana}/.
  • ReConPlot draws rearrangement and copy-number figures per CN/SV caller pair: ASCAT + Severus, Wakhan + Severus, and SAVANA on its own, with per-chromosome figures, a genome-wide strip, an optional region-focus panel (gene labels, BAF track) and the harmonised CN/SV tables. Output under <sample>/reconplot/{severus_ascat,severus_wakhan,savana}/.

Both run by default and are switched off with --skip_padfoot / --skip_reconplot. PAIRED_SAVANA and TUMORONLY_SAVANA gain three emits (somatic BEDPE, fitted purity/ploidy, het-SNP allele counts) that the two steps consume; no other change to the SAVANA stack from #189.

Design notes

  • Sources. Neither tool is on bioconda. Padfoot is fetched once per run from a pinned commit of Tim-Yu/Padfoot, a fork of KolmogorovLab/Padfoot that adds SAVANA input (I will propose it upstream; --padfoot_dir takes a local checkout). The ReConPlot wrapper (run_reconplot.R + parsers for ASCAT, Wakhan, Severus and SAVANA) ships in assets/reconplot/; the ReConPlot R package is fetched from a pinned upstream commit because -profile conda installs it at run time.
  • Containers. ghcr.io/tim-yu/padfoot-repeatmasker (Padfoot deps + RepeatMasker 4.2.4 + Dfam 4.0) and ghcr.io/tim-yu/reconplot (R + ReConPlot), both digest-pinned in the module container directives so nf-core pipelines download stages them. Recipes under containers/. Under conda, RepeatMasker uses the curated Dfam subset bundled with bioconda.
  • Layout decisions. Caller-pair folders are <sv_caller>_<cn_caller> for both tools. SAVANA's own output stays where Add savana module #189 put it (variants/savana).
  • Skipped samples, not failures. A sample missing one side of a pair (no Wakhan integer-CN VCF, no SAVANA fit, tumour-only for ASCAT) is dropped from that pair by the joins; the run continues.

Testing

  • nextflow lint: no new warnings (15 on dev, 15 here). pre-commit/prettier clean. nf-core pipelines lint: 0 failed.
  • tests/default.nf.test passes under Singularity with the snapshot unchanged (Padfoot and ReConPlot need whole-genome caller output, so -profile test and test_full skip them).
  • New -stub nf-tests for both subworkflows (tag small, so they run on every PR) check that each caller pair is formed for the right samples and that samples lacking one side are dropped. Module-level nf-tests for PADFOOT and RECONPLOT (stub).
  • Fresh-environment re-verification (UCL CS cluster, Singularity 3.8.5, Nextflow 25.10.7, this branch rebased onto dev @ 0cf00bf, 2026-09-23): nextflow lint identical to dev; nf-test --tag small --profile +singularity 39/39; the two modules driven through their real subworkflows on whole-genome Severus/Wakhan/ASCAT/SAVANA output of a 40x ONT pair reproduce the earlier run byte-for-byte (tables, gene lists, figure counts); chr22 paired + tumour-only end-to-end 75/75 tasks. A whole-genome paired run from raw uBAMs with every tool on (Padfoot, ReConPlot, SAVANA, Wakhan, ASCAT, VEP, modkit, report) is in progress. Hopufully finishing soon and will update.
  • Real data, Singularity: full pipeline on a chr22 subset of a paired ONT tumour/normal plus the same tumour as tumour-only (minimap2 -> Clair3/ClairS/ClairS-TO -> LongPhase -> Severus + SAVANA -> Padfoot -> ReConPlot), twice on this branch. On the merged tree SAVANA finds no acceptable purity/ploidy fit for either chr22 sample (it did for the tumour-only sample on the pre-merge ClairS-TO 0.4.2 calls), so these runs prove the skip path: the joins drop the samples and the pipeline completes. The positive path is proven on whole-genome caller output of a 40x ONT pair (Severus, Wakhan, ASCAT, SAVANA): Padfoot with RepeatMasker on both pairs, ReConPlot on all three pairs including the focus panel with gene labels and BAF track, using the vendored wrapper.

Known limitations / follow-ups

  • No real-data module tests for Padfoot and ReConPlot yet: they need small Severus/Wakhan/ASCAT/SAVANA fixtures. I can prepare a chr22 set for IntGenomicsLab/test-datasets if you want them.
  • The HTML report does not yet show Padfoot or ReConPlot output. After the full testing, I/We can work on adding those?
  • A whole-pipeline -stub-run currently fails before reaching these steps (the CRAMINO stubs lack *.arrow, the nf-core longphase/phase stub runs tabix on a file it never creates, and the WAKHAN and SEVERUS stubs create fewer or differently named files than they declare). Out of scope here; happy to send a separate PR.
  • PADFOOT_ANNOTATION and RECONPLOT_FIGURES fetch their sources (90 MB + 30 MB) at run start even when no caller pair forms for any sample; harmless, could be guarded on non-empty input.
  • Separate proposal to come for the SAVANA stack: deriving --chromosomes from the contigs list, and a few unexposed CNA knobs (which was implemented in my previous attempts of local savana module).
  • Fine tunning the resource label after the real data run.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

For sure generated with Claude Code and then audited by Boyu.

@Tim-Yu

Tim-Yu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

The whole-genome test has run through, so here is the update promised in the description.

Setup: fresh environment on a UCL CS cluster node (Singularity 3.8.5, Nextflow 25.10.7, local executor, work dir on node-local disk which I think is nvme ssd), one paired 40x ONT tumour/normal from raw uBAMs (126 + 113 GB), pipeline defaults with every tool on only --skip_signatures, run with singularity profile.

Outcome: Pipeline completed successfully, 69/69 tasks, no failures or retries, 11 h 32 m end to end. However, if run on SAN or slower disks time usage will differ a lot and will trigger retires. And it will get more complex when run using executor as HPC.

The two new steps RSS labels were assigned fine: Padfoot 14.2 GB RSS in 1.5 min on process_medium, ReConPlot 0.3 GB in 4-5 min on process_low.

Two things not about this PR that may interest you (I may raise them in the SAVANA follow-up issue): SAVANA_RUN peaked at 149 GB resident against its process_high 72 GB label and SAVANA_CNA is the long step (2 h 28 m at 12 threads); SEVERUS peaked at 49.6 GB against process_medium 36 GB. Both would be killed on a scheduler that enforces memory based on my tuning of my previous savana config. This will affect massively when run using HPC.

@Tim-Yu
Tim-Yu marked this pull request as ready for review September 23, 2026 15:47
Copilot AI lite review requested due to automatic review settings September 23, 2026 15:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings remain in ReConPlot handling and Padfoot pairing.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity · 2 Low severity

Open (5)
What changed in this PR

Adds default-on Padfoot annotation and ReConPlot visualization for SV/CNA caller pairs, with SAVANA output wiring, containers, parsers, configuration, documentation, and tests.

Changes:

  • Adds Padfoot and ReConPlot workflows and modules.
  • Extends paired and tumour-only SAVANA outputs.
  • Adds R parsers, containers, parameters, documentation, and nf-tests.

Review findings:

  • Critical: fix uncompressed VCF handling and preserve ReConPlot pipeline failures.
  • Moderate: handle missing SV mates, empty Padfoot inputs, and unconditional downloads.
  • Nits: correct changelog placeholders, container documentation, output documentation, and usage admonitions.
File Description
workflows/​lrsomatic.nf Integrates the new downstream workflows.
tests/​.nftignore Ignores generated figures and logs.
subworkflows/​local/​utils_nfcore_lrsomatic_pipeline/​main.nf Validates new parameters.
subworkflows/​local/​tumor_only/​tumoronly_savana.nf Emits additional SAVANA products.
subworkflows/​local/​tests/​reconplot_figures.nf.test Tests ReConPlot pairing.
subworkflows/​local/​tests/​padfoot_annotation.nf.test Tests Padfoot pairing.
subworkflows/​local/​reconplot_figures.nf Builds ReConPlot caller-pair inputs.
subworkflows/​local/​paired/​paired_savana.nf Emits additional paired SAVANA products.
subworkflows/​local/​padfoot_annotation.nf Builds Padfoot inputs and pairing logic.
README.md Documents new outputs and workflow stages.
nextflow.config Adds runtime parameters.
nextflow_schema.json Defines parameter schema entries.
modules/​local/​reconplot/​tests/​nextflow.config Configures module tests.
modules/​local/​reconplot/​tests/​main.nf.test.snap Stores ReConPlot snapshots.
modules/​local/​reconplot/​tests/​main.nf.test Tests ReConPlot I/O.
modules/​local/​reconplot/​meta.yml Documents the module interface.
modules/​local/​reconplot/​main.nf Runs ReConPlot.
modules/​local/​reconplot/​environment.yml Defines R dependencies.
modules/​local/​padfoot/​tests/​nextflow.config Configures module tests.
modules/​local/​padfoot/​tests/​main.nf.test.snap Stores Padfoot snapshots.
modules/​local/​padfoot/​tests/​main.nf.test Tests Padfoot I/O.
modules/​local/​padfoot/​meta.yml Documents the module interface.
modules/​local/​padfoot/​main.nf Runs Padfoot annotation.
modules/​local/​padfoot/​environment.yml Defines Padfoot dependencies.
docs/​usage.md Documents parameters and usage.
docs/​output.md Documents generated outputs.
containers/​reconplot/​README.md Documents ReConPlot image builds.
containers/​reconplot/​Dockerfile Builds the ReConPlot image.
containers/​padfoot/​README.md Documents Padfoot image builds.
containers/​padfoot/​Dockerfile Builds the Padfoot image.
conf/​test.config Disables unsupported full-genome steps in tests.
conf/​test_full.config Disables unsupported full-genome steps in full tests.
conf/​modules.config Configures arguments and publishing.
CITATIONS.md Adds tool and dependency citations.
CHANGELOG.md Records the new functionality.
assets/​reconplot/​VERSION Pins the wrapper revision.
assets/​reconplot/​run_reconplot.R Provides the plotting CLI.
assets/​reconplot/​README.md Documents the vendored wrapper.
assets/​reconplot/​R/​utils_vcf.R Provides VCF helpers.
assets/​reconplot/​R/​utils_common.R Provides shared validation utilities.
assets/​reconplot/​R/​plotting.R Renders and saves figures.
assets/​reconplot/​R/​parsers/​wakhan.R Parses Wakhan CN output.
assets/​reconplot/​R/​parsers/​severus.R Parses Severus SV output.
assets/​reconplot/​R/​parsers/​savana.R Parses SAVANA output.
assets/​reconplot/​R/​parsers/​registry.R Registers parsers.
assets/​reconplot/​R/​parsers/​lrsomatic.R Provides the lrsomatic parser shortcut.
assets/​reconplot/​R/​parsers/​generic.R Provides generic table parsing.
assets/​reconplot/​R/​parsers/​ascat.R Parses ASCAT CN output.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread assets/reconplot/R/utils_vcf.R
Comment thread modules/local/reconplot/main.nf Outdated
Comment thread subworkflows/local/padfoot_annotation.nf Outdated
Comment thread CHANGELOG.md Outdated
Comment thread docs/usage.md Outdated
Tim-Yu added a commit to Tim-Yu/lr_somatic that referenced this pull request Sep 23, 2026

@ljwharbers ljwharbers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The pairing logic is carefully done. I checked the wiring against dev @ 0cf00bf:

  • SEVERUS, ASCAT and WAKHAN use the same meta subMap for paired and tumour-only samples, and SAVANA only joins with itself, so the joins hold.
  • A sample missing one side of a pair is dropped rather than failing the run.
  • pipefail is set, so the Rscript | tee pipes fail when Rscript fails.
  • Empty SV sets pass through validate_sv.
  • Moving the wiring into subworkflows is the right fix for the JVM's 64 KB method-size limit.

CI is green. Before merge, a few things need attention.

Should fix before merge

  1. Every default run now needs internet at start-up. PADFOOT_WGET and RECONPLOT_PKG_WGET download about 120 MB from GitHub at run start whenever the steps are enabled, and they are enabled by default. This happens even when no caller pair forms. nf-core pipelines download does not stage these tarballs, so on compute nodes without internet an upgrade breaks runs for users who never asked for these tools.

    • The ReConPlot package is already built into the container, so the download is only needed under -profile conda. Please guard it.
    • Padfoot already runs in a custom image. Could it be built into that image at the pinned commit? That removes the WGET, and --padfoot_dir stays for conda and development. It also means runs no longer depend on a personal fork being reachable at run time.
    • Related question for the maintainers: should skip_padfoot / skip_reconplot default to true until real-data fixtures exist?
  2. docs/output.md says ASCAT does not run on tumour-only samples, but it does. On dev, ASCAT runs on tumour-only samples because ClairS-TO Verdict uses its output (ascat_ch mixes in branched_minimap.tumor_only). So reconplot/severus_ascat/ is produced for tumour-only samples too. That path is not covered by any test: the stub test only gives ASCAT output to paired sample1. Please fix the sentence and add a tumour-only ASCAT + Severus sample to reconplot_figures.nf.test.

Strongly suggested

  1. The stub tests count tasks but never check which files were staged. The comments say solution_1's integer VCF and HP BEDs "must be picked", but the assertions only count tasks per sample. A regression that stages solution_2, or the subclonal BEDs, would still pass. One option: have the stubs write their staged inputs, e.g. ls cn_input sv_input > ${prefix}/staged.txt, and assert on that file. Same for PADFOOT.

  2. A crash in either step fails the whole run. Both steps inherit errorStrategy 'finish', so a failure in an optional figure or annotation step at the end of a ~12 h whole-genome run marks the entire run as failed. Two known ways to trigger it:

    • sanitize_regions stops when --reconplot_regions names a contig that has no CN.
    • run_reconplot.R exits 1 when it writes no files.

    Worth a deliberate decision. We have been bitten by blanket 'ignore' hiding failures in CI before, so an explicit warn-and-continue may be the better option.

  3. The Padfoot genome gate is written twice (see the inline comment).

Nits

  • containers/reconplot/Dockerfile: a comment still refers to the removed params.reconplot_url.
  • run_reconplot.R: the help for --baf-track says "SAVANA only", but the ASCAT parser supports it and the pipeline passes it for ASCAT.
  • nextflow.config: the comment says "containers ship full Dfam 4.0", but the image has only the root and curated-consensus partitions. The container README is accurate.
  • CHM13 is a reference we run routinely, and Padfoot skips it with only a warning. A short note in docs/usage.md on supplying --padfoot_gff / --padfoot_rm for CHM13 would help.
  • Out of scope, but thanks for the numbers: SAVANA_RUN peaked at 149 GB against process_high's 72 GB, and SEVERUS at 49.6 GB against process_medium's 36 GB. Worth a separate issue.

padfoot_src = channel.value([[id: 'padfoot'], file(params.padfoot_dir, type: 'dir', checkIfExists: true)])
}
else {
PADFOOT_WGET( channel.value([[id: 'padfoot'], params.padfoot_url]) )

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs on every default run, even when no caller pair forms, and needs GitHub access at start-up. nf-core pipelines download won't stage it. Could Padfoot be built into the (already custom) image at this commit, with --padfoot_dir kept for conda and development?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished, wget is removed and all embedded in the image. One step towards the offline ready, I hope..

Comment thread subworkflows/local/reconplot_figures.nf Outdated
reconplot_pkg = channel.value([[id: 'reconplot_pkg'], file(params.reconplot_pkg_dir, type: 'dir', checkIfExists: true)])
}
else {
RECONPLOT_PKG_WGET( channel.value([[id: 'reconplot_pkg'], params.reconplot_pkg_url]) )

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The container already has ReConPlot installed, so this download is only needed under conda. Suggest guarding it on the conda profile or on !workflow.containerEngine and passing [] otherwise. The module's library(ReConPlot) check already copes with the package being absent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Luuk, I’m back! Two quick questions: 1. Regarding the recent PRs (like #196 dropping the conda directive and CI no longer testing for conda), is conda becoming no longer a supported profile for the pipeline as a whole? 2 . am I right to assume lrsomatic is moving toward being fully offline-ready (no run-time downloads under containers)?
Thanks!

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Yep, we we havent prioritized conda as an option for a while already since one of the earlier tools didn't allow for it. So if a tool doesn't work with conda, don't worry about it. As long as it works with singularity and docker its fine!
Same goes for run-time downloads, ideally we limit this but I do not think it is fully offline-ready, we for example download clair3 models at runtime (although you can specify custom ones).

Comment thread docs/output.md Outdated
└── multiqc
```

The `padfoot` and `reconplot` directories are only present when the corresponding step is enabled (`--skip_padfoot`, `--skip_reconplot`); SAVANA's own output lives under `variants/savana`. Within them, each caller-pair subdirectory requires both of its callers to have produced output for that sample: `severus_wakhan` needs `--skip_wakhan false`, `severus_ascat` needs `--skip_ascat false` and a matched normal (ASCAT is not run for tumour-only samples), and the `savana` subdirectories additionally need SAVANA copy number, which is only produced when an SNP source is available (the phased germline VCF for paired samples, or the bundled 1000G panel for tumour-only samples) and SAVANA finds an acceptable purity/ploidy fit.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On dev, ASCAT does run on tumour-only samples (Verdict uses its output), so severus_ascat will appear for them too. Please update this sentence and cover that case in the stub test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, also added the corresponding tests. All passed.

Comment thread workflows/lrsomatic.nf Outdated
//
def padfoot_genome = params.padfoot_genome ?:
(params.genome == 'GRCh38' ? 'hg38' : params.genome == 'CHM13' ? 'chm13' : null)
def padfoot_annot_ok = padfoot_genome && ((padfoot_genome in ['hg38', 'mm10']) || (params.padfoot_gff && params.padfoot_rm))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same logic lives in validateSvAnnotationParams(), with a comment saying the two must mirror each other. Could it be one helper in utils_nfcore_lrsomatic_pipeline that both places call, so they can't drift apart?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now all sourcing from the utils file

Tim-Yu added a commit to Tim-Yu/lr_somatic that referenced this pull request Sep 30, 2026
@Tim-Yu
Tim-Yu force-pushed the feat/padfoot-reconplot-on-savana-pr189 branch from 8049f36 to 3abf2f4 Compare September 30, 2026 08:10
@ljwharbers

Copy link
Copy Markdown
Collaborator

@claude review


tag "modules"
tag "modules_local"
tag "padfoot"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new module test has no small tag, so PR CI (.github/workflows/nf-test.yml picks tests by small) never runs it. The PR description says these module tests exist and run, but CI skips them. Please add tag "small" like the two subworkflow tests have.


tag "modules"
tag "modules_local"
tag "reconplot"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same problem as the PADFOOT module test: there is no small tag, so PR CI never runs this test. Please add tag "small".

Comment thread modules/local/padfoot/main.nf Outdated
// dependencies (guard in `script:`). Built from containers/padfoot/Dockerfile: Padfoot + RepeatMasker 4.2.4 with the
// Dfam 4.0 root and curated-consensus partitions. Padfoot update = new commit in the Dockerfile, rebuild, re-pin the digest.
// Override per site with `process { withName: '.*:PADFOOT_(SEVERUS_WAKHAN|SAVANA)' { container = ... } }`.
container "ghcr.io/tim-yu/padfoot-repeatmasker@sha256:bf134e2ffca5ae5f0b0c85c7f9d189ab0a886c0ba389c5ff7f1d39a39bd58467"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This container directive breaks the pipeline convention in REVIEW.md:

  • It is one ghcr.io reference, without the workflow.containerEngine == 'singularity' ... ternary used across modules/.
  • Custom images should be an oras://docker.io/...-sif / docker.io/... pair.
  • REVIEW.md asks for Docker Hub, not ghcr, for large Apptainer images. This one bundles RepeatMasker plus the Dfam 4.0 root and curated partitions, so it is large.

As written, Singularity users convert the ghcr Docker image at run time instead of pulling a prebuilt SIF. Please publish to Docker Hub with a matching -sif tag and use the standard ternary. Keep both branches on the same Padfoot commit.

Comment thread modules/local/reconplot/main.nf Outdated
// No conda: the image ships the ReConPlot R package itself (guard in `script:`); the wrapper is pipeline glue in
// assets/reconplot/. Built from containers/reconplot/Dockerfile. Package update = new commit there, rebuild, re-pin the digest.
// Override per site with `process { withName: '.*:RECONPLOT_(SEVERUS_ASCAT|SEVERUS_WAKHAN|SAVANA)' { container = ... } }`.
container "ghcr.io/tim-yu/reconplot@sha256:1145fc5aebe0227bec371f4c59b08b9a09871498e403c01b83f83973149ae9e7"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same container problem as PADFOOT: this is one ghcr.io reference with no singularity/docker ternary. The pipeline convention for custom images is an oras://docker.io/...-sif / docker.io/... pair on Docker Hub, with both branches on the same ReConPlot commit.

Comment thread conf/test.config
skip_ascat = true
skip_modkit = true
// Padfoot/ReConPlot need whole-genome caller output; covered by the module- and subworkflow-level nf-tests
skip_padfoot = true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Padfoot and ReConPlot skipped here and in test_full, the only small tests for this feature are -stub runs. So no CI test runs:

  • the real script: blocks of either module;
  • the ~1,800 lines of R under assets/reconplot/ (parsers for ASCAT, Wakhan, Severus and SAVANA, plotting, region parsing);
  • Padfoot itself.

A break in the wrapper or in the Padfoot CLI arguments (--specie, --skip_RepeatMasker, --cna-caller savana) would still pass CI. Under REVIEW.md, please run the Slurm test array on real Severus/Wakhan/ASCAT/SAVANA output before merge. Ideally, also add the chr22 fixtures to test-datasets so a non-stub small test can cover at least one pair.

Comment thread docs/usage.md Outdated
- `severus_wakhan/` -- Wakhan top-ranked solution CN + Severus somatic SVs
- `savana/` -- SAVANA absolute CN + SAVANA classified somatic SVs

Each pair produces `per_chromosome/` (one figure per chromosome), `genome_wide/` (all chromosomes in one strip), the harmonised CN/SV tables, and, when `--reconplot_regions` is set, a `focus/` multi-panel figure with optional gene labels and BAF track. Both the wrapper and the ReConPlot R package (neither on conda) are staged as source from GitHub (or local checkouts for offline systems); the default container ships the package pre-installed, while `-profile conda` installs it at run time.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence contradicts the code and the rest of this section:

  • It says the wrapper and the ReConPlot package are "staged as source from GitHub (or local checkouts for offline systems)". In fact the wrapper ships in assets/reconplot/ and the package is inside the image (see line 409 and line 439: "Neither step downloads anything at run time").
  • It says "-profile conda installs it at run time". In fact RECONPLOT raises error under conda/mamba (modules/local/reconplot/main.nf:31-33).

It looks left over from an earlier design. Please remove it or correct it. The PR description has the same leftovers: --padfoot_dir and "fetch their sources (90 MB + 30 MB) at run start".

Comment thread docs/output.md
│ │ │ └── samtools
│ │ └── whatshap_stats
│ ├── reconplot
│ │ ├── severus_wakhan

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tree is for tumour-only samples, and it leaves out reconplot/severus_ascat. ASCAT runs for tumour-only samples and RECONPLOT_SEVERUS_ASCAT forms for them: the subworkflow test asserts sample3, and line 112 of this file says so. Please add severus_ascat here.

Comment thread nextflow_schema.json
"description": "Padfoot genome preset (`hg38`, `chm13`, `mm10`). Default: inferred from `--genome`.",
"help_text": "GRCh38 -> hg38, CHM13 -> chm13. Padfoot bundles gene and repeat annotations for hg38 and mm10 only; chm13 and other presets need `--padfoot_gff` and `--padfoot_rm`, otherwise Padfoot is skipped with a warning."
},
"padfoot_gff": {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEW.md asks for a help_text on every new parameter. Only padfoot_genome has one. Some key behaviour is only in docs/usage.md, so it is missing from --help and the nf-core launcher:

  • padfoot_gff and padfoot_rm are needed together with a non-hg38/mm10 preset, or Padfoot is skipped.
  • reconplot_regions is syntax-checked at start-up, and regions on contigs without copy number are skipped.
  • reconplot_baf_track applies only to ASCAT and SAVANA.
  • reconplot_genome falls back to hg38 for custom genomes.

Please add help_text for the other padfoot_* / reconplot_* and skip_padfoot / skip_reconplot params.

@claude

claude Bot commented Sep 30, 2026

Copy link
Copy Markdown

Automated review: 8 inline findings.

  • Tests: the new PADFOOT and RECONPLOT module tests have no small tag, so PR CI does not run them.
  • Containers: both module containers are single ghcr.io images, not the singularity/docker (oras://docker.io / docker.io) pair the pipeline uses.
  • Docs: docs/usage.md (download and conda behaviour) and docs/output.md (missing tumour-only severus_ascat) don't match the code, and most new params have no help_text in the schema.

The channel joins looked correct to me: Severus, ASCAT, Wakhan and SAVANA all use the same meta subMap, and the joins cannot pair records from different samples.

A human reviewer needs to run the real code. All small tests for this feature are -stub, and test/test_full skip both steps. CI never runs Padfoot, the real module scripts, or the ~1,800 lines of R in assets/reconplot/. Please run the Slurm test array (or a whole-genome run with Severus, Wakhan, ASCAT and SAVANA) with Padfoot and ReConPlot on, and check that padfoot/* and reconplot/* contain real, non-empty outputs for every pair. Also check a --genome CHM13 run: ReConPlot should use T2T, and Padfoot should warn and skip without --padfoot_gff/--padfoot_rm.

Tim-Yu added a commit to Tim-Yu/lr_somatic that referenced this pull request Sep 30, 2026
@Tim-Yu
Tim-Yu force-pushed the feat/padfoot-reconplot-on-savana-pr189 branch from 29e9321 to 1c5168d Compare September 30, 2026 11:11
@ljwharbers

Copy link
Copy Markdown
Collaborator

@claude review

Comment thread modules/local/padfoot/main.nf Outdated
Comment on lines +9 to +11
container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container
? 'oras://ghcr.io/tim-yu/padfoot-repeatmasker-sif:4.2.4-dfam4-padfoot-e7eaed7'
: 'ghcr.io/tim-yu/padfoot-repeatmasker:4.2.4-dfam4-padfoot-e7eaed7'}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Container registry does not follow the repo convention. REVIEW.md asks custom images to use the oras://docker.io/...-sif / docker.io/... pair, i.e. Docker Hub rather than ghcr, for large Apptainer images. This image ships RepeatMasker plus the Dfam 4.0 root and curated-consensus partitions, so it is exactly the large Apptainer case that rule is about. modules/local/reconplot/main.nf:8-10 (ghcr.io/tim-yu/reconplot) has the same problem.

The PR description also says both images are "digest-pinned in the module container directives". They are pinned only by mutable tags (4.2.4-dfam4-padfoot-e7eaed7, 0.2-r4.4).

Please push both images to Docker Hub (ideally under the lab org rather than a personal namespace), point these directives at them, and update the description, or pin by @sha256: digest if digest pinning is intended.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let me know if eventually all the images need to be uploaded to Docker hub. I will then upload there.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We noticed that ghcr hosted images, if they're too big, they very often (unexpectedly) time out, without any good fix on the pipeline side. That's why most of our larger images are hosted on dockerhub now. So ideally, yes, on dockerhub would be better

Comment thread conf/test.config
Comment on lines +73 to +75
// Padfoot/ReConPlot need whole-genome caller output; covered by the module- and subworkflow-level nf-tests
skip_padfoot = true
skip_reconplot = true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real script: path of both new modules runs in no CI test. test and test_full skip Padfoot and ReConPlot. Every new small test (both module tests, both subworkflow tests) uses -stub, so CI checks only the channel pairing and file staging.

None of the following ever runs on a PR:

  • the Padfoot command line (--sv-caller/--cna-caller savana from the fork, --skip_RepeatMasker, --specie mouse for mm10, the RepeatMasker/Dfam setup in the image);
  • the three run_reconplot.R calls, including the Wakhan mv cn_input/*.bed cn_input/solution_1/bed_output/ re-layout and the mv ${prefix}/per_chromosome/*.reconplot_{cn,sv}.tsv step;
  • the roughly 2,000 lines of vendored R under assets/reconplot/.

A break in any of these would still show green CI. Per REVIEW.md, please run the Slurm test array or a whole-genome run with Padfoot and ReConPlot on, and attach the results. Better still, add a small real-data (non-stub) module test once the chr22 fixtures offered in the PR description exist in IntGenomicsLab/test-datasets.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working on adding the testing module. And spotted a small bug on Padfoot not correctly annotating some TE event, fixing now.

@claude

claude Bot commented Sep 30, 2026

Copy link
Copy Markdown

Automated review: 2 findings (inline).

  1. Both new containers are on ghcr.io, but REVIEW.md asks for Docker Hub for large Apptainer images. They are also pinned by tag, not by digest as the description says.
  2. Every new test is -stub, and test/test_full skip both steps. So no CI test runs the real Padfoot or ReConPlot script: blocks or the vendored R wrapper.

A human reviewer needs to check (I cannot run these):

  • Run the Slurm test array, or a whole-genome run with Padfoot and ReConPlot enabled, to confirm the real command lines and R wrapper work. This includes the Wakhan solution_1 re-layout, the SAVANA single-source mode and the focus/BAF panel.
  • If possible, check CHM13 (ReConPlot T2T preset; Padfoot is skipped unless --padfoot_gff/--padfoot_rm are given).
  • The PR description is partly out of date compared with the code: it mentions --padfoot_dir, sources fetched at run start (the 90 MB + 30 MB item under known limitations), RepeatMasker under conda, and digest pinning. The modules now ship everything in the container and fail under -profile conda. Please update the description.

The channel pairing and joins (meta keys, SAVANA remainder join, Wakhan solution_1 selection), the modules.config entries, the schema/usage docs and the small tags look consistent.

Tim-Yu and others added 20 commits October 2, 2026 20:08
…run-time downloads

Every default run fetched Padfoot (90 MB, from a personal fork) and the ReConPlot R
package (30 MB) from GitHub at start-up, even when no caller pair formed, and
`nf-core pipelines download` could not stage them. Following IntGenomicsLab#196: Padfoot is now
installed into `ghcr.io/tim-yu/padfoot-repeatmasker` at a pinned, checksum-verified
commit (`/opt/padfoot`), the ReConPlot package was already in `ghcr.io/tim-yu/reconplot`,
and both modules drop `conda` with the same guard as `LRSOMATICREPORT`.

- `PADFOOT_WGET`/`PADFOOT_UNTAR`, `RECONPLOT_PKG_WGET`/`RECONPLOT_PKG_UNTAR` and the
  params `padfoot_url`, `padfoot_dir`, `reconplot_pkg_url`, `reconplot_pkg_dir` are removed;
  the module inputs for the staged sources go with them.
- The environment.yml files move under `containers/`, where the Dockerfiles use them.
- Docs: Padfoot/ReConPlot options, the offline paragraph, a CHM13 note, both container
  READMEs; the Dfam wording says root + curated-consensus partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s for tumour-only samples

The subworkflow stub tests only counted tasks per sample, so a regression that
staged Wakhan's solution_2 or the subclonal BEDs would still pass. Both stubs now
list their staged inputs in `<prefix>/staged.txt` (declared as a stub-only optional
output, published like the other files) and the tests assert on it: the top-ranked
solution's integer-CN VCF and HP BEDs are picked, nothing from solution_2 or the
subclonal set is staged, and SAVANA samples carry their allele counts only when they
have them.

ASCAT runs for tumour-only samples on dev (Verdict uses it), so `reconplot/severus_ascat`
forms for them too: `docs/output.md` said the opposite; fixed, and the ReConPlot stub
test gains a tumour-only sample with a tumour-only ASCAT file set (no normal BAF).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e --reconplot_regions at start-up

A crash in ReConPlot at the end of a whole-genome run failed the whole run through
the inherited `errorStrategy 'finish'`. Two ways to trigger it were known:
`sanitize_regions()` stopped when every requested region lacked copy-number data,
and `run_reconplot.R` exited 1 when it wrote no file. Both are now warnings with
exit 0, so an optional focus panel can be absent without failing the task; the
required per-chromosome and genome-wide outputs still fail the task when a sample
has nothing plottable, which is a real problem and should stop the run. No blanket
`errorStrategy 'ignore'`.

`--reconplot_regions` is now checked at start-up in `validateSvAnnotationParams()`
with the wrapper's grammar (`chr` or `chr:start-end`, start < end), with function
tests. Also: the `--baf-track` help no longer says SAVANA only (ASCAT supports it
and the pipeline passes it for ASCAT).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…validator

The same preset/annotation logic lived in `workflows/lrsomatic.nf` and in
`validateSvAnnotationParams()`, with a comment asking that the two be kept in sync.
`padfootGenome()` and `padfootAnnotationOk()` in `utils_nfcore_lrsomatic_pipeline`
are now the single source, with function tests for the three presets and the
gff/rm requirement.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the option tables

The REVEL licence notice was still on one line with its `[!IMPORTANT]` marker (the
first of the two collapsed blocks was fixed in e793b61), so GitHub did not render it
as an alert. Tables realigned with the prettier build pre-commit installs (3.9.6).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
nf-core lint's template_strings check flags `{{index .RepoDigests 0}}` in the
`docker inspect` example as an unrendered template; `docker images --digests`
shows the same digest without braces.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…_text to every new parameter, fix two doc leftovers

- The PADFOOT and RECONPLOT module tests now carry the `small` tag that
  `.github/workflows/nf-test.yml` selects on, so PR CI runs them.
- Every `padfoot_*`, `reconplot_*`, `skip_padfoot` and `skip_reconplot` parameter has a
  `help_text` describing the behaviour that was only in docs/usage.md (gff+rm needed
  together for non-hg38/mm10 presets, region syntax check and skipping, BAF track
  applicability, hg38 fallback, conda not supported).
- docs/usage.md: removed the sentence that still described the wrapper and package as
  staged from GitHub and installed under conda.
- docs/output.md: the tumour-only tree lists `reconplot/severus_ascat`, which forms for
  tumour-only samples too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…T with prebuilt SIF twins

Both modules referenced a single `ghcr.io/...@sha256:` digest, so under Singularity/Apptainer the
Docker image was converted at first pull, unlike every other custom image in the pipeline (REVIEW.md,
"Containers"; automated review findings 3-4). They now use the same ternary as LRSOMATICREPORT and
SIGPROFILER, with both branches pinned to one tag:

- `oras://ghcr.io/tim-yu/padfoot-repeatmasker-sif:4.2.4-dfam4-padfoot-e7eaed7` /
  `ghcr.io/tim-yu/padfoot-repeatmasker:4.2.4-dfam4-padfoot-e7eaed7`
- `oras://ghcr.io/tim-yu/reconplot-sif:0.2-r4.4` / `ghcr.io/tim-yu/reconplot:0.2-r4.4`

The SIFs were built from the published digests (bf134e2f, 1145fc5a) and pushed with ORAS. The
container READMEs describe the build + push of both images, and the docs, CHANGELOG and the
Dockerfile comment now say "bump the tags" instead of "re-pin the digest".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The test profiles produce no copy-number caller output, so no caller pair forms and the two new
modules ran only as stubs in CI. tests/fixtures/padfoot_reconplot/ adds invented Severus, Wakhan,
ASCAT and SAVANA outputs in each caller's exact file layout on a 2 Mb slice of GRCh38 chr19 (a
deletion over STK11 exons with a matching one-copy loss, a tandem duplication with a matching
gain, a templated and an Alu insertion, two breakend pairs; regenerated by make_fixtures.py).

Five `small` tests run the real script blocks on them: Padfoot on Severus + Wakhan with
RepeatMasker on and on SAVANA + SAVANA, ReConPlot on ASCAT + Severus, Wakhan + Severus (the
solution_1 re-layout) and SAVANA single-source with the BAF track. They assert on gene hits,
impact classes, strand notation and figure names, and snapshot the stable tables and versions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, consistent filters

Found by an in-depth review of the vendored wrapper on the real P215003155 output and synthetic
fixtures:

- `--write-tables` wrote coordinates and lengths in scientific notation (`1.2e+08`, `5e+05`);
  fwrite now uses scipen = 50.
- An SV with an undetermined orientation (Severus record without STRANDS and a type other than
  INS/DEL/DUP, SAVANA notation outside the four symbols) aborted the run in validate_sv instead of
  being dropped with the others.
- Mate records are collapsed on the caller's mate ID when it gives one, with the positional key
  (now including the orientation) as the fallback; insertions are never collapsed. Mates 1 bp
  apart no longer draw twice, a ++ and a -- junction on the same breakpoints no longer merge.
- `--min-svlen` now applies to SAVANA as well, so the three RECONPLOT tasks of a sample filter alike.
- `--genes` symbols are resolved against ReConPlot's gene table first: a symbol with several rows
  made the package error and lose the whole focus panel; unknown and duplicated symbols are now
  skipped with a warning.
- A failing BAF-track reader (e.g. a het-SNP BED with fewer than 12 columns) skips the track
  instead of failing the task; `--genome` is validated up front; thousands separators are accepted
  in `--regions`; regions with end <= start are dropped, duplicates collapsed; the Wakhan BED
  header is found with or without its comment block; `--source X --cn-input DIR` without `--input`
  gets a clear error.

Cosmetic: CN tables sort chr1, chr2, ... and an empty SV table keeps the parser's column order.
VERSION records the local changes on top of the vendored commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…idation, skip under conda

- `--specie mouse` is ambiguous for RepeatMasker in the shipped Dfam 4.0, so every mm10 run with
  RepeatMasker on failed; pass "Mus musculus".
- `--format` and `--genes` were interpolated unquoted into the ReConPlot command; a value with a
  space killed the task after the required figures had been written.
- validateSvAnnotationParams() accepts thousands separators in --reconplot_regions like the
  wrapper, no longer throws a stack trace on an unparsable number, and rejects --reconplot_genes
  with whitespace and --reconplot_format outside pdf/png (also a schema pattern), which would
  otherwise fail every task with "Missing output file(s)".
- Both steps ship only inside their containers: under -profile conda/mamba they are now skipped
  with a start-up warning instead of failing hours in.
- Annotation and figures are downstream of every caller, so a sample they cannot process must not
  stop a cohort: errorStrategy 'ignore' for PADFOOT_* and RECONPLOT_*.
- Tests: the chm13 inference is asserted on its own; the ReConPlot stub tests stage the real
  wrapper directory.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t genome-wide clipping, label copy number against the tumour ploidy

Images (requested in review: large Apptainer images time out on ghcr):
- docker.io/timmy9527/padfoot-repeatmasker:4.2.4-dfam4-padfoot-25fd6d4 and
  docker.io/timmy9527/reconplot:0.2-r4.4-clipfix, each with an oras:// SIF twin, both branches of
  the ternary on the same tag.
- Padfoot 25fd6d4 bundles the cancer-gene table it used to download on every run (the "nothing is
  downloaded at run time" claim is now true), pads unlisted Wakhan segments instead of letting genes
  inherit a neighbour's copy number, keeps the RepeatMasker hits of the last insertion, and fixes
  the bundled annotation's mislabelled last gene per chromosome.
- The ReConPlot Dockerfile pins the package tarball by checksum and patches its clipping step,
  which compared against the whole chr_selection vector and truncated chr1/chr2 in every genome-wide
  figure; the build fails if the two lines are not found. Dfam is fetched from the versioned
  release directory; both Dockerfiles declare the frontend `ADD --checksum` needs.

Ploidy: PADFOOT takes the CN caller's fitted purity/ploidy table (Wakhan solutions_ranks.tsv,
SAVANA *_fitted_purity_ploidy.tsv) and Padfoot labels each haplotype's integer copy number against
ploidy/2, rounding SAVANA's fractional values first. On P215003155 the SAVANA pair's by_gene.tsv
went from 17,209 to 30 genes with both haplotypes "deleted" and the Wakhan pair from 11,941 to 450;
the SV tables are unchanged. The subworkflow joins the fit tables with remainder: true, so a missing
table never drops a pair; the harness and all tests pass them; the real Padfoot test asserts the
RepeatMasker class and the ploidy-driven labels.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/usage.md: Docker Hub references, SIF sizes in the offline paragraph, how gene copy number is
labelled against the fitted ploidy. docs/output.md: the SV table's real columns, by_gene semantics.
README: the tumour-only tree gains reconplot/severus_ascat (ASCAT runs tumour-only). meta.yml of
both modules describes the stub-only `staged` output and the new ploidy input. Container READMEs
describe building and publishing both images on Docker Hub.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…abels and known-fusion detection

Two more Tim-Yu/Padfoot commits on top of 25fd6d4, both fixing behaviour inherited from upstream:

- f8c9e5d: the breakpoint-to-gene lookup compared two bisect positions on lists that are not
  both sorted, so on P215003155 it left 66 of 660 Severus + Wakhan and 44 of 560 SAVANA
  breakpoints without the gene they lie in (FGFR3, LETM1, SPPL2B, ...). The new containment search
  agrees with an independent brute-force lookup on every breakpoint. It also finishes relabelling
  the bundled annotations' misplaced last genes (FRG2B, MZF1, PRMT2, RABL2B).
- 3846215: flank breakpoints were labelled as introns and intron numbers kept only their last
  digit (132 of 256 labels wrong), which could turn a multi-exon deletion into `intronic`; the
  in-frame test used `//` instead of `%`; the known-fusion lookup never fired. On P215003155 both
  callers now report COL1A1::PDGFB as an oncogenic fusion.

Image docker.io/timmy9527/padfoot-repeatmasker:4.2.4-dfam4-padfoot-3846215 (+ -sif). The real
PADFOOT tests now pin the STK11 deletion as `between_exons` (intron 3 -> exon 10) and the
minus-strand SBNO2 flank as `downstream`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…thout a solution

Found in the 23-pair cohort run:

- ReConPlot adds one plot layer per SV, so on a chromosome with a few thousand SVs R runs out of
  its node stack ("node stack overflow"; the stack size is fixed at build time and disabling the
  JIT does not help). The wrapper already logged such a figure and went on, but genome_wide was a
  required output, so the task failed and its other figures were lost (P215002781: 44 drawn
  per-chromosome figures discarded). All figure outputs are now optional and the task fails only
  when no figure at all was drawn.
- ASCAT writes an empty segments.txt when it finds no solution (P215002180); the parser crashed in
  fread. RECONPLOT_FIGURES now skips the ASCAT pair for such a sample, and the wrapper stops with
  a clear message if it is run on one directly.

The subworkflow stub test gains a sample whose ASCAT found no solution; docs/output.md describes
both behaviours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dev moved WAKHAN to 0.5.0 (IntGenomicsLab#214), whose module no longer has `vcf_files` / `bed_files`: each
solution is one `solution_<ploidy>_<purity>_<conf>/` directory with a merged
`integer_profile.{bed,vcf}`, and `solution_rank_N` links rank them. With Wakhan on (the default)
the pipeline aborted at launch ("No such property: vcf_files" / "bed_files"); CI could not see it
because the test profiles skip Wakhan.

- Both subworkflows take `WAKHAN.out.solution_dirs` and use `solution_rank_1`: Padfoot gets its
  `integer_profile.vcf` (same CN1/CN2/COV1 format Padfoot reads), ReConPlot its
  `integer_profile.bed` plus `solutions_ranks.tsv`.
- The wrapper's Wakhan parser reads the merged profile (hp1_/hp2_copynumber_state; unphased
  copynumber_state too) and keeps the 0.4 layout as a fallback; the module's bed_output relayout
  is gone.
- PADFOOT_ANNOTATION: a fit table without a matching SV/CN pair arrives as a right-only row of
  the remainder join and crashed the 4-argument closure; both branches now filter on row[1].
- Fixtures regenerated in the 0.5.0 layout (`wakhan/solution_rank_1/integer_profile.{bed,vcf}`,
  `wakhan/solutions_ranks.tsv`), headers as Wakhan 0.5.0 writes them; the stub tests use ranked
  solution directories and cover the right-only rows (a sample with ranks but no rank-1 profile,
  a SAVANA fit table without copy number).
- tests/sv_annotation_compose.nf.test (`small`): `-preview` of the test profile with Wakhan,
  ASCAT, Padfoot and ReConPlot on, so a renamed or removed caller output fails CI at launch. It
  fails on the previous wiring with exactly the launch error.

Checked on a real Wakhan 0.5.0 run of P215003155 (the dev module on the September inputs): both
subworkflows run through the real solution_rank_1 link; the ReConPlot CN table and Padfoot's
gene table differ from the 0.4-based ones only by Wakhan's own calling changes (a chr8 haplotype
swap, 7 segment states on chr12/chr17).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…NA figures without SVs, check --padfoot_genome

From the independent review of the branch:

- errorStrategy 'ignore' on PADFOOT_* / RECONPLOT_* had replaced base.config's retry on resource
  kills (130-145, 104, 175-177); those are now retried first, then ignored.
- Padfoot passes the RepeatMasker species unquoted, so "Mus musculus" split into two arguments
  ("Ambiguous search term 'Mus'"); mm10 now uses mus_musculus, which the image resolves to taxid
  10090 (the Dockerfile smoke test uses the same term).
- SAVANA writes an empty classified.somatic.bedpe for a sample without somatic SVs; the wrapper
  stopped ("fewer than 7 columns") and the sample lost its SAVANA figures. An empty BEDPE is now
  an empty SV set.
- --padfoot_genome on another assembly than --genome (e.g. hg38 on CHM13) silently annotated T2T
  coordinates with hg38 genes; it is now an error at start-up, and the skip warning no longer
  suggests it. Two function tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…repeats, robust Wakhan input

Six Tim-Yu/Padfoot commits on top of 3846215, from the independent review and its follow-ups:

- --gff / --rm (the route for CHM13 and other genomes without bundled annotations) crashed three
  ways: relative paths after Padfoot's chdir, the RepeatMasker .out / BED parsing, and a gzip
  read/write mismatch. They now take a GENCODE-style GFF3 (plain or gzipped) and a RepeatMasker
  .out or a chr/start/end/class BED; on the fixture they reproduce the bundled hg38 annotation.
- External commands are quoted, so a species such as "Mus musculus" no longer splits.
- temp_bps.bed was not closed before bedtools read it, so breakpoint repeat classes were lost
  after the last flushed 8 KiB block: on P215003155, 28 Severus and 44 SAVANA SVs gain their
  Repeat_Anno, checked against an independent scan of the bundled repeat file; nothing else in
  the tables changes.
- A header-only Wakhan VCF (copy-neutral tumour) and a very low per-copy coverage no longer crash;
  contig lengths for the copy-number padding fall back to the reference .fai; the mm10 hint names
  mus_musculus; insertion alignments are written in a fixed order (they followed the hash seed).

Image docker.io/timmy9527/padfoot-repeatmasker:4.2.4-dfam4-padfoot-1748e84 (+ -sif). A new real
PADFOOT test runs the custom-annotation route with the chm13 preset on fixtures in
tests/fixtures/padfoot_reconplot/custom_annotation/; the docs and schema describe the accepted
formats and that --padfoot_genome must match --genome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de /home, non-fatal focus panel, cache and gate fixes

- PADFOOT stub: `printf '%s\n'` in the GString put a real newline into .command.sh, so the
  indented heredoc never closed and versions.yml was malformed; any -stub run with Padfoot failed
  in softwareVersionsToYAML. The stub snapshots recorded the broken file.
- Padfoot image 4.2.4-dfam4-padfoot-1748e84-r2 (same Padfoot commit): Dfam moved from
  /home/mambauser/dfam to /opt/conda/envs/padfoot/share/dfam. With /home bound into the container
  (site singularity.conf or runOptions) RepeatMasker failed and every Padfoot result was silently
  ignored; checked with -B <empty>:/home on both images. The image records PADFOOT_COMMIT, which
  versions.yml now reports (Padfoot's own version string never changes).
- ReConPlot: the optional focus panel no longer fails the task (a failure is logged and the
  per-chromosome and genome-wide figures are kept, as the docs say); an empty region list is no
  longer a crash in the wrapper and is rejected at start-up; --reconplot_genes is checked against
  a symbol pattern (a quote broke the command line) and both it and --reconplot_baf_track warn
  when set without --reconplot_regions.
- -resume: the wrapper directory is cached on its top level only, so a change under R/ reused stale
  figures; a digest of the wrapper files now goes into its meta map.
- Padfoot/ReConPlot are skipped (with a start-up warning) when no container engine is in use, not
  only under the conda/mamba profiles.
- Tests: two ReConPlot tests run the pipeline's own arguments with a focus panel (thousands
  separators, an unknown gene, the BAF track) and a failing focus panel; two validator tests.
  Docs: the Severus SV table columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…graph

The -preview tests only asserted that the run composed, which is also true when a gate silently
drops Padfoot or ReConPlot. They now write the workflow graph (-with-dag dag.mmd) and assert on its
process nodes: with Padfoot on, both PADFOOT_* and all three RECONPLOT_* steps and their callers must
be present; with --skip_padfoot, no PADFOOT_* step and all three RECONPLOT_* steps.

Checked against deliberately broken copies of the branch: the tests now fail when Padfoot's genome
gate or the container-engine gate is wrongly false and when --skip_padfoot is ignored (the old
tests passed the first of these), and still fail at launch on the pre-Wakhan-0.5.0 wiring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tim-Yu
Tim-Yu force-pushed the feat/padfoot-reconplot-on-savana-pr189 branch from 1c5168d to d7c7008 Compare October 5, 2026 22:44
…test

Re-align the fixture README table as prettier writes it, keep the Wakhan
fixture BED's empty last column (a trailing tab, as Wakhan writes it) out of
the whitespace trimmer, and exempt the -preview pipeline test from the
versions.yml snapshot rule: a preview run publishes nothing, so it asserts on
the workflow graph instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tim-Yu

Tim-Yu commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

thanks for the approval. The containers are now on Docker Hub. Before you merge, though, the head has changed a lot since you approved.

  1. The major reason is that after your approval, Update nf-core modules and local module tools #212 brought Wakhan 0.5.0 (Update Wakhan to 0.5.0 #214) into dev. Wakhan 0.5.0 replaces the vcf_files/bed_files emits with ranked solution_dirs. So that the module used those files need to change.
  2. And the previous mentioned 'Padfoot not correctly annotating' fixed on forked repo.
  3. Yes, now images are under docker hub
  4. added the required testing modules.

Thank you so much

SAVANA writes <id>.classified.somatic.bedpe only from its model classifier
(ONT). With --pb, `savana classify` and `savana to` run classify_pacbio,
which writes the classified somatic VCF alone. RECONPLOT_FIGURES joined on
the BEDPE, so every PacBio sample, paired or tumour-only, silently lost its
reconplot/savana figures (seen on the C8 PacBio pair).

- RECONPLOT_FIGURES takes the classified somatic VCF, which SAVANA writes on
  every platform and Padfoot already reads; the now-unused somatic_bedpe
  emits leave PAIRED_SAVANA / TUMORONLY_SAVANA.
- Wrapper: --sv-format defaults to auto (the BEDPE when staged, else the
  VCF). On a real ONT sample (BL1, 2,084 junctions) both give the same
  junctions and CN table; only an insertion's second position differs by
  the 1 bp SAVANA's BEDPE adds. VERSION bumped.
- New RECONPLOT module test for the VCF-only case.
- The PADFOOT_ANNOTATION / RECONPLOT_FIGURES stub tests call their local
  closures with .call(): Nextflow >= 26.04 strict syntax rejects mk(...)
  ("`mk` is not defined"), which failed the latest-everything shards
  (hidden by continue-on-error).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ljwharbers

Copy link
Copy Markdown
Collaborator

I pushed 5453919 to this branch. It fixes something a real-data check turned up: PacBio samples never got reconplot/savana/ figures.

Cause. SAVANA writes <id>.classified.somatic.bedpe only from its model classifier, which is the ONT path. With --pb, both savana classify and savana to run classify_pacbio (savana 1.3.8, classify.py). That function writes the classified somatic VCF alone. RECONPLOT_FIGURES joined on the BEDPE, so every PacBio sample, matched normal or tumour-only, was silently dropped. The stub tests didn't catch it because the stub always creates a BEDPE.

Change.

  • RECONPLOT_FIGURES now takes SAVANA's classified somatic VCF, which SAVANA writes on every platform and which Padfoot already reads. The unused somatic_bedpe emits are removed from PAIRED_SAVANA and TUMORONLY_SAVANA.
  • In the wrapper, --sv-format now defaults to auto: it reads the BEDPE when one is staged and the VCF otherwise. VERSION is bumped.
  • On a real ONT sample with both files (2,084 junctions), the VCF and BEDPE paths give the same junctions and an identical CN table. The only difference is the second position of each insertion, which is off by the 1 bp that SAVANA's BEDPE adds.
  • There is a new RECONPLOT module test for the VCF-only case.

Also in this commit. The PADFOOT_ANNOTATION and RECONPLOT_FIGURES stub tests now call their local closures with .call(). Nextflow ≥ 26.04 strict syntax rejects mk(...) with "mk is not defined". That failure already shows in this PR's latest-everything shards 1/7 and 5/7, but continue-on-error reports them as green.

Checked on Mindwell (Apptainer).

  • The two subworkflows ran on real whole-genome caller output: a PacBio matched pair on GRCh38 and an ONT tumour-only sample on CHM13. All 7 Padfoot/ReConPlot tasks completed, including the PacBio SAVANA figures.
  • The Padfoot/ReConPlot nf-tests passed 33/33 on Nextflow 26.04.3.
  • nf-core pipelines lint and pre-commit pass.

🤖 Generated with Claude Code

@ljwharbers

Copy link
Copy Markdown
Collaborator

@claude review

Comment thread workflows/lrsomatic.nf
// Padfoot and ReConPlot ship only inside their containers (validateSvAnnotationParams() warns under conda/mamba)
def sv_annot_container_ok = workflow.containerEngine != null

if (!params.skip_padfoot && padfoot_annot_ok && sv_annot_container_ok) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test coverage (REVIEW.md → Tests): no small test runs a task through this wiring. conf/test.config and conf/test_full.config skip Padfoot/ReConPlot, tests/sv_annotation_compose.nf.test uses -preview only (it builds the DAG and runs no tasks), and the subworkflow tests feed hand-made channels under -stub.

So these paths are never exercised with real upstream output:

  • the real WAKHAN.out.solution_dirs, where solution_rank_1 is a symlink to solution_<ploidy>_<purity>_<conf>/. The subworkflows call best.resolve('integer_profile.vcf'|'.bed').exists() on it on the head node, and if that resolves to false the pair is silently dropped rather than failing;
  • the real ASCAT.out.bafs / segments file names that the wrapper picks by name;
  • errorStrategy 'ignore' in conf/modules.config. A Padfoot/ReConPlot failure in a real run shows up as missing output, not as a red CI.

Please run the Slurm test array (whole-genome paired + tumour-only, GRCh38 and CHM13) with --skip_padfoot false --skip_reconplot false. Then confirm that <sample>/padfoot/{severus_wakhan,savana} and <sample>/reconplot/{severus_ascat,severus_wakhan,savana} contain real tables and figures, and that no task was ignored.

@claude

claude Bot commented Oct 7, 2026

Copy link
Copy Markdown

Automated review: 1 finding.

  • Test coverage: no small test runs Padfoot or ReConPlot on real upstream caller output. The test profiles skip both, the pipeline test is -preview only, and the subworkflow tests are -stub. Combined with errorStrategy 'ignore', a broken join or failed task would go unnoticed in CI. See the inline comment.

For a human reviewer to check by running the pipeline:

  • Run the Slurm test array (GRCh38 and CHM13, paired and tumour-only) with Padfoot and ReConPlot on. Confirm that every caller-pair folder under padfoot/ and reconplot/ has real outputs and that no PADFOOT_* or RECONPLOT_* task was ignored.
  • The module-level real-data tests (tag small) pull large images from Docker Hub. Check that they fit within the GitHub runner's disk and time limits.

Non-blocking notes:

  • The PR description is out of date relative to the code. It says the sources are fetched at run time, the images are on ghcr.io and digest-pinned, and conda is supported. The code ships both tools inside tag-pinned Docker Hub images and errors out under conda.
  • Both images live under a personal Docker Hub account (timmy9527). Consider moving them to a lab-owned namespace.

@ljwharbers
ljwharbers merged commit 99d00fc into IntGenomicsLab:dev Oct 7, 2026
20 checks passed
robert-a-forsyth added a commit that referenced this pull request Oct 12, 2026
Keeps both sides in every conflict: CoRAL/ecDNA alongside Padfoot,
ReConPlot (#202), the Severus whitelist (#215) and the VEPPLUGIN fix (#218).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants