Skip to content

About

Snakemake workflow, benchmark harness and analysis scripts reproducing the GSNetAct manuscript (companion to BMGLab/GSNetAct)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

GSNetAct benchmark and analysis pipeline

Companion analysis repository for the article GSNetAct: task-aware network weighting for single-cell gene-set activity scoring (Garan & Kaymaz, Current Genomics).

The GSNetAct tool itself lives at https://github.com/BMGLab/GSNetAct and on PyPI (pip install gsnetact). This repository holds the Snakemake workflow, the benchmark harness and the analysis scripts that produce the tables and figures of the article, including scripts/revision/, which contains the parameter sweep, the topology-versus-selection ablation, the held-out-donor comparison with published scoring methods, the gene-level weight audit and the differential-activity analyses.

Input data are not redistributed here: every cohort is public and cited in the article, and the MSigDB gene-set files are covered by the MSigDB licence. results/, work/ and data/ are therefore excluded; the workflow regenerates them.

Licensed CC BY-NC-SA 4.0, matching the GSNetAct package.


One reproducible Snakemake workflow that produces every table and figure for the GSNetAct manuscript.

It asks whether network-weighted gene-set activity carries cell-clustering information beyond individual-gene expression — and, once cells are grouped, what that activity means biologically. Those are two analysis arms over one shared core:

                    ┌─ acquire → prepare ─┐   one prepared H5AD per dataset
  MSigDB GMTs ──────┤                     ├── shared by both arms
  STRING network ───┘                     │
                                          ├─► clustering arm
                                          │     matched-representation benchmark
                                          │     collection × feature-count sweep
                                          │     thinning · topology · donor bootstrap
                                          │
                                          └─► biological-insight arm
                                                donor-level differential activity
                                                network-driver decomposition
                                                dropout robustness

The clustering arm withholds the curated biological programs from its discovery library; the insight arm tests exactly those programs. A program may not both build a cluster and validate it — but it may explain what the cluster is.

What it compares

Key Representation Purpose
expression_hvg PCA of highly variable genes Identity baseline
expression_gsgenes PCA over library genes only Controls for curated gene selection
meanset Unweighted per-set mean Controls for pathway aggregation alone
aucell Internal rank-AUC score Rank-based control (not the R AUCell package)
gsnetact Network-weighted activity, robust-z across cells GSNetAct-Standalone
hybrid Block-standardized weighted concat → one graph GSNetAct-Fusion

Published gene-set scorers run as additional arms on the same retained sets:

Key Scorer Provenance
dc_aucell · dc_ulm AUCell, ULM decoupler — the authors' implementations
ucell Mann-Whitney U on capped ranks Reimplementation (UCell is R/Bioconductor)
scse Signature share of the cell total Reimplementation (SCSE ships as a Go binary)
jasmine Mean expressed rank × enrichment odds Reimplementation (JASMINE is R)

The three reimplemented arms follow the published formulae and are unit-tested against hand-computed values, but they are this repository's code: a difference from the authors' tool is a difference here, not evidence about the method. jasmine min-max scales across cells, so unlike every other arm its scores are population-relative and must not be compared across differently-composed subsets.

Two decoupler arms are supported but not run, both switched off in decoupler_baselines. dc_ssgsea because decoupler 2.x has no ssgsea entry point and the key no longer falls back to gsea; dc_gsea because it measures 0.027 s per cell per feature — 3.4–16.7 h for one analyze job against 0.5–14 s for every other method — which is ~379 h across the sweep. Enable either by adding it back to that list.

Every representation sees the same cells in the same order, the same 30 latent dimensions, the same 15-neighbour graph, the same seeds and resolution grid. The representation is the only variable.

Each runs across 5 MSigDB collections (Hallmark, C8 full, C8 label-blind, KEGG MEDICUS, combined non-redundant) × 4 feature-count modes (native, matched 50/200/500), on three datasets: 10x CITE-seq PBMC (identity), Kang GSE96583 IFN-β PBMC (CD14 state), and the Lung Cancer Atlas (paired neutrophil state).

Install

micromamba create -f environment.yml     # or: conda env create -f environment.yml
micromamba activate gsnetact-ms
pip install -e . --no-deps

--no-deps is deliberate. environment.yml already installs every dependency in pyproject.toml from conda-forge, and conda-forge ships igraph under the dist name igraph while pyproject.toml requires python-igraph (upstream renamed the PyPI project). Without --no-deps pip does not see that requirement as satisfied and installs PyPI wheels over the conda C-extension builds, downgrading leidenalg in the process.

If import scanpy then fails with zarr-python major version > 2 is not supported — or any other version error naming a package you did not install here — a user-level ~/.local/lib/pythonX.Y/site-packages is shadowing the environment. Make the environment ignore it:

mkdir -p "$CONDA_PREFIX/etc/conda/activate.d"
echo 'export PYTHONNOUSERSITE=1' > "$CONDA_PREFIX/etc/conda/activate.d/zz_no_user_site.sh"

Re-activate afterwards. This applies on activate only; invoking the environment's bin/python directly still needs PYTHONNOUSERSITE=1 in the environment.

Then provide your licensed MSigDB GMTs:

cp config/local.example.yaml config/local.yaml
# edit msigdb.files to point at your GMT paths

config/local.yaml is git-ignored. GMT contents never enter a commit or the release archive.

Run

./run.sh quick   --cores 8      # exploratory: 3 seeds, 20 perms, 2 thinning reps
./run.sh full    --cores 16     # manuscript: 20 seeds, 200 perms, 5 reps, 1000 bootstraps
./run.sh release --cores 16     # full + release gating + results.zip

./run.sh quick --dry-run        # resolve the DAG without running
./run.sh full --arm clustering  # one arm only
./run.sh full --slurm           # via profiles/slurm/config.yaml

The explicit --cores values above suit the laptop compute class. Omit --cores to let run.sh size the run to the machine; that is the safe default. compute.profile defaults to auto, so on a machine with >=32 logical CPUs and >=128 GB the hpc class is selected, and hpc forks 10 Leiden workers per analyze job — so --cores 8 there is rejected up front by the thread-oversubscription guard in workflow/rules/common.smk. Pass at least --cores 10, or force the smaller class with GSNETACT_COMPUTE=laptop.

quick output is marked exploratory. Manuscript numbers come from full. release refuses to run on a dirty git tree, requires the tests to pass, and sets submission_ready=true only after verifying every required figure and table exists and the archive passes an integrity check.

Outputs

results/
├── metrics/          all_clustering_metrics · primary_configuration_metrics
│                     resolution_curves · seed_stability · collection_comparison
│                     fusion_weight_sensitivity · heldout_signature_validation
├── donor_bootstrap/  metric_differences (estimate + CI, paired across methods)
├── thinning/         replicate-level robustness curves
├── topology/         permutation nulls with BH q-values
├── runtime/          per-stage seconds and peak memory
├── qc/               gene-set QC, removed-set audits, small-cluster audit
├── bioinsight/       per dataset: pseudobulk · differential · drivers · dropout
├── figures/          Figures 1-7 and Supplementary S1-S10 (PDF + 300-dpi PNG)
├── figure_source_data/   the exact table behind every panel
├── RESULTS_SUMMARY.md · MANUSCRIPT_RESULTS_DRAFT.md · MANUSCRIPT_FIGURE_LEGENDS.md
└── reproducibility/analysis_manifest.json

Every figure is reproducible from the tables in docs/master_tables.md without re-reading an H5AD. Unavailable combinations are labelled unavailable — never plotted as zero.

Design commitments

  • The donor is the biological replicate. Cells are not replicates; seeds are technical and reported as stability, never as biological uncertainty.
  • Pre-specified primary configuration. Fixed in config/manuscript.yaml before results were inspected, and never itself a reason to state a positive claim. Results are reported across the whole resolution grid.
  • Controls that can kill the hypothesis. MeanSet, rank-AUC, gene-set-gene expression, and a within-set weight-permutation null all exist to explain away an apparent GSNetAct gain. A network-specific claim requires beating both the permuted null and MeanSet.
  • Generated claims. The Results draft is produced by an auditable rule engine over observed tables, not written by hand.

Full reasoning: docs/analysis_plan.md.

Documentation

Document Contents
docs/analysis_plan.md Hypothesis, controls, leakage policy, claim policy
docs/master_tables.md The table contract every figure reads
docs/figure_plan.md Panel-by-panel figure specification
docs/extending.md Add a dataset, representation, metric, figure, or claim
docs/data_sources.md Dataset and software identifiers
docs/hpc.md Running on the HPC server: install, compute profile, LUCA, storage
docs/validation.md What has been checked, and how
docs/methods_score_scale.md Score scales, effect sizes and topology: Methods text
docs/changelog_score_scale_fix.md What the score-scale/effect/topology corrections changed
docs/validation_score_scale_fix.md Verification record for those corrections
docs/briefs/ Original design briefs, kept verbatim

Validation

python -m pytest -q
./run.sh quick --dry-run

The GSNetAct implementation is checked numerically against gsnetact==0.0.20; a scoring change that breaks that agreement fails the test suite.

Limitations

CITE-seq is a single-donor orthogonal-label benchmark. The rank-AUC control is not the R AUCell package. STRING is context-agnostic, so a topology benefit is evidence for informative prior structure, not tissue-specific causality. The driver decomposition interprets the scoring model and is not causal network inference. Public dataset licences and MSigDB terms apply to all downloaded resources.

About

Snakemake workflow, benchmark harness and analysis scripts reproducing the GSNetAct manuscript (companion to BMGLab/GSNetAct)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages