Skip to content

Audit fixes, shared folds and faster cross-validation (3.17.2) - #125

Draft
ellispatrick wants to merge 54 commits into
develfrom
cv-overhaul
Draft

Audit fixes, shared folds and faster cross-validation (3.17.2)#125
ellispatrick wants to merge 54 commits into
develfrom
cv-overhaul

Conversation

@ellispatrick

@ellispatrick ellispatrick commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

For review, use the stack #126 → #133 instead. It splits this branch into eight parts that can be accepted or rejected one at a time, each with its reasons and evidence. This draft holds all eight together for testing them in one install. Its files are identical to #133's.

Fixes the bugs found in the October 2026 audit (AUDIT.md on branch audit-2026-10) and speeds up the cross-validation machinery. The project notes, regression harness and decisions are in SydneyBioX/ProjectClassifyR (private).

Results that change on purpose

  • Shared folds: all assays, classifiers and combinations in one crossValidate call use the same splits, so they can be compared sample by sample (Ellis's decision). Later cross-validations in a call give different results from before.
  • Bug fixes that change predictions:
    • DLDA's prior had the wrong sign;
    • GLM had no intercept;
    • CoxNet ranked features by signed coefficients;
    • predicted classes were relabelled by position;
    • precision pathway costs are now cumulative, and models are matched to classifiers by name;
    • the criss-cross modelTest diagonal was leaky.
  • Inputs that were silently ignored are now used: selectionMethod for MultiAssayExperiment and list input, and tuning ranges.

Speed: identical results for a fixed seed

  • One pool of workers per crossValidate call, cached per session, given a few chunks of interleaved tasks per worker. Each split sets the random stream it had under bpmapply, so results are the same for any number of cores. macOS honours nCores.
  • Data conversion: DataFrame → data.frame without per-column S4 dispatch (0.2 s → <1 ms per call). Numeric design matrices are built directly. The SVM is fitted to a matrix.
  • CoxNet: lambda tuning computes the C-index of all lambdas at once and matches cv.glmnet exactly.
  • Metrics: sample-wise C-index, AUC and ROC curves vectorised, with bit-identical output.
  • Precision pathways: prediction applies the stored models directly.

Full-size timings on verona (devel → this branch):

Run Devel This branch
Procedure 1, random forest 100 × 5, 1 core 184.5 s 107.5 s
Procedure 1, 8 cores 41.3 s 31.1 s
Procedure 2 step 4, CoxNet 7 views, 1 core 734.5 s 191.0 s
Procedure 2 step 4, 8 cores 139.9 s 36.4 s
Procedure 2 step 7, merge with 31 combinations, 8 cores 31.1 min 18.2 min

Checks

  • A testthat suite: 469 tests, all passing.
  • R CMD check: one pre-existing NOTE (ggplot column names; the ensemble selection code, which is left for a decision).
  • The vignettes build.
  • Regression harness: 18 seeded scenarios, identical between 1 and 8 cores.

Open questions

docs/QUESTIONS.md in ProjectClassifyR. Examples:

  • the random forest's second, importance-only forest;
  • 2 dp rounding of AUC;
  • precision pathway confidence.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4

ellispatrick and others added 30 commits October 9, 2026 19:18
@ellispatrick @claude
Add a testthat suite with smoke tests for every keyword; drop the sta…
b7bba8d 
…le Packaged field

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
DLDA: subtract twice the log prior and compute posteriors on the log …
4b07494 
…scale

predict.dlda added log(prior) to the score it minimises, which favoured the
smaller class; the discriminant is sum((x - mean)^2 / var) - 2 log(prior).
The posterior multiplied per-feature densities, which underflowed to NaN with
many features; it is now normalised with log-sum-exp. Training drops
non-numeric columns as prediction does, and the "both" return type orders
the posterior columns by class level, as "score" does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
GLM interface: fit the logistic model with an intercept
19e5c9d 
The formula class ~ . + 0 forced the linear predictor through the origin, so
a feature far from zero could not separate the classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
penalisedFeatures: rank survival coefficients by absolute value
b97a652 
Protective features (negative Cox coefficients) ranked below features with
zero coefficients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fix the cross-validation driver, train/predict and data preparation
4b5c364 
- crossValidate passes selectionMethod on for MultiAssayExperiment and list input.
- Parallel parameters are chosen by .Platform.type, so macOS honours nCores and the seed.
- extraParams are applied to each stage by one helper; a tuneParams list is tuned rather than
  passed on as a setting, and tuning ranges without a tuning mode are an error.
- Tuning keeps the user's other training settings; nested CV tuning of classifier parameters works.
- train() fits each model with runTest, as runTests fits its final model, for one or several assays and
  for merge, prevalidation and PCA; predict() names new data as train() does and gives each model
  the features it was trained with.
- runTest: the MultiAssayExperiment method works; prepared training data is used for survival too.
- Feature importance uses the cross-validation's performance type.
- Percentage splits use the outcome given (and balance events for survival); nFeatures = 1 works.
- prepareData: topNvariance and maxSimilarity work; the clinical-features warning can fire.
- The kNN keyword works.
- Tests for each of these in tests/testthat/test-core.R.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
CoxNet and penalised GLM: encode test data with the training columns
595627e 
CoxNet trained on MatrixModels::model.Matrix but predicted on glmnet::makeX,
which encode factors differently and name non-syntactic features
differently. Penalised GLM rebuilt the test design matrix on its own, so a
factor level absent from the test samples gave a non-conformable error, and
a different column order could silently use the wrong features.

.encodeTrain records the features, their factor levels and the design
matrix columns; .encodeTest puts test features in the training order,
applies the training levels and returns exactly the training columns, with
absent columns as zero. Training matrices are unchanged (same columns,
values and groups), so fitted models are identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Penalised GLM: choose lambda from one predict call over all lambdas
f6a8f6b 
The resubstitution choice of lambda called predict once per lambda. One
call with every candidate lambda gives the same predictions and the same
chosen lambda (checked on 6 asthma subsets with alpha 1, 0.5 and 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
XGB: use the xgboost 3 interface (xgb.DMatrix and xgb.train)
f62688b 
xgboost 3 removed the xgboost(x, y, ...) call, so XGB failed for every
input. The interface builds an xgb.DMatrix from the same sparse one-hot
matrix and calls xgb.train with the same booster parameters (objective,
num_class, colsample_bynode, nrounds, nthread = 1). Extra arguments are
passed to xgb.train or xgb.DMatrix when they are theirs and are booster
parameters otherwise, so a user-supplied nthread replaces the default
instead of giving a "matched by multiple actual arguments" error. Test data
use the training encoding (.encodeTest). Survival predictions are a named
numeric vector.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
SVM: predict on the data frame instead of a model matrix
ff48b0a 
The model is fitted with a formula, but prediction passed
model.matrix(~ ., test), which adds an intercept column, expands factors and
backtick-quotes non-syntactic names, so prediction failed for categorical
features or names such as "g-1". Predicting on the data frame lets e1071
encode the test data as it encoded the training data. Predictions for
numeric features are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
kNN: return a factor with the training levels; handle one test sample
5b3e043 
The predicted classes were a character vector, unlike every other
classifier. With a single test sample (leave-one-out), apply() over the
neighbour indices returned a vector and the voting step failed. Neighbour
classes are built as a character matrix directly, and the predicted classes
are a factor with the training levels. Votes and scores are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fisher discriminant, k-TSP and Poisson LDA: use their own arguments
0643461 
The three classifiers referred to trainingMatrix and testingMatrix, which
were never defined, so they failed on any input. They now build them from
their arguments. Poisson LDA passes the classes to PoiClaClu as the numbers
1 to K and maps its predictions back to a factor with the training levels.
Fisher's pooled variance weighted each class's variance by the number of
features minus one; it is weighted by the class size minus one. Fisher and
k-TSP use only the first two levels of the classes, so they stop with an
error for any other number of classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Naive Bayes and mixtures of normals: crossover distance weighting
7c04a64 
Both classifiers failed with weighting = "crossover distance". Naive Bayes
used an undefined object `test` for the test samples. The horizontal
distances were computed only when difference was "weighted" (naive Bayes)
or when difference equalled "crossover distance", which never happens
(mixtures), but they are always needed with that weighting, because they
decide which features vote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Vectorise the sample-wise C-index, AUC and the merged ROC curve
532600e 
The sample-wise C-index counts concordant and discordant pairs with outer()
per group. ROC rates come from cumulative sums over sorted scores, used by
AUC and ROCplot's merge mode, and .calcArea works on vectors. Values are
identical to the loop implementations, including tie handling and rounding.
A class absent from a group gives a clear error instead of 'missing value
where TRUE/FALSE needed'; a sample never predicted gets NA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
colCoxTests: keep one feature as a matrix; make the slow option work
16561fd 
Sorting the samples dropped a single-feature matrix to a vector, so CoxPH
ranking failed for one feature. The "slow" option called coxph on a plain
matrix response, compared class() with ==, filled failures with two values
for three columns, did not transpose the apply() result and named the rows
by sample. It now converts a time/event matrix to Surv, fits
survival::coxph per feature and returns one row per feature, agreeing with
the fast option. The documented return value is corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
subtractFromLocation: work for one numeric feature; keep feature names
944c9f2 
Subsetting the numeric columns without drop = FALSE, and transposing the
row-wise apply(), failed for a single numeric feature. The subtraction uses
sweep() on the numeric matrix, and the result keeps the original feature
names (DataFrame() had made them syntactic, so "g-1" became "g.1"). Test
columns are matched to the training columns by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Precision pathways: match models by assay name, cumulative costs, pre…
0b4d3b7 
…dict with stored models

- .permutations keeps a matrix (drop = FALSE), so clinical plus one assay works in the default mode.
- fixedAssays may be NULL or several assays; list input no longer calls colData().
- Models are named and looked up by assay; prediction uses each result's own predictParams.
- precisionPathwaysPredict predicts every new sample with every stored fold model instead of
  re-splitting the new cohort with runTests; classes of new samples are optional.
- A sample is charged for every tier it passed through (cumulative cost).
- summary() matches weights by name and checks they sum to 1.
- The confidence cut-off is a minimum (>=, with a floating point tolerance).
- Each pathway carries its assay sequence; no strsplit on '-'.
- bubblePlot and strataPlot no longer call theme_set(); bubblePlot keeps pathways below 0.5;
  strataPlot draws the true-class strip as its own layer.
- Fold models drop the forImportance forests, which prediction does not use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
crissCrossValidate: honest modelTest diagonal, shared features, TOP o…
9493b64 
…nly when used

- TOP is required only when runTOP = TRUE.
- Data sets are restricted to shared features (with a message) before training and testing.
- The modelTest diagonal is the crossValidate run on that data set, not a leaky re-test.
- Outcome type is checked for every data set; performanceType is resolved before tuning params.
- Random features use max(nFeatures) when several values are tuned.
- crissCrossPlot no longer attaches the result, labels the diagonal, and returns the plot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Parameter classes: informative errors, SelectParams for 'none' and fu…
aa46278 
…nctions, show fixes

- PredictParams on a one-step classifier or unknown keyword gives a clear error.
- SelectParams('none') returns NULL; SelectParams accepts plain ranking functions.
- TrainParams(balancing = ) is an error pointing to ModellingParams.
- show methods no longer error on missing characteristics and end lines with newlines.
- Remove the unused misspelled class union; fix the 'higer' typo for AUC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
METABRICclinical documentation: six features plus survival outcome, c…
119b81d 
…orrect source URL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fix calcExternalPerformance, grouping by fold and easyHard
5898d1b 
- calcExternalPerformance matches predicted classes to actual classes by
  name instead of relabelling levels by position, errors on unknown
  predicted classes, and calculates every requested metric (the loop used
  the whole vector of metric names).
- grouping = "fold" averages C-index and AUC within each permutation into a
  plain named numeric vector with permutations in numerical order.
- easyHard drops samples with missing performance and matches assay rows by
  sample name; its assay and performanceType defaults work (they are no
  longer generic arguments); MultiAssayExperiment assays and "clinical"
  are usable and a matrix assay is converted to a DataFrame.
- Documentation: the definitions of precision and recall were swapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Prevalidation and PCA: more than two classes, inner seeds, one-featur…
963f524 
…e assays

- extractPrevalidation simplified the per-assay score matrices with
  sapply(), which only works for one column per assay, so prevalidation
  failed for more than two classes. The columns are bound instead and named
  <assay>_<class> when an assay contributes several.
- The inner cross-validation seed was .Random.seed[1], the RNG kind code,
  which is the same for every call (and missing in a fresh session). It is
  drawn from the random number stream with sample.int().
- The clinical model was fitted twice; the second fit was never used. Only
  the non-clinical models are refitted for prediction.
- Splitting the table by assay dropped a one-feature assay to a vector.
- A first assay without a selection step (selectionMethod "none") made the
  tuning check fail on a NULL selectParams; also in selectMulti.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
limmaRanking: pass extra arguments to lmFit as named arguments
b1a858f 
append(fitParams, ...) matched the first extra argument to append()'s
`values` (or failed for a named one such as weights) and the second to
`after`. The extra arguments are now added with c(fitParams, list(...)).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
edgesToHubNetworks: accept matrix input
1c894e2 
class(x) == "matrix" has length two for a matrix (c("matrix", "array")), so
the if() failed. is.matrix() is used instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
previousSelection: compute the overlap with the current data and warn
8fcb03a 
The previous features were intersected with the current ones before the
overlap was computed, so the overlap was always 100%, and
signalCondition() without a handler does nothing. The overlap is now the
percentage of previously selected features present in the current data,
and a warning is given when it is below minimumOverlapPercent. The selected
features are the same as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
KS, KL and pairs-differences rankings: stop for other than two classes
6adc5ec 
These rankings compare the first two class levels only, so with more
classes they silently ignored the rest. They now stop with an error that
says they are for two classes (as Fisher discriminant and k-TSP do).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Run the splits of all of crossValidate's cross-validations in one poo…
e38d367 
…l of workers

runTests is split into preparing a cross-validation (data checks, splits, final model), running its
splits and assembling the ClassifyResult. crossValidate prepares every assay, classifier and combination
first and runs all of their splits in one pool, handing tasks out one at a time. Each split uses the random
number stream that bpmapply with the cross-validation's seed gives it, so results are identical to before
for any number of cores. Nested cross-validations within a split run serially rather than forking again.
Missing values are checked per column and constant features found without converting to a matrix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Levene ranking: one F-test over all features instead of car per feature
6d370e9 
Levene's test centred on the median is a one-way F-test of the absolute
deviations from the class medians. Computing the deviations for all features
and calling genefilter::rowFtests gives the same ranking as calling
car::leveneTest for each feature (identical order on the full asthma data,
10 subsets of 120 samples and a random three-class split), 17 times faster.
car is no longer needed for this ranking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
SelectParams: name user ranking functions 'User-specified Ranking'; f…
840c857 
…ix PredictParams show test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fit the final models of crossValidate's cross-validations in the work…
781663c 
…er pool; share folds and seed

All assays, classifiers, selection methods and combinations in one crossValidate call use the first one's
seed and training/test splits, so their results can be compared sample by sample. The final model of each
is fitted in the worker pool from the random number state that follows making the splits. Worker pools are
made once per session and reused, because making one takes about half a second on servers that allow many
open files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick
Merge branch 'fix-interfaces' into cv-overhaul
4c433b9
ellispatrick and others added 18 commits October 9, 2026 22:45
@ellispatrick @claude
samplesMetricMap: match samples by name and fix failing options
b54f490 
- Each result's sample-wise metric is matched to the first result's samples
  by name, and results must predict the same set of sample names.
- comparison = "Cross-validation" uses the characteristic like any other.
- showLegends = FALSE, a single result and a class with one sample work.
- More than two classes get a white-to-dark gradient per class by default,
  one legend per class; too few user colours give an error.
- The matrix method's class and group bars keep their first and last
  tiles (limits = c(1, n) dropped them) and its clustering works.
- Legends are taken from small plots with one tile per colour; rendered
  output of the list method is byte-identical to before.
- Rd: metric values listed "Sample Error" twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
ROCplot: fix average mode, keep the user's theme, use linewidth
d9b4296 
Average mode split a DataFrame and subset() with class = aClass (an unused
argument). The plot theme is added to each plot instead of theme_set().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
performancePlot: respect yLimits when rotated, keep the user's theme
70a9a6a 
rotate90 = TRUE replaced user yLimits by c(0, 1) and turned a character x
characteristic into NA. Documentation of the box plot default, aggregate
and the return value corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
rankingPlot: use sizesList fonts, row/column characteristics and shor…
8bc49c3 
…t rankings

Fonts were read from sizesList$fontSizes instead of the documented
$fonts; row and column values compared to the whole list; ordering by
performance lacked the metric; a ranking shorter than top counted NA as a
shared feature. Theme added to the plot instead of theme_set().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
selectionPlot: fix importance mode and commonality tables
7deb99f 
Importance mode: undefined plot and colourVariable, keepFeatures compared
names to counts, y mapped to a constant string, facets were subset by one
characteristic at a time. Commonality: duplicate overlap column, and with a
reference level the characteristics were looked up in the whole table and
repeated the wrong number of times. alpha reaches the geoms; rotate90
works for importance. Theme added to each plot instead of theme_set().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
plotFeatureClasses errors on unknown features; featureSetSummary mess…
3dba16c 
…age spacing

plotFeatureClasses returned an empty list after printing a message for a
useFeatures not in the data; it now stops. Its theme is added to each plot.
Adds tests for all plotting fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick
Merge branch 'fix-metrics' into cv-overhaul
6a09c56
@ellispatrick @claude
Merge metric fixes; Cox elastic net tuning with the C-index of all la…
465b39c 
…mbdas at once

.cvCoxnetC reproduces cv.glmnet(family = 'cox', type.measure = 'C'): the same folds, fits and lambda.min.
It counts concordant pairs for every lambda of a fold at once instead of calling survival::concordance per
lambda and fold. cv.glmnet is still used if settings of its own cross-validation are given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
SVM and glmnet/XGB encodings: numeric features skip building a model …
d33d19d 
…frame

An SVM is fitted to a matrix of numeric features, which gives the same model as the formula interface
in less time; the design matrix of numeric features is built directly, the same as model.matrix makes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fold models of random forests don't keep the forest grown to rank fea…
d23f224 
…tures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Regenerate help pages
43842f4 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Check notes: derive split streams without ':::', import stats and uti…
51b354c 
…ls functions, drop the C++11 request

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
NEWS for 3.17.2; version 3.17.2
877b82d 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Vignette: state the computed balanced accuracy inline
ef019c5 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Give tasks to workers in a few chunks each, interleaving cross-valida…
f596d39 
…tions

MulticoreParam forks a process for every chunk of tasks; one task per chunk made a full-size Procedure 1 run
on 8 cores slower than serial. Results don't depend on the order or chunking, since each task sets its
own random number stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fork each worker once per crossValidate call instead of once per chun…
f9de8a8 
…k of tasks

BiocParallel's MulticoreParam forks a process for every chunk of tasks, and each forked process copies much
of the main session's memory (about 0.5 GB with ClassifyR's dependencies loaded) when it first collects
garbage. On Linux and macOS, .forkApply now forks the workers once, after the data are prepared, keeps the
task function in the package environment so that only task numbers and results are sent, and hands tasks
out two at a time as workers become free. Procedure 1 of the protocol at 20 cores: 16.5 s -> 10.6 s, with
identical predictions. Windows still uses BiocParallel's SnowParam, and runTests still uses the
BiocParallel parameters it is given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
One worker pool for every platform with the parallel package; fixes a…
b697fa8 
…nd removals from a maintainability review

Parallel: crossValidate's workers are made once per call with the parallel package, forked on Linux and
macOS and socket workers on Windows (sent the task function and its data once each), with tasks handed out
two at a time. Results are the same as serial, and the same with forked or socket workers. runTests still
uses the BiocParallel parameters it is given.

Review fixes, without changes to results:
- an unknown multiViewMethod is an error naming the choices;
- the merge, prevalidation and PCA branches of crossValidate are one;
- CV loses its unused and broken independent train/test path;
- train and predict align the samples of a list of tables by name;
- prepareDataFormals includes maxSimilarity;
- prepareData's maxSimilarity uses stats::cor (dcanr dropped);
- unused mergeTrainInterface, fastCox and simplifyResults removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Untrack Rplots.pdf written by the plot tests; one entry per line in .…
04862e9 
…gitignore

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
ellispatrick and others added 3 commits October 10, 2026 05:14
@ellispatrick @claude
Hand tasks to workers in about eight chunks each instead of two tasks…
40e64ad 
… at a time

Why: a timeline of the protocol's Procedure 1 on 20 cores showed each worker busy for only 7.5-8.3 s of the
12.8 s the tasks took: with 1500 short tasks (median 75 ms) handed out two at a time, workers waited for the
main process between chunks. With about eight chunks per worker: Procedure 1 13.2 s -> 8.9 s on 20 cores,
9.4 s -> 8.1 s on 32; Procedure 2 step 4, with longer tasks, unchanged within noise. Results are identical
(each task sets its own random number stream).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Cox elastic net: a type.measure given by the user replaces the C-inde…
29f337e 
…x instead of clashing with it

cv.glmnet was always called with type = 'C', so extraParams = list(train = list(type.measure = 'deviance'))
stopped with 'formal argument matched by multiple actual arguments'. Tuning by C-index remains the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Cox elastic net: end the path of lambda at 0.05 of its largest value …
e5a1348 
…by default (changes results)

Why: lambdas near zero give nearly unpenalised Cox models, which overfit the few events of typical omics
cohorts (39 relapses in the protocol's METABRIC data) and converge slowly. On the protocol's Procedure 2
step 7 (merge of five views, 31 combinations, 20 x 5 CV, 16 cores), compared with glmnet's default path:
- mean C-index over the 31 combinations 0.602 vs 0.599; best in 21 of 31 combinations;
- 1.4 min vs 9.2 min;
- cross-validation had chosen a lambda below 0.05 of the largest in 12% of fold models; clipping those
  did not lower the C-index.
Tuning by partial likelihood deviance was also compared and gave lower C-indexes (0.587), so tuning by
C-index stays. ncvreg uses the same 0.05 when there are more features than samples. Users can still set
lambda or lambda.min.ratio through extraParams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick ellispatrick mentioned this pull request Oct 10, 2026
Draft

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ellispatrick