[ellispatrick]

Follow-up to the stack (#126 → #134 → this). Based on #134. It can be accepted or rejected on its own: closing it keeps resubstitution tuning and the first-class lambda filter.

Summary

Changes results (Ellis Patrick's decision, on the evidence below). The penalised GLM classifiers (ridgeGLM, elasticNetGLM, LASSOGLM) choose lambda by 5-fold cross-validation of the balanced error within each training set, instead of by the balanced error of the training samples themselves (resubstitution).

• Resubstitution stays available, and gives the previous results exactly: extraParams = list(train = list(lambdaTuning = "resubstitution")).
• The lambda filter now considers every class (bug fix).

Why

• Statistical: resubstitution rewards fitting the training samples, so it picks lambdas near the unpenalised end of the path. On the protocol data the lambdas it chose were 10–30× smaller than those chosen by cross-validation (asthma: median 0.00075 against 0.016). On the larger cohorts this overfits: PROMAD GSE48581 (262 samples) kept 53 features, with balanced accuracy 0.638 against 0.713.
• Precedent: CoxNet in ClassifyR already tunes by cv.glmnet.
• Lambda filter (bug): lambdas were considered only where the signed sum of the first class's coefficients was non-zero. For three or more classes, that dropped lambdas at which only the first class had no coefficients, so the choice depended on the order of the factor levels. A user-given path made only of such lambdas stopped with a Matrix dimnames error. In a synthetic 3-class example, 12 of 100 usable lambdas were dropped. With two classes the first class's coefficients are the negative of the second's, so two-class results don't change.

Evidence

20 × 5 CV, seed 1, 4 cores; the penalised GLM is the classifier in every run.

• Elastic net: asthma; Procedure 1 (ghist, 3 views); Procedure 3 (melanoma TCGA, 3 views).
• LASSO: Procedure 4 (PROMAD cohorts, 100 features, cross-validation within each cohort).

resubstitution is this PR with lambdaTuning = "resubstitution", and it is identical to #134 on every data set.

Data set	View	Balanced accuracy, resub → CV	AUC, resub → CV	Median features, resub → CV
asthma	RNA	0.812 → 0.815	0.887 → 0.890	19 → 10
Procedure 1	clinical	0.711 → 0.714	0.692 → 0.687	2 → 1
Procedure 1	Inferred Protein	0.790 → 0.771	0.890 → 0.831	19 → 9.5
Procedure 1	Microarray RNA	0.808 → 0.797	0.907 → 0.861	17 → 7
Procedure 3	clinical	0.683 → 0.684	0.722 → 0.713	3 → 1
Procedure 3	miRNA	0.623 → 0.628	0.661 → 0.679	15 → 12
Procedure 3	RNA	0.548 → 0.549	0.539 → 0.565	16 → 11.5
Procedure 4	GSE129166	0.925 → 0.913	0.981 → 0.967	9 → 5
Procedure 4	GSE192444	0.952 → 0.942	0.984 → 0.974	21 → 8
Procedure 4	GSE48581	0.638 → 0.713	0.758 → 0.792	53 → 18
Procedure 4	GSE93658	0.881 → 0.868	0.938 → 0.940	5 → 9

• Accuracy: mean balanced accuracy +0.002 and mean AUC −0.006. Without GSE48581: −0.005 and −0.010.
• Where it loses: the two small omics views of Procedure 1 (54 samples, AUC −0.05), where inner 5-fold cross-validation is noisy.
• Where it gains: the largest, most imbalanced cohort (GSE48581: 40 rejections against 222 controls).
• Sparsity: models keep about 40% fewer features.

Run time per crossValidate call (s)	Resubstitution	CV	Ratio
asthma	8.5	12.5	1.5×
Procedure 1 (3 views)	9.7	16.5	1.7×
Procedure 3 (3 views; selection dominates)	67.9	70.7	1.04×
Procedure 4 (4 cohorts)	54.4	62.3	1.15×

Times are on a shared server; repeat runs of the same setting varied by up to 35%.

Caveat: the average gain is within noise. The case rests on resubstitution being a biased choice of lambda, sparser models, the fix on larger cohorts, and consistency with CoxNet, all for seconds of extra time.

What changes

• penalisedGLMtrainInterface has new arguments lambdaTuning = c("CV", "resubstitution") and nFoldsLambda = 5:

  • cv.glmnet uses the same lambda path and class weights as the fit, with folds stratified by class and type.measure = "class". Weighted by inverse class size, misclassification is the balanced error.
  • The largest lambda with the smallest error is chosen, among lambdas at which any class has a coefficient.
  • With a class of fewer than three training samples, it falls back to resubstitution with a warning.

• The inner folds use the random stream, so results change for any seed. This includes precisionPathwaysTrain, whose default classifier for clinical data is elasticNetGLM (Procedure 3).

• ?crossValidate describes the tuning and the option.

• Tests:

  • CV tuning is reproducible;
  • the resubstitution option picks the lambda with the smallest training balanced error;
  • a path with only first-class-empty lambdas works (it errored before);
  • the small-class fallback.

  552 expectations pass (test_branch.sh).

Scripts and results: ProjectClassifyR scripts/glm_cv_benchmark/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4