Skip to content

2/8 Fixes to classifier and ranking interfaces (DLDA prior, GLM intercept, XGB for xgboost 3, ...) - #127

Merged
DarioS merged 19 commits into
review/1-core-fixesfrom
review/2-interface-fixes
Oct 28, 2026
Merged

DarioS merged 19 commits into
review/1-core-fixesfrom
review/2-interface-fixes

Conversation

@ellispatrick

@ellispatrick ellispatrick commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Part of a stack of 8 pull requests from the October 2026 audit of ClassifyR (AUDIT.md on branch audit-2026-10). Each part is based on the one before, so they need to be merged in order. Each one can be accepted, changed or rejected on its own merits. Draft PR #125 holds all eight together. Every part installs and passes its own tests.

Stack, merge in order: #126 tests and core fixes → #127 interface fixes → #128 metrics and plots → #129 pathways and criss-cross → #130 maintenance → #131 faster cross-validation (identical results) → #132 shared folds (changes results) → #133 release notes. The regression harness and timings are in SydneyBioX/ProjectClassifyR (private).

Summary

Fixes to the classifier, feature ranking and helper interfaces: wrong results, and code paths that failed on any input.

Why and what (one commit each, each with a test that failed before)

Wrong results with no error:

  • DLDA: the class prior had the wrong sign. It added log(prior) to a score that is minimised, which favoured the smaller class: on pure noise with 90/10 classes, 97% of samples were predicted to be the minority class. It now subtracts 2 log(prior), and posteriors are computed on the log scale (they underflowed with many features).
  • GLM: fitted without an intercept (class ~ . + 0), so a shifted feature gave chance accuracy. Now fitted with an intercept.
  • CoxNet feature ranking used the signed coefficient, so protective features ranked below features with zero coefficients. Now uses the absolute value.
  • CoxNet, penalised GLM and XGB encoded test data separately from training data. A factor level missing from a test fold gave an error, or silently misaligned columns. Test data are now encoded with the training columns and levels.

Failed on any input:

  • XGB with xgboost 3: ported to xgb.DMatrix/xgb.train.
  • SVM with factor features.
  • kNN returned characters, and failed for one test sample.
  • Fisher discriminant, k-TSP and Poisson LDA referred to an undefined trainingMatrix.
  • Naive Bayes and mixtures of normals with "crossover distance" weighting.
  • colCoxTests with one feature, and its "slow" option.
  • subtractFromLocation with one feature.
  • Prevalidation with more than two classes.
  • limmaRanking's ....
  • edgesToHubNetworks with a matrix.

Surprising behaviour:

  • Prevalidation and PCA seeded their inner CV with .Random.seed[1] (the RNG kind code, the same in every fold); they now draw a seed.
  • previousSelection's overlap check could never fire, and now warns.
  • KS, KL and pairs-differences rankings silently used only the first two classes, and now stop.

Speed, with identical results:

  • Levene ranking uses one F-test over all features: the same order as car::leveneTest, 36.8 s → 2.1 s on asthma.
  • Penalised GLM chooses lambda from one predict call: the same choice.

Do results change?

Yes, where a fix applies: DLDA predictions with unbalanced classes, GLM predictions, CoxNet rankings and XGB (which works now). Rankings and selections of the other methods are unchanged (regression harness).

Evidence

157 tests pass. Each commit's message gives its test.

Questions (left unchanged on purpose)

  • xgboost 3 estimates base_score from the data.
  • The penalised GLM lambda filter uses the signed sum of the first class's coefficients.
  • previousSelection applies make.names, so non-syntactic names never match.
  • The naive Bayes crossover uses unscaled densities.
  • The likelihood-ratio ranking has a 25× faster closed form, but it would reorder two features whose statistic is exactly 0 (ties), so it is not included.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4

ellispatrick and others added 19 commits October 10, 2026 04:44
…scale

predict.dlda added log(prior) to the score it minimises, which favoured the
smaller class; the discriminant is sum((x - mean)^2 / var) - 2 log(prior).
The posterior multiplied per-feature densities, which underflowed to NaN with
many features; it is now normalised with log-sum-exp. Training drops
non-numeric columns as prediction does, and the "both" return type orders
the posterior columns by class level, as "score" does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
The formula class ~ . + 0 forced the linear predictor through the origin, so
a feature far from zero could not separate the classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
Protective features (negative Cox coefficients) ranked below features with
zero coefficients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
CoxNet trained on MatrixModels::model.Matrix but predicted on glmnet::makeX,
which encode factors differently and name non-syntactic features
differently. Penalised GLM rebuilt the test design matrix on its own, so a
factor level absent from the test samples gave a non-conformable error, and
a different column order could silently use the wrong features.

.encodeTrain records the features, their factor levels and the design
matrix columns; .encodeTest puts test features in the training order,
applies the training levels and returns exactly the training columns, with
absent columns as zero. Training matrices are unchanged (same columns,
values and groups), so fitted models are identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
The resubstitution choice of lambda called predict once per lambda. One
call with every candidate lambda gives the same predictions and the same
chosen lambda (checked on 6 asthma subsets with alpha 1, 0.5 and 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
xgboost 3 removed the xgboost(x, y, ...) call, so XGB failed for every
input. The interface builds an xgb.DMatrix from the same sparse one-hot
matrix and calls xgb.train with the same booster parameters (objective,
num_class, colsample_bynode, nrounds, nthread = 1). Extra arguments are
passed to xgb.train or xgb.DMatrix when they are theirs and are booster
parameters otherwise, so a user-supplied nthread replaces the default
instead of giving a "matched by multiple actual arguments" error. Test data
use the training encoding (.encodeTest). Survival predictions are a named
numeric vector.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
The model is fitted with a formula, but prediction passed
model.matrix(~ ., test), which adds an intercept column, expands factors and
backtick-quotes non-syntactic names, so prediction failed for categorical
features or names such as "g-1". Predicting on the data frame lets e1071
encode the test data as it encoded the training data. Predictions for
numeric features are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
The predicted classes were a character vector, unlike every other
classifier. With a single test sample (leave-one-out), apply() over the
neighbour indices returned a vector and the voting step failed. Neighbour
classes are built as a character matrix directly, and the predicted classes
are a factor with the training levels. Votes and scores are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
The three classifiers referred to trainingMatrix and testingMatrix, which
were never defined, so they failed on any input. They now build them from
their arguments. Poisson LDA passes the classes to PoiClaClu as the numbers
1 to K and maps its predictions back to a factor with the training levels.
Fisher's pooled variance weighted each class's variance by the number of
features minus one; it is weighted by the class size minus one. Fisher and
k-TSP use only the first two levels of the classes, so they stop with an
error for any other number of classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
Both classifiers failed with weighting = "crossover distance". Naive Bayes
used an undefined object `test` for the test samples. The horizontal
distances were computed only when difference was "weighted" (naive Bayes)
or when difference equalled "crossover distance", which never happens
(mixtures), but they are always needed with that weighting, because they
decide which features vote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
Sorting the samples dropped a single-feature matrix to a vector, so CoxPH
ranking failed for one feature. The "slow" option called coxph on a plain
matrix response, compared class() with ==, filled failures with two values
for three columns, did not transpose the apply() result and named the rows
by sample. It now converts a time/event matrix to Surv, fits
survival::coxph per feature and returns one row per feature, agreeing with
the fast option. The documented return value is corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
Subsetting the numeric columns without drop = FALSE, and transposing the
row-wise apply(), failed for a single numeric feature. The subtraction uses
sweep() on the numeric matrix, and the result keeps the original feature
names (DataFrame() had made them syntactic, so "g-1" became "g.1"). Test
columns are matched to the training columns by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
…e assays

- extractPrevalidation simplified the per-assay score matrices with
  sapply(), which only works for one column per assay, so prevalidation
  failed for more than two classes. The columns are bound instead and named
  <assay>_<class> when an assay contributes several.
- The inner cross-validation seed was .Random.seed[1], the RNG kind code,
  which is the same for every call (and missing in a fresh session). It is
  drawn from the random number stream with sample.int().
- The clinical model was fitted twice; the second fit was never used. Only
  the non-clinical models are refitted for prediction.
- Splitting the table by assay dropped a one-feature assay to a vector.
- A first assay without a selection step (selectionMethod "none") made the
  tuning check fail on a NULL selectParams; also in selectMulti.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
append(fitParams, ...) matched the first extra argument to append()'s
`values` (or failed for a named one such as weights) and the second to
`after`. The extra arguments are now added with c(fitParams, list(...)).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
class(x) == "matrix" has length two for a matrix (c("matrix", "array")), so
the if() failed. is.matrix() is used instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
The previous features were intersected with the current ones before the
overlap was computed, so the overlap was always 100%, and
signalCondition() without a handler does nothing. The overlap is now the
percentage of previously selected features present in the current data,
and a warning is given when it is below minimumOverlapPercent. The selected
features are the same as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
These rankings compare the first two class levels only, so with more
classes they silently ignored the rest. They now stop with an error that
says they are for two classes (as Fisher discriminant and k-TSP do).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
Levene's test centred on the median is a one-way F-test of the absolute
deviations from the class medians. Computing the deviations for all features
and calling genefilter::rowFtests gives the same ranking as calling
car::leveneTest for each feature (identical order on the full asthma data,
10 subsets of 120 samples and a random three-class split), 17 times faster.
car is no longer needed for this ranking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4

@DarioS DarioS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some good catches.

@DarioS
DarioS merged commit c607584 into review/1-core-fixes Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants