[ellispatrick]

Based on #134; independent of the other post-#134 PRs. It can be accepted or rejected on its own.

Summary

Changes results slightly (decided by Ellis Patrick). These are metric and plot definitions; no predictions change.

• No rounding inside metrics. AUC is no longer rounded to 2 dp per class before averaging; the sample-wise C-index, crissCrossValidate's performance matrices and runTest's variable importance are no longer rounded either. Plots round only their labels.
• Sample-wise C-index: a pair with tied risk scores counts as half concordant, as in the C-index (survival::concordance). It used to be dropped.
• grouping = "fold": every class metric (e.g. Balanced Accuracy, Macro F1) is now calculated per fold and averaged within each permutation, as AUC and C-index already were. It used to return one value per fold (500 values for 100 × 5).
• Macro precision and F1: a class that is never predicted has precision 0, not NaN.
• AUC for a group lacking a sample of some class: NaN with a warning, not an error.
• performancePlot's chance line: the metric's expected value under uniformly random predictions. That is 1/k for accuracy, balanced accuracy, precision, recall and F1, 1 − 1/k for error metrics, 0.5 for AUC and C-index, and 0 for MCC; no line otherwise. It used to be 0.5 for every metric.
• rankingPlot: overlap divides by the length of the shorter list of top features, not by top, when a ranking has fewer features. Axis labels now say what is measured: "Average Top Features in Common (%)" for rankingPlot and "Jaccard Similarity (%)" for selectionPlot. Both used to say "Common Features (%)" although they compute different things.
• crissCrossPlot: the diagonal is hidden by default for both training types; showDiagonal = TRUE shows it. showResubMetric still works, with a deprecation warning.

Why

Ellis's answers to the open questions (ProjectClassifyR docs/DECISIONS.md, 10 Oct 2026; docs/QUESTIONS.md Q6–Q12, Q19):

• Rounding before averaging biases the averaged value and creates artificial ties.
• Ignoring tied risks makes the sample-wise C-index disagree with the C-index.
• The fold grouping was documented as making no difference to accuracy metrics, yet it returned a different shape.
• NaN macro precision made a whole permutation's value missing.
• A missing class in one fold should not stop the calculation for the other folds.
• With three or more classes, 0.5 is not the chance level of accuracy.
• The diagonal is not a cross-cohort result, so the cross-cohort heatmap now hides it by default.

What changes

• R/calcPerformance.R, R/utilities.R (.calcArea), R/ROCplot.R, R/performancePlot.R (new .chanceLevel), R/rankingPlot.R (new .topOverlap), R/selectionPlot.R, R/crissCrossValidate.R, R/runTest.R; help pages regenerated.

• Tests:

  • tied risks in the sample-wise C-index;
  • unrounded AUC;
  • AUC NaN with a warning;
  • macro precision and F1 with a never-predicted class;
  • fold grouping for class metrics;
  • chance levels;
  • top-feature overlap;
  • the hidden and shown criss-cross diagonal and the deprecated argument.

  The reference implementations in the tests are updated to the new definitions.

Evidence

• Tests and check: 570 test expectations pass. R CMD check gives only the existing NOTEs (.doSelection globals; .git from building in a worktree).

• Regression harness (18 scenarios, the build of 9/9 Cox elastic net: lambda path ends at 0.05 of the largest value by default (changes results; 6.6x faster) #134 against this one): 17 are identical. proc4_crisscross differs only by the removed rounding, e.g. 0.98 → 0.9844 and 0.96 → 0.9569.

• Metric values on results computed once, evaluated under both builds (asthma DLDA 20 × 5; asthma random forest 5 × 5; Procedure 1 random forest; METABRIC clinical CoxPH 20 × 5):

  Metric	Change
  AUC	at most 0.005 per permutation (removed rounding); means within 0.001
  Sample C-index	at most 0.005 per sample (rounding and tied risks); mean 0.6707 → 0.6706
  Balanced Accuracy, grouping = "fold"	now one value per permutation (e.g. 100 → 20 values); mean unchanged
  Balanced Accuracy, C-index, macro precision and F1, sample accuracy (default grouping)	identical

  Scripts: ProjectClassifyR scripts/metrics_defaults/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4

[DarioS]

More edge cases fixed.