Skip to content

3/8 Metrics and plots: vectorised C-index, AUC and ROC (identical values); fixes - #128

Merged
DarioS merged 8 commits into
review/2-interface-fixesfrom
review/3-metrics-plots
Oct 28, 2026
Merged

3/8 Metrics and plots: vectorised C-index, AUC and ROC (identical values); fixes#128
DarioS merged 8 commits into
review/2-interface-fixesfrom
review/3-metrics-plots

Conversation

@ellispatrick

@ellispatrick ellispatrick commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Part of a stack of 8 pull requests from the October 2026 audit of ClassifyR (AUDIT.md on branch audit-2026-10). Each part is based on the one before, so they need to be merged in order. Each one can be accepted, changed or rejected on its own merits. Draft PR #125 holds all eight together. Every part installs and passes its own tests.

Stack, merge in order: #126 tests and core fixes → #127 interface fixes → #128 metrics and plots → #129 pathways and criss-cross → #130 maintenance → #131 faster cross-validation (identical results) → #132 shared folds (changes results) → #133 release notes. The regression harness and timings are in SydneyBioX/ProjectClassifyR (private).

Summary

Performance metrics and plots: three slow calculations vectorised with bit-identical output, and fixes to options that failed or gave wrong results.

Why and what

Speed, with identical values (checked with identical() against the previous code, including ties, absent classes and three classes):

Calculation Before After
Sample-wise C-index 22.7 s 0.19 s
AUC 9.3 s 0.16 s
ROCplot merged curve 102 s 0.28 s

Each was measured per result: 165 samples, 100 × 5 CV.

Wrong results:

  • calcExternalPerformance relabelled predicted classes by position, so a perfect prediction whose factor levels were in another order scored accuracy 0. It now matches classes by name, and it accepts several performance types.
  • samplesMetricMap aligned samples by position across results and labelled them with the first result's names. It now matches samples by name.
  • easyHard dropped rows by position. It now matches samples by name.

Failed on any input:

  • ROCplot(mode = "average").
  • samplesMetricMap with comparison = "Cross-validation", showLegends = FALSE, a single result, or more than two classes.
  • selectionPlot importance mode.
  • rankingPlot: fonts from sizesList were ignored, and there was a row/column copy-paste error.
  • performancePlot replaced the user's yLimits when rotate90 = TRUE.

Behaviour:

  • Plotting functions no longer call theme_set(), which changed the user's global ggplot2 theme. The theme is applied to the plot itself, giving byte-identical PNGs.
  • grouping = "fold" returns a numeric vector, not a by object.
  • Documentation had precision and recall swapped.

Do results change?

Metric values do not. Results change only where a bug is fixed (mismatched factor levels, sample order).

Evidence

211 tests pass (test-metrics.R, test-plots.R).

Questions (left unchanged on purpose)

  • AUC and the sample-wise C-index are rounded to 2 dp.
  • The sample-wise C-index drops tied pairs.
  • Balanced accuracy is not averaged per permutation with grouping = "fold".
  • Macro precision and F1 are NaN when a class is never predicted.
  • The chance line in performancePlot is drawn at 0.5 for every metric.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4

ellispatrick and others added 8 commits October 10, 2026 04:45
@ellispatrick @claude
Vectorise the sample-wise C-index, AUC and the merged ROC curve
0c52ca5 
The sample-wise C-index counts concordant and discordant pairs with outer()
per group. ROC rates come from cumulative sums over sorted scores, used by
AUC and ROCplot's merge mode, and .calcArea works on vectors. Values are
identical to the loop implementations, including tie handling and rounding.
A class absent from a group gives a clear error instead of 'missing value
where TRUE/FALSE needed'; a sample never predicted gets NA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
Fix calcExternalPerformance, grouping by fold and easyHard
e5a9b3e 
- calcExternalPerformance matches predicted classes to actual classes by
  name instead of relabelling levels by position, errors on unknown
  predicted classes, and calculates every requested metric (the loop used
  the whole vector of metric names).
- grouping = "fold" averages C-index and AUC within each permutation into a
  plain named numeric vector with permutations in numerical order.
- easyHard drops samples with missing performance and matches assay rows by
  sample name; its assay and performanceType defaults work (they are no
  longer generic arguments); MultiAssayExperiment assays and "clinical"
  are usable and a matrix assay is converted to a DataFrame.
- Documentation: the definitions of precision and recall were swapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
samplesMetricMap: match samples by name and fix failing options
bfb9906 
- Each result's sample-wise metric is matched to the first result's samples
  by name, and results must predict the same set of sample names.
- comparison = "Cross-validation" uses the characteristic like any other.
- showLegends = FALSE, a single result and a class with one sample work.
- More than two classes get a white-to-dark gradient per class by default,
  one legend per class; too few user colours give an error.
- The matrix method's class and group bars keep their first and last
  tiles (limits = c(1, n) dropped them) and its clustering works.
- Legends are taken from small plots with one tile per colour; rendered
  output of the list method is byte-identical to before.
- Rd: metric values listed "Sample Error" twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
ROCplot: fix average mode, keep the user's theme, use linewidth
b19cba0 
Average mode split a DataFrame and subset() with class = aClass (an unused
argument). The plot theme is added to each plot instead of theme_set().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
performancePlot: respect yLimits when rotated, keep the user's theme
a20f71c 
rotate90 = TRUE replaced user yLimits by c(0, 1) and turned a character x
characteristic into NA. Documentation of the box plot default, aggregate
and the return value corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
rankingPlot: use sizesList fonts, row/column characteristics and shor…
afa5242 
…t rankings

Fonts were read from sizesList$fontSizes instead of the documented
$fonts; row and column values compared to the whole list; ordering by
performance lacked the metric; a ranking shorter than top counted NA as a
shared feature. Theme added to the plot instead of theme_set().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
selectionPlot: fix importance mode and commonality tables
dc0e004 
Importance mode: undefined plot and colourVariable, keepFeatures compared
names to counts, y mapped to a constant string, facets were subset by one
characteristic at a time. Commonality: duplicate overlap column, and with a
reference level the characteristics were looked up in the whole table and
repeated the wrong number of times. alpha reaches the geoms; rotate90
works for importance. Theme added to each plot instead of theme_set().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
@ellispatrick @claude
plotFeatureClasses errors on unknown features; featureSetSummary mess…
3992871 
…age spacing

plotFeatureClasses returned an empty list after printing a message for a
useFeatures not in the data; it now stops. Its theme is added to each plot.
Adds tests for all plotting fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfsjwkbDbHkX4kJZNe4ER4
This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
@DarioS
DarioS marked this pull request as ready for review October 28, 2026 00:35
DarioS
DarioS reviewed Oct 28, 2026
View reviewed changes

@DarioS DarioS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to personally try all of the changes to samplesMetricMap. It looks good at a glance, but it's a lot.

@DarioS
DarioS merged commit 8d32493 into review/2-interface-fixes Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarioS DarioS DarioS left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ellispatrick @DarioS