Adding pred_measure feature

[florence-bockting]

Fixes #281

Summary

Adds a unified predictive performance measures API on top of loo-v3.0.0. Density scores, scoring rules, and point-prediction metrics can be computed under in-sample, PSIS-LOO, k-fold, and holdout evaluation via *_pred_measure().

Design log: developer-notes.md (edit: developer-notes.Rmd)

Issues addressed

This PR consolidates work that was tracked across several related issues:

• Fixes (loo_)(s)crps could ask for only one argument with predictions #223: scoring rules use a single ypred draw matrix (measure_rps(), measure_srps())
• Fixes loo_predictive_metric and loo_crps could accept psis objects to reduce computation #213: loo_pred_measure() accepts a pre-computed psis_object (or extracts one from a loo result with save_psis = TRUE)
• Fixes Add R2 #201 / user defined loss/utility functions #135: custom measures via user-supplied functions; measure_r2() as a built-in
• Fixes loo_compare for crps and loo_crps #220 (partial): CRPS/RPS and other measures under in-sample, LOO, k-fold, and holdout via *_pred_measure() — loo_compare integration still outstanding (see Known limitations)

How to navigate this PR

This is a large PR. These documents explain design, scope, and current status:

• developer-notes.md:
  central reference: status, open tasks, design decisions, numerical comparisons
• overview-measures.Rmd: supported scores and metrics, with mathematical definitions
• pred-measure-workflow.Rmd: conceptual introduction and end-to-end workflow examples
• migration-guide.Rmd: mapping from deprecated functions to the new API

What's new

• Entry points: insample_pred_measure(), loo_pred_measure(), kfold_pred_measure(), test_pred_measure(), pred_measure()
• Built-in measures: measure_*() + supported_measures_list() (incl. r2, classification metrics, Brier, MAE/MSE/RMSE, ELPD/IC)
• Unified scoring rule: measure_rps() / measure_srps() — single ypred matrix, PWM/ECDF estimator; covers continuous and ordered categorical outcomes (CRPS/RPS and scaled variants in the new API)
• Custom measures: pass a function (or named list of functions) to measure
• PSIS reuse: supply psis_object or reuse weights from a prior loo_pred_measure() result via predperf
• S3 print methods for pred_measure objects (incl. Pareto-k-hat for LOO)
• Docs: migration-guide vignette + website-only overview-measures and pred-measure-workflow articles
• Tests & CI: test suite, pre-fitted .Rds fixtures, pkgdown vignette-data generation step

Deprecations (implementations retained)

Existing functions are deprecated but still work with their current APIs:

Deprecated	New workflow
elpd()	measure_elpd() / *_pred_measure()
crps(), scrps() (x, x2)	measure_rps(), measure_srps() (ypred)
loo_crps(), loo_scrps()	loo_pred_measure(..., measure = "rps" | "srps")
loo_predictive_metric()	loo_pred_measure()

In *_pred_measure(), elpd is computed automatically when ylp is supplied; request measure = "ic" separately for the information criterion.

The new RPS functions use a different estimator (one draw matrix, PWM) than deprecated CRPS (two draw matrices, permutation). Results are highly correlated but not identical — see the comparison section and figures in developer-notes.Rmd. Comparison tests are in tests/testthat/test_crps.R.

Dependencies

Vignettes, test-data generation, and pkgdown CI install brms from GitHub master (paul-buerkner/brms), because k-fold support for categorical / multinomial models (brms#1890, fixing brms#1889) is not yet on CRAN. Once a CRAN release includes that fix, the GitHub pin can be dropped.

Known limitations

• group_ids grouping not yet implemented
• loo_compare integration still outstanding
• Additional documentation still in progress (formula derivations article, glossary extension)
• brms GitHub pin remains until a CRAN release includes the categorical k-fold fix

See developer-notes.Rmd for open design questions and
remaining tasks.

Acknowledgements

This work builds on an initial prototype of the pred_measure feature implemented by @VisruthSK (see VisruthSK/loo-sandbox) under guidance from @jgabry.

Test plan

• devtools::test() and devtools::check() pass
  • devtools::test(): 1 failure — unrelated test_psislw.R snapshot drift (Warning in psislw() → Warning:)
  • R CMD check: 1 error — kfold_pred_measure() roxygen example uses measure = "rmse" without mupred
• Deprecated functions warn (test_deprecated_measures.R)
• Deprecated functions return same results (snapshot tests in test_crps.R; not re-diffed against loo-v3.0.0 branch)
• CRPS/RPS comparison tests pass (test_crps.R, 32 tests)
• New *_pred_measure() examples run (migration guide + workflow vignette; rendered locally with NOT_CRAN=true)
• pkgdown site builds with GitHub brms (workflow + migration-guide articles render; full pkgdown::build_site() not verified locally — existing docs/ blocks build)

[github-actions]

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 884df6e is merged into loo-v3.0.0:

• ✔️loo_function: 1.57s -> 1.57s [-0.69%, +0.41%]
• ❗🐌loo_matrix: 1.54s -> 1.56s [+0.95%, +2.05%]
  Further explanation regarding interpretation and methodology can be found in the documentation.

[florence-bockting]

Linking here to Issue #179 as it seems to be about loo scores. We should see whether it is still relevant and can be implemented within loo refactoring.

• Add issue Calibration/Refinement Decomposition of LOO Scores #179 to PR description text if we can address it.

[florence-bockting]

There is an open issue requesting a subset() method for psis objects #343 and an (unrelated) open PR #110. Might be worth considering whether we can address this as well in this refactoring PR. If yes, we should

• close PR Subsetting psis objects #110 with link to this PR and also
• [ ] mark issue Feature Request: public subset() method for psis objects #343 as fixed in the description text of this PR.

Moved this to PR #379

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.51002% with 214 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.45%. Comparing base (fe16a43) to head (884df6e).

Files with missing lines	Patch %	Lines
R/pred_measure-builtin.R	73.31%	111 Missing ⚠️
R/pred_measure-helpers.R	74.56%	59 Missing ⚠️
R/pred_measure-compute.R	91.13%	25 Missing ⚠️
R/print.R	75.64%	19 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           loo-v3.0.0     #363      +/-   ##
==============================================
- Coverage       92.70%   89.45%   -3.26%     
==============================================
  Files              31       35       +4     
  Lines            3029     4124    +1095     
==============================================
+ Hits             2808     3689     +881     
- Misses            221      435     +214     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.