Implements real APS & dynamic scoring for Conformal Prediction - #1189
Merged
Conversation
Adds pyhealth/calib/predictionset/scores.py, a shared score module
implementing genuine Adaptive Prediction Sets (Romano, Sesia, and
Candes 2020): nonconformity score = cumulative sum of predicted
probabilities for classes ranked above the target, plus a randomized
U*p(target) term (U ~ Uniform(0,1), one draw per example, shared
across all candidate classes). Previously, BaseConformal's
score_type="aps" was silently just an alias for "threshold" and did
not implement APS at all.
Threads a new score_type parameter ("threshold" [default, unchanged
behavior] or "aps") through BaseConformal, LABEL, ClusterLabel,
CovariateLabel, and NeighborhoodLabel, each with an optional
random_state for reproducible APS randomization. SCRIB and FavMac are
intentionally excluded since their calibration isn't a score-then-
quantile pattern.
Verified via numpy-only synthetic tests: both score types hit ~90%
empirical coverage at alpha=0.1, for marginal and class-conditional
coverage, in both nonconformity and conformity sign conventions.
Adds >>> usage examples to the 4 public functions in pyhealth/calib/predictionset/scores.py (verified against real computed output). Adds a score_type="aps" usage example to the docstrings of BaseConformal, LABEL, ClusterLabel, CovariateLabel, and NeighborhoodLabel. Adds tests/core/test_scores.py covering both score types: threshold backward-compatibility, the APS formula's hand-computable non-randomized case, monotonicity, reproducibility under a seeded RNG, nonconformity/ conformity complementarity, and empirical marginal coverage at the target alpha. Extends test_cluster_label.py, test_covariate_label.py, and test_neighborhood_label.py with score_type="aps" end-to-end cases. Documents the score_type argument and adds the previously-missing BaseConformal entry to docs/api/calib/pyhealth.calib.predictionset.rst.
CI's ruff install (pip install 'ruff~=0.15') resolves to the latest 0.x release under PEP 440 compatible-release semantics, which enabled more pyupgrade/ruff-specific default rules than the older cached ruff used for local verification. Fixes all 13 flagged violations: Optional[X]/Union[X, Y] -> X | Y, typing.Dict -> dict (including the now-modernized pre-existing forward() return annotations this forced), an unused unpacked variable, and an unsorted __all__. Verified by reproducing the CI's exact environment: a clean venv with `pip install 'ruff~=0.15'` (which also resolves to 0.16.3), confirming `tools/check_pr_rules.py` now passes.
Collaborator
|
Any idea on how long these tests take? |
Copilot stopped reviewing on behalf of
jhnwu3 due to an error
August 17, 2026 14:28
jhnwu3
approved these changes
Aug 18, 2026
jhnwu3
left a comment
jhnwu3
left a comment
Collaborator
There was a problem hiding this comment.
lgtm. Lmk if any issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.