Skip to content

Reuse cumulative counts when building classification tables - #1997

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:perf/classification-table-cumulative-counts-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:perf/classification-table-cumulative-counts-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

calculate_pr_table and calculate_lift_table recompute true-positive counts from the start of the sorted population at every percentile. This repeatedly scans approximately 20 and 100 overlapping prefixes, respectively, even though each table advances monotonically through the same binary target sequence.

Change

Accumulate TP counts only over the newly included interval. Keep sorting, stable tie order, percentile boundaries, thresholds, rounding and returned rows unchanged. The change uses only the current interval slice rather than allocating a full-length cumulative-count array.

Validation

  • 28 new numerical and integration cases pass on both the original and changed implementations; this is a performance change, not a numerical bug fix.
  • With the four existing classification calculation tests: 32 passed on CPU.
  • Coverage includes empty/singleton inputs, percentile boundary sizes, non-divisible tails, tied scores, and native binary/multiclass current/reference Reports. Expected rows independently sum their selected populations.
  • Repository-pinned Ruff 0.3.7 lint/format and git diff --check pass.

Local CPU benchmark, one warm-up and five timed calls per case, seeded shuffled binary targets and scores, including each function's sorting but excluding construction/copying of its input list:

Input rows PR table before/after Lift table before/after
10,000 2.729 / 1.576 ms 8.585 / 2.758 ms
100,000 31.454 / 18.258 ms 101.109 / 20.950 ms
500,000 642.536 / 138.943 ms 1293.687 / 129.687 ms

These are median local measurements, not end-to-end Report or training benchmarks. Every benchmark returned an identical table representation before and after. Sorting still determines the overall O(N log N) complexity; the change eliminates repeated target-prefix scans. The full repository and notebook smoke suites were not run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant