feat(de): add ls_scheme to choose how the shared length scale is estimated - #22
feat(de): add ls_scheme to choose how the shared length scale is estimated#22settylab-dotto-bot[bot] wants to merge 2 commits into
Conversation
…mated
The two conditions have always been smoothed at a shared length scale so
that their fitted surfaces are comparable. That part is right: giving each
condition its own makes the two surfaces differently smooth, and on an
exchangeable null the mismatch alone roughly doubles the false-positive
rate -- worse than any shared value.
What was undocumented is that the shared value is estimated from condition
1's cells only (`ls_for_model2 = self.model1.ls`), so it is a function of
n_condition1 and de(adata, condition1=X, condition2=Y) is not equivalent to
de(adata, condition1=Y, condition2=X). Measured on a cell-level exchangeable
null (2,917 cells, 16,285 genes, truth = 0 differential genes), the two
orientations of one partition differ in false-positive rate by 0.059 at 2:1,
0.082 at 3:1 and 0.079 at 6:1, against a floor of ~0.004.
`ls_scheme` makes the choice explicit without changing anything by default:
"condition1" (default) exactly today's behaviour
"symmetric" size-weighted geometric mean of the two conditions'
own estimates -- the same estimator, with neighbours
looked up within each condition rather than across
the union; swap-invariant
"pooled" estimated from both conditions' cells together; also
swap-invariant, but systematically smaller, since the
union is denser than either condition and the
estimate tracks nearest-neighbour distance
"separate" each condition estimates its own; diagnostics only
It is ignored, with a log line, when `ls` or a `cov_func` is supplied. It
also joins the run-parameter match list, so re-running under a different
scheme is not mistaken for a matching rerun.
Refs settylab/dotto-nexus#310
The `ls_scheme` gate excluded `cov_func` / `cov_func_curry` from the automatic path. Under the default scheme that dropped the historical `ls_for_model2 = self.model1.ls` inheritance, so a caller supplying a custom kernel and never touching `ls_scheme` silently got `separate` behaviour -- the worst-measured scheme in the comparison this branch is based on (tail mass 0.322 vs 0.172 at 2:1 on an exchangeable null). `function_kwargs` is public and the CLI forwards into it, so the path is reachable from the shipped API. The gate is now exactly the historical one -- `ls is None and "ls" not in function_kwargs`. A supplied kernel no longer disables scheme resolution: the resolved value is handed to the custom kernel via `estimator_defaults["ls"]`, which is what the old inheritance did too, so both conditions keep a shared scale. Verified against 19ee1a1 in a clean worktree, default scheme, `ls=None`: no function_kwargs old 92.589588/92.589588 new 92.589588/92.589588 cov_func_curry=Matern52 old 92.589588/92.589588 new 92.589588/92.589588 Three tests added covering the curry path for the default, the shared schemes and `separate`. Caught by an independent skeptic review, which also noted that 17/17 green missed it because no test exercised `function_kwargs`.
Regression fixed —
|
function_kwargs |
old model1.ls / model2.ls |
new |
|---|---|---|
| none | 92.589588 / 92.589588 | 92.589588 / 92.589588 |
cov_func_curry=Matern52 |
92.589588 / 92.589588 | 92.589588 / 92.589588 |
Three tests added for the curry path (default, shared schemes, separate); 20/20 green. The
reason 17/17 missed it is that no test exercised function_kwargs at all — only explicit ls.
The docstring's "ignored when ... cov_func is supplied" was true of the scheme and false of the
behaviour; it now says the resolved value is handed to the custom kernel.
Full skeptic findings are on settylab/dotto-nexus#310. Everything else in the review reproduced or
strengthened the measurements; this was the one merge-blocking item.
Closes the code half of settylab/dotto-nexus#310. Default behaviour is unchanged; this adds
the knob and documents what the current default actually does.
The problem
kompot/differential/differential_expression.pygave condition 2 condition 1's fitted lengthscale:
Sharing one length scale is right — see below — but estimating it from condition 1 alone makes
it a function of
n_condition1, sode(adata, condition1=X, condition2=Y)andde(adata, condition1=Y, condition2=X)are not equivalent. Nothing at the call site says so;sigmais documented as a fixed hyperparameter andls's provenance was invisible.What was measured
Cell-level exchangeable null, 2,917 Young HSPC cells, 16,285 genes, truth = 0 differential genes,
n_landmarks=2500,null_genes=2000,null_seed=42. 236 DE runs. Metric is tail mass =mean(pvalue < 0.05); a correct null gives 0.05.Swap invariance — hold one partition fixed, exchange which side is
condition1, take|Δ tail mass|:
condition1(today's default)symmetricpooledseparateThe floor is measured, not assumed: pinning a fixed shared
lsand swapping gives 0.0007–0.0049across a 0.20–1.20 grid. Every alternative reaches it.
Sharing is right, and
separateshows why. Letting each condition fit its own gives tail mass0.322 at 2:1 — roughly double the default's 0.172, and above every point on the shared-
lsresponse curve, which spans 0.004 (ls=0.20) to 0.235 (ls=1.20). Its pair is (0.593, 0.822), whose
members sit individually at 0.146 and 0.193 on that curve, so ~+0.15 comes from the mismatch
between the two surfaces' smoothness rather than from either level. Hence
separateis documentedas diagnostics-only.
pooledis swap-invariant but systematically small.compute_lsis a geometric mean ofwithin-cloud nearest-neighbour distances, and the union is denser than either condition, so the
estimate shrinks by cell count alone — 24% below what each condition supports at balance, 57%
below what the smaller condition supports at 6:1. Measured shrink at balance is 0.758 =
(2917/1458)^(-1/2.5), i.e. exactly density scaling at an effective dimension of ~2.5.symmetricavoids that: because kompot's estimator isexp(mean(log nn)) * const, thesize-weighted geometric mean of the two per-condition fits is exactly the same estimator
applied to the two conditions' within-condition neighbour distances. It stays close to the
balanced value across the ladder (0.677 -> 0.608 over a 6x ratio) where the default swings
0.543–1.188.
Full comparison, including absolute calibration and the response curve, is on
settylab/dotto-nexus#310. This does not fix settylab/dotto-nexus#309 — the imbalance
dependence survives every scheme (the best still runs 0.006 -> 0.117 -> 0.217 -> 0.342).
What changed
GPSettings.ls_scheme, default"condition1"(byte-identical behaviour), plus"symmetric","pooled","separate". Validated in__post_init__._auto_ls/_resolve_ls_schemeindifferential_expression.py; the historicalls_for_model2 = self.model1.lsinheritance now runs only under"condition1".lsorcov_func/cov_func_curryis supplied.de(), the deprecatedcompute_differential_expression()shim andbuild_params_dict; added to the run-parameter match list, so re-running under a differentscheme is not treated as a matching rerun.
GPSettings.ls/ls_schemeandDifferentialExpression.fitnow state where theshared length scale comes from and that the default makes the contrast depend on argument order.
tests/test_ls_scheme.py— 17 tests, including the default's non-invariance pinned as acharacterisation test so a future change to it is deliberate.
[Unreleased].Verification
The measurement harness reached these schemes by handing kompot an explicit
ls(plus a probethat forces self-fitting); this PR reaches them through
ls_scheme. Both paths were run on thesame split and agree bit-identically:
condition10.140375,pooled0.121216,separate0.340436,
symmetric0.166779. So the PR is the thing that was measured.Full suite: 2021 passed, 49 failed, 32 skipped. All 49 failures are pre-existing — the same
49 fail at
19ee1a1in a clean worktree, all intest_cli*, allRuntimeError: anndata.settings.allow_write_nullable_strings is Falsefrom anndata 0.12.6. Filedseparately; untouched here.
Not decided here
Which scheme should be the default. Swap invariance does not discriminate — every alternative
reaches the floor — and absolute calibration separates them only through a monotone response to
the number they pick, which is
#309-adjacent territory. I would default to"symmetric", butthe default is deliberately left alone pending @katosh's call on settylab/dotto-nexus#310.