Skip to content

feat(de): add ls_scheme to choose how the shared length scale is estimated - #22

Open
settylab-dotto-bot[bot] wants to merge 2 commits into
mainfrom
dominik/310-ls-scheme
Open

feat(de): add ls_scheme to choose how the shared length scale is estimated#22
settylab-dotto-bot[bot] wants to merge 2 commits into
mainfrom
dominik/310-ls-scheme

Conversation

@settylab-dotto-bot

Copy link
Copy Markdown
Contributor

Closes the code half of settylab/dotto-nexus#310. Default behaviour is unchanged; this adds
the knob and documents what the current default actually does.

The problem

kompot/differential/differential_expression.py gave condition 2 condition 1's fitted length
scale:

ls_for_model2 = ls
if ls is None and "ls" not in function_kwargs and self.model1.ls is not None:
    ls_for_model2 = self.model1.ls

Sharing one length scale is right — see below — but estimating it from condition 1 alone makes
it a function of n_condition1, so de(adata, condition1=X, condition2=Y) and
de(adata, condition1=Y, condition2=X) are not equivalent. Nothing at the call site says so;
sigma is documented as a fixed hyperparameter and ls's provenance was invisible.

What was measured

Cell-level exchangeable null, 2,917 Young HSPC cells, 16,285 genes, truth = 0 differential genes,
n_landmarks=2500, null_genes=2000, null_seed=42. 236 DE runs. Metric is tail mass =
mean(pvalue < 0.05); a correct null gives 0.05.

Swap invariance — hold one partition fixed, exchange which side is condition1, take
|Δ tail mass|:

scheme 1:1 2:1 3:1 6:1
condition1 (today's default) 0.0015 0.0586 0.0820 0.0788
symmetric 0.0013 0.0045 0.0016 0.0016
pooled 0.0006 0.0061 0.0016 0.0017
separate 0.0015 0.0037 0.0046 0.0028

The floor is measured, not assumed: pinning a fixed shared ls and swapping gives 0.0007–0.0049
across a 0.20–1.20 grid. Every alternative reaches it.

Sharing is right, and separate shows why. Letting each condition fit its own gives tail mass
0.322 at 2:1 — roughly double the default's 0.172, and above every point on the shared-ls
response curve, which spans 0.004 (ls=0.20) to 0.235 (ls=1.20). Its pair is (0.593, 0.822), whose
members sit individually at 0.146 and 0.193 on that curve, so ~+0.15 comes from the mismatch
between the two surfaces' smoothness rather than from either level. Hence separate is documented
as diagnostics-only.

pooled is swap-invariant but systematically small. compute_ls is a geometric mean of
within-cloud nearest-neighbour distances, and the union is denser than either condition, so the
estimate shrinks by cell count alone — 24% below what each condition supports at balance, 57%
below what the smaller condition supports at 6:1. Measured shrink at balance is 0.758 =
(2917/1458)^(-1/2.5), i.e. exactly density scaling at an effective dimension of ~2.5.

symmetric avoids that: because kompot's estimator is exp(mean(log nn)) * const, the
size-weighted geometric mean of the two per-condition fits is exactly the same estimator
applied to the two conditions' within-condition neighbour distances. It stays close to the
balanced value across the ladder (0.677 -> 0.608 over a 6x ratio) where the default swings
0.543–1.188.

Full comparison, including absolute calibration and the response curve, is on
settylab/dotto-nexus#310. This does not fix settylab/dotto-nexus#309 — the imbalance
dependence survives every scheme (the best still runs 0.006 -> 0.117 -> 0.217 -> 0.342).

What changed

  • GPSettings.ls_scheme, default "condition1" (byte-identical behaviour), plus "symmetric",
    "pooled", "separate". Validated in __post_init__.
  • _auto_ls / _resolve_ls_scheme in differential_expression.py; the historical
    ls_for_model2 = self.model1.ls inheritance now runs only under "condition1".
  • Ignored, with a log line, when ls or cov_func/cov_func_curry is supplied.
  • Threaded through de(), the deprecated compute_differential_expression() shim and
    build_params_dict; added to the run-parameter match list, so re-running under a different
    scheme is not treated as a matching rerun.
  • Docstrings on GPSettings.ls / ls_scheme and DifferentialExpression.fit now state where the
    shared length scale comes from and that the default makes the contrast depend on argument order.
  • tests/test_ls_scheme.py — 17 tests, including the default's non-invariance pinned as a
    characterisation test so a future change to it is deliberate.
  • CHANGELOG under [Unreleased].

Verification

The measurement harness reached these schemes by handing kompot an explicit ls (plus a probe
that forces self-fitting); this PR reaches them through ls_scheme. Both paths were run on the
same split and agree bit-identically: condition1 0.140375, pooled 0.121216, separate
0.340436, symmetric 0.166779. So the PR is the thing that was measured.

Full suite: 2021 passed, 49 failed, 32 skipped. All 49 failures are pre-existing — the same
49 fail at 19ee1a1 in a clean worktree, all in test_cli*, all
RuntimeError: anndata.settings.allow_write_nullable_strings is False from anndata 0.12.6. Filed
separately; untouched here.

Not decided here

Which scheme should be the default. Swap invariance does not discriminate — every alternative
reaches the floor — and absolute calibration separates them only through a monotone response to
the number they pick, which is #309-adjacent territory. I would default to "symmetric", but
the default is deliberately left alone pending @katosh's call on settylab/dotto-nexus#310.

…mated

The two conditions have always been smoothed at a shared length scale so
that their fitted surfaces are comparable. That part is right: giving each
condition its own makes the two surfaces differently smooth, and on an
exchangeable null the mismatch alone roughly doubles the false-positive
rate -- worse than any shared value.

What was undocumented is that the shared value is estimated from condition
1's cells only (`ls_for_model2 = self.model1.ls`), so it is a function of
n_condition1 and de(adata, condition1=X, condition2=Y) is not equivalent to
de(adata, condition1=Y, condition2=X). Measured on a cell-level exchangeable
null (2,917 cells, 16,285 genes, truth = 0 differential genes), the two
orientations of one partition differ in false-positive rate by 0.059 at 2:1,
0.082 at 3:1 and 0.079 at 6:1, against a floor of ~0.004.

`ls_scheme` makes the choice explicit without changing anything by default:

  "condition1" (default)  exactly today's behaviour
  "symmetric"             size-weighted geometric mean of the two conditions'
                          own estimates -- the same estimator, with neighbours
                          looked up within each condition rather than across
                          the union; swap-invariant
  "pooled"                estimated from both conditions' cells together; also
                          swap-invariant, but systematically smaller, since the
                          union is denser than either condition and the
                          estimate tracks nearest-neighbour distance
  "separate"              each condition estimates its own; diagnostics only

It is ignored, with a log line, when `ls` or a `cov_func` is supplied. It
also joins the run-parameter match list, so re-running under a different
scheme is not mistaken for a matching rerun.

Refs settylab/dotto-nexus#310
The `ls_scheme` gate excluded `cov_func` / `cov_func_curry` from the
automatic path. Under the default scheme that dropped the historical
`ls_for_model2 = self.model1.ls` inheritance, so a caller supplying a
custom kernel and never touching `ls_scheme` silently got `separate`
behaviour -- the worst-measured scheme in the comparison this branch is
based on (tail mass 0.322 vs 0.172 at 2:1 on an exchangeable null).

`function_kwargs` is public and the CLI forwards into it, so the path is
reachable from the shipped API.

The gate is now exactly the historical one -- `ls is None and "ls" not in
function_kwargs`. A supplied kernel no longer disables scheme resolution:
the resolved value is handed to the custom kernel via
`estimator_defaults["ls"]`, which is what the old inheritance did too, so
both conditions keep a shared scale.

Verified against 19ee1a1 in a clean worktree, default scheme, `ls=None`:

    no function_kwargs        old 92.589588/92.589588  new 92.589588/92.589588
    cov_func_curry=Matern52   old 92.589588/92.589588  new 92.589588/92.589588

Three tests added covering the curry path for the default, the shared
schemes and `separate`. Caught by an independent skeptic review, which
also noted that 17/17 green missed it because no test exercised
`function_kwargs`.
@settylab-dotto-bot

Copy link
Copy Markdown
Contributor Author

Regression fixed — 487a02e

An independent skeptic review found a real defect in the first commit, and it would have shipped.

The ls_scheme gate excluded cov_func / cov_func_curry from the automatic path. Under the
default scheme that dropped the historical ls_for_model2 = self.model1.ls inheritance, so a
caller who supplied a custom kernel and never touched ls_scheme silently got "separate"
behaviour — the worst-measured scheme in the comparison this branch is based on. function_kwargs
is public and kompot/cli/de.py forwards into it, so the path is reachable from the shipped API.
It falsified this PR's claim that nothing changes unless ls_scheme is set.

The gate is now exactly the historical one, ls is None and "ls" not in function_kwargs. A
supplied kernel no longer disables scheme resolution — the resolved value reaches the custom kernel
through estimator_defaults["ls"], which is what the old inheritance did as well, so both
conditions keep a shared scale.

Verified against 19ee1a1 in a clean worktree, default scheme, ls=None:

function_kwargs old model1.ls / model2.ls new
none 92.589588 / 92.589588 92.589588 / 92.589588
cov_func_curry=Matern52 92.589588 / 92.589588 92.589588 / 92.589588

Three tests added for the curry path (default, shared schemes, separate); 20/20 green. The
reason 17/17 missed it is that no test exercised function_kwargs at all — only explicit ls.

The docstring's "ignored when ... cov_func is supplied" was true of the scheme and false of the
behaviour; it now says the resolved value is handed to the custom kernel.

Full skeptic findings are on settylab/dotto-nexus#310. Everything else in the review reproduced or
strengthened the measurements; this was the one merge-blocking item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant