Replace spectraldiff's extension hyperparameters with detrend + odd extension?

[pavelkomarov]

Idea

spectraldiff currently manages the FFT's implicit periodicity with two boolean hyperparameters, even_extension and pad_to_zero_dxdt, the second of which pads 100 samples on each end and smooths them in. Section 7.5.1 of the taxonomy paper already notes the weakness: an even extension removes the value discontinuity across the wrap but "can still result in a corner if the signal edges are not flat," and a corner in the signal is a jump in its derivative.

A more principled treatment removes both problems in three steps, each with a reason:

1. Detrend — subtract the straight line joining the endpoints, so x[0] == x[N-1] and the periodic wrap closes.
2. Odd-extend through the last endpoint — x[N+k] = 2*x[N-1] - x[N-2-k], which continues value and slope, so no corner. Reflecting x[0] as well would repeat it across the wrap, a flat spot rather than a continuation, so that one sample is left out and the extension is 2N-2.
3. Differentiate, then add the slope back — exact, since the derivative of a line is a constant.

This drops both hyperparameters and the padding block, and the extension is 2N-2 rather than the 2N of even extension.

Results

On noiseless analytic signals at a fixed cutoff_freq=0.2, it is much better (max relative error):

signal	even + pad	detrend + odd ext
sin	1.52e+00	5.66e-15
ramp + wiggle	1.47e+00	2.00e-02
exp	1.41e+00	6.80e-03
quadratic	1.43e+00	3.38e-03

But that is not the case that matters, and the honest result is worse. Optimizing both variants with pynumdiff.optimize over six sims x two step sizes x two noise levels, the new version is a slight net loss: geometric mean of new/old RMSE 1.028, better in 11 of 24, median ratio 1.029. It wins clearly on lin_auto (0.495, 0.593) and on coarse dt generally, and loses worst on cruise at high noise (2.063) and pop_dyn (1.396, 1.576).

The reconciliation: on noisy data with the cutoff optimized, the low-pass filter dominates and edge treatment is second order, while the two booleans give the optimizer freedom to pick whichever treatment suits each signal. Removing two degrees of freedom costs about what the better extension gains.

Open questions

• Keep the odd extension but retain a flag, so the optimizer can still choose? Recovers the freedom, keeps some complexity.
• Is detrending alone the win, independent of the extension change? It has the cleanest justification and may be carrying the lin_auto improvement by itself.
• Six error-bound cells move in each direction, so spectraldiff's test bounds need regenerating either way.

A branch, spectraldiff-odd-extension, has a working implementation with the search space trimmed and bounds regenerated; 210 tests pass. Not merged, given the above.

---

Investigated by Claude Opus 5 (Claude Code) on behalf of @pavelkomarov, PyNumDiff maintainer.

[pavelkomarov]

Re-ran the comparison at much larger scale, since the first pass (24 configs) had wide error bars. Six sims x three step sizes (0.005, 0.01, 0.04) x three noise levels (0.5, 1, 2) x four seeds = 216 paired configs, 432 optimizations, each variant optimized independently with pynumdiff.optimize.

The result is worse than the small sample suggested, and now significant:

geomean new/old  1.0823   95% CI [1.0248, 1.1431]
median           1.0582   new better in 86/216

The interval excludes 1.0, so this is not a wash; it is about an 8% loss.

The interesting part is that the loss is not uniform. Broken out by signal:

sim	geomean new/old	new better
lin_auto	0.6703	33/36
sine	0.9369	21/36
triangle	1.0745	11/36
lorenz	1.1260	9/36
cruise	1.1996	5/36
pop_dyn	1.7636	7/36

Detrend-plus-odd-extension is clearly better on lin_auto and sine, and clearly worse on pop_dyn and cruise. Logistic growth is the worst case, which fits: it is monotone and saturating, with near-flat ends and its curvature in the middle, so subtracting the endpoint chord leaves a large hump that the reflection then extrapolates badly. Flat ends are exactly the case where an even extension produces no corner and so has nothing to fix.

That suggests the two treatments have complementary failure modes, and that the optimizer's freedom to pick between them is doing real work rather than absorbing noise. Removing even_extension and pad_to_zero_dxdt outright therefore costs more than the better extension gains.

Options that remain live:

• Add the detrend-plus-odd-extension as a third choice alongside the existing two, keeping the search space categorical. Gains the wins on lin_auto and sine without giving up pop_dyn, at the cost of not simplifying anything.
• Test detrending alone, with the existing even extension. It is the piece with the cleanest justification and may carry the lin_auto improvement by itself.

The spectraldiff-odd-extension branch still holds a working implementation if either is worth pursuing.

---

Investigated by Claude Opus 5 (Claude Code) on behalf of @pavelkomarov, PyNumDiff maintainer.

[pavelkomarov]

Hi, me the human.

I decided on a design that allows no extension, detrending alone, even extension, and odd extension after detrending (so your odd extension doesn't create a big gap at the ends of the signal). It purely gives the method more possibilities to choose from, making it more powerful, although a bit harder to optimize, though spectraldiff runs so fast it's honestly not a concern with optimization.

I came at this while interrogating waveletdiff a little more, because it has an extension scheme of its own.