Lineardiff first principles - #227
Merged
Merged
Conversation
…ative Every row of X = A*integral_X + C*B above the last is a tautology, because integral_X[i] comes out bitwise identical to X[i-1] when both are a cumulative_trapezoid of the row beneath, so a shift matrix solves them at exactly zero residual with zero integration constants; since the objective is separable by row and the reconstruction reads only the bottom row, the old code was solving for 2*order^2 unknowns in order to use 2*order of them, and fitting x = a*integral_X + c*B directly reproduces the derivative to 1e-12 relative at orders 2 and 3. The output path had two more inherited layers worth removing: the backward pass and its linear-ramp crossfade, which measured statistically indistinguishable from the forward pass alone (RMSE 0.2038 vs 0.2038 over 24 signals, no positional gradient) because slide_function's centered overlapping windows average away the per-window left-anchoring that made a window directional in the first place, and the closing finitediff of the blended x_hat, which is exactly a [1,2,1]/4 binomial smoother of the derivative and was therefore an untunable low-pass rather than a reconstruction; averaging the per-window derivatives the fit actually produces and integrating once at the end is both simpler and, on 15 signals across three gammas, 22% more accurate at 3.5x the speed. See #223 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_lineardiff already returns both a smoothed x_hat and the derivative the fit produces, so slide_function can kernel-average both directly instead of averaging one and reconstructing the other; integrating the averaged derivative back to x_hat was redundant work on top of an estimate the windows had already made. See #223 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
slide_function hands _lineardiff a slice of the working vector, which numpy makes a view rather than a copy, so the in-place x -= mean introduced in 00f622b subtracted each window's local mean from the shared array and left every later overlapping window fitting progressively corrupted data. Restoring the copying form fixes six test_diff_method failures and one test_multidimensionality failure that are red on master right now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The centered window gets its own name so the reason it must not be an in-place subtraction is visible at the point of use. Fitting six parameters to eleven samples was never a defensible test point, and the pipeline change makes it worse rather than papering over it with a binomial smoother, so the test now slides a 21-sample window over the 31-sample signals at a gamma suited to their 0.05 noise; the regenerated bounds are tighter than the old ones in nine cells, notably every noisy-derivative column, and looser in three. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
|
Strong results from the optimization run. Actually improved error correlation quite a bit in the large |
pavelkomarov
added
research
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In digging into the math behind
lineardiffI realized there is some weirdness that I think was largely buried by the old code's structure:rtsdiff, because it's not like we can trust one of those sweeps more at one end of the array than the other. Rather, they're both biased by stepping in a single direction, and their biases should in theory help balance each other out, but with all theslide_functionaction going on you really can't see an effect, says Claude, and it doubles the convex solve cost, so I've dropped it.slide_function's derivative estimate was being thrown away, so the elegant formula to extract a derivative from the answer wasn't being properly used. Instead, the smoothed signal estimate was being passed tofinitediff, which basically applies another filter on top. I believe this is why we were seeing a different RMSE-Error-Correlation tradeoff at largedt, when the smoothing caused by thefinitedifffilter "bites", as Claude would say. "The mechanism is exact:finitediff(cumtrapz(d))is[1,2,1]/4 ⊛ dto 6.6e-15, with gain cos²(πf·Δt) — 0.998 at dt=0.005, 0.655 at dt=0.04, f=5. A dt-dependent low-pass wearing a derivative's clothes." -ClaudeAproperly encoding a shift between derivative/integral orders andCzero for those rows, because it has nothing left to fit there. Then because all the norms used, both Frobenius and 1-norm, are essentially pointwise sums, you can split the whole thing however you like and find the convex problem was solving for way more variables than necessary,ris like 2 or 3, but still. The math can be simpler, that is unless you use a matrix norm that entangles all the rows, in which case the tautological ones might have a regularizing effect, but Claude finds no advantage to a spectral norm on whatever subset of the full experiment it could run on my laptop.I'm going to run this version on the server and compare results, and if things look sane, I think this should become the canonical path.