Add InterpCRPSDiT for CRPS temporal interpolation - #987
Conversation
…nterface DataReplay is a weightless prognostic adapter (torch.nn.Module + PrognosticMixin) that steps any DataSource through the create_iterator interface, yielding the source's own reanalysis/analysis frames instead of a forecast rollout -- e.g. supplying observed frames to a downstream model, providing a reference trajectory to score forecasts against, or sub-sampling a finer source in time. Complements Persistence (which echoes the initial state forward); for the forecast-as-trajectory case, prefer ForecastSource. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Greptile SummaryThis PR introduces two new prognostic model classes —
|
| Filename | Overview |
|---|---|
| earth2studio/models/px/interpcrpsdit.py | New 1110-line InterpCRPSDiT model; core interpolation logic, halo/sub-domain handling, and coordinate bookkeeping are correct, with P2 issues around a misleading variance comment in _gaussian_blur_valid and the placeholder load_default_package URL |
| earth2studio/models/px/datareplay.py | New DataReplay prognostic adapter; batch-dim handling, grid validation, hook contract, and step arithmetic are all correct and well-tested |
| test/models/px/test_interpcrpsdit.py | Comprehensive 1238-line test suite covering gap validation, endpoint pinning, drop_variables, set_domain, amp_dtype, seeded noise, and multi-time init; test math and dummy DiT contracts are correct |
| test/models/px/test_datareplay.py | Good coverage of step validation, grid-mismatch guard, finite-value guard, hook contracts, and coordinate ordering; all test assertions are mathematically correct |
| examples/02_medium_range/07_crps_temporal_interpolation.py | Clear worked example; post-construction mutation of sub.num_interp_steps is an undocumented API pattern, and the example will fail at runtime until weights are hosted |
| pyproject.toml | Correctly adds the interp-crps-dit extra and uv conflict declarations to prevent resolution failures with da-healda's physicsnemo cap |
| earth2studio/models/px/init.py | Adds DataReplay and InterpCRPSDiT imports in alphabetical order; no issues |
| test/conftest.py | Correctly registers test_interpcrpsdit.py with the interp-crps-dit optional-dependency guard; DataReplay has no optional deps so test_datareplay.py is correctly left unregistered |
Comments Outside Diff (2)
-
earth2studio/models/px/interpcrpsdit.py, line 1163-1171 (link)_trimwithbot=0orright=0— behavior is correct but a comment would helpThe slice
x[..., top : h - bot, left : w - right]correctly handlesbot=0andright=0via integer arithmetic (h - 0 = h,w - 0 = w), avoiding any negative-index issue. The early-return guard fires only when all four halo values are zero. This is fine as-is, but a short comment noting thatbot=0/right=0are intentionally handled byh-0=harithmetic would help future readers distinguish the correct intent from an apparent off-by-one.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
-
examples/02_medium_range/07_crps_temporal_interpolation.py, line 1678 (link)Direct attribute mutation post-
set_domainas an undocumented API patternsub.num_interp_steps = 24mutates a public attribute after object construction. Becauseset_domaininheritsnum_interp_stepsfrom the parent, users who want a different cadence for the sub-domain must mutate it afterwards. The example is the primary documentation of this workflow, implicitly establishing mutation as the intended API. Ifnum_interp_stepsis later cached or used to derive other state at construction time, this pattern could silently break. Consider acceptingnum_interp_stepsas a parameter toset_domain, or documenting the mutation explicitly in theset_domaindocstring.
Reviews (1): Last reviewed commit: "Add InterpCRPSDiT: one-shot endpoint-pin..." | Re-trigger Greptile
| Real cells added per side then trimmed off the output (boundary-artifact guard), by default 0. | ||
| min_cells : int | None, optional | ||
| Per-side floor on the run grid (NATTEN kernel must fit the latent). Defaults to the model's | ||
| ``_min_domain_cells`` (derived from the architecture at load, ``attn_kernel x patch``; 64 before loading). | ||
|
|
||
| Returns | ||
| ------- | ||
| InterpCRPSDiT | ||
| A fixed sub-domain model with its own ``input_coords`` (run grid) / ``output_coords`` (bounding box). | ||
|
|
||
| Raises | ||
| ------ | ||
| ValueError | ||
| If the bounding box is degenerate, ``halo`` is negative, ``min_cells`` is less than 1, the |
There was a problem hiding this comment.
Misleading unit-variance claim in blur normalizer
The comment states "unit-variance white noise stays unit-variance", but this is mathematically incorrect. A separable 2D Gaussian blur (two successive 1D convolutions, each with normalized kernel k) multiplies the input variance by (sum(k_i²))², not sum(k_i²). Dividing by (k * k).sum() = sum(k_i²) only removes one factor, leaving output variance = sum(k_i²) — which equals 1 only for a Dirac kernel. For the largest noise scale (scale=64, σ=32, r=96), sum(k_i²) is roughly 1/(2·96+1) ≈ 0.005, so the "unit-variance" noise is actually much weaker than claimed.
This does not affect inference correctness if the training code used the identical normalization (as stated by "Reproduces the noise generator used at training"), but the incorrect claim could mislead future maintainers into "fixing" the normalization in a way that breaks the training/inference match.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
For a normalized 1d kernel, each blur pass scales the marginal white-noise variance by the sum of the squared kernel weights. The two perpendicular passes therefore apply this factor twice. Dividing the output values by that factor applies the inverse factor twice to the variance, restoring unit marginal variance. This matches the training-time noise generator and is retained for consistency.
There was a problem hiding this comment.
You're correct, and my original comment was wrong. I conflated scalar value division with variance scaling: dividing the output values by c divides the variance by c², not c. So dividing by sum(k_i²) restores unit marginal variance exactly — two convolution passes each multiply variance by sum(k_i²), giving sum(k_i²)², and dividing values by sum(k_i²) divides variance by sum(k_i²)², which fully cancels it. The comment and normalization are correct.
Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.
| x0n[:, ti, lti], x1n[:, ti, lti], cond, tau, z[:, ti] | ||
| ) | ||
| emitted = self._trim(out * self.scale + self.center) | ||
| if len(self.drop_idx): # emit only the present (non-dropped) variables | ||
| emitted = emitted[..., self.present_idx, :, :] | ||
| yield (emitted, sub) | ||
|
|
||
| # ------------------------------------------------------------------ public | ||
| @batch_func() | ||
| def __call__( | ||
| self, x: torch.Tensor, coords: CoordSystem | ||
| ) -> tuple[torch.Tensor, CoordSystem]: | ||
| """Run one step: return the initial condition (step 0). | ||
|
|
||
| Parameters | ||
| ---------- | ||
| x : torch.Tensor | ||
| Input tensor. | ||
| coords : CoordSystem |
There was a problem hiding this comment.
Placeholder URL will cause a runtime failure for any user calling
load_default_package
load_default_package currently returns hf://nvidia/earth2studio-interp-crps-dit, which does not yet exist. Any user who calls InterpCRPSDiT.load_default_package() (e.g. by following the example as written) will get a hard runtime error. The TODO comment makes this clear to developers, but the public example links directly to load_default_package() without a visible guard. Before removing the draft label, the URL should be pinned to the real commit hash, or the example and class docstring should prominently state that this method will fail until the weights are hosted.
…ope (Greptile P2) - output_coords: compute lead_time in one step (the intermediate value was read, not dead code, but inlining removes the confusing two-step assignment). - __call__: Notes section stating front_hook/rear_hook fire only in create_iterator (matching Persistence; hooks are iterator-scoped per PrognosticMixin). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
InterpCRPSDiT upsamples a base model's coarse trajectory to finer sub-steps (6 h -> hourly, down to sub-hourly) with a DiT backbone (natten2d_rope), one shot per frame (no iterative solver). The bracket endpoints are the base frames (verbatim); the interior is a CRPS-trained correction whose sin(pi tau) envelope vanishes at the endpoints, with ensemble spread from a per-member latent. Regional sub-domain inference via set_domain. A base that supplies only VARIABLES minus the optional channels (e.g. Pangu's 69) drops in via drop_variables. Adds the interp-crps-dit optional-dependency extra (natten + physicsnemo>=2.2.0), the temporal-interpolation example, and docs. Depends on DataReplay (used by the example and tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er package) load_default_package() now emits a runtime warning and a prominent docstring/class warning that the returned Hugging Face URL is a placeholder and will fail to download until the weights are published -- pass a local Package to load_model until then. Addresses Greptile P2 review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ile P2) The separable blur multiplies marginal (per-pixel) white-noise variance by S**2 (each 1D pass contributes S=sum(k**2)); dividing by S restores unit marginal variance. The prior wording could be misread as leaving variance ~sum(k**2). Comment-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Description
Add
InterpCRPSDiT, an endpoint-pinned CRPS temporal-interpolation model that refines a base model’s coarse trajectory to hourly and sub-hourly resolution. For a 6 h bracket,num_interp_steps=6, 24, and 36 produce 1 h, 15 min, and 10 min output, respectively. The requested cadence must be a whole number of minutes and evenly partition the coarse gap.For each interior fraction
tauthe field is produced in a single forward asout(tau) = (1−tau)·x0 + tau·xT + sin(pi·tau)·f(x0, xT, cond, z)consisting of a linear base plus a DiT correction whosesin(pi·tau)envelope vanishes at both endpoints. The bracket endpoints are the base model's own frames and only the interior is learned. A regional sub-domain is available viaset_domain. A base does not have to provide all 73 variables. If the only ones it lacks are the four optional channels (sp, u100m, v100m, tcwv), pass them todrop_variables. The model then no longer requires them from the base model but it fills them internally and omits them from the output.Closes #986
Blocked by:
nvidia-physicsnemo2.2.0 on PyPI — needed to resolveuv.lock(2.2.0 is the first release withnatten2d_rope+proj_reshape_2d_conv, PR #1731). Until thenuv lockcannot resolve the extra, so this PR stays a draft.load_default_packagecurrently returns a placeholderhf://nvidia/earth2studio-interp-crps-dit; needs the bundle hostedModel details
natten2d_ropeneighborhood attention (physicsnemoModule)num_interp_steps(trained gaps 3–10 h; e.g. 6 h/6 = 1 h). Base gap auto-detected from the wrapped model.73 − drop_variablesdrop_variableshf://nvidia/earth2studio-interp-crps-dit(TODO: host +@commit)CRPSModel.mdlus376 MiB, gridset_phys.nc5.6 MiB, plus norm.npy+config.json)Dependencies added
The model has the
interp-crps-ditoptional-dependency extra.nattennatten2d_ropeattentionnvidia-physicsnemo>=2.2.0natten2d_ropelayer +Modulecheckpoint format (first in 2.2.0 / PR #1731)Checklist
models_px.rst,install.md).CHANGELOG.mdis up to date with these changes.DataReplay) merged — this PR is stacked on it (example + tests importDataReplay).load_default_packageURL pinned with@commit.nvidia-physicsnemo2.2.0 available on PyPI (unblocksuv.lock).