From 305ec600b96ac38728a639a3f3affcdd3adfaec3 Mon Sep 17 00:00:00 2001 From: seabbs-bot Date: Wed, 30 Sep 2026 15:34:15 +0100 Subject: [PATCH 1/4] feat: add the composable recurrences approach page Describes Recurrence and Convolution, how they relate to the other two approaches, and the package map. The map colours come from the theme variables, with a new --epi-accent in both themes. Co-authored-by: Sam Abbott --- approaches/composable-recurrences.qmd | 281 ++++++++++++++++++++++++++ styles.css | 35 ++++ theme-dark.scss | 1 + theme-light.scss | 1 + 4 files changed, 318 insertions(+) create mode 100644 approaches/composable-recurrences.qmd diff --git a/approaches/composable-recurrences.qmd b/approaches/composable-recurrences.qmd new file mode 100644 index 0000000..b18e232 --- /dev/null +++ b/approaches/composable-recurrences.qmd @@ -0,0 +1,281 @@ +--- +title: "Composable recurrences" +description: "Build renewal processes, autoregressions and reporting delays from two fast, differentiable operators that compose." +toc: true +toc-location: left +page-layout: full +--- + +:::::: {.side-space} + +::: {.what-is-turing-grid .approach-hero .section-start-space .section-end-space} + +::: {.feature-text-column} + +::: {.feature-item} +::: {.feature-item-header} + +```{=html} + +``` + +##### Two operators, many models + +::: + +::: + +::: {.feature-item} +::: {.feature-item-header} + +```{=html} + +``` + +##### Fast gradients on every backend + +::: + +::: + +::: {.feature-item} +::: {.feature-item-header} + +```{=html} + +``` + +##### No epidemiology built in + +::: + +::: + +::: + +::: {.code-example-column} +::: {.code-window .box} +::: {.code-window-header} +::: {.code-window-dots} + +```{=html} + +``` + +::: + +::: {.code-window-title} + +```{=html} +renewal_then_delay.jl +``` + +::: + +::: + +```{.julia .code-overflow-scroll} +using ComposableRecurrences + +# Generation interval, lag 1 first: weights on y[t-1], y[t-2], y[t-3] +g = [0.6, 0.3, 0.1] +# Two strata mixed by a contact matrix +K = [0.9 0.1; 0.2 0.8] +infections = Recurrence(g; coupling = K) + +# Reporting delay, lag 0 first: P(delay = 0, 1, 2, 3) +reports = Convolution([0.0, 0.5, 0.3, 0.2]) + +# Operators are callable. R_t is the gain, one column per day +R = fill(1.1, 2, 10) +I = infections(R; history = ones(2, 3)) +Y = reports(I) + +# Stop, keep the state, and resume for a forecast +I1, state = infections(R[:, 1:5]; history = ones(2, 3), + return_state = true) +I2 = infections(R[:, 6:end]; history = state) +I ≈ hcat(I1, I2) +``` + +::: +::: + +::: + +## The approach + +Renewal processes, random walks, autoregressions and reporting delays share one structure. +Each step reads a window of past values, weights it by a kernel, and produces the next value. +This approach builds all of them from two operators rather than one bespoke loop per model. + +A `Recurrence` feeds its output back into its own history. +At each step it convolves the last `L` values with its kernel, mixes strata through a coupling, scales by a gain, adds an input, and passes the result through a tuple of modifiers. +A renewal process is a `Recurrence` with the generation interval as kernel and `R_t` as gain. +A random walk is a `Recurrence([1.0])` with the noise as the added input. + +A `Convolution` has no feedback. +It weights current and past inputs by a kernel, which covers reporting delays, moving averages and occupancy. + +The package has no epidemiological assumptions. +It takes numbers in (kernels, gains, histories) and returns series out. +Modelling packages build their own named steps on top. + +### Lag-first kernels, callable operators + +Every kernel is written lag first, the way a generation interval or AR coefficients are written down. +A `Recurrence` kernel starts at lag 1 and a `Convolution` kernel at lag 0, so nothing is ever reversed by hand. +Operators are built once, outside the model, and called with the per-day inputs that are usually sampled. +Chaining is function composition, as in `reports(infections(R; history))`. + +### Strata, time variation and coupling + +Wrappers name the extra axis of a coefficient. +`PerStratum` gives each stratum its own kernel, `TimeVarying` lets a kernel or coupling change by day, and `Pairwise` gives each pair of strata its own lag kernel. +The coupling mixes strata before the gain, as a contact or mobility matrix does, and can be dense, sparse or structured. +Any array type and any real element type is accepted, so `Float32`, sparse matrices and dual numbers all pass through. + +### Modifiers + +Modifiers are applied in order after each step and carry their own state. +They handle what the linear core cannot, such as susceptible depletion or moving cases between strata. +A modifier is a plain type with an `apply!` method, so a modelling package can add its own. + +### Plug-in adjoints + +Reverse-mode AD is slow when each step copies its lag window into a new state. +The operators own a history buffer instead, so plain AD is already fast. +Hand-written adjoints will then plug in per operator, modifier or coupling by adding a method, with no AD declarations from the user. +`NoAdjoint(op)` turns an adjoint off, so its gain can be timed and the adjoint dropped if a backend catches up. + +In prototypes the buffer alone gave 1.75 to 15 times faster gradients with no custom rules, and analytic adjoints gave 8 to 25 times for delays and renewals. +These were measured on isolated kernels (200 days, 20 lags) under Mooncake and Enzyme, not on whole models. + +### How it fits with the other approaches + +[Composable Turing models](composable-turing-models.qmd) will build on these operators. +Its renewal, delay and latent process steps become calls to `Recurrence` and `Convolution`, with the priors and observation models staying in the Turing layer. +The [composed distributions](composed-distributions.qmd) stack supplies the delay pmfs that become `Convolution` kernels. +The operators have no dependency on either, so they can also be used in other models. + +## Package map + +Solid arrows are data or code that will flow once the package is adopted. +Dashed grey arrows are test-time links. +The dashed amber arrow is an optional future backend, not a dependency. +The ComposableRecurrences box lists the planned first release. + +```{=html} + +``` + +The package has no runtime dependency on any other EpiAware package. +Today each model above runs its own renewal and delay loops, and several of them duplicate each other. +Moving them onto one set of operators gives one forward kernel and one adjoint to maintain and test. + +## Status + +This approach is under development and its public API is landing now. +`Recurrence`, `Convolution`, the `TimeVarying`, `PerStratum` and `Pairwise` wrappers, and the coupling and modifier interfaces exist on a development branch. +The test suite differentiates them with ForwardDiff, Mooncake and Enzyme through plain AD. +Use-case tests compare them against reference implementations of real models. + +Planned next are the built-in modifiers (`Depletion`, `Imports`, `Redistribute`, `Clamp`), the native Mooncake and Enzyme adjoints, and a first registered release. +After that come the ports of ComposableTuringIDModels and BVDOutbreakSize onto the operators. +An FFT method for long fixed kernels and a Reactant backend are later extensions. + +## Where next + +- The [ComposableRecurrences.jl repository](https://github.com/EpiAware/ComposableRecurrences.jl) + tracks development, and its documentation will follow the first release. +- Building a full model with priors and observations? See the + [composable Turing models approach](composable-turing-models.qmd). +- Need the delay distributions that feed a `Convolution`? See the + [composed distributions approach](composed-distributions.qmd). + +:::::: diff --git a/styles.css b/styles.css index 0df387d..cb80219 100644 --- a/styles.css +++ b/styles.css @@ -304,6 +304,41 @@ h1, h2, h3 { letter-spacing: -0.02em; } } .library-links a { font-weight: 600; font-size: 0.92rem; } +/* Package map on the composable recurrences page. Colours come from the + theme variables and currentColor so the diagram follows light and dark + mode. */ +.cr-map { + overflow-x: auto; + border: 1px solid var(--border-color); + border-radius: 12px; + margin: 1rem 0 1.5rem; +} +.cr-map svg { display: block; min-width: 760px; width: 100%; height: auto; } +.cr-map text { font-family: inherit; fill: currentColor; } +.cr-map .cr-sub { fill-opacity: 0.68; font-size: 11px; } +.cr-map .cr-lbl { fill-opacity: 0.68; font-size: 10.5px; } +.cr-map .cr-node { + fill: none; stroke: var(--border-color); stroke-width: 1.2; +} +.cr-map .cr-node.cr-focus { + fill: var(--epi-primary); fill-opacity: 0.12; + stroke: var(--epi-primary); stroke-width: 1.6; +} +.cr-map .cr-band { + fill: none; stroke: var(--border-color); stroke-dasharray: 3 4; +} +.cr-map .cr-band-t { + fill-opacity: 0.6; font-size: 10.5px; letter-spacing: 0.08em; +} +.cr-map .cr-e { stroke: var(--epi-primary); stroke-width: 1.5; fill: none; } +.cr-map .cr-e.cr-dash { + stroke: currentColor; stroke-opacity: 0.5; stroke-dasharray: 5 4; +} +.cr-map .cr-e.cr-warn { stroke: var(--epi-accent); stroke-dasharray: 5 4; } +.cr-map .cr-ah { fill: var(--epi-primary); } +.cr-map .cr-ah.cr-muted { fill: currentColor; fill-opacity: 0.5; } +.cr-map .cr-ah.cr-warn { fill: var(--epi-accent); } + /* Value propositions + foundations grid ----------------------------------- */ .feature-grid { display: grid; diff --git a/theme-dark.scss b/theme-dark.scss index ed719c6..dd83d67 100644 --- a/theme-dark.scss +++ b/theme-dark.scss @@ -34,6 +34,7 @@ $border-radius: 0.6rem; --epi-primary-dark: #2a9a94; --epi-primary-light: #4fc9c0; --border-color: #26303b; + --epi-accent: #e8965a; } .navbar { diff --git a/theme-light.scss b/theme-light.scss index c6fd437..3737ea9 100644 --- a/theme-light.scss +++ b/theme-light.scss @@ -35,6 +35,7 @@ $border-radius: 0.6rem; --epi-primary-dark: #0b5e6d; --epi-primary-light: #1a9aab; --border-color: #e6ebf0; + --epi-accent: #c46f2f; } .navbar { From 80ef08e8e211023628b3313cf6442cb84092db29 Mon Sep 17 00:00:00 2001 From: seabbs-bot Date: Wed, 30 Sep 2026 15:34:15 +0100 Subject: [PATCH 2/4] docs: list composable recurrences in the overview, navbar and packages Co-authored-by: Sam Abbott --- README.md | 2 +- _includes/navbar.html | 1 + approaches/index.qmd | 9 ++++++++- packages/index.qmd | 8 ++++++++ 4 files changed, 18 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index cc439f0..71d1661 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ request. - `index.qmd` — home - `packages/index.qmd` — the ecosystem inventory (one filterable table) - `approaches/` — the modelling approaches (composed distributions, composable - Turing models) and their shared overview + Turing models, composable recurrences) and their shared overview - `tutorials/index.qmd` — a filterable listing of runnable tutorials, each living in its own package's documentation - `gallery.qmd`, `team.qmd`, `funding.qmd`, `get-involved.qmd`, `community.qmd` diff --git a/_includes/navbar.html b/_includes/navbar.html index 9264ab9..2789bd3 100644 --- a/_includes/navbar.html +++ b/_includes/navbar.html @@ -50,6 +50,7 @@
  • Overview
  • Composed distributions
  • Composable Turing models
    • +
  • Composable recurrences
  • diff --git a/approaches/index.qmd b/approaches/index.qmd index 37bfd8b..3da745f 100644 --- a/approaches/index.qmd +++ b/approaches/index.qmd @@ -44,7 +44,7 @@ This is our first cut, and we would like to be told where it is wrong. ## The approaches We are not sure yet which of these will work best, so we are trying more than one and keeping them interoperable. -Both work with the same underlying packages, and they can be combined in a single analysis. +They share the same underlying packages, and they are designed to combine in a single analysis. ::: {.feature-grid} ::: {.feature-card} @@ -61,6 +61,13 @@ Any piece can be swapped for another without rewriting the rest. [Read the approach →](composable-turing-models.qmd) ::: +::: {.feature-card} +### Composable recurrences +Build renewal processes, autoregressions and reporting delays from two fast, differentiable operators, one that feeds back on its own history and one that does not. +They carry no epidemiology of their own, and composable Turing models are planned to run on them. + +[Read the approach →](composable-recurrences.qmd) +::: ::: More approaches may join these as the ecosystem grows. diff --git a/packages/index.qmd b/packages/index.qmd index 867df1f..98d73e5 100644 --- a/packages/index.qmd +++ b/packages/index.qmd @@ -40,6 +40,14 @@ Status reflects where a package sits in its release cycle rather than a judgemen Assemble infectious disease models from interchangeable infection and observation parts as a single Turing model. The package that ties the ecosystem together. + + + ComposableRecurrences.jl + + experimental + + Fast, differentiable recurrences and causal convolutions, the renewal, autoregressive and delay steps inside a model, with plug-in adjoints. Under development, with no public API released yet. + CensoredDistributions.jl From 18bb09f243a5f303e2266f57c4cfe8c705a20532 Mon Sep 17 00:00:00 2001 From: seabbs-bot Date: Wed, 30 Sep 2026 15:35:57 +0100 Subject: [PATCH 3/4] docs: tighten status claims on the composable recurrences page Co-authored-by: Sam Abbott --- approaches/composable-recurrences.qmd | 16 ++++++++-------- approaches/index.qmd | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/approaches/composable-recurrences.qmd b/approaches/composable-recurrences.qmd index b18e232..50d0534 100644 --- a/approaches/composable-recurrences.qmd +++ b/approaches/composable-recurrences.qmd @@ -32,7 +32,7 @@ page-layout: full ``` -##### Fast gradients on every backend +##### Built for fast reverse-mode gradients ::: @@ -88,14 +88,14 @@ reports = Convolution([0.0, 0.5, 0.3, 0.2]) # Operators are callable. R_t is the gain, one column per day R = fill(1.1, 2, 10) -I = infections(R; history = ones(2, 3)) -Y = reports(I) +y = infections(R; history = ones(2, 3)) +cases = reports(y) # Stop, keep the state, and resume for a forecast -I1, state = infections(R[:, 1:5]; history = ones(2, 3), +y1, state = infections(R[:, 1:5]; history = ones(2, 3), return_state = true) -I2 = infections(R[:, 6:end]; history = state) -I ≈ hcat(I1, I2) +y2 = infections(R[:, 6:end]; history = state) +y ≈ hcat(y1, y2) ``` ::: @@ -148,7 +148,7 @@ The operators own a history buffer instead, so plain AD is already fast. Hand-written adjoints will then plug in per operator, modifier or coupling by adding a method, with no AD declarations from the user. `NoAdjoint(op)` turns an adjoint off, so its gain can be timed and the adjoint dropped if a backend catches up. -In prototypes the buffer alone gave 1.75 to 15 times faster gradients with no custom rules, and analytic adjoints gave 8 to 25 times for delays and renewals. +In prototypes the buffer alone gave gradients 1.75 to 15 times faster than today's scan-based steps with no custom rules, and analytic adjoints gave 8 to 25 times for delays and renewals. These were measured on isolated kernels (200 days, 20 lags) under Mooncake and Enzyme, not on whole models. ### How it fits with the other approaches @@ -263,7 +263,7 @@ Moving them onto one set of operators gives one forward kernel and one adjoint t This approach is under development and its public API is landing now. `Recurrence`, `Convolution`, the `TimeVarying`, `PerStratum` and `Pairwise` wrappers, and the coupling and modifier interfaces exist on a development branch. The test suite differentiates them with ForwardDiff, Mooncake and Enzyme through plain AD. -Use-case tests compare them against reference implementations of real models. +Tests check them against copies of the ComposableTuringIDModels steps they will replace. Planned next are the built-in modifiers (`Depletion`, `Imports`, `Redistribute`, `Clamp`), the native Mooncake and Enzyme adjoints, and a first registered release. After that come the ports of ComposableTuringIDModels and BVDOutbreakSize onto the operators. diff --git a/approaches/index.qmd b/approaches/index.qmd index 3da745f..c0a7470 100644 --- a/approaches/index.qmd +++ b/approaches/index.qmd @@ -44,7 +44,7 @@ This is our first cut, and we would like to be told where it is wrong. ## The approaches We are not sure yet which of these will work best, so we are trying more than one and keeping them interoperable. -They share the same underlying packages, and they are designed to combine in a single analysis. +They are designed to work together and to combine in a single analysis. ::: {.feature-grid} ::: {.feature-card} From 8ee378012597803c5f7974b51adeae17c42f2fc7 Mon Sep 17 00:00:00 2001 From: seabbs-bot Date: Wed, 30 Sep 2026 16:26:48 +0100 Subject: [PATCH 4/4] docs: drop internals from the recurrences map and match main Remove BVDOutbreakSize and internal names from the package map and text, and show a Reactant optional backend in its place. Drop the unsourced prototype speed-ups, the unlabelled EpiAwareADTools edge, and note the package is on main but unregistered. Status now says the core operators are merged and the modifiers and native adjoints are in progress. Co-authored-by: Sam Abbott --- approaches/composable-recurrences.qmd | 90 +++++++++++++-------------- packages/index.qmd | 2 +- styles.css | 3 + 3 files changed, 48 insertions(+), 47 deletions(-) diff --git a/approaches/composable-recurrences.qmd b/approaches/composable-recurrences.qmd index 50d0534..20bbe39 100644 --- a/approaches/composable-recurrences.qmd +++ b/approaches/composable-recurrences.qmd @@ -103,6 +103,9 @@ y ≈ hcat(y1, y2) ::: +The example runs on the main branch of [ComposableRecurrences.jl](https://github.com/EpiAware/ComposableRecurrences.jl). +The package is not registered yet, so install it from the repository to try it. + ## The approach Renewal processes, random walks, autoregressions and reporting delays share one structure. @@ -132,8 +135,8 @@ Chaining is function composition, as in `reports(infections(R; history))`. Wrappers name the extra axis of a coefficient. `PerStratum` gives each stratum its own kernel, `TimeVarying` lets a kernel or coupling change by day, and `Pairwise` gives each pair of strata its own lag kernel. -The coupling mixes strata before the gain, as a contact or mobility matrix does, and can be dense, sparse or structured. -Any array type and any real element type is accepted, so `Float32`, sparse matrices and dual numbers all pass through. +The coupling mixes strata before the gain, as a contact or mobility matrix does. +Couplings can be dense, sparse or diagonal matrices, and any real element type is accepted, so `Float32` and dual numbers pass through. ### Modifiers @@ -144,12 +147,10 @@ A modifier is a plain type with an `apply!` method, so a modelling package can a ### Plug-in adjoints Reverse-mode AD is slow when each step copies its lag window into a new state. -The operators own a history buffer instead, so plain AD is already fast. -Hand-written adjoints will then plug in per operator, modifier or coupling by adding a method, with no AD declarations from the user. +The operators own a history buffer instead, which avoids that copy. +Native adjoints will plug in per operator, modifier or coupling by adding a method, with no AD declarations from the user. `NoAdjoint(op)` turns an adjoint off, so its gain can be timed and the adjoint dropped if a backend catches up. - -In prototypes the buffer alone gave gradients 1.75 to 15 times faster than today's scan-based steps with no custom rules, and analytic adjoints gave 8 to 25 times for delays and renewals. -These were measured on isolated kernels (200 days, 20 lags) under Mooncake and Enzyme, not on whole models. +A [benchmark script](https://github.com/EpiAware/ComposableRecurrences.jl/blob/main/benchmark/gradients.jl) in the repository times plain AD gradients of the operators under Mooncake and Enzyme. ### How it fits with the other approaches @@ -161,9 +162,8 @@ The operators have no dependency on either, so they can also be used in other mo ## Package map Solid arrows are data or code that will flow once the package is adopted. -Dashed grey arrows are test-time links. +The dashed grey arrow is a test-time link. The dashed amber arrow is an optional future backend, not a dependency. -The ComposableRecurrences box lists the planned first release. ```{=html} ``` The package has no runtime dependency on any other EpiAware package. -Today each model above runs its own renewal and delay loops, and several of them duplicate each other. +Today ComposableTuringIDModels runs its own renewal, delay and latent process loops. Moving them onto one set of operators gives one forward kernel and one adjoint to maintain and test. ## Status -This approach is under development and its public API is landing now. -`Recurrence`, `Convolution`, the `TimeVarying`, `PerStratum` and `Pairwise` wrappers, and the coupling and modifier interfaces exist on a development branch. -The test suite differentiates them with ForwardDiff, Mooncake and Enzyme through plain AD. -Tests check them against copies of the ComposableTuringIDModels steps they will replace. +This approach is under development. +The core operators are merged on the main branch of ComposableRecurrences.jl. +These are `Recurrence`, `Convolution`, the `TimeVarying`, `PerStratum` and `Pairwise` wrappers, and the coupling and modifier interfaces. +The test suite differentiates them with ForwardDiff, ReverseDiff, Mooncake and Enzyme through plain AD. +Use-case tests against copies of the ComposableTuringIDModels steps they will replace are written but not yet switched on. -Planned next are the built-in modifiers (`Depletion`, `Imports`, `Redistribute`, `Clamp`), the native Mooncake and Enzyme adjoints, and a first registered release. -After that come the ports of ComposableTuringIDModels and BVDOutbreakSize onto the operators. +The built-in modifiers (`Depletion`, `Imports`, `Redistribute`, `Clamp`) and the native Mooncake and Enzyme adjoints are in progress. +A first registered release follows them, and then the port of ComposableTuringIDModels onto the operators. An FFT method for long fixed kernels and a Reactant backend are later extensions. ## Where next diff --git a/packages/index.qmd b/packages/index.qmd index 98d73e5..5cad9f1 100644 --- a/packages/index.qmd +++ b/packages/index.qmd @@ -46,7 +46,7 @@ Status reflects where a package sits in its release cycle rather than a judgemen experimental - Fast, differentiable recurrences and causal convolutions, the renewal, autoregressive and delay steps inside a model, with plug-in adjoints. Under development, with no public API released yet. + Fast, differentiable recurrences and causal convolutions, the renewal, autoregressive and delay steps inside a model. The core operators are on the main branch but not yet registered, and native adjoints are in progress. diff --git a/styles.css b/styles.css index cb80219..831820f 100644 --- a/styles.css +++ b/styles.css @@ -330,6 +330,9 @@ h1, h2, h3 { letter-spacing: -0.02em; } .cr-map .cr-band-t { fill-opacity: 0.6; font-size: 10.5px; letter-spacing: 0.08em; } +.cr-map .cr-node.cr-opt { + stroke: var(--epi-accent); stroke-dasharray: 5 4; +} .cr-map .cr-e { stroke: var(--epi-primary); stroke-width: 1.5; fill: none; } .cr-map .cr-e.cr-dash { stroke: currentColor; stroke-opacity: 0.5; stroke-dasharray: 5 4;