Skip to content

Rewrite - #44

Open
beykyle wants to merge 170 commits into
mainfrom
rewrite
Open

beykyle wants to merge 170 commits into
mainfrom
rewrite

Conversation

@beykyle

@beykyle beykyle commented Sep 11, 2026 •

Copy link
Copy Markdown
Owner

Significant re-write, superseding #43 and #42

Remove the 0.x tests, notebooks, in-package samplers, drivers and the
declaration classes.  The eight modules the rewrite harvests stay in
place for one commit so that git mv can carry their history into the
new layout.  The 0.x package remains at v0.1.0 and on legacy/0.x.
git mv only, no edits: likelihood_model -> likelihood, covariance ->
terms, observation_from_measurement -> units, model_comparison ->
diagnostics, the two reaction observation modules -> reactions/.  Their
bodies are rewritten in milestones 1, 5 and 6.
Minimal package init and smoke test; pytest fast/slow tiers (slow marker,
deselected by default); py312 lint targets; pandas and scikit-learn out of
the runtime requirements and into the examples extra with dill; docs
reduced to the design and recipes pages; a README for the branch; CI with
format, tests and docs jobs on main and rewrite plus a scheduled slow
job; a trusted-publishing workflow on version tags.
…prior

Frozen dataclass with eq=False, so sharing is by object; bounds validated
and coerced; prior, unit and latex carried for Problem and plotting.
Drop the contextual flag, the context keyword and per_observation_scaling;
scale() uses the new Parameter signature.  Transform, composition, log/exp
and the finite-difference derivative are unchanged.
The measurement adapters return inside from_measurement in milestone 5.
Gaussian, StudentT(nu=None) with a bounded default parameter, Chi2 and the
log_likelihood helper; the Cholesky helper moves to covariance in
milestone 3.  Test helpers gain a dense reference assembly used by the
term tests now and the structured covariance tests later.
Term is a frozen dataclass: fn, params, kind, on, coords, constant.  No
bind, no support indices, no caches; value() evaluates on a given support
and compile resolves on= to rows.  TermContext gains meta(key) for
per-point dataset metadata.  Factories keep their bodies with support=
renamed on= and the _term suffix dropped; kernel() accepts params= to
share hyperparameters between comparisons.  Tests port TermKinds, Coords,
Factories, Kernel and the alpha+Ca study forms against a hand-built dense
reference, plus the term-level halves of recipes 19 and 27.
x4i3 converts every EXFOR cross section to barns while parsing and
exfor_tools labels the result from a three-entry table, so measurement
unit labels come from a small fixed vocabulary.  parse_unit(label) maps
that vocabulary (plus the obvious spellings) to a factor into the
internal unit and a quantity kind, and rejects anything else loudly.
The 0.x pint path could not parse the real labels (barns/ster, b/Sr);
it only worked for hand-built measurements.
Frozen, identity-hashed; x is opaque beyond its first dimension; y_err
validated; norm_err and offset_err stored as a float or an (n,) array;
meta copied.  No comparison transform, mask or solver state.
Model(fn, params) binds to a grid by closing over it; subclasses override
bind() to build solvers from meta.  Composition happens on the bound
predictors: | applies a mean transform after the prediction, + and * are
x-dependent corrections, parameters concatenated left to right.
polynomial(order) is the generic example.
Comparison binds a model to a dataset in a parameter-free space, with the
delta-method errors and the per-point log-Jacobian as constants;
reported_terms() turns the dataset's offset and normalisation
systematics into rank-one modes, propagated at the data and at the
prediction respectively.  Constraint holds the comparisons, terms,
likelihood, weight and per-comparison masks; masked, masked_where and
complement share every object with the original.  Every term's on= is
resolved eagerly and an array term's shape is checked against its
support.
The study labels (L0, E0, L1, ...) meant nothing to a reader who had not
seen the study.  test/helpers.py now carries STUDY_LEGEND, one line per
label, and study_form(label, x, y, ym) builds the terms, values and the
hand-built dense reference; the term tests parametrise over it and the
structured covariance tests will too.  Recipe 18 and the design document
point at the legend.
Sigma = diag(D) + blockdiag(M_b) + U U^T over the active rows of a
constraint: diag terms into D, mode terms as columns of U (a mode across
comparisons is one column with entries in several blocks), matrix terms
into their block, and a matrix term that crosses blocks forces the dense
path.  Per-block Cholesky plus a rank-r correction gives (d2, logdet);
constant pieces are evaluated once and a fully constant covariance is
factored once, with a singular block reported by label at construction.
Verified against the hand-built dense reference on every study form,
across blocks, with masks, and on the dense fallback.
Recipe 18 now carries the full legend table; test/helpers.py keeps an
identical copy and says so, and the design document cites the recipe.
A docs reader never has to open the test suite.
ParameterIndex assigns each distinct Parameter a slot in first-seen order
(predictors, terms, likelihood, constraint by constraint) and rejects
duplicate names with a hint.  The prior is assembled so every slot is
covered exactly once: a marginal truncated to its bounds, the uniform on
finite bounds, or one joint block; a frozen multivariate normal gets a
whitening unit-cube map, other joints supply their own.  Compiled
constraints stack the comparisons, check finiteness on the active points,
stack per-point metadata for terms, resolve every term's rows and gather,
and build the structured covariance.  Problem exposes log_prior,
log_likelihood (weighted sum), log_posterior (prior first), prior_transform,
sample_prior, predict, chi2, columns and the black-box-bayes aliases, and
pickles with dill.  Tests cover the index, the prior rules, compile
errors, masked views, and emcee, dynesty and dill drivers.
One test file per recipe under test/recipes, docstring quoting the
recipe's intent, assertions its expected-behaviour bullets that need no
converged chain; a linear-Gaussian oracle supplies closed-form posteriors
for the linear recipes.  offset() and normalization() now take the
nuisance parameter as their first positional argument, like every other
nuisance factory and like the recipes spell them; magnitude= stays a
keyword.
…heck

Recipe tests for external samplers, custom terms, preprocessing, the
singular-covariance error, shared hyperparameters, a GP across energies
and the hyperprior block.  Two library changes they exposed: a joint prior
block may introduce a parameter no model or term uses (its
hyperparameter), which now gets a slot after the constraints' parameters;
and the block factor B is checked at construction whenever every diag and
matrix term is constant, since modes never enter B, so a block covered
only by modes fails by label before any likelihood call.  Term is
exported from the package.
SafeBayes prefix differences equal the Gaussian conditional; KDUQ deltas
are linear fractions (log=False, also in the recipe); Peelle's puzzle is
pinned by the two-point closed form and by the ordering data-built <
live prediction-built < t0 across replicates, with the recipe text
corrected to say the live mode keeps a log-determinant pull that the t0
refit removes; cut posteriors by closure and by module weights; an
emulator term seeing the model's values; MAP + Laplace against the
oracle; error scale and USU modes; energy-dependent parameters;
Legendre modes; and the eight-schools model in its three spellings.
GLS with Sigma = sigma^2 I + s^2 y y^T collapses by Sherman-Morrison to
t = ybar / (1 + (s/sigma)^2 sum (y_i - ybar)^2), exact per dataset and
sampler-free; the test asserts it, that identical points give no bias,
and that replicate means lie between the leading-order expectation
1/(1 + (n-1) s^2) and the truth.  The recipe text now quotes that form
instead of the 1/(1 + n s^2) factor it had copied unverified.
ElasticXS and IsobaricAnalogPN build their jitr workspace from the
dataset's kinematics in bind() and return a Predictor closing over it;
the elastic basis is cached per model and energy so a plotting grid or a
held-out view reuses it, and the cache is dropped on pickling.  The
harvested set_up_solver functions take one grid; rutherford() is the
closed form of the kinematics and momentum_transfer() a free function.
No compound-correction hook.  Real-solve tests with lmax=10 pin
finiteness, units, the cache, composition, solver-setting forwarding,
the Lane-term IAS, and a dill round trip of a reaction Problem.
Reads the exfor_tools fields, converts angles to radians and cross
sections to b/sr through the fixed label table, converts dXS/dA to and
from dXS/dRuth with the closed-form Rutherford cross section (a
per-angle norm), scales the dimensionful errors with the data and leaves
the fractional normalisation error alone, and fills meta with the
kinematics a reaction model needs.  Every failure names what is wrong.
Reported systematics become modes only when asked for; an EXFOR
measurement lands in b/sr and radians with the Rutherford conversion as
a per-angle norm; one elastic model serves the data grid and a plotting
grid from one cached basis; a cross section and an analysing power from
one measurement share an angle-calibration mode across comparisons.
Recipe 37 now notes that a spanning basis sees the gathered stack and
must split a finite difference at the seam by per-point metadata.
A term on several comparisons sees their rows gathered into one stack,
so a basis that differentiates or smooths along the grid would straddle
the seam.  TermContext now carries segments (the slice of the support
belonging to each spanned comparison, in constraint order), labels, and
split(a); the covariance computes them once per entry from the block
offsets.  Outside a problem a term has one segment.  Recipe 37 takes its
angle-calibration slope per segment, and the design and recipe docs
describe the new fields.
A kernel term closed over its sklearn kernel and nothing on the Term
exposed it, but the predictive band promised by recipe 7 must condition
the discrepancy from the term alone.  KernelTerm is a Term subclass
carrying the kernel, the count of free hyperparameter elements (so the
log-theta and amplitude columns follow from params), the amplitude and
the jitter; the covariance treats it as any matrix term.
… score

predictive_draws and heldout_log_predictive take a compiled Problem and
chain rows in problem.names order; coverage, sharpness, the log posterior
predictive and the evidence bookkeeping are lifted from 0.x.  A held-out
problem built from fit.complement() scores the marginal of its rows,
which is wrong when a term spans the split (a GP over several
experiments); given=fit_problem computes the Gaussian conditional under
the full covariance instead, and equals the marginal when nothing spans.
split_samples and the module-level log_jacobian are gone: columns come
from problem.columns and the Jacobian from problem.log_jacobian().
total_predictive_band(problem, term, predictor, x_pred, samples) locates
the kernel term's constraint and covariance entry, takes its training
rows, comparison space, kernel and amplitude columns from the problem,
and conditions the discrepancy with the rest of the declared covariance
as the regression noise, so a chain with nuisance columns needs no
column arithmetic.  The amplitude is evaluated at the prediction grid
through a TermContext; physical=True maps draws back through the
space's inverse.  GP conditioning against sklearn is lifted from 0.x.
beykyle and others added 30 commits November 4, 2026 19:51
The two long-NUTS references were accepted under the classic R-hat; rerun
with the same seeds under the rank-normalised one they pass (1.001 and
1.001, min bulk ESS 2475 and 3920) with means and widths identical to
before, so no row scored against them changes.
…98 s -> 518 s)

The cross-section ladder's posterior has one mode, so pocomc runs at the
benchmark's single-mode setting, 1024/512, rather than the 4096/2048 that
separated modes need.  Gate against the committed run at 4096/2048 and
other seeds: log Z -467.18 / 96.73 / 103.88 against -467.21 / 96.78 /
103.91, Delta log Z 563.9 and 7.1 with the same verdicts, all twelve
medians within 0.07 posterior sd, envelope fraction 0.56 against 0.58.
The caveat now says the two runs agree, and that two runs of one
algorithm cannot reveal a shared bias.

Intervals are 68 %, as in the sibling notebooks; the corner plots are
density-normalised; two plot labels use mathtext.
…(3845 s -> 1342 s)

- The log-space error model is median one: point_to_point = noise(log
  sigma_A) and common = offset(log sigma_B), log-space standard deviations
  with log-uniform priors on [0.05, 2], in place of the jitr mean-one
  moment map written as custom terms.  The derivation says why median one
  and gives E[y/ym] correctly; sigma_A = 0.197 [0.176, 0.222], a factor of
  about 1.22 per angle; log Z 198.9, still about 730 above the linear model.
- The NUTS cross-check, unconverged by design, is gone; the benchmark's
  numbers say why a gradient sampler started at the MAP is no check here.
- One log-space run (n_seeds = 1, raise it for the spread); its island
  split is 15 %, as the benchmark's reference.
- The linear model, with one narrow mode, runs pocomc at 1024/512.  Gate
  against the committed run: log Z -526.69, Vv 45.4 [45.1, 45.6] at 68 %
  (the old 90 % interval scaled), chi2/N 35.2, coverage 0.06 / 0.26.
- predictive_draws(physical=True) replaces the hand-written exp; 68 %
  intervals; coverage on the 0.05 grid; the predictives are 'model only',
  '+ statistical', '+ point-to-point' and 'full predictive'; the duplicate
  coverage table goes.  Recipes 10, 11, 14, 15, 16, 17, 18, 21, 40.
The check against the earlier run at 4096/2048 shows the resolution is
enough; both runs used seeds 10, 11 and 12, as 3ab75f6's message also
wrongly said otherwise.  Text only.
…53 s -> 1765 s)

- The error model is median one, T.noise(log sigma_A) + T.offset(log
  sigma_B) with log-uniform priors on [0.05, 2], replacing the jitr
  moment map; sigma_A, sigma_B and the GP amplitude A are all log-space
  standard deviations and are compared directly.  Their posterior
  correlations are printed: log sigma_A and log A are only weakly
  correlated (-0.10, -0.14), so the data do tell the uncorrelated error
  and the discrepancy apart.
- The growth power p is uniform on [0, 16]; its posterior, 7.5 [4.4,
  11.5], no longer presses against the bound.  The verdict holds:
  Delta log Z = 6.96 +/- 0.05, power of q favoured.
- One pocomc run per model (n_seeds = 1).  A scratch run with n_seeds = 3
  (outside the repo; its first runs reproduce these exactly) gave per-run
  log Z within 0.09 and 0.04, Delta log Z 6.95 +/- 0.05, and family
  weights 0.45/0.45/0.45 and 0.73/0.75/0.73; the prose quotes it, from
  this model and prior.
- 68 % bands, coverage on the 0.05 grid, 'model only' and 'full
  predictive', the Rutherford-ratio label, 'power of $q$' in plot labels,
  jnp.sin in power_of_q, and the coverage title naming its 73 points.
  Recipe 19 is no longer cited (no custom Term remains).
- gp_discrepancy: the growing-amplitude toy fit had landed at R-hat 1.010
  (not converged by the notebook's own rule) while the prose said every fit
  passed; the toy emcee runs are now 40 000 steps and all converge (R-hat
  at most 1.005, runs of at least 174 tau).  The half-widths at x = 0.2
  and 5, the toy envelope fractions (81 % / 68 %) and the toy intervals
  are read from this run; a sentence says the single-mode pocomc setting
  can bias GP hyperparameters on the benchmark.  588 s.
- local_optical: the Rutherford-ratio label follows the shared
  convention; the benchmark's omp problem is called its version of this
  problem, with an earlier parametrisation of the two errors, not 'this
  very problem' (Phase 4 makes them identical); the island-weight
  sentence cites the benchmark's 4096/2048 runs, not stale two-run
  numbers; the counting-error shift is at most 0.15 %; 'model only'
  throughout.  1324 s.
- alpha_ca: alpha_0 for the starting parameters (theta is the angle); the
  three-run numbers are labelled as an uncommitted n_seeds = 3 run, with
  LaTeX minus signs; 'a factor of 1.6 to 1.7'.  1760 s.
- docs/benchmarks.md describes omp as using the earlier parametrisation.
- local_optical: the benchmark's omp problem differs in the
  parametrisation *and the priors* of the two errors, so its posterior
  and evidence differ a little (log Z -6.63 against -6.71); said so here
  and in design.md.  Recipe 11 is dropped from its list: the notebook
  only points to it, it does not hold data out.
- alpha_ca: the three-run spread (0.09 for the constant amplitude)
  is about three times pocomc's quoted 0.03, so the quoted errors
  understate it; the verdict still does not depend on the seed.
Text only; no rerun.
- omp is now exactly local_optical's log-space problem (median-one
  noise + offset errors on log-uniform log-space sds), and two problems
  join it: alpha_gp, exactly alpha_ca's power-of-q GP problem, and
  n40ca_gp, exactly gp_discrepancy's GP rung (via n40ca_ladder(), which
  builds all three rungs).  Each was checked against the notebook's own
  cells: identical log posterior at five prior draws (to 4e-11 for omp).
- Their references pool three pocomc runs at 8192/4096 and are
  corroborated by one dynesty run (2000 live points); a disagreement is
  recorded (corroborated: false) rather than refused.
- Parallel-safe output: --references gives each reference job its own
  file; merge_results.py merges per-job files into the committed results,
  refusing two inputs with the same entry.
- sweep_pocomc.py sweeps pocomc's resolution on a problem (saving each
  run, resumable, mode share where there are two modes); gp_seed_check.py
  reruns gp's ladder at its settings with three other seed sets.
- benchmarks/hpc/: a SLURM job script, site settings, submit_all.sh (31
  jobs) and the runbook.  Tests for the problems and the merge; the
  drivers end to end in the slow tier.  docs/benchmarks.md lists the
  problems and notes the pending rows.
The references, sweeps and seed check ran their pieces in sequence inside
one job; now every independent run is an array task (72 in 10 array
jobs): a reference's three pocomc seeds and its dynesty corroboration,
each sampler, each (setting, seed) of a sweep, each (seed set, rung) of
the gp check.  benchmarks/hpc/task.py maps $SLURM_ARRAY_TASK_ID to its run
and writes results/hpc/<kind>/<problem>/<index>.json; assemble.py pools
the reference parts (reference.pool_parts, split out of long_pocomc, which
is unchanged in behaviour) and joins the sweep and seed-check pieces,
skipping any problem with a missing piece.  submit_all.sh sizes each
array from task.py --count.  Tests cover the index mapping and assembly.
benchmarks/README.md maps the directory: the problems, the density and
sampler benchmarks, the references, sweeps, seed check, merging and the
cluster runner.

The hard-coded submit_all.sh gives way to hpc/submit.py, which submits any
subset as array jobs: --problems (default every benchmark problem),
--kinds ref/run/sweep/seed, and within a kind --samplers, --settings,
--seeds or --seed-sets.  A selection keeps each task's array index, so a
result lands in the same file whichever subset produced it, and a failed
task is rerun by selecting it again.  Walltimes come from a per-(kind,
problem) table, overridable with --time.  task.py now handles every
problem: a reference that is not pooled (quadrature, a long NUTS run) is
one task, and assemble.py takes it whole.  A test pins the selection.
The coverage curves of the six notebooks that draw them now use
np.linspace(0.05, 0.95, 19), not np.linspace(0, 1, 21), whose levels 0 and
1 coverage_curve does not take (level 1 is the draws' min to max, so it
plotted an under-coverage that is not there).  robust and normalization
look the levels they quote up by value.  local_optical keeps each model's
pocomc setting next to its problem; gp_discrepancy gives the doubled emcee
run only to the growing-amplitude toy fit (the other three converge at
20 000 steps, R-hat at most 1.004).

All seven reran; every printed number is unchanged (the quoted coverages
are computed at explicit levels, with the same seeds).  design.md has the
runtimes of the uncontended runs (linear 15 s, gp 582 s, local_optical
1332 s, alpha_ca 1763 s); robust, normalization and hierarchical were
timed while other work shared the machine, and keep their earlier
runtimes, their computation being unchanged.
- References: one dynesty run could not have corroborated a pocomc pool
  (single runs were 0.12 to 0.37 sd off on omp), so three dynesty runs
  (rslice, 1000 live points) are pooled the way the pocomc runs are, as
  three more array tasks, and the pools must agree on log Z within twice
  the combined error, on the mode share within 0.03 and on the moments
  within 0.2 sd; the checks are recorded and named, not enforced.  The
  mode share (problems.mode_share, moved from sweep_pocomc) is computed
  for every reference run, dynesty included (run_dynesty gains the
  summary hook).
- assemble.py no longer aborts on one problem's failed pool: it keeps it,
  marked accepted: false, with its runs.  Pooled references keep each
  run's seed; whole references record their wall time; references carry
  their parameter names, and a report refuses to score a run against a
  reference of other parameters.
- merge_results.py: the docstring's command is the right one; anything
  that is not a reference or sampler file (a sweep) is refused; a
  problem whose parameters changed drops the rows no input replaces.
- gp_seed_check.py records log Z as the notebook prints it (with the
  Jacobian).  env.sh gives each task its own PyTensor compile cache and
  drops two thread settings nothing reads.  submit.py validates problem
  names and asks for --problems or --all.
…_FLAGS

intra_op_parallelism_threads is not an XLA flag; JAX 0.12 starts with it
but ignores it.  --xla_cpu_multi_thread_eigen=false and OMP_NUM_THREADS
remain.
The end-to-end driver test ran pocomc on the GP problems (about 19 ms per
likelihood call), which took most of an hour.  The sweep is now tested
end to end on linear with short runs (28 s), resume included; the seed
check's summaries, the mode share and the merge are tested directly.
run_pocomc passes n_total and n_evidence, if given, to pocomc's run (its
defaults, 4096 each, otherwise), so short runs are possible.
What the cluster tasks compute:
- gp_seed_check's weighted quantiles centre each draw's weight on its
  value (equal weights give the midpoint median, 2.5 of 1..4, not 2.0).
- job.sbatch runs each task through srun --cpu-bind=cores: XLA does not
  read OMP_NUM_THREADS, and binding keeps its threads on the task's cores
  when tasks share a node.  env.sh's comment no longer claims otherwise.
- A seed task builds only its own rung of the ladder, and n40ca_gp only
  the GP rung (n40ca_ladder(rungs=...)), not all three.

What scores them, later:
- bench_samplers.usable_reference refuses a reference that does not name
  its parameters or names others, and flags one not accepted or not
  corroborated (also marked in the tables); sweep_pocomc's report uses it.
  The committed references now carry their names; the committed omp
  reference names the earlier log_s/log_b, so it can no longer score the
  redefined omp.  The committed tables are unchanged.
- merge_results drops a redefined problem's stale reference along with
  its rows, unless an input brings a new one.
- sweep_pocomc and gp_seed_check create their output's directory; seed
  tasks refuse any problem but gp.  A dead line in a test is gone.
omp now defines local_optical's exact problem, so the earlier problem's
reference and sampler rows describe nothing the benchmark runs; they are
removed (git keeps them) and come back from the cluster refresh.  The
results table is regenerated without them; the references table states
the new population-reference method for omp, alpha_gp and n40ca_gp; the
correct/fastest row and the note above Results say omp is pending, and
that the omp numbers the text still quotes are of the earlier version.
gradient_based_sampling:
- the hierarchy's MLE search is L-BFGS-B, not Nelder-Mead;
- emcee is "cheaper and agrees with NUTS" on the GP, not "as good":
  agreement between two samplers is not a check, and on the benchmark's
  exact gp neither gets the widths within the tolerance;
- the benchmark's NUTS had 31 divergences, not "a similar number" to
  this run's 85;
- the Laplace widths on b are standard deviations the tails dominate;
- "about twelve seconds"; "wrong means or widths", not "biased";
- the recipes link.

normalization_and_covariance_structure:
- ignoring the systematics is the most overconfident answer, not the
  most wrong (Peelle's is further off);
- treatment 6 is close to the truth, its 68 % interval missing a0 = 1;
- leaving a dataset out is recipe 30, not 28;
- the start ball is wider than most posterior widths, and the modes
  reference is itself NUTS;
- the covariance cells fill theta by name through problem.columns;
- rerun (figures unchanged).

The rest:
- hierarchical_calibration: over-coverage from a nominal 0.3 up; the
  sentence on energies between datasets rewritten.
- robust_likelihoods: KDUQ cited at first use; a note that its scaled
  term needs model_only=True on a grid.
- linear_calibration: Bayes' rule over log sigma.
- Recipe lists: robust adds 17, 33, 40 and normalization 33, 40, 41, in
  cell 0, design.md and the index test.
- design.md: runtimes re-timed on a quiet machine (robust 64 s,
  normalization 318 s, hierarchical 294 s); emcee unmixed in tau too.
job.sbatch and env.sh no longer carry a site's account, partition or venv.
submit.py takes --account, --partition, --constraint, --venv (exported to
env.sh as RXMC_VENV) and --sbatch for any other option, and reads them from
an @file kept outside the repository.
Three dynesty runs put 0.55, 0.69 and 0.80 of alpha_gp's mass in the deeper
family, too scattered to test the pocomc pool's 0.740 against the 0.03 bar;
six more seeds (3003-3008) take task indices 6-11 of its ref job.
84 tasks on amd20 nodes at bf6e086/12db2ca: references, the samplers, the
pocomc sweeps and gp_discrepancy's seed check (results/hpc/), merged into
results/samplers.json and references.json.  All three references are
accepted; omp's and n40ca_gp's are corroborated by dynesty.  alpha_gp's is
not: nine dynesty runs pool to a deeper-family share of 0.787 +/- 0.035
against pocomc's 0.740, a gap within dynesty's own scatter, and log Z and
the moments agree; it is kept, marked corroborated: false, and the docs say
why.  docs/benchmarks.md: the new rows, references and correct/fastest
entries; the discussion's omp numbers are still the earlier version's.
…esults

The omp text quoted the earlier parametrisation; it now reads from the
cluster refresh, and covers alpha_gp and n40ca_gp too.  The sweep table is
the new one for all three problems (4096/2048 correct on every seed but
one, omp's seed 100); the discussion's bullets on separated modes, the
ensembles, the evidence, PyMC against blackjax and the cost per call carry
the new numbers.  A new section sets gp_seed_check's three seed sets
against the gp_discrepancy notebook: its log Z, Delta log Z and volume
medians hold to a small fraction of their intervals.
proportional_error and normalization are fractions of the prediction
(or of the data and prediction averaged), which is a relative error only
in linear space.  On a comparison with space=log the term sees log(ym),
so the "fraction" scaled with log(sigma) in whatever units the data
carry.  A Constraint now refuses them there (Term(relative=True)), with
a message naming the log-space equivalents: in log space a multiplicative
error is additive, so an uncorrelated relative error is noise() and a
normalisation is offset().  reported_terms() already maps reported
uncertainties by the delta method and is unaffected.

This is a change of public behaviour: code that combined either term
with a log-space comparison now fails at Constraint construction.

The guard found two such uses besides gp_discrepancy's (fixed with its
rerun): recipe 18's L2y ("a normalisation mode, sys * ym") in log space
is L2n's offset mode, so L2y is gone from the legend and recipe 18 and
the design example use L2n; a test_problem test now builds its log-space
mode with offset.  design.md's terms section gains the linear/log table.
On the three optical-model problems the reference is pooled pocomc,
checked by pooled dynesty, and the two differ by up to 0.07 (omp), 0.16
(alpha_gp) and 0.15 (n40ca_gp) sd in their moments, which is coarser
than the 0.1 tolerance.  That gap is now the reference's
resolution r (bench_samplers.resolution): an error more than r above the
tolerance is wrong (✗), one within r of it either way is unresolved (?),
and a run is correct unless shown wrong.  Exact and NUTS references keep
r = 0, so the other problems' verdicts are unchanged.  The sweep report
uses the same verdict; the uncorroborated-reference warning goes to
stderr, out of the Markdown.

docs/benchmarks.md, from the three reviews and the regenerated tables:
- every n40ca_gp sampler but emcee is within resolution; HMC is fastest,
  pocomc fastest with log Z; omp and alpha_gp's pocomc are ?;
- 4096/2048 is wrong on one seed of nine (omp's 100), every lower
  setting on at least one seed per problem but alpha_gp's 2048/1024;
- the dynesty/pocomc settings rule stated as it is (every setting swept
  is correct on elastic, none on both funnels; rslice 250 on gp alone),
  the sweep's data noted as not in results/;
- the scoring, references and alpha_gp-corroboration text says what r
  is and why alpha_gp's verdicts are ?; pooled log Z error explained;
- elastic's vmapped row is 3.0 ms, not 3.9; the PyMC and per-call
  bullets merged and shortened; the funnel bullet shortened;
- history wording removed ("an earlier run", "with three runs the gap
  was"); prose math in LaTeX.
The gp seed-check section follows with the gp_discrepancy rerun.
Narrative that referred to earlier runs, versions or process now states
the result:
- local_optical: omp is this notebook's problem, not "its version ...
  which differs in the parametrisation and priors"; the gradient-sampler
  numbers are the refreshed ones (HMC and PyMC 0.30 to 0.32 off at R-hat
  1.005 to 1.007; blackjax NUTS the one flagged, 1.69); the island share
  13 % in the reference, 14 to 16 % at this setting, 23 to 85 % at a
  quarter of it; the linear model's setting justified by elastic and by
  its evidence, 726 below.
- alpha_ca: the n_seeds = 3 run that was never committed is replaced by
  the benchmark's three runs of the same problem (log Z -84.50 to -84.55;
  71 to 74 % in the deeper family, dynesty's check 79 %); the reference's
  self-agreement is stated; "a few minutes" is half an hour.
- design.md: the stale omp parenthetical, "a three-run result quoted",
  "the 0.x quartic", "the 0.x pins", "was removed", "JAX rxmc", "as it
  is"; the performance summary covers alpha_gp and n40ca_gp, and the
  rule for the notebooks' samplers is stated as applied; the release
  section states the release path, not the milestone history; the
  hierarchical "in detail" paragraph, a restatement of the notebook, is
  gone; the gradient row's 50-tau claim corrected.
- docs/groundup_design.md, the plan the rewrite was built from, moves
  out of the repository with the other plans; its pointers are gone.
- README: pocomc by preference for optical models (not nested sampling),
  pocomc among the samplers, one link to the documentation.
- recipes.md: the "rewrite must support" sentence, "pre-0.1", "the
  authors have confirmed", section numbers (§6); recipe 12's coverage
  claim qualified.  index.rst points to installation for 0.x.
- Notebooks: "the 0.x normalisation example", KDUQ's tempering as the
  authors' private communication, black-box-bayes's name, robust's
  scaled term on comp.y_err as recipe 34 has it.
- Prose math in design.md, recipes.md and README in LaTeX.
The cross-section ladder's "proportional error" rung put
proportional_error on a log-space comparison, so its error scaled with
log(sigma in b/sr): about 0.26 gamma forward and 5.5 gamma backward, in
whatever units the data carry.  In log space a relative uncorrelated
error is noise(); the rung (and the benchmark's copy of the ladder) now
uses it.

pocomc now runs the ladder at 4096/2048: on n40ca_gp, this notebook's GP
rung, the sweep finds 1024/512 wrong on two seeds of three (0.29 and 0.40
sd, beyond the reference's resolution), while every 4096/2048 seed is
within it.

Rerun (1256 s): Delta log Z 558.3 (proportional over bare) and 12.8 (GP
over proportional, was 7.1); the proportional error is 6 sd out in V_v,
4 in R_v and 10 in W_v, the GP 2.7 in V_v and 8 to 9.6 elsewhere; the
envelope holds the defect over 58 % of the grid.  The toy half is
unchanged.  The seed check reruns at the same settings: log Z within
0.06, Delta log Z 12.8 in every set, medians within 0.16 of a half-width,
and the GP rung within 0.12 sd of n40ca_gp's reference.  Its nine
cluster pieces, of the old rung, are removed; the JSON is the local run.
- docs/conf.py enables MyST's dollarmath and amsmath.  Without them no
  page but the API reference had a math node, so MathJax never loaded and
  every $...$ in the docs and the notebooks showed as literal text.
- benchmarks: verdict() calls a NaN error wrong, not unresolved;
  resolution()'s docstring says which references have r = 0.
- The guard's message names the refused space and says every non-identity
  space is refused; its test covers on=None across mixed spaces and on= a
  dataset.
- Statements about pocomc at 4096/2048 say what the benchmark shows (not
  shown wrong; wrong on one omp seed of three) in recipes.md,
  local_optical and gp_discrepancy; design.md's rule for the notebooks'
  samplers asks for evidence that agrees with the reference.
- docs/benchmarks.md: the gp seed-check section from the rerun; dynesty's
  share range 0.55 to 0.90; pocomc's ESS on gp about 4500; the density
  baseline named as rxmc 0.x (v0.1.0); prose uncertainties in LaTeX.
- Pointers to the removed groundup_design.md in terms.py and pyproject.
Rerunning for the figure titles ("vermillion", for Wong's colour, not
"orange") showed that the GP's NUTS at target acceptance 0.95 is not
reliably divergence-free: this run gave 17 divergences and R-hat 1.010
where the committed one gave none, and three seeds gave 4 to 17.  The
run is deterministic within an environment (the same seed twice gives
the same chains), so the committed run was a favourable one.  At 0.99
three seeds gave no divergences; with 2000 draws per chain the R-hat
is 1.004 to 1.007.

The notebook now runs the GP's NUTS at 0.99 with 2000 draws: 904
effective draws in 60 s, R-hat 1.004, no divergences; emcee is about six
times faster (96 against 15 per second); the 0.8 run gives 41
divergences.  The text and the design row follow (runtime 366 s).
…ces; dynesty's settings swept

- reference.py: alpha_gp and n40ca_gp pool fifty dynesty runs (seeds
  3000-3049), not nine and three: single runs scatter about 0.1 in
  alpha_gp's mode share against a 0.03 bar, and the pools' moment gaps (0.16,
  0.15 sd) set those references' resolution.  Existing task indices keep
  their runs (new ones are 12-52 and 6-52).
- sweep_pocomc.py sweeps dynesty's settings as well as pocomc's: on gp,
  modes and elastic (PRIOR_PROBLEMS) seven settings, dynesty rwalk 200/500,
  rslice 250/500 and pocomc 512/256 to 2048/1024, so the docs' settings
  table can be regenerated from committed data.  Settings are named by
  label ("pocomc 1024/512", "dynesty rslice 250"); runs saved before keep
  reading as pocomc.  --grid prints several problems' sweeps side by side.
- hpc: sweep tasks come from settings_for(problem); submit.py selects sweep
  settings by label and any kind by --indices; ref walltimes for alpha_gp
  and n40ca_gp follow their cluster runs (2 h, 1 h).
- Tests for the labels, the grid, --indices and a dynesty sweep end to end.

Claude-Session: https://claude.ai/code/session_012XSD4kQVCWxa2N29LhXjnu
…p's data committed

- alpha_gp: fifty dynesty runs pool to a deeper-family share of 0.756
  (standard error 0.015) against pocomc's 0.740, within the 0.03 bar; the
  moment gap falls from 0.16 to 0.05, so the reference is corroborated and
  its ? marks go.  n40ca_gp's fifty move its gap to 0.07 / 0.12.
- The dynesty/pocomc settings table on gp, modes and elastic is regenerated
  from cluster runs (sweep_pocomc.py --grid, seeds 100-102).  Every setting
  is correct on elastic; none on gp or modes on more than one seed, and the
  earlier local result for rslice 250 on gp does not reproduce.
- docs: the pooled log Z error (half the runs' range) grows with the number
  of runs; against the standard error, dynesty is about three errors below
  pocomc on n40ca_gp.

Claude-Session: https://claude.ai/code/session_012XSD4kQVCWxa2N29LhXjnu

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant