Skip to content

alloc_greedy: deterministic rank-and-fill allocator - #65

Merged
mmyrte merged 6 commits into
claude/gifted-lamport-41q8gv-06-deterministic-training-orderfrom
claude/gifted-lamport-41q8gv-07-alloc-greedy
Oct 10, 2026
Merged

mmyrte merged 6 commits into
claude/gifted-lamport-41q8gv-06-deterministic-training-orderfrom
claude/gifted-lamport-41q8gv-07-alloc-greedy

Conversation

@mmyrte

@mmyrte mmyrte commented Oct 6, 2026

Copy link
Copy Markdown
Member

Stack 7, on top of #64.

What

alloc_greedy() / db$alloc_greedy() / alloc_greedy_one_period() is a deterministic allocator. Each transition's demanded count (trans_rates_t$count, or rate × n_anterior where the count is missing) goes to the cells with the highest adjusted potential (adjusted_trans_pot_v()). Every cell changes at most once, and no transition exceeds its demand. There is no randomness and no patch geometry. Entry points and arguments mirror alloc_clumpy().

Two ways to settle the competition between transitions for a cell (arbitration):

  • "joint" (default): one ranking of all (cell, transition) pairs by adjusted potential. The adjusted potentials are scaled to each transition's rate, so they are comparable across transitions.
  • "ordered": transitions claim cells in the priority given by order (a vector of id_trans), each taking its best remaining cells. This is the Ordered procedure of Fuchs et al. (2013) as lulcc implements it (lulcc:::.ordered), with transitions in place of classes.

The allocator works on id_coord rows only and never builds a raster, so it applies to any tessellation coords_t can describe. The sequential walk is a small Rcpp kernel (greedy_fill_cpp). Ties are broken by id_trans, then id_coord, so the result does not depend on the order DuckDB returns rows in.

Why

It is a comparator for the stochastic allocators: the same family as lulcc's Ordered model and SEALS' rank-and-fill. On the PIE benchmark (mmyrte/evoland-experiments, 2026-09-model-comparison), single-map figure of merit orders the allocators by how greedy they are:

allocator (same potentials) figure of merit
CLUMPY 0.026 (logistic regression) / 0.029 (random forest)
Dinamica 0.034
greedy 0.038–0.043

An ensemble-level score reverses that order.

Tests

  • test_alloc_greedy.R: unit tests of the kernel (order, one change per cell, quotas, index checks).
  • test_integ_allocation.R: the allocator is deterministic, returns a complete map, never exceeds the demanded counts, and within each transition the changed cells have at least the potential of every untouched candidate; the ordered mode is exercised too.
  • Full tinytest suite locally: 644 results, all pass. air format --check is clean.

Note: roxygen also wanted to rewrite man/ducklake_db.Rd, man/lulc_data_t.Rd and man/pred_data_t.Rd. They are already out of sync on develop, so I left them out of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D1LLswXe3jLjGTFrBsxgPf


Generated by Claude Code

Fills each transition's demanded count with the cells of highest adjusted
potential; every cell changes at most once. Competition between transitions is
settled either by one joint ranking ('joint') or by a transition priority
('ordered', the Ordered procedure of Fuchs et al. 2013 as in lulcc). Works on
id_coord rows only, so it is independent of the tessellation. The walk is a
small Rcpp kernel (greedy_fill_cpp); ties are broken by id_trans, id_coord.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1LLswXe3jLjGTFrBsxgPf
@mmyrte

mmyrte commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

@claude I'm sure that the c++ approach really is not the right one; the R based logic in evoland-experiments is not idiomatic at all and shouldn't be sped up by transplanting this flawed logic to C++. Is there a way the same change could be modelled using a top-n query in data.table?

…walk

greedy_fill() takes each transition's top-n remaining candidates, keeps the earliest pick
per cell, and accepts the picks before the first candidate a losing transition would take
next; repeated until quotas or candidates run out. This gives exactly the sequential walk's
result (randomised check against a reference walk in the tests; identical PIE maps for
both estimators) without the C++ kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1LLswXe3jLjGTFrBsxgPf

mmyrte commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

Yes. 8e9e3d2 replaces the C++ walk with greedy_fill(), which runs repeated top-n queries in data.table, and deletes src/alloc_greedy.cpp.

A single top-n per transition isn't enough on its own. Once a cell goes to a higher-ranked transition, the other transitions that wanted it need their next-best cell, and that cell can in turn take a cell another transition was counting on. So the query runs in rounds:

  1. Each transition takes its top quota remaining candidates: rowid(id_trans) <= quota on the candidates sorted by priority.
  2. A cell picked by several transitions keeps the earliest pick: ord == min(ord), by = id_coord.
  3. The picks are final up to the "horizon": the first candidate that a transition which lost a cell would take next. Before that point, the sequential walk would see exactly these picks. Picks before the horizon are accepted; their cells and quotas are removed, and the next round starts.

Each round accepts at least the earliest pick, and usually all of them, so only a few rounds are needed. The joint and ordered modes share this code; the only difference is the sort order of the candidates.

Checks:

  • The tests compare the result with a plain R reference walk on 200 random cases in both modes.
  • On the PIE benchmark, the rewrite gives the same map as the C++ version, cell for cell, for both the logistic regression and the random forest runs. Each period takes about 2 s, including the DB reads.
  • The full suite passes with at_home = TRUE (643 tests).

I've merged the change forward into the #66 branch, where RcppExports is regenerated and the suite also passes (650 tests).


Generated by Claude Code

…order' into claude/gifted-lamport-41q8gv-07-alloc-greedy
…order' into claude/gifted-lamport-41q8gv-07-alloc-greedy
@mmyrte

mmyrte commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

@claude I misformulated: I did not think that a C++ implementation in general is wrong, just that the R side of your implementation was unidiomatic and hard to follow. the current function interface would have made the cpp much more legible. if the current implementation is indeed as fast as the c++ version, even with some tied ranks, let's stick with this. otherwise you can go back to a cpp implementation.

the current function has a good description in the roxygen tag, but that's not where developers are going to need the explanation: there are not enough comments (and too little whitespace structure) to follow along. be a little more generous with explanations in the code.

claude added 2 commits October 7, 2026 15:36
The top-n rounds in data.table were about 10x slower than the C++ loop on 4 M cells
(6.9 s vs 0.7 s, identical results incl. tied potentials). greedy_fill(candidates, quota)
keeps its interface and now hands id_coord straight to greedy_fill_cpp() as the cell index
(no remapping), translating only id_trans to quota rows. More comments through
alloc_greedy_one_period() and the kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1LLswXe3jLjGTFrBsxgPf
"joint" ranks equal potentials by id_trans, then id_coord, so it is not symmetric in the
transitions; "ordered" ranks ties within a transition by id_coord. Both are needed for
determinism and carry no meaning; say so in the Description, @PARAM arbitration and the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1LLswXe3jLjGTFrBsxgPf

mmyrte commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

I benchmarked the data.table version against the C++ walk on synthetic candidates with tied potentials (values rounded to 1–2 digits). Both give identical results. The data.table rounds are about 10× slower:

cells candidates data.table C++
0.1 M 0.13 M 0.16–0.25 s 0.04 s
1 M 1.3 M 1.3–1.4 s 0.13–0.16 s
4 M 5.4 M 6.9 s 0.6–0.8 s

So 355fd80 goes back to C++ and keeps the greedy_fill(candidates, quota) interface:

  • greedy_fill() passes id_coord directly to greedy_fill_cpp() as the cell index, so there is no unique()/match() remap. That remap had taken most of the time on the R side (2.3 s down to 0.7 s at 4 M cells). Only id_trans is mapped to rows of quota.
  • I added comments and blank-line structure to greedy_fill(), to alloc_greedy_one_period() (sections for quota, candidates, order and walk, with what each step does and why), and to the C++ kernel.
  • The tests keep the randomised comparison against the plain-R reference walk, plus the out-of-range checks on the kernel.
  • The PIE maps are still identical cell for cell, and the full suite passes with at_home = TRUE.

d05e437 documents the tie-breaking, in the Description, in @param arbitration and in the code comments. To stay deterministic, "joint" ranks equal potentials by id_trans, then id_coord. Neither key means anything, and the Description now says so. Where two transitions tie at a cell, the lower id_trans gets it, so renumbering the transitions can change the result. "ordered" breaks ties within a transition by id_coord, which usually follows raster row order. The Description also notes that ties are common with coarse potentials such as classification trees or rounded inputs.

Both commits are merged forward into #66.


Generated by Claude Code

@mmyrte
mmyrte merged commit cecdded into develop Oct 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants