set_neighbors: stream neighbourhoods into DuckLake instead of materialising the edge list

[mmyrte]

Last of the stack (#59–#65 are merged); now based on develop.

What

set_neighbors() used to build the whole neighbour edge list in memory with distance_neighbors_cpp(), key it, add a cut() factor, and only then commit it in chunks. At 500 m around 100 m cells that is 79 pairs per cell. So the edge list, not DuckLake, set evoland's memory ceiling.

distance_neighbors_chunked_cpp() uses the same spatial hash index but flushes as it goes:

• It hands each chunk of complete neighbourhoods (all neighbours of a set of origins, in both directions) to an R callback. set_neighbors() commits each chunk inside one transaction: the first chunk with overwrite, the rest with append.
• Chunks are disjoint in id_coord_origin, so the table stays unique by construction.
• Distance classes are coded in C++ with the semantics of cut(right = FALSE, include.lowest = TRUE), and each chunk is labelled with the labels cut() would give.
• The pair count is returned as a double, since it can exceed R's 32-bit integers.

The in-memory distance_neighbors_cpp() and the exported create_neighbors_t() are removed. They were only used in tests. A caller who wants the whole table in memory can pass an accumulating callback. Input validation (max_distance, distance_breaks) moved to set_neighbors().

Memory is bounded by chunksize. Its default goes from 1e8 to 1e7 pairs, since it now bounds the computation as well as the writing. With no pairs within max_distance, an empty table is still committed.

Effect

Measured on the PIE benchmark tiled k × k (mmyrte/evoland-experiments 2026-09-model-comparison/080-scaling; 4 cores, 16 GiB; logistic regression; whole calibration step: set-up, neighbours, fit, prediction):

cells	before	after
1.0 M	204 s, 10.1 GB peak	198 s, 3.7 GB (neighbours 80 s → 55 s)
1.8 M	out of memory	290 s, 5.6 GB
4.1 M (≈ Swiss 100 m grid)	killed for memory	599 s, 9.8 GB

At 4.1 M cells, the remaining peak comes before the neighbour step, from the benchmark's own data ingestion.

Tests

• test_neighbors_t.R tests distance_neighbors_chunked_cpp() directly, without a DB, using an accumulating callback. It covers:
  • symmetry and distances;
  • distance classes, including the closed last break and NA outside the breaks;
  • empty results, dense points and hash-cell boundaries;
  • progress output;
  • chunking: chunks are disjoint in id_coord_origin, every chunk except the last reaches chunk_rows, and their union equals the single-chunk result.
• test_db_evoland.R checks only the DB behaviour: an empty table for a max_distance with no pairs, then many small chunks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D1LLswXe3jLjGTFrBsxgPf

[mmyrte]

In general: drop the non-chunked/non-callback version. the callback could always be an accumulator if the users really wished to construct the full table in memory.