diff --git a/docs/figures/spatial-chunk-alignment.png b/docs/figures/spatial-chunk-alignment.png new file mode 100644 index 0000000..d3bdd25 Binary files /dev/null and b/docs/figures/spatial-chunk-alignment.png differ diff --git a/docs/image/improvement_spatial/1792006397731.png b/docs/image/improvement_spatial/1792006397731.png new file mode 100644 index 0000000..4efce35 Binary files /dev/null and b/docs/image/improvement_spatial/1792006397731.png differ diff --git a/docs/improvement_dtype.md b/docs/improvement_dtype.md new file mode 100644 index 0000000..eb7d68d --- /dev/null +++ b/docs/improvement_dtype.md @@ -0,0 +1,458 @@ +# A shared standard for data types and missing values + +The objectives are: + +- **Storage:** store each layer in the smallest type that holds its values, without changing them. +- **Accuracy:** never change a value silently, and never lose track of missing data during a calculation. +- **Meaning:** for every cell, tell apart a **value** (zero included), a **null** (the source says it doesn't know), a cell **outside the source's coverage**, and a cell that is **not available** because processing failed or hasn't run yet. + +The proposal is split into three contracts, which can be adopted and tested one after the other: + +1. **Values:** what is stored in each cell, in which type, and how it was transformed. +2. **Availability:** where a source has data, and whether processing of a tile, variable and year is complete. +3. **Calculations:** which inputs each result needs, what happens when one is missing, and what each total covers. + +## How Zarr stores data + +A Zarr array is a grid of cells, all of the same type, cut into blocks and saved as separate objects in a bucket: + +```text + one 1° tile = one shard (one object) one chunk = 1,000 × 1,000 cells + ┌──────┬──────┬──────┬──────┐ each cell: 1 byte in uint8 + │chunk │chunk │chunk │chunk │ → 1 MB before compression, + ├──────┼──────┼──────┼──────┤ much less after + │chunk │chunk │empty │empty │ + ├──────┼──────┼──────┼──────┤ the index at the end of the shard + │chunk │empty │empty │empty │ says where each chunk starts, + ├──────┼──────┼──────┼──────┤ or that it is empty + │empty │empty │empty │empty │ + └──────┴──────┴──────┴──────┘ + + index +``` + +- **Chunks.** The array is cut into chunks of 0.25° (1,000 × 1,000 cells on the 30 m grid). A chunk is the smallest piece that is read or written, and it is compressed before it is saved. +- **Shards.** The 16 chunks of one 1° tile are saved together in one object, the shard, with a small index at its end that says where each chunk is. +- **Size of a cell.** Every cell takes the size of the array's type, whatever its value: 1 byte in `uint8`, 2 in `uint16`, 4 in `float32`. Storing 0, 2 or 255 costs the same. +- **Metadata.** Next to the data, a small `zarr.json` file gives the type, the shape, the chunk size, the `fill_value` and free-form attributes (units, meaning of codes, source). + +**Empty costs almost nothing.** If every cell of a chunk equals the `fill_value`, the chunk is not saved: the shard index just marks it as empty (16 bytes). If all 16 chunks of a tile are empty, the shard itself is not saved, and the tile costs nothing. Reading an empty chunk or tile gives back the `fill_value` for every cell. For example, one year of GPW grassland on the grid is stored in 16,962 shards, out of 64,800 1° tiles over the globe: the other tiles, mostly ocean, are not stored at all. + +**Partly empty chunks are cheap too.** A chunk with some data is saved whole, so each of its cells needs a value: the cells without data hold the `fill_value`. A long run of the same value compresses to almost nothing, so these cells add very little. + +**What costs space is variety, not the values themselves.** A chunk full of different values compresses less than a uniform one. Reserving codes for missing data (e.g. 254 and 255 in `uint8`) therefore doesn't make an array bigger. It only decides what those bytes mean. + +### Folder structure and where the metadata is written + +A Zarr v3 store is a folder (or a prefix in a bucket). Each **group** and each **array** is a sub-folder with its own `zarr.json`. The data of an array is under its `c/` folder, with **one file per shard**. Chunks don't have their own files: they are inside their shard. + +```text +raw.zarr/ +├── zarr.json ① root group: the grid +├── gpw/ one source = one group +│ ├── zarr.json ② source group: where the data comes from +│ ├── grassland_class/ one variable = one array +│ │ ├── zarr.json ③ array: type, shape, shards, fill value, codes, mapping +│ │ └── c/ data: one file per shard +│ │ ├── 10/82/279 year index 10 (2010), tile x279y082 +│ │ ├── 10/82/280 the next tile to the east +│ │ └── … empty tiles (ocean): no file at all +│ ├── grassland_class_coverage/ +│ │ ├── zarr.json ④ coverage array: where the source applies +│ │ └── c/… +│ ├── year/ zarr.json + c/0 coordinates: one small array each +│ ├── y/ zarr.json + c/0 +│ ├── x/ zarr.json + c/0 +│ └── _done/ ⑤ completion markers (plain files, not Zarr) +│ └── grassland_class/2010/x279y082.json +├── gnw_tcl/ same layout for every source +│ └── … +└── … +``` + +The name of a shard file is its position in the array: `c/{year}/{row}/{column}`, counted in shards. With 1° shards, row 82 and column 279 is tile `x279y082`. A reader goes straight to the file it needs, without listing the folder. + +Each level holds a different part of the metadata: + +| File | Level | What it holds | +| ------------------------------------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ① `raw.zarr/zarr.json` | root group | the grid shared by every source: coordinate system, cell size, origin, grid name and version | +| ② `gpw/zarr.json` | source group | the source: product, release, download URL, ingest date, code version | +| ③ `gpw/grassland_class/zarr.json` | array | the Zarr fields (type, shape, shard and chunk size, compression, `fill_value`) and the attributes that describe the values: codes and their meanings, units, null code, mapping from the source, resampling | +| ④ `gpw/grassland_class_coverage/zarr.json` | array | the same for the coverage array: one per variable, with the same dimensions (year, y, x) | +| ⑤ `gpw/_done/…/x279y082.json` | plain file | completion of one tile, variable and year: `complete` or `failed`, with the source files used and the reason | + +Zarr readers ignore `_done/`, because it has no `zarr.json`. Only the project reader uses it. + +The array's `zarr.json` (③) in full. The first part is defined by Zarr; `attributes` is ours: + +```json +{ + "zarr_format": 3, + "node_type": "array", + "shape": [25, 720000, 1440000], + "data_type": "uint8", + "dimension_names": ["year", "y", "x"], + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [1, 4000, 4000]}}, + "chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}}, + "fill_value": 255, + "codecs": [{ + "name": "sharding_indexed", + "configuration": { + "chunk_shape": [1, 1000, 1000], + "codecs": [ + {"name": "bytes"}, + {"name": "blosc", "configuration": {"cname": "zstd", "clevel": 5, "shuffle": "bitshuffle", "typesize": 1, "blocksize": 0}} + ], + "index_codecs": [{"name": "bytes", "configuration": {"endian": "little"}}, {"name": "crc32c"}], + "index_location": "end" + } + }], + "attributes": { + "_FillValue": 255, + "missing_value": [254], + "flag_values": [0, 1, 2], + "flag_meanings": "not_grassland cultivated_grassland natural_grassland", + "state_codes": {"null": 254, "nothing_written": 255}, + "mapping": "identity", + "resampling": "none" + } +} +``` + +- `shape`: 25 years × 720,000 rows × 1,440,000 columns, i.e. the whole globe at 1/4000°. +- `chunk_grid`: the size of one **shard**, one year of one 1° tile (4,000 × 4,000 cells). This is what becomes one file. +- `sharding_indexed` → `chunk_shape`: the size of one **chunk** inside the shard, 0.25° (1,000 × 1,000 cells), and how each chunk is compressed. +- `index_location: end`: the shard's index (where each chunk starts, or that it is empty) is at the end of the file. +- `fill_value`: what a reader gets for a chunk or shard that was never written. + +`run.zarr` follows the same layout: one group per step (e.g. `emit/`), one array per result layer, with the same root grid and the same kind of metadata. + +## Motivation + +### Everything becomes float32 today + +`geo.unify_dtype_and_no_data` turns every band's nodata into NaN and casts the band to float32 before harmonize writes it. Most source bands are small integers: + +| Dataset | Source type | Content | +| --------------------------------------------- | ----------- | ------------------------------------------ | +| `gnw_tcl` | `uint8` | year of tree-cover loss | +| `gpw_grassland` | `uint8` | grassland class | +| `liao_gaced30` | `uint8` | cropland class | +| `gnw_global_peatlands` | `uint8` | peat flag | +| `ipcc_climate_zones` | `uint8` | climate zone code | +| `usda_nass_cdl` | `uint8` | crop class | +| `gnw_harris_agb` | `uint16` | aboveground biomass | +| `descals_oil_palm` | `uint16` | planting year | +| `soilgrids_ocs` | `int16` | soil organic carbon stock | +| `huang_bgb`, `gpw_livestock`, `ifpri_mapspam` | `float32` | belowground biomass, livestock, crop areas | + +Casting a `uint8` band to float32 makes it 4× bigger before compression, and floats usually compress worse than small integers. Harmonize is 92% of the cache (1.2 TiB on the continent-scale benchmark), and this cast is expected to be a large part of it. Class codes also become floats (`3` becomes `3.0`), so a reader can no longer tell a code from a measurement. For large IDs, float32 is not just bigger but wrong: above 16,777,216, two different integers can become the same float. + +### One NaN for several meanings + +After harmonize, a NaN can mean any of these: + +- the source does not cover this place (outside its published extent, ocean); +- the source covers it but has no value there (cloud, sensor gap, water or rock in SoilGrids); +- the source file was missing or corrupted, so nothing was ingested; +- the layer has not been processed yet for this tile. + +These call for different responses. The first is a methodology question (what to do where a source doesn't apply). The second is a scientific gap (keep it missing, or apply a declared assumption). The last two are processing failures: the work is incomplete and should be retried, not published as a result. The pipeline cannot tell them apart today. + +### Missing data is silently turned into zero + +Because NaN is ambiguous, emit replaces it with zero to keep the arithmetic going: + +- `get_mineral_soil_emissions` uses `soil_organic_carbon.fillna(0)`, so a SoilGrids gap counts as "no soil carbon". The code comment explains why: otherwise the NaN would also discard the cell's vegetation emissions. +- `get_grassland_carbon` and `get_mineral_soil_emissions` use `climate_zones.fillna(0)`. Zone 0 is not in the lookup table, so a cell with no climate zone gets zero carbon. + +Each choice may be right, but it is hidden in the code, applies to every reason a value can be missing, and is not recorded in the results. A total computed this way looks complete even when part of its area had no input. + +### No-data conventions differ by dataset + +The `no_data` field of each dataset uses a different convention: 255 for most `uint8` layers, 0 for CDL (where 0 is CDL's "background"), 65535 for Harris AGB, 32767 for SoilGrids, −1 or NaN for MapSPAM, and `None` (no nodata at all) for TCL, Descals, Huang BGB and GPW livestock. Nothing defines which value means what. + +### The Zarr fill-value trap + +In Zarr, a chunk that was never written is not an error: reading it returns the array's `fill_value`. This is useful (empty ocean costs nothing), but it hides the reason. With `fill_value: 0`, a chunk that was never written and a chunk measured as "not grassland" (0) read back exactly the same. The Xu peat layer rasterized on the grid shows this: 0 means "not peat", so a cell outside the source's coverage and a non-peat cell look the same. + +Even with a fill value that can't be confused with data, **an unwritten chunk can't say why it is empty**: the source doesn't cover it, the file failed, or ingest hasn't run yet. That answer has to come from a separate record. + +## Contract 1: values + +### 1.1 Keep the source type, losslessly + +Each band is stored in its source type, with its values unchanged: + +| Kind of data | Stored as | Example | +| -------------------- | ------------------------------------------- | ------------------------------------------- | +| masks and flags | `uint8`: 0, 1 | peat, protected area | +| classes | `uint8` (or `uint16`), codes unchanged | GPW grassland, GACED30, climate zones, CDL | +| years | source type (`uint8` or `uint16`) | TCL loss year, Descals planting year | +| IDs | `uint16` or `uint32` | admin units | +| integer measurements | source type | Harris AGB (`uint16`), SoilGrids (`int16`) | +| float measurements | `float32` | Huang BGB, livestock density, MapSPAM areas | +| per-cell results | `float32` by default; `float64` if declared | t CO₂e/ha | + +Rules: + +- **Values are bit-equal to the source** for every band that isn't transformed. This is a test at ingest. +- **Class codes, flags, IDs and years are never cast to float**, in storage or when read. +- **float32 is the default for measurements and results**, not an absolute rule. A layer may use float64 when its declared error budget requires it. +- **`bool` is not used for stored layers.** It has no room for a missing marker, and `uint8` costs about the same once compressed. + +### 1.2 The metadata describes the values and the mapping from the source + +Each array's metadata (its `zarr.json`) says two things: + +1. **What each stored value means:** units for a measurement, or the list of codes and their meanings for classes and flags. +2. **The mapping from the source:** which source value became which stored value. For most layers it is the identity (every value is kept as it is); for some, a few codes are changed. + +The metadata is the definition: the stored data must follow it, and a reader needs nothing else to interpret a cell. + +Identity, e.g. GPW grassland, whose source codes are kept as they are: + +```json +"attributes": { + "flag_values": [0, 1, 2], + "flag_meanings": "not_grassland cultivated_grassland natural_grassland", + "mapping": "identity" +} +``` + +A few codes changed, e.g. CDL, whose 0 ("background", outside the United States) becomes our standard 255: + +```json +"attributes": { + "flag_values": [1, 5, 24, ...], + "flag_meanings": "corn soybeans winter_wheat ...", + "mapping": {"0": 255, "other values": "unchanged"} +} +``` + +Ingest checks that the data follows the metadata: with an identity mapping, every cell is exactly the source's value; with a table, applying the table to the source gives exactly the stored data. + +**Moving a source onto the grid is declared too.** A source on another grid (e.g. GACED30, at 1/3710.65° instead of 1/4000°) has to be resampled: each cell of our grid takes the value of the closest source cell. The values are the same codes, but some source cells are used twice and others not at all. The metadata records the method (`"resampling": "nearest"`), and ingest checks that each 1° tile keeps the same share of each class as the source, within an agreed tolerance. + +**A scale factor and an offset are allowed, as a declared mapping.** A measurement can be stored as an integer, with two numbers in the metadata that turn it back into the real value: + +```text +stored = round((value − add_offset) / scale_factor) +value = stored × scale_factor + add_offset +``` + +For example, biomass 85.37 Mg/ha with `scale_factor: 0.1` and `add_offset: 0` is stored as `854` in `uint16`, and read back as 85.4. xarray, GDAL and QGIS apply the two numbers by themselves when they read. + +```json +"attributes": { + "units": "Mg/ha", + "scale_factor": 0.1, + "add_offset": 0, + "mapping": {"scale": "round to nearest", "max_error": 0.05} +} +``` + +- **An offset alone is lossless.** Years from 2000 to 2253 fit in `uint8` with `add_offset: 2000` and `scale_factor: 1`: 2015 is stored as 15, and nothing is rounded. +- **A scale factor rounds the value**, after the division. The rounding is always to the **nearest** integer, never a cut (85.37 → 854, not 853), because cutting always goes down and would make every total slightly too small. The largest error is half a step: 0.05 with a scale of 0.1. +- **The whole range must fit** below the reserved codes. With `uint16` and a scale of 0.1, the largest value is 6,553.3. Ingest checks the source's maximum, and refuses the layer if it doesn't fit, rather than cutting the largest values. +- **Missing values are mapped first.** A source NaN or nodata becomes the null code before the scale is applied, so it is never rounded into a number. +- **Ingest checks the result.** It reads the stored integers back, compares them with the source, and records the largest difference in the metadata. It must be at most half a step. + +Rounding changes values, and a small change can move a cell across a threshold (e.g. 25.04 rounded to 25.0 under a threshold of 25). So for now, **no layer uses a scale factor**. One may be used later, layer by layer, when the storage gain is measured and the emissions results are shown not to change. An offset alone, being lossless, can be used at any time. + +### 1.3 Accumulate totals in float64 + +Per-cell results are float32, but a jurisdiction total adds up millions of cells, and float32 accumulation loses digits. Rollups use float64 accumulators. + +## Contract 2: availability + +### 2.1 Four answers for every cell + +| State | Meaning | Example | Where it comes from | +| -------------------- | ------------------------------------------------------- | ------------------------------------------------ | ----------------------------------------- | +| **value** | the source gives a value (zero included) | class 2 = natural grassland; 0 = not grassland | the data | +| **null** | the source covers the cell but says it doesn't know | cloud, sensor gap, SoilGrids water/rock/ice | the data, declared per dataset | +| **outside coverage** | the source does not apply here, by its published extent | CDL outside the United States; ocean | the coverage record of the source release | +| **not available** | processing failed or hasn't run for this tile | file corrupted or missing; tile not ingested yet | the completion record | + +Zero is an ordinary value. Each of the other three needs a different response: + +- **null:** keep it missing, or apply an assumption that the calculation declares (contract 3); +- **outside coverage:** apply the methodology's coverage rule (e.g. another layer is used outside the United States); +- **not available:** the work is incomplete. It is retried, and by default blocks publication of the results that need it. It never becomes an ordinary partial result silently. + +### 2.2 How a cell is stored + +The fill value is never a data value, and it only means "nothing was written here". Two designs can mark null and "nothing written" in the data: + +| Design | How | Benefit | Cost | +| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| **A. Reserved codes** | integers keep the top two codes (`uint8`: 254 = null, 255 = fill); floats use NaN plus a state array | the data alone shows which cells are missing | different handling for integers and floats; a source using 254 or 255 must be widened or remapped | +| **B. Native values + state array** | the data keeps the full source range; a companion `uint8` state array per variable says value, null or nothing written | the full source range is kept; the same handling for every type | one more array to store and read | + +Neither design is clearly better on paper. **Decide by benchmarking both on the same tiles**, with identical data, compression, chunks and shards, and compare compressed size, number of requests per read, and implementation complexity. In both designs: + +- the stored marks are value, null and nothing written; +- **nothing written** is resolved into "outside coverage" or "not available" with the records of section 2.3, never from the data alone. + +### 2.3 Coverage and completion records + +Two records answer why a cell has nothing written: + +- **Coverage** says where a source release applies. It is a property of the source, and it is stored with the same dimensions as the data it describes: per variable and per year. If 2010 grassland was ingested and 2011 failed, these are two different answers. One mask may be shared by several variables or years only when their coverage is checked and found identical. +- **Completion** says, for each tile, variable and year, whether processing finished: `complete`, `failed` (with the reason and the source files involved) or absent (not run yet). A `complete` tile with nothing written in a cell means the cell is outside coverage. Anything else means the cell is not available. + +### 2.4 Publishing a tile + +Data, coverage and completion are separate writes, so a reader could see a mix of old and new. The rule is: + +1. write the data; +2. write its coverage (and state array, in design B); +3. write the completion marker **last**, with an identifier of what was written (e.g. the run name and source release). + +A reader only trusts a tile whose completion marker exists and matches what it expects. A rerun overwrites the marker last, after the data. A run is published only when every tile it needs is complete, or explicitly as **partial**, with the list of incomplete tiles. + +### 2.5 What is written in `zarr.json` + +An integer layer, design A: + +```json +{ + "data_type": "uint8", + "fill_value": 255, + "attributes": { + "_FillValue": 255, + "missing_value": [254], + "flag_values": [0, 1, 2], + "flag_meanings": "not_grassland cultivated_grassland natural_grassland", + "state_codes": {"null": 254, "nothing_written": 255}, + "source": "gpw/grassland@v2_beta", + "mapping": "identity", + "resampling": "none" + } +} +``` + +A float result: + +```json +{ + "data_type": "float32", + "fill_value": "NaN", + "attributes": { + "_FillValue": "NaN", + "units": "t CO2e/ha", + "state_array": "emit_state" + } +} +``` + +There are two kinds of reader, with two expectations: + +- **Generic readers** (xarray's default decoding, GDAL, QGIS) must at least **mask every missing code**. That's why both 254 (`missing_value`) and 255 (`_FillValue`) are declared with their usual CF names: a generic reader then shows both as empty, and never draws or scales 254 as a measurement. It can't tell null from nothing written, and isn't expected to. +- **The project reader** keeps all the states. It opens arrays without automatic masking and decodes them with one function (e.g. `jdluc.values.decode`), which returns the **values in their own type** (integers stay integers) and a **state array** (value, null, outside coverage, not available), using the completion and coverage records. Measurements are converted to float only when a calculation needs it. The project reader doesn't rely on xarray's fill-value masking, whose behaviour for Zarr depends on the version. + +`state_codes`, `state_array`, `mapping` and `resampling` are our own attributes; generic tools ignore them. + +## Contract 3: calculations + +### 3.1 Missing values follow the inputs actually used + +A result is missing only if an input **that its calculation actually needs** is missing. Which inputs are needed often depends on the cell. Today's emissions code already has these cases: + +- **Peat replaces the mineral soil term** (`emit.get_conversion_emissions`). On a peat cell, missing mineral soil carbon doesn't make the soil emissions missing. +- **Forest biomass is only needed for a forest conversion.** A missing biomass value doesn't affect a grassland conversion. +- **Destination layers have a priority.** If a higher-priority layer has already set the destination, a missing lower-priority layer doesn't matter. If a higher-priority layer is null, the destination is unknown, because it might have claimed the cell. + +So each calculation declares, for each branch, its **required inputs**, and the result's state comes from those inputs only: + +| Required inputs of the branch used | Result | +| ---------------------------------- | --------------------------------------------------------------------------------------------- | +| all values | value | +| any not available | not available | +| any null (none not available) | null | +| any outside coverage | the coverage rule of that input (e.g. another layer, a declared default, or outside coverage) | + +The rules also cover operations other than arithmetic: + +- **comparisons** (`x > threshold`, `class == CROPLAND`): a null input gives a null result, not `false`; +- **selections** (first match wins): a null before the first match gives null; a null after it is ignored; +- **time series** (last departure year): a null year makes the result null only if it could change the answer. + +### 3.2 Exceptions are declared, not hidden + +A calculation can declare, for a given input, that null is treated as a value, e.g. that a SoilGrids gap (water, rock, ice) counts as zero soil carbon. This replaces today's `fillna(0)` with a setting in the run config, recorded in the run's manifest: + +```yaml +emit: + inputs: + soilgrids_ocs/ocs: {null_as: 0} # gaps are water, rock or ice: no soil carbon + ipcc_climate_zones/zone: {null_as: error} +``` + +`null_as` applies to null only. "Not available" is never replaced by a value: the tile is retried, or the result is published as partial. + +### 3.3 Totals say what they cover + +Every rollup (attribution, country totals) reports, next to the total, the share of its area that is value, null, outside coverage and not available. A total with a not-available share is flagged as incomplete. + +## Worked example: one corrupted input file + +A layer is ingested from several source files. Shard `x279y082` (1°, 4 × 4 chunks of 0.25°) sits across two of them: file A covers its western half, file B its eastern half. File A fails its checksum at ingest; file B is fine. + +The calculation is deliberately simple: **grassland area in the shard**, i.e. the area of the cells whose class is 1 (cultivated) or 2 (natural grassland). Each square below is one 0.25° block of cells (design A codes): `255` = nothing written, `254` = null, `0` = not grassland, `1` and `2` = grassland. + +```text + 1. source files 2. stored shard 3. availability + gpw/grassland_class (uint8) (data + records) + + ┌─────────┬─────────┐ ┌───┬───┬───┬───┐ ┌───┬───┬───┬───┐ + │ file A │ file B │ │255│255│ 0 │ 2 │ │ ✗ │ ✗ │ v │ v │ + │ │ │ ├───┼───┼───┼───┤ ├───┼───┼───┼───┤ + │ corrupt │ valid │ ingest │255│255│ 1 │254│ decode │ ✗ │ ✗ │ v │ n │ + │ ✗ │ ✓ │ ──────► ├───┼───┼───┼───┤ ──────► ├───┼───┼───┼───┤ + │ │ │ │255│255│ 1 │254│ │ ✗ │ ✗ │ v │ n │ + │ │ │ ├───┼───┼───┼───┤ ├───┼───┼───┼───┤ + │ │ │ │255│255│ 0 │ 0 │ │ ✗ │ ✗ │ v │ v │ + └─────────┴─────────┘ └───┴───┴───┴───┘ └───┴───┴───┴───┘ + west: never written v = value n = null + east: copied from B ✗ = not available + (completion: failed, file A) +``` + +1. **Ingest.** File A is rejected, and nothing is written for it. File B is written as it is: its classes, its zeros and its nulls (254, a cloud gap in the source). The completion record for this tile, variable and year says `failed: file A, checksum`, and no completion marker is written. +2. **Decode.** On the east, the data alone gives values and nulls. On the west, the data only says "nothing written". The completion record says why: the tile failed, so these cells are **not available**, not outside coverage. If file A were simply outside the source's extent and the tile were complete, the same 255 would decode as outside coverage. +3. **Result.** Grassland area over the eastern half is computed, with 2 null blocks reported separately. The shard's result is marked **incomplete** (8 of 16 blocks not available), so it is not published as an ordinary result: the tile is retried, or the run is published explicitly as partial, listing this tile. +4. **Retry.** When file A is fixed and re-ingested, its blocks are written, then the completion marker. The tile becomes complete, and the result is published. + +## Expected gains, to be measured + +- **Storage.** `uint8` bands are 4× smaller than float32 before compression. The actual gain after compression must be **measured**: write the same tile as float32 and in its native types, with identical data, compression, chunks, shards and coverage arrays, and compare. The same benchmark decides between designs A and B. +- **Accuracy.** No stored value changes except through a mapping or resampling declared in the metadata and checked at ingest; integers stay integers when read; totals are added up in float64. +- **Meaning.** A missing or corrupted file shows up as an incomplete tile, not as zero emissions or a smaller total. Every total says which part of its area had values, nulls, no coverage, or no data yet. + +## Implementation and acceptance + +Implement in stages, one contract at a time: + +1. **Values.** Stop the float32 cast; store each band in its source type; the reader keeps integer types. Measure the storage gain on one tile. +2. **Availability.** Benchmark designs A and B on the same tiles and choose. Declare null codes and coverage for every dataset; write coverage per variable and year; write completion markers last; add the project decoder. +3. **Calculations.** Declare the required inputs of each branch in emit; replace each `fillna` with a declared `null_as`; add the four shares to every rollup; block or flag publication when a needed tile is not complete. + +Acceptance requires evidence that: + +- Bands stored in their source type are bit-equal to the source, and integers (including IDs above 16,777,216) are read back unchanged. +- A failed file gives "not available" cells, an incomplete tile and a result that is not published as complete; a cell outside a complete tile's coverage gives "outside coverage". +- Coverage that differs between two years of the same variable is recorded separately. +- A reader never treats a tile as complete before its data and coverage are written. +- Generic readers mask both null and nothing-written codes. +- On a peat cell, missing mineral soil carbon doesn't make the soil result missing; on a grassland conversion, missing forest biomass doesn't make the result missing. +- Totals are the same as today's where no needed input is null, outside coverage or not available (apart from float64 accumulation). + +The decisions for discussion are: the four states and who resolves them, design A or B (after the benchmark), coverage per variable and year, the publication rule, required inputs per calculation branch, and `null_as` replacing `fillna`. + +## References + +- [Zarr v3 core specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html), for `data_type` and `fill_value`. +- [CF conventions](https://cfconventions.org/), for `_FillValue`, `missing_value`, `scale_factor`, `add_offset`, `flag_values` and `flag_meanings`. +- [xarray `open_zarr`](https://docs.xarray.dev/en/stable/generated/xarray.open_zarr.html), for its masking options and their version-dependent behaviour. diff --git a/docs/improvement_spatial.md b/docs/improvement_spatial.md new file mode 100644 index 0000000..6b9c568 --- /dev/null +++ b/docs/improvement_spatial.md @@ -0,0 +1,339 @@ +# A shared spatial standard for raster ingestion and comparison + +**Status: proposal for discussion.** This document brings together improvements 1–3 in [improvements.md](improvements.md): the definition of the lattice, the processing tiles, and the storage chunks and shards. They form one spatial contract, which should be defined before changing the source store or the execution model. + +The objective is to define **two permanent, nested analysis grids**. Every raster admitted to an analysis grid must use its exact cell boundaries, indexing and coordinates. For a given grid and area, every layer then has the same rows, columns and cell footprints, regardless of its source, year, ingest date or processing run. Comparing layers becomes an operation on corresponding cells. + +Between the two grids, the cell counts differ by a fixed factor: each `L4000` cell contains exactly nine `L12000` cells. That relationship is part of the standard too. + +## Motivation + +### The current grid exists implicitly + +The pipeline already harmonizes rasters onto EPSG:4326 at 1/4000° per cell. Its definition is distributed across the implementation: + +| Component | Current responsibility | Limitation | +| ----------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------ | +| `tiling.TileResolution.GLAD` | Defines 40,000 × 40,000 pixels per 10° tile. | The spatial standard is expressed through one source's tile dimensions. | +| `harmonize.Grid` | Builds an EPSG:4326 transform from a tile's north-west corner. | Origin, extent and resolution conventions are coupled to 10° processing. | +| Dataset adapters | Crop, mosaic or warp sources into ingest tiles. | Each adapter must reproduce the intended spatial conventions. | +| `geo.validate_geotiff` | Checks CRS, dtype, positive dimensions and transform direction. | Does not check exact lattice alignment, cell spacing or tile bounds. | +| `geo.get_chunk_size` | Chooses a power-of-two chunk from a memory budget and band dtypes. | Block boundaries depend on the stack and need not divide a tile evenly. | +| `emit.get_hectares_per_pixel` | Infers spacing from coordinates and estimates area using latitude. | Grid geometry and cell area are reconstructed downstream. | + +`geo.exact_merge` already requires identical coordinates when combining datasets. This is useful, but it checks agreement between those datasets rather than proving that each follows an independently defined global standard. Two equally shifted layers could agree with one another while violating the intended grid. + +### Choose the standard from the source grids + +Most of the main 30 m layers already share the same grid: EPSG:4326 at 1/4000°, with cell edges on whole degrees. This is the case for Hansen tree-cover loss, GLAD land cover, Harris aboveground biomass, GNW peatlands and GPW grassland. The other layers use a different resolution or projection and must be resampled once: GACED30 and Descals oil palm (1/3710.65°, Earth Engine's "30 m"), USDA CDL (30 m in Albers), SoilGrids (250 m in Homolosine), Huang belowground biomass (1/120°), GPW livestock (1 km in Homolosine), MapSPAM (1/12°) and the IPCC climate zones (1/2°). + +### A common grid exists, but it is not defined or checked + +The pipeline already puts every layer on a common grid of 1/4000° before combining them. The problem is how that grid exists: + +- **it is never clearly defined**: the documentation describes it in one sentence ("EPSG:4326 at 0.00025°, tiled on the 10° graticule"), but without saying where it starts, whether coordinates mark cell centres or corners, or how cells are numbered. The code has no definition either: it only knows that a 10° tile is 40,000 pixels wide, and reads each tile's position from its name (e.g. `20N_090W`), so each step rebuilds the grid from these two pieces. And the stored Zarr files do not record it at all: they keep only lists of coordinates, without even the coordinate system, which the reading code adds back by assumption; +- **layers are aligned at different moments**: some when they are ingested, others later, each time the harmonized stack is rebuilt; +- **nothing checks the result**: a layer is accepted as long as it has the right number of rows and columns. + +This matters because the same number of pixels, or the same "30 m" label, does not guarantee that pixels cover the same ground. A layer shifted by half a pixel still has 4,000 × 4,000 pixels. Huang belowground biomass is a real case: its coordinates are read as pixel centres when they are probably pixel corners, so the layer lands about one pixel away from its true position, and nothing in the pipeline notices. When two shifted layers are overlaid, each pixel is compared with its neighbour, which creates false disagreements between maps, or false changes between years, along every field and forest edge. + +The standard keeps the same grid but makes it explicit: the grid is written down once, every layer is aligned to it **once** at ingest, and the result is **checked**. After that, any step can ask for "this tile" and get exactly the same pixels from every layer, with nothing left to align or resample. A common grid only makes layers comparable pixel by pixel; differences in definitions, units, years or coverage still have to be handled separately. + +### No grid exists for finer sources + +Today there is, implicitly, one grid of about 30 m, and nothing finer. A 10 m source, such as a Sentinel-2-based map, has no defined target: it would either be degraded to 30 m, losing most of its detail, or brought in on its own grid, which would not line up with the other layers. + +### Work and storage units are too large and not aligned + +The unit of work is the 10° tile: 1.6 billion pixels per band. A single task needs a large machine, runs for hours, and loses all its work if it fails. + +Storage blocks are not aligned with the tiles either. Their size is computed from the number of bands, so it changes when a band is added: today it is 2,048 × 2,048 pixels for the main stack of 56 bands, but 16,384 × 16,384 for a stack of one band. And 2,048 does not divide a tile evenly: a 10° tile of 40,000 pixels holds 19 full blocks plus a partial one of 1,088 pixels, and a degree of 4,000 pixels is not a whole number of blocks either, so block edges never fall on degree lines. As long as each tile is written to its own store, this goes unnoticed; once neighbouring tiles are written into one shared array, two workers would have to write the same block. + +## Proposed spatial contract + +### 1. Define two versioned global grids + +| Property | `L4000` | `L12000` | +| ---------------------------- | ------------------ | ------------------ | +| Coordinate reference system | WGS 84, EPSG:4326 | WGS 84, EPSG:4326 | +| Cells per degree, `N` | 4,000 | 12,000 | +| Exact angular cell size | 1/4000° | 1/12000° | +| Informal scale | Approximately 30 m | Approximately 10 m | +| Global columns, `W = 360N` | 1,440,000 | 4,320,000 | +| Global rows, `H = 180N` | 720,000 | 2,160,000 | +| Cells in a 1° shard | 4,000 × 4,000 | 12,000 × 12,000 | +| Cells in a 0.25° chunk | 1,000 × 1,000 | 3,000 × 3,000 | +| Spatial chunks in a 1° shard | 4 × 4 | 4 × 4 | + +![1792006397731](image/improvement_spatial/1792006397731.png) + +**Why these two grids.** `L4000` is the grid the pipeline already uses, and the one the main 30 m layers are published on, so their pixels are kept as they are. `L12000` is its finer sibling for future 10 m sources: each `L4000` cell is exactly 3 × 3 `L12000` cells, with the same origin and the same degree lines. Two fixed grids give every current and future source a target, without creating a new grid for each dataset. + +The grids are defined in degrees. "About 30 m" and "about 10 m" are only approximations: a cell gets narrower towards the poles, and a source published on a true 10 m grid in metres will still need to be reprojected onto `L12000`. + +The standard fixes the following conventions: + +- Cell edges lie on multiples of `1/N` degrees, so both grids pass through the graticule origin `(0°, 0°)`. +- The global indexing origin is the north-west corner `(-180°, 90°)`. +- Columns increase eastward; rows increase southward. Arrays use spatial dimension order `(y, x)`. +- Coordinates are interpreted as longitude and latitude respectively. Library axis handling must preserve that convention. +- Cells represent areas; GeoTIFF exports declare `AREA_OR_POINT=Area`. The affine transform refers to outer edges, while xarray coordinates refer to cell centres. +- Grid definitions carry a version, for example `L4000-v1`. Changing origin, spacing, indexing or ownership conventions creates a new grid version. + +The global shape describes the address space. It does not require allocating or writing every cell. Regional and sparse stores may contain only covered windows, provided their global offsets and grid identity are explicit. + +### 2. Make cell addresses unambiguous + +For column `I` and row `J`, with `0 <= I < W` and `0 <= J < H`: + +```text +west edge = -180 + I/N +north edge = 90 - J/N +centre lon = -180 + (I + 1/2)/N +centre lat = 90 - (J + 1/2)/N +cell_id = J*W + I +``` + +A full cell reference is `(grid_id, cell_id)`, or equivalently `(grid_id, I, J)`. A numeric ID alone is insufficient because the two grids have different widths. IDs require a 64-bit integer representation at these global dimensions. + +Use half-open cell footprints: + +```text +longitude: [west, east) +latitude: (south, north] +``` + +A shared vertical edge belongs to the cell on its east; a shared horizontal edge belongs to the cell on its south. This implements ownership of each cell's west and north edges without double counting. Treat `+180°` as `-180°` when assigning a point; do not create a duplicate dateline column. The exact south pole is outside the half-open cell domain and should be rejected by point-to-cell lookup. Geographic windows crossing the dateline are split into two integer windows. + +Generate coordinates directly from integer indices using the rational definition. Do not accumulate floating-point steps or derive a new grid from a requested bounding box. Floating-point coordinates are an interchange representation of the standard, rather than its definition. + +### 3. Fix the relationship between the grids + +A coarse cell `(I, J)` contains the fine cells: + +```text +fine columns: 3I, 3I+1, 3I+2 +fine rows: 3J, 3J+1, 3J+2 +``` + +Conversely, a fine cell's parent is `(I_fine // 3, J_fine // 3)`. All parent edges coincide with fine-cell edges. The coarse-cell centre also coincides with the centre of its middle child. + +Because the two grids share the same origin and their ratio is a whole number (3), comparing a layer on one with a layer on the other needs no reprojection, only a reshape of the array: + +- **Fine → coarse:** group the fine pixels in blocks of 3 × 3 and summarize each block (mean for a density, majority for a class, sum for a total). In xarray it's one line: `coarsen(x=3, y=3)`. +- **Coarse → fine:** repeat each coarse value into its 3 × 3 block. That doesn't create new information, it just puts the coarse value on the fine layout. + +This only works with a whole-number ratio. A grid of 10,000 cells per degree, for example, would be 2.5 times finer than `L4000`: some fine cells would straddle two coarse cells, and comparing them would again require resampling. + +The ratio is recorded in the grid metadata of each file (for example "`L12000`, parent `L4000`, ratio 3"), so a reader comparing two files knows immediately how to line them up. Chunks line up too: a 0.25° chunk is 1,000 pixels on `L4000` and 3,000 on `L12000`, so chunk *k* of one file covers exactly the same ground as chunk *k* of the other, and the two can be compared chunk by chunk, in parallel, without loading whole layers. + +This eliminates geometric ambiguity between the two grids, but moving values between them still requires a declared rule: + +| Quantity | Coarse to fine | Fine to coarse | +| ------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| Class or mask | Replicate the parent if that is the intended interpretation. | Use a declared rule such as majority, any, all or class fractions. Define ties and missing values. | +| Intensive value, such as carbon per hectare | Replicate for a piecewise-constant interpretation; record any alternative interpolation. | Use an area-weighted mean over valid children. | +| Extensive value, such as total emissions | Allocate the parent total by an explicit rule, such as child area fractions. | Sum child totals. | + +Replication creates no new observations. A coarse estimate represented on the fine grid retains its original effective resolution. Preserve that information in metadata so the finer array is not mistaken for finer evidence. + +### 4. Standardize the 1° work tile + +A 1° tile holds 16 million pixels per band instead of 1.6 billion for a 10° tile: the unit of work becomes 100 times smaller, so tasks fit on small machines, finish in minutes, are cheap to retry, and spread evenly across workers. + +Define 360 tile columns and 180 tile rows, independently of raster resolution: + +```text +0 <= tx < 360 +0 <= ty < 180 +id = x{tx:03d}y{ty:03d} +west = -180 + tx +north = 90 - ty +extent = [west, west+1) × (north-1, north] +``` + +For grid `N`, a tile occupies the integer window: + +```text +columns: [tx*N, (tx+1)*N) +rows: [ty*N, (ty+1)*N) +``` + +For example, `x279y082` covers longitude 99°E–100°E and latitude 7°N–8°N on both grids. It has 4,000 rows and columns on `L4000`, and 12,000 on `L12000`. + +Within the same grid, a cropland layer and a forest-loss layer requested for that tile must return identical spatial coordinates and dimensions, even if one covers only part of the tile. Missing coverage is represented by a mask and declared missing-value semantics, rather than by shifting or shrinking the cell layout. + +Publisher tiles remain acquisition units. A 10° source tile can feed 100 work tiles without changing the analysis addresses. Country boundaries select intersecting work tiles and mask their cells; they never establish a country-specific origin. + +A 1° tile is a scheduling and retry unit, not a requirement to hold its entire stack in memory. Processing remains chunked. `L12000` has nine times as many cells per tile, so its worker memory and runtime must be measured separately. Skip a tile only when the relevant coverage or land mask proves it has no work; an absent storage object alone does not establish this. + +### 5. Align chunks and shards with tiles + +Use spatial chunks of 0.25° and spatial shards of 1°. Each tile contains 16 spatial chunks, and no spatial shard crosses a work-tile boundary. Chunk and shard dimensions are derived from the grid definition, independent of the number of layers. + +Each level has one role: **cells** say where things are, **tiles** are the units of work, **chunks** are the blocks that are read and compressed, and **shards** group chunks into one storage file. Because each level divides the one above it exactly, a tile is always a whole number of chunks and one shard. + +![Two separate one-degree shard files, each containing four 1,000-pixel chunks across; 2,048-pixel chunks cannot divide a 4,000-pixel shard.](figures/spatial-chunk-alignment.png) + +**A concrete example: two neighbouring shard files at 7–8°N.** The footprint `x279y082` covers 99–100°E, and `x280y082` covers 100–101°E. Each footprint corresponds to one spatial shard per array and temporal shard block. Each shard contains 4,000 × 4,000 cells on `L4000`. Here, “tile” names the geographic processing footprint of a shard; it is not another object inside Zarr. + +With 1,000 × 1,000 inner chunks, four chunks fit along each side of a shard. Those 16 chunks are stored in that shard's file. Worker A exclusively writes shard A; worker B exclusively writes shard B. No chunk belongs to both files. + +The reason to choose 1,000 rather than 2,048 is **exact divisibility**. The [Zarr sharding codec](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html#configuration-parameters) requires each inner chunk dimension to divide the corresponding shard dimension evenly. `4,000 / 1,000 = 4`, whereas `4,000 / 2,048` is not an integer. Two 2,048-pixel chunks would require 4,096 pixels. That configuration cannot be used for a 4,000-pixel shard; chunks do not overflow into another shard or become shared between files. + +With one exclusive writer per shard, neighbouring workers do not write the same storage object. The reason for the 1,000-pixel chunk size is compatibility with the fixed 1° shard and a predictable read granularity. Other exact divisors are possible; 1,000 is the proposed choice because it gives 0.25° chunks and 16 spatial chunks per shard. Concurrent writes would require coordination if several workers were assigned to the same shard, regardless of inner chunk size. + +A fully populated global layer has 1,036,800 spatial chunks, grouped into 64,800 spatial shards for each stored temporal block. These are layout counts, not predictions of written objects: sparse coverage may omit fill-only shards, while additional time blocks create additional objects. + +A 0.25° float32 chunk contains about 4 MB on `L4000` and 36 MB on `L12000`, before compression and temporary computation buffers. Memory planning must account for concurrent chunks, bands and time slices. + +For time-dependent arrays, declare both temporal chunk depth and temporal shard depth. A shard spanning several years makes its entire temporal block part of one writer's responsibility. Temporal layout is a storage choice; it must preserve the spatial addresses defined here. + +Zarr v3 sharding groups independently encoded inner chunks within a storage object. Its [sharding specification](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html) describes index-based access and shard updates. Request counts depend on the reader, caching and access pattern, so this proposal does not promise a fixed number of requests per read. + +Assign one active writer to each `(array, spatial tile, temporal shard block)`. Create shared array metadata before dispatching workers, and prevent overlapping retry attempts from writing simultaneously. A completion marker is written only after the output region succeeds. Shard alignment establishes disjoint ownership across neighbouring tiles; removing `COMPUTE_LOCK` also requires checking the remaining scheduler, metadata and retry interactions. + +### 6. Declare the grid in the Zarr metadata + +A Zarr store is a folder. Next to the data, each group and each array has a small JSON file, `zarr.json`, with an `attributes` section that can hold any metadata. That is where the grid should be written. + +**What happens today.** A store only carries `x` and `y` coordinate arrays: one longitude per column and one latitude per row. A reader must work out the grid from these numbers: the cell size from the gap between two values, the origin from the first value. This is fragile: + +- the values are floating-point numbers, so `0.00025` is never stored exactly; +- nothing says whether a value marks the centre or the corner of a cell, and mixing the two shifts the layer by half a cell; +- the CRS is not stored in the file; the code that reads it assumes EPSG:4326. + +**What we propose.** The grid is **declared once**, in the attributes of the store's root group, and never inferred: + +```json +{ + "zarr_format": 3, + "node_type": "group", + "attributes": { + "proj:code": "EPSG:4326", + "spatial:dimensions": ["y", "x"], + "spatial:transform": [0.00025, 0, -180.0, 0, -0.00025, 90.0], + "lattice": { + "name": "L4000", + "version": 1, + "cells_per_degree": 4000, + "origin": [-180, 90], + "pixel_convention": "area" + } + } +} +``` + +- `proj:code` gives the coordinate system. +- `spatial:transform` gives the position of the top-left corner and the size of a cell, in the usual GDAL order. It refers to cell **edges**, not centres. +- `lattice` names the grid and its version, so a reader can check that two stores use the same one by comparing a name instead of comparing numbers. + +The `proj:` and `spatial:` keys follow the GeoZarr conventions being prepared at the OGC, so other tools can read them. + +**Each array** then only describes what is specific to it: + +```json +"attributes": { + "units": "Mg/ha", + "fill_value_meaning": "no_data", + "source": "gnw/harris-agb@v1.4", + "resampling": "none" +} +``` + +**Keep `x` and `y`, but as a convenience.** xarray and QGIS expect them, so they stay in the store. They are computed from the grid definition (cell centres), not the other way round, and the conformance check verifies that they agree with the declared transform. + +The rule is simple: **a reader takes the grid from the metadata, it never reconstructs it from the coordinates.** + +### 7. Enforce the contract at raster ingestion + +Each dataset adapter declares its target grid, source interpretation and any permitted transformation. Ingestion has two explicit paths: + +1. **Already conforming:** verify the source against the target grid, then copy or window it without resampling. Values and valid-data masks must be preserved. +2. **Transformation required:** reproject or resample deliberately onto a target window derived from integer grid indices. Record the operation, then apply the same conformance checks to its output. + +Do not accept a layer based only on a “10 m” or “30 m” label. Check its actual CRS, affine transform, shape and bounds. + +| Source situation in this project | Proposed treatment | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| GLAD and Hansen inputs expected to match the existing analysis lattice | Verify each release's metadata against `L4000`; preserve samples when it conforms. | +| GPW grassland ingest already targets the GLAD resolution | Verify output alignment and separately establish whether acquisition preserved native samples. | +| GACED30 | Its adapter already resamples publisher tiles onto the GLAD target. Record this transformation and check the result against `L4000`. | +| CDL and Descals oil palm | Their adapters already map sources onto the GLAD target. Declare the method and verify `L4000` output. | +| Future finer-resolution imagery or thematic maps | Target `L12000` when justified; verify native alignment or record reprojection. | +| Coarse sources such as MapSPAM, livestock, biomass or soil layers | Retain source resolution and grid metadata; define an explicit analytical conversion or aggregation before cellwise comparison. | + +The two grids are the standard **fine analysis targets**. Keeping an original coarse source is compatible with this proposal, but that source is not directly cell-comparable with a fine-grid layer until a relationship has been established. + +MapSPAM's 1/12° spacing and the current EPSG:4326 livestock ingest's 1/120° spacing divide `L12000` evenly: 1,000 and 100 fine cells per side respectively. Neither divides `L4000` evenly. Spacing alone does not prove nesting: origins and coverage edges must also align. The livestock publisher's native raster is projected, so the 1/120° grid is already a derived product. + +One-degree and quarter-degree boundaries can provide common aggregation windows for aligned coarse grids without expanding their values into fine cells. Avoid treating coarse totals as per-fine-cell totals: replication would multiply their sum. + +### 8. Define conformance as a checkable result + +Before publishing a standardized raster, verify: + +- **Identity:** declared grid ID and version, with the intended EPSG:4326 interpretation. +- **Orientation:** north-up, positive x spacing, negative y spacing and no rotation or shear. +- **Spacing:** both absolute steps equal `1/N` within the standard's numerical tolerance. +- **Origin:** the outer top-left corner corresponds to integer global cell indices. +- **Extent:** dimensions and bounds match the declared integer window and remain inside the global domain. +- **Coordinates:** centres are generated by the canonical index formula and are consistent with the transform. +- **Coverage:** valid data, uncovered cells and unprocessed regions have distinguishable meanings. +- **Transformation record:** whether values were copied or transformed, with source grid, method and processing identity. + +Use a documented, versioned tolerance measured in cell units. A proposed starting value is `1e-6` of a cell, to be checked against real source metadata before adoption. Check corner and far-edge residuals, not just the top-left corner, because a small spacing error can accumulate across a large raster. A failed check reports the mismatch and refuses publication; it must not silently fix a half-cell shift by relabelling coordinates. + +On successful ingest, record the grid ID, global window, source resolution, transformation and coverage information. Downstream reads verify these declarations and compare canonical integer windows. The contract applies equally to COG or Zarr storage; adopting a shared raw store is a subsequent change. + +### 9. Derive cell area from the grid + +Compute cell areas from their parallel and meridian boundaries on the WGS 84 ellipsoid. All cells in one row have the same area, so a one-dimensional area-per-row array is sufficient and can be broadcast when needed. + +This replaces inferring spacing from coordinate differences and applying the current circumference-and-cosine approximation. The area calculation must use the actual angular cell footprint; a polygon routine that connects corners with geodesics should not be assumed to represent the same parallel-bounded footprint exactly. + +Both grids use one area definition. The sum of the nine children's areas must match their parent's area within numerical tolerance. Area-weighted comparisons and conversion of per-hectare emissions to totals then share the same geometry. Changing the area calculation can change reported hectare and emissions totals, so evaluate and report that effect separately from the storage and tiling migration. + +## What comparison becomes + +For two layers on `L4000`, a request for `x279y082` produces two 4,000 × 4,000 windows with identical cell footprints. Their common valid-data mask defines the comparison domain. Counts, disagreements and differences can be calculated directly, with no additional spatial alignment. + +For an `L12000` layer compared with an `L4000` layer over the same tile, the fine window has 12,000 × 12,000 cells. Apply the declared 3 × 3 reduction to the coarse footprints, or the declared coarse-to-fine expansion. Record that choice as part of the comparison. The geometry is fixed; the meaning of the aggregation remains a scientific decision. + +For repeat runs, the same `(grid_id, tile_id, local row, local column)` always identifies the same ground footprint. A changed value can therefore be traced to inputs or methodology without also reconstructing a run-specific grid. + +The same holds between **two separate Zarr files**, even if they were produced by different runs, at different dates, or by another project such as GeoCollision. Each file declares its grid in its metadata, so comparing them starts with a simple check: same grid name and version. If they match, pixel (row, column) of one file is the same piece of ground as pixel (row, column) of the other, and the two can be compared directly, with no reprojection and no alignment. Today this is not possible: a Zarr file only carries lists of coordinates, so comparing two files means comparing those numbers one by one and assuming they use the same coordinate system. + +## Implementation and acceptance + +Introduce `jdluc/lattice.py` as the authoritative implementation of the two grid definitions, index conversions, transforms, tile windows, chunk and shard shapes, and row areas. Keep the source-specific grids in dataset metadata. Existing `TileResolution` and `harmonize.Grid` interfaces can initially delegate to this module while callers migrate. + +Publish a short, versioned grid specification alongside the implementation, with reference cells and tiles. Compatibility with GeoCollision should be demonstrated against its actual specification and fixtures before claiming interchangeable IDs. + +Implement in stages: + +1. Define the grids and conformance checker while retaining current processing and storage. +2. Make existing ingest adapters declare and validate their output grid; record transformations. +3. Replace inferred coordinates and areas with canonical grid-derived va +4. lues, measuring any numerical changes. +5. Move country selection and processing from 10° source footprints to 1° work tiles. Check that assembled outputs reproduce the former spatial extent and values. +6. Adopt aligned chunks and shards, then verify disjoint writes and retries before changing locking or cloud execution. + +Acceptance requires evidence that: + +- Independently ingested layers on the same grid and window have identical coordinates and dimensions. +- A half-cell offset, incorrect spacing, rotation, wrong bounds or undeclared CRS change is rejected. +- A conforming source retains identical samples and valid-data masks through ingest. +- Adjacent tiles and chunks cover their parent extent exactly, with no duplicate or missing row or column. +- Every coarse cell maps to exactly nine fine children; their area sum matches the parent. +- A tiled result agrees with the same computation over its assembled extent, including tile edges and coarse-grid aggregation boundaries. +- Parallel tasks own disjoint shards, and a failed or overlapping retry cannot leave an apparently complete mixed result. + +The proposed decisions for discussion are the two angular resolutions, the fixed global origin and indexing, the 1° work tile, and the 0.25° chunk within a 1° shard. Source resampling methods, missing-value policies and temporal shard depths require explicit dataset or processing choices within that spatial standard. + +## References + +- [Original improvements 1–3](improvements.md#1-define-the-lattice-explicitly). +- [Current architecture](architecture.md), particularly the grid, harmonization and caching sections. +- [GDAL geotransform conventions](https://gdal.org/en/stable/tutorials/geotransforms_tut.html), for the distinction between outer corners and pixel centres. +- [Zarr v3 sharding specification](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html), for the storage layout. diff --git a/docs/improvements.md b/docs/improvements.md new file mode 100644 index 0000000..3b30634 --- /dev/null +++ b/docs/improvements.md @@ -0,0 +1,428 @@ +# Improvements + +Proposed changes to the pipeline, to be taken up one at a time. Each entry has a **description** of the topic and how it works today, the **motivation** for changing it, and the **proposed change**. An entry is removed once it lands. + +They're listed in dependency order: the lattice (1) comes first because the tiling (2) and chunking (3) are defined in terms of it. The raw store (5), the run store (6) and the execution model (7) assume all three. The vector format (8) is independent; the rasterized vector layers (9) build on the raw store, and the config file (10) on the run store. Bucket access (18) is independent, but pays off most once chunks are sharded (3) and the work is spread over many machines (7). + +## Summary + +| # | Name | Description | Motivation | +| --- | ----------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| 1 | Define the grid explicitly | Write down, in one place, the shared global grid every dataset sits on. | A single shared grid makes combining data simple, reliable and checkable. | +| 2 | Smaller work units | Process the world in small 1° squares instead of large 10° squares. | Smaller jobs are faster, cheaper to retry and spread better across machines. | +| 3 | Tidy storage blocks | Store data in blocks that line up with the work squares. | Clean boundaries avoid conflicts between workers and cut storage costs. | +| 4 | Keep original data types | Store each layer in its original compact form instead of converting it. | Converting everything inflates storage several times over for no benefit. | +| 5 | One shared source store | Load each source once into a single shared store already on the grid. | Avoids keeping duplicate copies and redoing the same alignment every run. | +| 6 | One named folder per run | Give each run a readable name and keep all its results together. | Results become easy to find, share, compare and trace back to their settings. | +| 7 | Run on many cloud machines | Spread the work across many small cloud machines instead of one big one. | One machine is too slow to process the whole world in reasonable time. | +| 8 | Modern vector format | Store map shapes (e.g. borders) in the same format as the other tables. | It is faster to read and keeps one format for all non-grid data. | +| 9 | Turn shapes into grid layers | Convert every map shape layer into a grid layer when it is loaded. | Shapes then combine with other data the same simple way, without repeated work. | +| 10 | Rules in a config file | Move the methodology rules and thresholds out of the code into a settings file. | What makes one run differ from another is visible and recorded in one place. | +| 11 | Say what "no value" means | Record what an empty cell means for each layer. | "Not covered", "no data" and "zero" are currently mixed up and indistinguishable. | +| 12 | Read only what's needed | Load only the layers and areas a step actually uses. | Reading unused layers and empty ocean wastes time and money. | +| 13 | Track what produced a result | Identify results by the exact data and code that produced them. | Today an outdated result can be reused silently if someone forgets a manual step. | +| 14 | Reuse the processing engine | Start the internal processing engine once instead of for every save. | Restarting it each time is slow and forces saves to happen one at a time. | +| 15 | One command per run | Replace the five manual steps with one command, plus a status command. | It's easy to forget a step or mix settings, and hard to know what's done. | +| 16 | Check settings up front | Validate the settings file before the run starts. | Mistakes should fail in seconds, not hours into a run. | +| 17 | Settings from the environment | Read settings from the environment, with the local file as a fallback. | Cloud machines usually receive settings that way, not as a file. | +| 18 | Faster bucket access | Talk to the cloud buckets through obstore, a Rust library, instead of gcsfs. | Lower overhead per request, fewer dependencies, and native support in Zarr v3. | + +## 1. Define the lattice explicitly + +**Description.** The analysis grid is EPSG:4326 at 1/4000° (~30 m), anchored at the graticule origin, but the code never names it. It's implied in several places that have to agree with each other: + +- `tiling.TileResolution.GLAD` sets 40,000 pixels per 10° tile. +- `harmonize.Grid` hardcodes EPSG:4326 and builds a float geotransform from a tile ID's north-west corner. +- `liao_gaced30` derives its own `PIXELS_PER_DEGREE`. +- `emit.get_hectares_per_pixel` recovers the cell size by averaging coordinate differences. + +**Motivation.** + +- **A shared standard makes every other step simpler.** If every layer, every ingest and every output uses the same fixed lattice, then joining two layers is a lookup by cell index, with no resampling and no alignment check at each step. Tiles, chunks and shards (changes 2 and 3), the raw store (5), the run store (6) and the cloud tasks (7) are all defined from it. Most of the other changes in this list become simple because of this one. +- **Nothing checks alignment.** No code verifies that an ingested raster lands on the grid. Alignment holds only because every ingest happens to get it right, and a source that's off by a fraction of a cell would be resampled silently with no error. +- **The anchor and conventions aren't written down.** The pixel convention and cell indexing are undocumented, so a collaborator, or GeoCollision, has to reverse-engineer them to join data. +- **Cell area is approximate.** The spherical `cos(lat)` estimate is off by −0.75% to +0.5% against the WGS 84 ellipsoid, depending on latitude, and that error carries into every hectare and emissions total. +- **Changing resolution is a migration.** Moving to a finer grid would mean finding and editing every implicit copy of the cell size. +- **Coarse grids don't fit.** MapSPAM (1/12°) and GPW livestock (1/120°) don't divide evenly into 1/4000°, so they're stretched by a fractional factor today. + +**Proposed change.** Add `jdluc/lattice.py` as the only module where a cell size appears. It would define: + +- a `Lattice` class: EPSG:4326, integer `cells_per_degree`, cell size as the exact rational `1/N`, anchored at (0°, 0°) so cell edges sit at `lon = i/N`, `lat = j/N`; +- the pixel convention: `AREA_OR_POINT=Area`, north-up, with west and north edges owned; +- global integer indices `(I, J)` from the north-west corner `(-180°, 90°)`, with exact index-to-coordinate functions and no accumulated floating-point arithmetic; +- a conformance check (CRS, step, origin) that ingest runs before writing; anything off the lattice is refused unless it was warped on purpose; +- the ellipsoidal (WGS 84) cell area per row. + +Two instances would be declared: + +| Name | Cells per degree | Cell size | Role | +| -------- | ---------------- | ----------------------------- | ------------------------------------------------------------- | +| `L4000` | 4,000 | 1/4000° = 0.00025° (~30 m) | Today's grid: GLAD, Hansen TCL, GPW grassland, GACED30 | +| `L12000` | 12,000 | 1/12000° ≈ 0.0000833° (~10 m) | Finer sibling for 10 m sources, such as Sentinel-2-based maps | + +`L12000` is an exact 3× refinement of `L4000`: every 30 m cell is 3 × 3 cells of 10 m, with no edge moved, so moving between the two grids is replication rather than resampling. It's also the first of these grids that the coarse inputs fit into evenly: one MapSPAM cell is 1,000 cells of `L12000` and one GPW livestock cell is 100. + +**A predefined grid, like H3.** H3 is a global grid that is defined once for everyone: each cell has a fixed ID, so any two datasets indexed in H3 can be joined by cell ID, without discussing projections or pixel origins. The lattice should play the same role for this project, and for related ones such as GeoCollision: a published standard, with a fixed origin, a fixed cell size and fixed cell IDs, that every process uses. Each cell has a stable ID (`cell_id = J × width + I`), each tile has a stable name (e.g. `x279y082`), and a cell means the same place in every store and every run. Two differences with H3 work in our favour. The cells are the pixels of the main sources (Hansen, GPW), so those sources are used as they are, with no resampling. And the levels nest exactly: one `L4000` cell is exactly 3 × 3 `L12000` cells, while an H3 cell's children only approximately cover it. + +`TileResolution`, `harmonize.Grid` and the per-dataset pixel constants would then be derived from a `Lattice` instead of restating it. This matches GeoCollision's `grid.py` and `grid-spec.md`, which define the same 1/4000° lattice, so cell IDs stay interchangeable between the two projects. + +## 2. Process in 1° work tiles instead of 10° + +**Description.** The pipeline's unit of work, cache and retry is the 10° tile (e.g. `20N_090W`), the footprint of a Hansen GFC file. Harmonize, emit, downscale and attribute all run once per 10° tile. `attribute.workflow` gives each 10° tile its own thread. + +**Motivation.** + +- **Tasks are huge.** A 10° tile is 1.6 × 10⁹ pixels per band. The harmonize stack of ~56 bands is about 358 GB uncompressed as float32, so a task needs a large machine and takes hours. The continent-scale harmonize benchmark took ~53 h. +- **Failures are expensive.** If a spot VM is preempted, the whole tile's work is lost. +- **Load is uneven.** A full-land tile and a coastal sliver cost very different amounts, but both are one task, so a run waits on its largest tiles. +- **Too few tasks.** A few hundred tiles can't keep a cloud scheduler (Cloud Batch, GKE) busy across many small workers. +- **Coarse pixels don't fit 10°, and 0.25° is too small.** One degree is exactly 4,000 GLAD pixels, 120 GPW livestock pixels and 12 MapSPAM pixels, so a 1° tile edge never cuts through a coarse source pixel. Every 10° tile is also whole in coarse pixels, but it's far too big to be a task. Every 0.25° tile is whole too, but it's too small. + +**Proposed change.** + +- **Make the 1° tile the unit of processing, caching and retry.** A 1° tile is 1.6 × 10⁷ pixels per band, about 3.6 GB for the full float32 stack, which fits a small worker and finishes in minutes. +- **Keep 10° only for ingest.** It's the native footprint of the Hansen and GLAD sources. Ingested COGs are internally tiled, so a 1° window read from them is a cheap range request. +- **Use resolution-independent tile IDs.** Name tiles by index, e.g. `x279y082` as in GeoCollision, so the same ID means the same ground on `L4000` and `L12000`. +- **Skip empty tiles.** Most of the lattice is ocean, so a cheap land-mask check should skip a tile before any band is read. +- **Update the tile lookup.** `worldbank_jurisdictions` country-to-tile lookups would return 1° tiles. Attribute's one-thread-per-tile rule carries over unchanged. + +## 3. Align chunks to degree lines and shard by tile + +**Description.** `geo.get_chunk_size` picks the largest power-of-two chunk that keeps all bands under 1 GiB. For the ~56-band harmonize stack that's 2048 × 2048 pixels. Each chunk is stored as its own object. + +**Motivation.** + +- **Chunks fall in arbitrary places.** 2048 doesn't divide 40,000 (or 4,000), so chunk edges don't line up with tiles or degree lines. The last chunk in each row and column is ragged. +- **Chunk size depends on band count.** Adding a band can change the chunking of every store. +- **Writers can collide.** When several workers write neighbouring tiles, a chunk that straddles a tile edge has two writers. Today that's avoided only by `storage.COMPUTE_LOCK`, which serializes writes. +- **Too many objects.** At 30 m and 0.25° chunks, an unsharded global variable is about 1 million objects per year in GCS, and request count is what drives cost and latency. + +**Proposed change.** + +- **Fix the chunk at 0.25° (1,000 px on `L4000`, 3,000 on `L12000`).** Size it in degrees so the chunk count per year is the same at any resolution, and so it always lands on degree lines. +- **Use Zarr v3 sharding with one shard per 1° tile (4 × 4 chunks).** A global variable then drops to about 65,000 objects per year. Each read costs two requests: one for the shard index and one for the data. +- **One writer per shard.** Because a shard is exactly one work tile, parallel workers never write the same object, and `COMPUTE_LOCK` can go. +- **For multi-year arrays, put time first in the shard.** GeoCollision's benchmark measured a point time-series at 2 requests with time-major shards against 48 with per-year shards. + +## 4. Keep native dtypes in harmonize + +**Description.** `geo.unify_dtype_and_no_data` casts every harmonized band to float32 and turns nodata into NaN before the zarr is written. Most source bands are categorical `uint8` (land classes, CDL, masks, loss year) or small integers. + +**Motivation.** + +- **Harmonize dominates storage.** It's 92% of the cache, 1.2 TiB for the continent-scale benchmark. +- **The cast inflates it.** Storing a `uint8` band as float32 quadruples its uncompressed size. Floats also compress worse than small integers. +- **Codes and measurements get blurred.** Class codes become floats, so readers can't tell a code from a measured value. + +**Proposed change.** + +- **Store each band in its source dtype** with an explicit integer fill value declared per dataset (the `no_data` field already exists). +- **Convert in emit.** Emit would cast to float32 and turn fill into NaN when it reads, so the arithmetic doesn't change. +- **Expect a 2–4× smaller cache.** That's the estimate for harmonize's cache from the band mix. Measure it on one tile before adopting. + +## 5. Ingest straight into a raw zarr on the lattice + +**Description.** + +- **Ingest writes COGs.** Each source is downloaded, checked and converted to a Cloud-Optimized GeoTIFF under `INGEST_ROOT`, at its own resolution and on its own grid, with provenance tags (`watershed-*`). +- **Harmonize does the alignment later.** For each tile, harmonize builds one GDAL VRT per band that puts the COG on the tile grid, resampling when the source is not at 30 m. It then casts the result to float32 and saves it as a new zarr under `SCRATCH_ROOT`. +- **Emit, downscale and attribute read the harmonize zarr**, never the COGs. + +**Motivation.** + +- **The same values are stored twice.** For 30 m sources (GLAD, TCL, GPW grassland, GACED30), the harmonize zarr holds the same values as the COG, only in float32. This copy is most of harmonize's 1.2 TiB. +- **The alignment is redone again and again.** Harmonize's cache key includes the list of datasets in the stack. Adding one dataset, or bumping the version, recomputes every band of every tile: all the VRT reads and resampling run again, even for sources that did not change. +- **The COG's main reason is inspection, not processing.** `architecture.md` chose COGs so each ingested file can be opened in QGIS or with `gdalinfo` and checked by hand. That matters while a dataset is being added, but every run after that pays the processing cost. +- **COGs are the slow path for processing.** A COG is good for viewing in QGIS or GDAL, but every read goes through GDAL and a VRT. A zarr on the lattice is read directly by xarray and dask, chunk by chunk, with the chunks and shards of change 3. +- **Sources are used by every run.** They change rarely (a new release of a dataset), while runs change often (new thresholds, new processes). Work done once at ingest is saved for every run after it. + +**Proposed change.** + +- **Keep today's ingest code, change only its last step.** The per-dataset `save_tile_id_to_local_path` functions already build a GeoTIFF in a temporary folder, and they stay as they are. Only the end of `RasterDataset.ingest_a_tile` changes: + + ``` + today: source → temp GeoTIFF → validate → convert to COG → upload COG + proposed: source → temp GeoTIFF → validate → lattice check (change 1) → write region in raw.zarr → delete temp + ``` + + The lattice check is new: today nothing verifies that an ingested raster sits on the grid. + +- **Ingest writes into one shared `raw.zarr`.** Each source becomes a group, and each band an array (e.g. `gnw_tcl/lossyear`), on the lattice of change 1, with the chunks and shards of change 3 and the source dtype of change 4. + +- **Do the alignment once, at ingest.** + + - **Sources already on the lattice** (GLAD, TCL, GPW grassland): copied as they are, with no resampling. The values must be bit-equal to the source, which is a test. + - **30 m sources on another grid** (GACED30): warped once onto the lattice at ingest, and the method is recorded in the array's attributes. + - **Coarse sources** (SoilGrids, Harris AGB, Huang BGB, MapSPAM, GPW livestock): stored at their own resolution, on their own lattice, in their own group. Bringing them to 30 m is a methodology choice (copy each value or interpolate), so it belongs to the run (change 6), not to ingest. + +- **Emit reads `raw.zarr` directly.** There is no harmonize step and no harmonize copy for 30 m sources. + +- **Don't keep the COGs by default.** The temporary GeoTIFF is deleted once its region is written. To look at a layer in QGIS, export a COG from `raw.zarr` (`tools/dump-zarr-to-cogs.py` already does this for zarr stores); this export should be one simple command, since easy inspection was the main reason for COGs. For the few layers the team inspects often, the COG can be kept next to the zarr as an option. + +- **Move the provenance into the zarr.** The `watershed-*` tags (data version, git SHA, processing time, source and product, remote URL) become attributes of each group in `raw.zarr`, together with the source cell size and any resampling method. + +- **A new source release is a new group or a new version.** `raw.zarr` is never changed in place. A new release of a dataset gets its own group (e.g. `gnw_tcl@v1.12`), and each run records in its manifest which groups it read. + +**Measured.** We compared one year (2010) of GPW grassland, stored as the source GeoTIFFs and as a zarr on the 1/4000° lattice (from GeoCollision). The zarr is 1.9× smaller: + +| | Zarr (`adastra-zarr/gcg4000.zarr/gpw/grassland_class/c/10`) | GeoTIFF (`adastra-input-public-data/…/global_pasture_watch/v2_beta/2010/raw`) | +| ----------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------- | +| Size | **8.46 GiB** | **16.14 GiB** | +| Objects | 16,962 shards of 1° | 35 files | +| Compression | blosc zstd-5 + bitshuffle | deflate | +| Overviews | none | 9 levels | + +Part of the gap comes from the GeoTIFF overviews, which add about one third. Without them, the zarr is still about 1.4× smaller (estimated, not measured). The rest comes from better compression and from empty 1° tiles (ocean) not being written at all. The cost is that the zarr has no overviews, so a viewer needs an exported COG or a pyramid built from the zarr. + +## 6. One named store per run + +**Description.** + +- **Every cached call is its own store.** The decorators in `storage.py` write each cached result as its own store, named by a hash of the function, its `version` number and its arguments: `{SCRATCH_ROOT}/{hash}.zarr` or `.parquet`. +- **Harmonize, emit and downscale therefore write one zarr per tile.** A global run leaves hundreds of unrelated hash-named stores in one flat folder. + +**Motivation.** + +- **Outputs can't be found or browsed.** A hash name is deterministic but unreadable: it doesn't say which step, tile or country produced it, so a store can only be found by recomputing the hash or searching the logs. +- **A run has no identity.** Nothing groups together what one run produced, so a run can't be shared, archived or deleted as a whole. Nothing records which code and settings made a result. +- **Nothing can be opened as one dataset.** To look at emit over a whole region, you open many per-tile stores and merge them. +- **Completion is coarse.** "Done" means a tile's whole store exists, so a partial write is either complete or invisible. + +**Proposed change.** + +- **Name the run when it starts.** The user gives the run a name, and everything the run produces goes under `runs/{name}/`. The name identifies a version of the emissions layers, not a crop or a country: a run is global, and one name means one set of methods and settings (thresholds, datasets, processing choices). For example `v1`, then `v2` when a threshold or a process changes. The exact settings behind each name are written in `manifest.json`. + + ``` + SCRATCH_ROOT/ + raw.zarr/ ← ingested sources, shared by all runs (change 5) + runs/ + v1/ + run.zarr/ ← every layer the run computes + emit/… + downscale/… + attribution.parquet + emissions-factors.parquet + manifest.json ← git SHA, settings and thresholds, dates, layer sources + v2/ ← same layout, different thresholds or processes + … + ``` + +- **One zarr for the whole run, on the lattice.** `run.zarr` covers the whole globe on the lattice of change 1, with all layers and all dimensions (x, y, year). One `open_zarr` gives everything. A layer on another grid (e.g. at MapSPAM resolution) goes in its own group with its own coordinates. + +- **Read sources from `raw.zarr`.** Sources are not copied into the run; the run reads them from the shared `raw.zarr` (change 5), which is on the same lattice, so a cell (I, J) is the same place in both. The only source-derived layers stored in the run are the coarse sources brought to 30 m, because how to do that is a run setting; the manifest records the method. + +- **Each tile writes its own region.** The run creates `run.zarr` once with the global shape and no data, then each worker writes its tile with `to_zarr(region=...)`, aligned to its shard (change 3). No two workers write the same object. + +- **Track completion per tile with a marker**, written last (e.g. `runs/{name}/_done/emit/x279y082`). A rerun with the same name skips tiles that have one. A run covers the globe but can be filled step by step: computing more regions later just fills new parts of the same store. + +- **Reuse earlier runs.** A run can start from another one (e.g. `v2` started with `--from-run v1`). Layers whose code and inputs didn't change are copied or linked from it, and only the rest is recomputed. This keeps the time savings of today's cache (e.g. not redoing the ~21 h emit), and the manifest records where each layer came from. + +- **Keep tables as parquet.** Attribution rollups and the emissions-factor table aren't on the grid, so they stay as parquet files in the run folder, with readable names. + +## 7. Two-level execution model for cloud runs (proposed by Claude) + +**Description.** + +- **Each zarr write starts its own dask cluster.** `storage.write_dask_dataset_to_zarr` starts an in-process `distributed.LocalCluster` sized by `NUMBER_OF_DASK_WORKERS`. +- **Writes are serialized.** Every write holds a global lock (`COMPUTE_LOCK`). +- **Stages fan out with threads on one host.** Attribute and ingest use thread pools over tiles. +- **The pipeline is single-host by design** (see `architecture.md`). + +**Motivation.** + +- **One host can't run the global corpus quickly.** A continent-scale harmonize alone takes ~53 h on one machine. +- **The lock blocks more parallelism.** Spreading tiles across more threads doesn't help while every write holds `COMPUTE_LOCK`. +- **A single global dask graph won't work either.** The graph alone would take gigabytes in the client. The whole job would also become one retry unit, and "skip an empty tile" couldn't be decided before every read was already in the graph. + +**Proposed change.** + +- **Outer level.** A scheduler (Cloud Batch, GKE Jobs, or Dask on Kubernetes) hands out 1° tiles as small, serializable task descriptions carrying only identifiers. The tile is the unit of retry and of failure. +- **Inner level.** Inside each task, a small dask graph runs with an explicit local scheduler, never the cluster's, to avoid a worker submitting work to itself. GeoCollision's benchmark found processes faster than threads (3.3× vs 2.3×) and lighter on memory. +- **Infrastructure:** + - compute in the same region as the buckets; + - spot VMs of about 4–8 vCPU and 16–32 GB; + - writes made safe to retry: write the region first and the completion marker last; + - a service account on the VMs instead of user application-default credentials. +- **GDAL settings for `/vsigs`:** keep `GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR`, and add `GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES`, `GDAL_HTTP_MULTIPLEX=YES`, `VSI_CACHE=TRUE` and a per-worker `GDAL_CACHEMAX`. +- **Coarse layers.** MapSPAM and GPW livestock stay on their own coarse grids in their own stores. They're joined to the 30 m data at 0.25° or 1° boundaries, where all the grids meet exactly, rather than resampled to 30 m. + +## 8. Store vectors as GeoParquet + +**Description.** + +- **Vectors are FlatGeobuf.** `VectorDataset.ingest_a_tile` converts each vector source with `geo.convert_vector_to_flatgeobuf` and writes `vector/{source}/{product}/{version}/whole-world/world.fgb`, with a spatial index (`SPATIAL_INDEX=True`). +- **The only vectors are the World Bank boundaries** (admin 0, 1 and 2), world-wide. +- **They are always read in full.** `worldbank_jurisdictions` loads them with `geopandas.read_file(path_to_fgb)`, then filters by country or tile in memory. + +**Motivation.** + +- **The spatial index is never used.** FlatGeobuf's strength is reading a small area over HTTP with its index. Here the whole file is read every time, so that strength brings nothing. +- **Full reads are slow in FlatGeobuf.** It is read row by row through GDAL and is not compressed. GeoParquet is columnar, compressed and read through Arrow, which is usually much faster for a full read, and can read only the needed columns. +- **Two tabular formats.** Every other table in the pipeline (ingested tables, attribution, emissions factors) is already parquet. GeoParquet keeps one format for everything that isn't a raster. + +**Proposed change.** + +- **Write vectors as GeoParquet.** In `convert_vector_to_flatgeobuf` (renamed), `to_file(..., driver="FlatGeobuf")` becomes `to_parquet(...)`, and in `worldbank_jurisdictions`, `read_file` becomes `geopandas.read_parquet`. +- **Use GeoParquet 1.1 with a bbox column and rows sorted by space**, so a small-area read stays possible if it's ever needed. +- **Nothing is lost for inspection.** GeoParquet still opens in QGIS and GDAL (≥ 3.5). + +## 9. Rasterize every vector layer onto the lattice + +**Description.** + +- **Vectors are combined with rasters by clipping.** Today the only vector layers are the World Bank boundaries (admin 0, 1 and 2). For each tile and each jurisdiction that touches it, attribution clips the raster stack with the polygon (`geo.clip_dset`, which uses `rio.clip_box` then `rio.clip`), then sums the emissions inside it. See `jurisdictional_direct.py` and `statistical.py`. +- **There is no general rule for vectors.** `VectorDataset` only stores the polygons (change 8). How a polygon layer is turned into cells is decided by the code that uses it, at the moment it uses it. +- **More vector layers are likely.** Protected areas, concessions, farm or supply-shed boundaries, and peatland polygons are typical inputs for this kind of work. Each would today need its own clipping code. + +**Motivation.** + +- **Vectors are the only data that doesn't join by cell.** Every raster layer joins with the others by cell on the lattice (change 1). Vectors are the exception: they use a second, slower way of combining data, and each new vector layer adds more of it. +- **Clipping is repeated work.** Each run, each tile and each polygon clips the same geometry again, although vector sources change as rarely as raster sources. +- **Cell membership is decided silently.** Which 30 m cells belong to a polygon is decided inside `rio.clip` each time (a cell belongs to the polygon that contains its centre), and never stored or checked. Where the vector data has overlaps or gaps, a cell can be counted in two polygons, or in none, and nothing reports it. +- **It doesn't fit the raw and run stores.** With `raw.zarr` and `run.zarr` (changes 5 and 6), every input is an array on the lattice, except the vectors. + +**Proposed change.** + +- **Rasterize every vector layer at ingest.** Each vector source is burned into `raw.zarr` as an array on the lattice, in its own group, as GeoCollision does with `gdal_rasterize`. Examples: `worldbank/admin_0_id`, `worldbank/admin_1_id`, `worldbank/admin_2_id`, and later `protected_areas/id` or `concessions/id`. After ingest, a vector layer is used exactly like a raster layer. +- **Declare how each layer is burned.** The vector dataset definition states what the cell holds: + - **an ID** (default): each cell carries the integer ID of the polygon that contains its centre, for layers that partition space, such as jurisdictions; + - **a class or a flag**: each cell carries a value taken from an attribute (e.g. a protection category), for thematic layers; + - **a priority rule for overlaps**: when polygons overlap (e.g. protected areas), the definition says which one wins, or stores a flag per polygon type instead of one ID. +- **Keep a lookup table per layer.** A small parquet table maps each integer ID to the polygon's attributes (`id`, `name`, level, …). +- **Keep the vectors.** The GeoParquet (change 8) keeps the exact geometry, names and areas, for reports and checks. The array is what processing uses. +- **Processing becomes a group-by or a mask.** Attribution groups the emissions by the value of the jurisdiction array; a thematic layer becomes a mask or a class in emit. Both are exact joins by cell, like every other layer, with no clipping. +- **Check every rasterization.** A cell is assigned to one polygon only, so edges are approximate at 30 m, and polygons smaller than a cell can disappear. At ingest, compare each polygon's area from the array with its vector area, list polygons that got no cell, and record both in the array's attributes. + +**Measured.** We compared the Xu et al. global peatland map, stored as the source GeoPackage and as the rasterized layer on the 1/4000° lattice (from GeoCollision). The zarr is about 5.7× smaller: + +| | Zarr (`adastra-zarr/gcg4000.zarr/xu_peat`) | GeoPackage (`adastra-input-public-data/…/xu_peat/v1/_/raw`) | +| ------- | ------------------------------------------ | ----------------------------------------------------------- | +| Size | **176 MiB** | **1,009 MiB** | +| Objects | 6,557 shards of 1° | 1 file | +| Content | `uint8` per cell: 1 = peat, 0 = not peat | 2.6 million polygons, with geometry and attributes | + +The two don't hold the same information. The GeoPackage keeps the exact polygon edges and their attributes; the zarr keeps only "peat or not" per 30 m cell, so edges are rounded to the cell and polygons smaller than a cell can disappear. But that one value per cell is all processing needs, and it compresses very well. Shards with no peat are not written at all, because the fill value (0) is the "not peat" value. This also shows why each layer must say what its fill value means: here 0 means "measured, not peat", not "no data", so a cell outside the source's coverage and a non-peat cell look the same. + +## 10. Move the methodology rules to a config file + +**Description.** The rules of the emissions core are written in Python. For example, `emit.DestinationDataset` and `DESTINATION_DATASET_TO_PREDICATE` say which layer claims a pixel as cropland or pasture (Descals oil palm, GACED30, GPW), in which year, and with which priority. Thresholds and constants (assessment year, lookback, carbon fractions) are module constants. + +**Motivation.** + +- **The rules are parameters, not logic.** Each destination is "layer + year + condition → flag", then grouped with a priority. Adding one means editing an enum, a dict and an assert in `emit.py`. +- **They are what makes one run different from another.** A new threshold or a new destination layer is a new run (`v2`, change 6), but today those settings are spread through the code instead of in one place the manifest can record. + +**Proposed change.** + +- **A config file for the rules**, read at the start of a run and copied into its manifest: + + ```yaml + assessment_year: 2024 + destinations: # order = bit order in `destination-dataset` and priority + - {name: descals_oil_palm, group: cropland, layer: descals/planting_year, + condition: {operator: between, min: 1, max: "{assessment_year}"}} + - {name: liao_gaced30, group: cropland, layer: liao_gaced30/cropland, + year: "{assessment_year}", condition: {operator: "==", value: CROPLAND}} + - {name: gpw_grassland, group: pasture, layer: gpw/grassland_class, + year: "{assessment_year}", condition: {operator: "==", value: CULTIVATED}} + group_priority: [cropland, pasture] + ``` + +- **Small generic functions in code**, called from the config, as in GeoCollision: condition → flag (`AttributeValueMapper`), first match wins (`AttributeSelector`), last departure year (`find_transitions`), lookup tables such as IPCC carbon by climate zone (`AttributeReclassifier`), group-by for attribution (`AggregatedParquet`). + +- **Keep the real methodology in code.** Discounting, peatland occupation, dropped emissions and the statistical downscaling stay Python, with their tests. Only rules, thresholds and layer choices move to config. + +- **Don't build a second generic engine.** GeoCollision is already the generic engine (lattice, store, transformers, config). cornerstone_luc should be one methodology on top of it, and other uses become new configs there, not changes here. + +- **The destination order is part of the run.** It sets which bit each layer gets in `destination-dataset`, so it is recorded in the manifest, and changing it means a new run. + +## 11. Say what "no value" means + +**Description.** Harmonize turns every band's nodata into NaN (`geo.unify_dtype_and_no_data`), and emit treats NaN as "no data". + +**Motivation.** "No value" can mean three different things: outside the source's coverage, measured but no data, or not ingested yet. Some layers also use their fill as a real value (in the Xu peat layer, 0 means "not peat"). With one NaN for all of them, a result can't tell them apart. + +**Proposed change.** Each array in `raw.zarr` and `run.zarr` declares what its fill value means (e.g. `fill_is_a_value: true`), and each step declares how nulls spread (e.g. `null_policy: propagate` or `as_value`), as GeoCollision does. Coverage is recorded per layer, so "outside coverage" is a separate answer. + +## 12. Read only what's needed + +**Description.** Emit opens the whole harmonize stack for each tile, and every tile is read, including ocean. + +**Motivation.** Most of the lattice is ocean, and many steps use only a few of the ~56 bands. Reading the rest costs time and requests for nothing. + +**Proposed change.** + +- **Open only the arrays a step uses.** With `raw.zarr` (change 5), each step lists its inputs and opens only those. +- **Walk back from the outputs**, as GeoCollision does: an array no output depends on is never read. +- **Skip empty tiles with a gate layer.** Every step first reads a land or coverage mask, and skips the tile if it holds nothing, before reading anything else. + +## 13. Provenance by content + +**Description.** The cache key is the function, its arguments and a hand-written `version=` number (`storage.get_cache_decorator`). It ignores the input data and the code itself. + +**Motivation.** If someone changes the code or an input and forgets to bump `version`, the old result is returned with no error. `tools/measure-drift.py` warns about it: "a clean result can be a lie". + +**Proposed change.** Key each result on its content: the identity of each input (the `raw.zarr` group and its release) and the identity of the code (as GeoCollision's `code_identity` does). Record both in the run manifest (change 6) for each layer. The manual `version=` goes away. + +## 14. One dask scheduler per process (from Claude) + +**Description.** `storage.write_dask_dataset_to_zarr` starts a new `distributed.LocalCluster` for every write, under a global lock (`COMPUTE_LOCK`). + +**Motivation.** Starting a cluster for every write is slow, and the lock allows only one write at a time. + +**Proposed change.** Start one local scheduler per process, and reuse it for every write. Once each tile is one shard (change 3), drop `COMPUTE_LOCK`. + +## 15. One command per run + +**Description.** A run is five commands (ingest, harmonize, emit, attribute, trace) that must be called in the right order. The only way to know what's done is to read the logs or recompute hashes. + +**Motivation.** It's easy to forget a step, or to run one with different settings from the others. There's no simple answer to "what's done?". + +**Proposed change.** One CLI with a few commands: + +- `jdluc run v2 --config v2.yaml --countries BRA`: runs every step needed, in order; +- `jdluc status v2`: tiles done, missing and failed, per step; +- `jdluc export v2 emit --bbox …`: writes a COG for QGIS. + +## 16. Check the config before running + +**Description.** Errors in settings appear during the run, from `assert` statements, sometimes hours in. + +**Motivation.** A typo or a wrong value should fail in seconds, not after a long run. + +**Proposed change.** A `jdluc check v2.yaml` command validates the config from change 10 without opening any store: unknown keys, wrong types, unknown layers or classes, unreachable rules. `run` calls it first. This is the same idea as GeoCollision's `process --check`. + +## 17. Read settings from environment variables + +**Description.** `config.Config.from_dot_env` stops if there's no `.env` file (`raise_error_if_not_found=True`). + +**Motivation.** Cloud containers usually get their settings as environment variables, not as a file. + +**Proposed change.** Read environment variables first, and use `.env` only as a fallback for local work. + +## 18. Use obstore for bucket access + +**Description.** Every read and write to the GCS buckets goes through `fsspec` and `gcsfs` (a direct dependency in `pyproject.toml`), apart from GDAL: + +- `storage.path_exists` and `storage.put_file` call `fsspec.url_to_fs` to check for and upload ingested COGs, FlatGeobufs and parquet files (`datasets/base.py`). +- `storage.open_zarr_to_dask_dataset` and `storage.write_dask_dataset_to_zarr` pass a bare `gs://…` string to `xarray.open_zarr` / `to_zarr`, which zarr resolves through fsspec and gcsfs. +- `pandas.read_parquet` / `to_parquet` on `gs://` URIs (the `cache_to_parquet` cacher, FAOSTAT, USDA NASS) also go through fsspec and gcsfs. +- GDAL and rasterio don't use fsspec: `storage.to_gdal_path` rewrites `gs://` to `/vsigs/`, and GDAL reads the bucket itself. + +**Motivation.** + +- **gcsfs is a slow layer for chunked I/O.** gcsfs is pure Python on top of `aiohttp`, and each request goes through fsspec's async-to-sync bridge. obstore is a thin Python binding over the Rust `object_store` crate (the one used by Apache Arrow, DataFusion and Delta Lake), which makes requests natively, with less overhead per request and real parallelism outside the GIL. +- **The request count is about to grow.** With 0.25° chunks and 1° shards (change 3), each read is a shard-index request plus a range request, and a run makes many of them over many small workers (change 7). Per-request overhead is what limits throughput there, more than bandwidth. +- **Zarr v3 supports it natively.** `zarr.storage.ObjectStore` wraps an obstore store directly, with no fsspec in between, and does concurrent range reads for sharded arrays. +- **Fewer and smaller dependencies.** gcsfs pulls in `aiohttp`, `google-auth`, `google-cloud-storage` and their dependency trees; obstore is one wheel with no Python dependencies. That's a smaller container image for cloud workers (change 7). +- **One way to talk to the bucket.** Today the bucket is reached through fsspec in some places and implicitly through zarr's own URL handling in others. A single `store` object, built once from the config, makes it explicit which bucket, prefix and credentials a run uses. + +**Proposed change.** + +- **Build the store once in `storage.py`.** A helper turns a root URI into an obstore store (`obstore.store.from_url("gs://bucket/prefix")`, or `GCSStore` for `gs://` and `LocalStore` for local paths, so local runs and tests keep working). Credentials come from application-default credentials or the VM's service account, as today. +- **Zarr through `zarr.storage.ObjectStore`.** `open_zarr_to_dask_dataset` and `write_dask_dataset_to_zarr` pass `store=zarr.storage.ObjectStore(store)` instead of a URI string. The same applies to `raw.zarr` and `run.zarr` (changes 5 and 6) once they exist. +- **Replace the fsspec helpers.** `path_exists` becomes `obstore.head` (catching "not found"), and `put_file` becomes `obstore.put` from the local file; there are no directories to create in an object store. +- **Parquet through the fsspec adapter, at first.** pandas and pyarrow still expect an fsspec filesystem. obstore ships one (`obstore.fsspec.FsspecStore`), which can be registered for `gs://` so `read_parquet` / `to_parquet` keep their current calls. Later, parquet files can be read and written as bytes with `obstore.get` / `put`, or through pyarrow's own GCS filesystem. +- **Leave GDAL alone.** GDAL reads `/vsigs/` with its own HTTP client, so `to_gdal_path` and the GDAL settings of change 7 don't change. +- **Then drop `gcsfs` from the dependencies**, and the `gcsfs` logger line in `write_dask_dataset_to_zarr`. +- **Measure before switching.** Time a full read of one harmonize zarr and one emit write on the same machine, with gcsfs and with obstore, and keep the numbers here, as for changes 5 and 9.