what-to-id

Does the order of the iNaturalist identification queue change how much gets identified, and what? what-to-id tests this with unlabelled, randomised lists and reads the answer back from iNaturalist. First deployment: British Columbia, autumn 2026, with Blitz the Gap.

It is a method, tested first in BC. A result from one blitz holds for that blitz's region and identifiers; other regions have to run it to know.

Companion to where-to-blitz, which answers where to go and record. Both serve Blitz the Gap.

What the lists test

Every list holds the same kind of records and differs only in order. The orders fall into two families, each with its own question:

• Speed lists change how many IDs an hour of identifier effort gives. similarity puts look-alike photos together, on the hypothesis that fewer context switches make identifying faster.
• Value lists change which gaps the IDs fill. gap_first puts records from data-poor places first, aimed at the gap in where species are known to occur (the Wallacean shortfall, Hortal et al. 2015) and in how recent the records are, which the score weights by record age. Blitz the Gap calls these spatial and temporal gaps (Hébert et al. 2026). novelty puts the photos least like any verified photo of their group first, aimed at the gap in what verified photos cover.
• recency, newest first, is the control for both.

Each family is judged on its own measure: speed lists on species-level IDs per identifier, value lists on the same count weighted by how data-poor the record's place is (gap_first) or on new species per grid cell (novelty).

Experiment diagram

flowchart TB
  pool["<b>The day's pool</b><br/>BC records that need an ID<br/>and have a photo"]
  split(["`each record goes to one list 
  by a keyed hash of its id, 
  the same list every day`"])
  pool --> split
  split --> L1["<b>List 1</b><br/>newest first<br/><i>the control</i>"]
  split --> L2["<b>List 2</b><br/>data-poor places first"]
  split --> L3["<b>List 3</b><br/>look-alikes together"]
  split --> L4["<b>List 4</b><br/>unfamiliar photos first"]
  L1 & L2 & L3 & L4 --> page["<b>One page, lists unnamed</b><br/>each Next batch press<br/>serves the next list"]
  page --> work["Identifiers work each batch<br/>in iNaturalist, as usual"]
  work --> count["<b>Count</b> species-level IDs<br/>per identifier, per list<br/><i>each list on its own measure</i>"]
  count --> cmp["<b>Compare</b> each list with List 1<br/>within each identifier"]

Loading

Each record sits on exactly one list and keeps it in every daily build. Because the list comes from a keyed hash of the record id, every list gets a like mix of taxon groups and of the observers who posted the records on average, so the lists differ only in the order they show their records. That order is the one thing under test.

The lists are unlabelled rather than blind: the page never names them, but a batch of look-alike photos shows which list it came from.

Every identifier works every list. A few identifiers make most of the IDs. If each identifier kept one list, the list that drew the busiest identifier would win whatever its order. So each browser draws its own random cycle of the four lists once, and every press of Next batch takes the next list in that cycle:

	Their cycle	Press 1	Press 2	Press 3	Press 4	Press 5	Press 6	Over the blitz
Identifier who does 40 batches	3 → 1 → 4 → 2	3	1	4	2	3	1	10 batches from each list
Identifier who does 4 batches	2 → 4 → 1 → 3	2	4	1	3			1 batch from each list

IDs are credited to iNaturalist accounts and each record's list is fixed, so a second device only adds a second balanced cycle. Each identifier is then compared with themself: their IDs on List 2 against their IDs on List 1, and so on. A busy identifier adds the same effort to every list, so their skill cancels out. python -m what_to_id.power simulates both designs.

	One list per identifier	Every identifier works every list
A busy identifier's effort goes to	one list	all lists, equally
A list scores high because of	who landed on it	its order
Compared unit	list totals	each identifier with themself

How it works

1. Pull. Pull the needs-ID records with a photo, observed since a start date. The daily job then adds only the records created since its last run and, before each build, drops served records that no longer need an ID. The region is BC (iNaturalist place 7085), set in inat.py and batches.py.
2. Assign. Each record goes to one list by a keyed hash of its record id (--key-env), so it stays on that list in every daily build. The key is private. Without it, a one-off build randomises once with a seed, balanced within taxon group and observer.
3. Compose. Each arm orders its records into batches:
   • recency: newest first as of the build, close to what an identifier gets from iNaturalist today.
   • gap_first: records from data-poor places first, using the where-to-blitz cell scores.
   • similarity: look-alike photos together, from BioCLIP 2.5 image embeddings.
   • novelty: photos least like any Research Grade photo of the same group and region first.
   • surprise (opt-in, not in the default build): records whose proposed species is least expected where, or in what climate, it was seen, from a per-species tail probability against Research Grade records (surprise.py). Pass the scores with --surprise-scores.
4. Serve. A static page shows lists labelled A to D, each a stack of Identify links. The arm behind each letter is recorded only in the build's manifest.json, never on the page. By default each identifier takes one list; --design rotation serves the one-page rotation pictured above. Each daily build appends its served records, by letter, to a served log (--served-log).
5. Read back. Every served record, from every daily build, is fetched again. Identifications keep their own timestamps, so the per-identifier comparison is fixed when the blitz ends and is read back within a week of it; the record-level outcomes are read back 30 days after.

First deployment: British Columbia

The first run is a BC identification blitz in autumn 2026, with Blitz the Gap. The protocol and the outcomes fixed before the blitz are in docs/protocol.md, a draft for co-design with the BC team. The set-up is four lists, recency, gap_first, similarity and novelty, over four weeks. An earlier draft of the protocol proposed two lists, recency and gap_first, and left the other two to a later round.

Install

uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"      # add ,embed for the image-embedding step, ,similarity for look-alike batches

The similarity extra installs labelfirst, a private repository pinned by commit in pyproject.toml, so it needs access to that repository. Only the look-alike list's seed picker and the separability report use it. The four-list daily build needs this extra; an explicit recency,gap_first build still runs without it. CI uses the LABELFIRST_DEPLOY_KEY secret, the private half of a dedicated read-only deploy key on traitlab/labelfirst, to check out the pinned library and test the integration.

The base CI job installs exact, hash-pinned versions from requirements.lock (pip install --require-hashes -r requirements.lock, then run with PYTHONPATH=src). After changing the dependencies in pyproject.toml, regenerate it with uv pip compile pyproject.toml --extra dev --universal --python-version 3.11 --generate-hashes -o requirements.lock.

Use

# Freeze the needs-ID pool (hours for all groups; --groups and --cap-pages for a sample).
# An interrupted pull resumes from the groups already saved.
python -m what_to_id.inat --d1 2025-01-01 --freeze YYYY-MM-DD --out data/pool_YYYY-MM-DD.parquet

# Research Grade reference pool for the novelty arm
python -m what_to_id.inat --d1 2025-01-01 --freeze YYYY-MM-DD --quality research --out data/ref_YYYY-MM-DD.parquet

# Build assignment, batches, manifest and site. --max-batches serves only the first N
# batches per arm and taxon group, so the read-back denominator is fixed before the blitz.
# --design rotation (opt-in) writes one page where each "Next batch" press takes the identifier
# to the next list, so every identifier's work splits evenly across lists.
what-to-id build --pool data/pool_YYYY-MM-DD.parquet --freeze YYYY-MM-DD --d1 YYYY-MM-DD --seed <private-seed> --max-batches N --out out/build

# Daily build, the same script .github/workflows/daily.yml runs: fetch state and a prepared pool,
# build with the private key, drop served records that no longer need an ID, rebuild, log what was served,
# keep the day's pool and build record, check for leaks and replay the build. WTB_DIR is a where-to-blitz
# checkout at the commit in what_to_id.manifest.WHERE_TO_BLITZ_REF. OFFLINE=1 rehearses on a saved pool.
export WTB_DIR=../where-to-blitz/cluster_results/ca BLITZ_D1=<blitz-start>
export EMBEDDING_RELEASE=<versioned-bundle-tag>
scripts/daily.sh fetch
scripts/daily.sh fetch-embeddings
scripts/daily.sh build     # scripts/daily.sh save uploads the state

# Rerun any past day from its snapshot; exits 1 unless it serves exactly what the served log says
python -m what_to_id.replay --pool state/days/pool-YYYY-MM-DD.parquet --record state/days/build-YYYY-MM-DD.json --served-log state/served.parquet --webapp-dir "$WTB_DIR" --key-env WHAT_TO_ID_KEY --embedding-bundle data/embedding-bundle

# After the blitz, open the key: letter -> list map for the read-back and the analysis
python -c "import json; from what_to_id.manifest import blind_labels_keyed as b, key_from_env as k; print(json.dumps({v: a for a, v in b(['recency', 'gap_first', 'similarity', 'novelty'], k()).items()}))" > labels.json

# Read back outcomes after the blitz: within a week for the per-identifier comparison, again at 30 days
python -m what_to_id.readback --pool state/pool.parquet --assign state/served.parquet --label-map labels.json --out out/outcomes.parquet

# Build the participants file --users reads below, from project members or a sign-up form's logins
python -m what_to_id.participants --project ID_OR_SLUG --out participants.txt
python -m what_to_id.participants --logins signups.txt --out participants.txt

# Pre-registered per-identifier comparison for the rotation design: participants only, plus a pre-blitz placebo.
# The default --weight primary holds each list to its pinned count (weighted for gap_first, plain for similarity and novelty), one-sided with Holm over gap_first and similarity, novelty exploratory (two-sided, unadjusted), the other count as secondary; each run also reports a sign test (p_sign)
python -m what_to_id.analysis --idents out/outcomes_idents.parquet --served state/served.parquet --label-map labels.json --control recency --start <blitz-start> --cutoff <read-back-cutoff> --users participants.txt --placebo-start <placebo-start> --obs out/outcomes_obs.parquet

The read-back denominator is the served records only: assign.parquet covers the whole pool, but batches.parquet (one build) and the served log (every daily build) hold just the records placed in a served batch. Both work as --assign and --served; the served log needs --label-map.

Prepare embeddings separately from daily serving. slurm/stage_mila.sbatch downloads photos on CPU into resumable group archives; slurm/prepare_mila.sbatch then regenerates the frozen 10,000-row Research Grade reference and candidate embeddings on GPU using the verified BioCLIP 2.5 checkpoint. Each group archive is bound to the IDs and photo URLs of its own rows, so archives copied from a run whose pool differed only in other groups are resumed without re-downloading. Candidate records whose photo cannot be fetched from either iNaturalist host are excluded before assignment and listed in RUN/excluded.json, and the bundle's pool omits them; staging stops instead if more than 0.1% of a group fails or any reference photo fails. Ten minutes before either job's time limit, staging archives the group in progress without marking it complete and preparation stops its current step and persists completed vectors, so resubmitting resumes. The scripts take explicit input paths and never publish. The daily runner consumes the completed bundle.

For subsequent pool snapshots, reuse only an attested prior preparation. The command retains observations whose ID, taxon group and photo URL match, embeds new or changed rows, drops removed IDs and keeps the supplied reference files unchanged. slurm/prepare_mila.sbatch passes the same three inputs when PREVIOUS_POOL, CANDIDATE_CACHE and PREVIOUS_PREPARATION are all exported at submission; the earlier bundle is only read:

python -m what_to_id.prepare_embeddings --pool data/pool-next.parquet \
  --previous-pool data/previous-bundle/pool.parquet \
  --candidate-cache 'data/previous-bundle/embeddings_{group}.npz' \
  --previous-preparation data/previous-bundle/preparation.json \
  --reference-embeddings 'data/frozen-reference/emb_{group}_bioclip25.npz' \
  --reference-pool data/frozen-reference.parquet --model-snapshot data/frozen-model \
  --work-dir data/preparation-next --out data/bundle-next

--model-snapshot must contain the exact config and safetensors weights for revision 6e3d04e3d6522012c88181085c5ae666e14c45cd; hashes are checked before encoding. Cache reuse verifies the previous pool, model and vector file hashes. Legacy caches without that record require regeneration. URL identity does not detect image bytes changed at the same URL. A work directory resumes only the same inputs, and packing requires complete candidate coverage. Publish the resulting pool, provenance and embeddings together as a new immutable bundle.

To package already verified caches directly:

python -m what_to_id.artifacts pack --pool data/pool_YYYY-MM-DD.parquet --embeddings 'data/candidates/emb_{group}.npz' --reference-embeddings 'data/reference/emb_{group}.npz' --out data/embedding-bundle
python -m what_to_id.artifacts verify data/embedding-bundle
what-to-id build --pool data/pool_YYYY-MM-DD.parquet --freeze YYYY-MM-DD --d1 YYYY-MM-DD --seed 0 --key-env WHAT_TO_ID_KEY --arms recency,gap_first,similarity,novelty --design rotation --embedding-bundle data/embedding-bundle --out out/build

Run python -m what_to_id.bundle_transport stage-release --bundle data/embedding-bundle --out data/release-assets, then publish all files in data/release-assets together as assets of a versioned release. The transport uses hashed parts below GitHub's per-file size limit; the daily job verifies and reconstructs the original files. Configure EMBEDDING_RELEASE and, if hosted elsewhere, EMBEDDING_REPO. EMBEDDING_BUNDLE defaults to data/embedding-bundle; downloading requires a fresh destination. The operator must provide release access where needed. Production rollout still requires embedding preparation, release upload, secret configuration and deployment.

A build records each candidate/reference file's content hash, dimensions, row count and backbone by taxon group. Replay accepts the original bundle or matching --embeddings and --reference-embeddings inputs; it refuses changed files. Builds using score files also require the original --surprise-scores or --sinr-scores. Older image-list build records without complete per-file fingerprints must be rebuilt before this replay path can verify them. Production bundles require candidate embeddings for every pool ID and nonempty references for each group. The daily job uses the bundle's exact pool snapshot and removes served observations that no longer need identification. Add new observations during preparation, encode their images, and publish the updated pool and embeddings together as a new bundle. The daily job never adds observations ahead of their embeddings. Exploratory builds with individual cache inputs retain the arm's recency fallback for missing candidates.

The gap_first arm reads where-to-blitz's cluster_results/ca; point --webapp-dir or the WHERE_TO_BLITZ_CA environment variable at a checkout of it.

Publish

The live page is served at https://pollocklab.github.io/what-to-id/. The daily workflow (.github/workflows/daily.yml) rebuilds it every morning once the repository variable DAILY_ENABLED is true (and on demand from the Actions tab), and saves it as study.html on the pool-state release. It does not deploy. The map workflow (map.yml) runs after each successful daily run and is the only workflow that publishes the full site: a generated landing page of cards at the root, with the study page under Experimental, taken from that release while DAILY_ENABLED is true and from the committed site/study.html otherwise. It reads the key from the WHAT_TO_ID_KEY secret and the blitz start from the BLITZ_D1 variable, and runs scripts/daily.sh. The script defaults to four lists with rotation and requires a prepared embedding bundle. Configure its versioned release and LabelFirst repository access before enabling this workflow. It refuses to keep a page that names a list, and reruns the day's build from its snapshot and stops unless that matches the served log exactly. The pool-state release keeps the pool, the served log (letters only), and each day's pool snapshot and public build record. Set the repository variable CODE_REF to a tag to run the same code for the whole blitz; each build record names the commit that ran. Actions are pinned by commit SHA. GitHub turns a schedule off after 60 days without a push to the repository, so push at least once in that window while it runs.

A one-off build is published instead by copying out/build/site/*.html over site/, which map.yml publishes (the Pages workflow only starts it when site/ changes on main); a rotation build writes only study.html, so delete the old arm_*.html pages when switching. Both the daily and map workflows refuse any file that is not HTML and any page that names an arm. Keep the key, and a real build's manifest and seed, out of the repo: with the public code and the same pool, any of them recovers which list is which.

The current page is a rotation preview of all four lists, built from a 10,000-record sample frozen on 2026-09-11 (--max-batches 20), not the blitz build. Its lists are assigned with a private key, like the daily build, so the public code and pool do not show which list is which.

Tests

pytest              # network tests are skipped by default
pytest -m network   # hits the live iNaturalist API

References

• Hortal J, de Bello F, Diniz-Filho JAF, Lewinsohn TM, Lobo JM, Ladle RJ (2015). Seven shortfalls that beset large-scale knowledge of biodiversity. Annual Review of Ecology, Evolution, and Systematics 46:523-549. doi:10.1146/annurev-ecolsys-112414-054400.