update sdk with new adds

[luke-e-schaefer]

Summary — v0.19.0: EvaluationV2, benchmark paradigm

Brings the SDK to parity with the Nucleus EvaluationV2 UI. Evaluation creation is benchmark-scoped only — dataset/slice-scoped creation (an unreleased earlier iteration of this branch) was removed before release as the platform moved to the benchmark paradigm.

Added

• Benchmarks. create_benchmark() (members from item_ids, (dataset_id, ref_id) items pairs, a slice_id, or a dataset_id — exactly one required; membership frozen at creation), list_benchmarks(), get_benchmark(), update_benchmark(), delete_benchmark(), list_benchmark_items(), plus the Benchmark resource (refresh() / update() / delete() / items() / create_evaluation_v2()) in nucleus/benchmark.py.
• Benchmark evaluations. create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...) — every benchmark item is scored (uncovered items count as FN, keeping leaderboard scores comparable). EvaluationV2 exposes benchmark_id, rollup_groups, exclusion_rules, exclusion_stats. No only_items_with_predictions / eval-scope slice_id (invalid in the benchmark paradigm; the server rejects the former).
• Rollup groups. RollupGroup (class_name + labels) as the primary label configuration on benchmark eval create and presets (create/update_evaluation_v2_preset(rollup_groups=...), mutually exclusive with legacy allowed_label_matches; parsed from both key casings). Preset seeding prefers rollup_groups, falling back to legacy matches.
• Exclusion rules. MetadataExclusionRule / LabelExclusionRule / BoxAreaExclusionRule via exclusion_rules on eval create and presets.
• Presets. list/create/update/delete_evaluation_v2_preset + EvaluationV2Preset resource; preset= seeds benchmark eval creation (explicit args override).
• Results. EvaluationV2.charts() (mAP, per-class AP, confusion matrix, PR/F1 curves, TIDE, AP by size) and examples() (paginated TP/FP/FN; match_type optional) with EvaluationV2FilterArgs filtering; cancel() / retry(); Dataset.evaluation_label_schema().
• Benchmark leaderboards. leaderboard_ranking(metric_type, benchmark_ids, ...) (metrics: MAP_50, MAP_50_95, AP_SMALL/MEDIUM/LARGE, PRECISION, RECALL, F1; scope/collapse) and leaderboard_f1_curve(benchmark_ids, ..., top_n=5).
• Filter schema discovery. EvaluationV2.filter_schema() / get_evaluation_v2_filter_schema() return the evaluation's filter vocabulary (gt_labels, pred_labels, metadata_fields with inferred value types).
• New DTOs: EvaluationV2FilterSchema, LeaderboardRankingEntry, LeaderboardF1CurveEntry, BenchmarkItemsPage.

Removed (relative to earlier commits on this branch — never released)

• create_evaluation_v2 (dataset/slice-scoped), create_evaluations_v2_batch, BatchEvaluationResult, only_items_with_predictions. Reading existing evals (including old slice-scoped ones) still works.

Fixed

• CI build_test pylint failure (pre-existing W0718 on the per-job catch) — suppressed with a black-stable # pylint: disable-next line.

Tests / Version

• tests/test_benchmarks.py, tests/test_leaderboard.py, plus rollup/benchmark coverage in the eval/preset test files (59 unit tests, mock-based).
• pyproject.toml → 0.19.0; single consolidated CHANGELOG entry (the unreleased 0.18.9 section was folded in).

Server dependency: the leaderboard and filter-schema methods call new REST mirrors (POST /nucleus/leaderboard/ranking, POST /nucleus/leaderboard/f1Curve, GET /nucleus/evaluationsV2/:id/filterSchema) that ship separately in scaleapi. Unit tests pass regardless; live calls 404 until that deploys. All benchmark CRUD / benchmark-eval / rollup-group / preset endpoints are already live (DE-8209).

resolves https://linear.app/scale-epd/issue/DE-8209

Greptile Summary

This PR brings the Nucleus Python SDK to parity with the EvaluationV2 UI under the benchmark paradigm. It introduces benchmarks (frozen item sets for evaluation), benchmark evaluations with rollup groups, exclusion rules, presets, leaderboard ranking, filter-schema discovery, and the cancel/retry lifecycle actions.

• New modules added: nucleus/benchmark.py, nucleus/evaluation_v2_exclusions.py, nucleus/evaluation_v2_preset.py — each fully tested with 59 mock-based unit tests.
• Breaking removal: create_evaluation_v2 (dataset/slice-scoped) is gone; create_benchmark_evaluation_v2 via benchmarks/{id}/evaluationsV2 is the sole creation path.
• Connection layer: A new patch() method mirrors the existing post() and get() helpers, used for update_evaluation_v2_preset and update_benchmark.

Confidence Score: 5/5

Safe to merge — all new functionality is additive, the existing evaluation path is untouched, and the 59 mock-based unit tests cover the key creation, preset-seeding, and edge-case paths.

The benchmark and evaluation changes are well-structured and well-tested. The previous-thread concerns about camelCase parsing and preset-override ordering both appear addressed in the current code. The remaining findings are all minor quality or consistency issues that do not affect the correctness of any live API call.

Files Needing Attention: nucleus/evaluation_v2_preset.py (or-vs-is-not-None inconsistency in from_json) and nucleus/evaluation_v2_exclusions.py (BoxAreaExclusionRule missing bounds guard).

Important Files Changed

Filename	Overview
nucleus/benchmark.py	New file: Benchmark dataclass with CRUD delegates to NucleusClient; from_json correctly maps benchmark_id.
nucleus/init.py	Adds benchmark/preset/leaderboard methods and NucleusClient.patch(); preset seeding logic correctly gates on _UNSET sentinel.
nucleus/evaluation_v2.py	Adds RollupGroup, exclusion fields, cancel/retry, filter_schema; charts migrated from GET+urlencode to POST; wait_for_completion str-enum comparisons are sound.
nucleus/evaluation_v2_preset.py	EvaluationV2Preset from_json handles both camelCase and snake_case; allowed_label_matches uses or fallback which silently discards empty-list responses.
nucleus/evaluation_v2_exclusions.py	New exclusion rule types; BoxAreaExclusionRule lacks client-side validation that at least one bound is set.
nucleus/data_transfer_object/evaluation_v2.py	Adds filter fields and new DTOs consistent with server snake_case convention.
nucleus/connection.py	Minimal: adds patch() mirroring post() pattern.
nucleus/dataset.py	Adds evaluation_label_schema() consistent with existing dataset endpoint patterns.
tests/test_benchmarks.py	New: comprehensive mock-based tests for all benchmark CRUD and eval create/preset seeding.
tests/test_evaluation_v2.py	Updated: adds tests for new filter args, exclusion parsing, cancel/retry, and filter-schema methods.

Sequence Diagram

sequenceDiagram
    participant U as User
    participant NC as NucleusClient
    participant API as Nucleus API

    U->>NC: "create_benchmark(name, slice_id=...)"
    NC->>API: POST /benchmarks
    API-->>NC: "{benchmark_id, ...}"
    NC-->>U: Benchmark

    U->>NC: "create_benchmark_evaluation_v2(benchmark_id, model_run_id, preset=preset)"
    Note over NC: Seed from preset if no explicit label config
    NC->>API: "POST /benchmarks/{id}/evaluationsV2"
    API-->>NC: "{evaluation_id}"
    NC->>API: "GET /evaluationsV2/{id}"
    API-->>NC: EvaluationV2 payload
    NC-->>U: EvaluationV2

    U->>NC: evaluation.wait_for_completion()
    loop poll
        NC->>API: "GET /evaluationsV2/{id}"
        API-->>NC: "{status: ...}"
    end
    NC-->>U: EvaluationV2 (succeeded)

    U->>NC: "evaluation.charts(iou_threshold=0.5)"
    NC->>API: "POST /evaluationsV2/{id}/charts"
    API-->>NC: EvaluationV2Charts
    NC-->>U: EvaluationV2Charts

    U->>NC: leaderboard_ranking("MAP_50", [benchmark_id])
    NC->>API: POST /leaderboard/ranking
    API-->>NC: "[{rank, score, ...}]"
    NC-->>U: List[LeaderboardRankingEntry]

Loading

_{Reviews (8): Last reviewed commit: "Make the eval/benchmark docstrings user-..." | Re-trigger Greptile}

[jaypsiri]

Had a few questions, but mostly looks good to me! I would defer to @vinay553 and @edwinpav's review on this though

[jaypsiri]

Should these be Literals (or StrEnums) instead of bare strs?

[jaypsiri]

I have a general curiosity question for myself, that I think is outside the scope of this PR:

Is there a reason payload is a dict instead of a type-validated object? I guess the access/assignment of payload attributes by string keys just seems a little unsafe/random to me lol

Relatedly, I see that there's this convention in Claude.MD to add payload keys to constants.py:

nucleus-python-client/CLAUDE.md

Line 41 in 49289ab

- `nucleus/constants.py` — All API payload keys are constants here. When adding a new field, add a `*_KEY` constant first and reference it from `from_json` / `to_payload` rather than inlining the string.

And that the *_KEY constants are used with the payload, for example here:

nucleus-python-client/nucleus/dataset_item.py

Lines 181 to 189 in 49289ab

	if BACKEND_REFERENCE_ID_KEY in payload:
	payload[REFERENCE_ID_KEY] = payload[BACKEND_REFERENCE_ID_KEY]
	return cls(
	image_location=image_url,
	pointcloud_location=pointcloud_url,
	reference_id=payload.get(REFERENCE_ID_KEY),
	metadata=payload.get(METADATA_KEY, {}),
	phash=payload.get(PHASH_KEY),

Is there a reason to do/not do that? Like on lines 57-70 below, for example.

[jaypsiri]

kind of nit / probably also outside the scope of this PR to fix - is there a reason to not use the existing .get() from connection.py instead? I see that a lot of existing syntax here is make_request so it makes sense that it's used here as well though the rest of the PR uses the wrappers lol