feat(examples): add verified task examples - #223
Open
svonava wants to merge 3 commits into
Open
Conversation
Contributor
Author
|
@coderabbitai review |
Contributor
Author
|
Review update:\n\n- Fixed the provenance tests so the documented uv run workflow does not scan generated virtual-environment metadata.\n- Added all three uv.lock files and reran with the resolved SIE SDK and Ruff versions.\n- Verified 17 offline tests, artifact hashes, full-vector score recomputation, source offsets, Ruff lint, and Ruff format.\n\nNo model calls or recorded outputs were rewritten. |
📝 WalkthroughWalkthroughThe PR adds three standalone, verified SIE examples for multimodal search, named-entity extraction, and primary-source reranking, including runners, validation logic, offline tests, project configuration, documentation, and recorded-artifact integrity checks. ChangesVerified SIE examples
Estimated code review effort: 4 (Complex) | ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
examples/named-entity-extraction/tests/test_example.py (1)
68-95: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winFailure-path coverage is limited to one branch of
validate_response.Only the "missing required anchors" branch is tested.
validate_responsealso raises on unrequested labels, out-of-bounds/mismatched offsets, and invalid/out-of-range scores (Lines 163-177 ofrun.py), none of which have dedicated tests. Consider adding tests for at least one or two of these to guard the fail-closed contract as the recorded artifacts evolve.As per path instructions, "Check that commands match the implementation and that tests cover failure paths."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/named-entity-extraction/tests/test_example.py` around lines 68 - 95, Expand failure-path coverage for validate_response beyond the existing missing-required-anchors case by adding dedicated tests for at least one or two other validation failures, such as unrequested labels, invalid or out-of-range offsets, or invalid scores. Each test should construct an otherwise representative response, assert the expected ValueError and message, and preserve the existing artifact-based test setup.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/multimodal-search/tests/test_example.py`:
- Around line 25-93: Add failure-path tests to MultimodalSearchExampleTests
covering checksum tampering, incorrect image counts, a true
synthetic_or_generated_images flag, non-finite vectors passed to dense_vector,
and an unexpected top match from evaluate. Assert each relevant
load_and_verify_sources, _validate_sources_dict, dense_vector, or evaluate call
raises ValueError, using the existing fixtures and validation entry points
without weakening success-path assertions.
- Around line 9-11: Fix the Ruff E402 violation in the test module by avoiding
the runtime sys.path.insert followed by a top-level import run; use an import
approach that preserves access to the example’s run module without placing an
import after executable statements. Alternatively, add the targeted tests/* E402
per-file ignore in pyproject.toml if retaining the current import pattern.
In `@examples/named-entity-extraction/tests/test_example.py`:
- Around line 9-11: Fix Ruff E402 in the test module by adding a targeted
per-file ignore for this file in the nearest pyproject.toml, rather than
changing the runtime path setup or import order. Keep the existing ROOT,
sys.path.insert, and import run behavior unchanged.
In `@examples/rerank/run.py`:
- Around line 147-150: Reject boolean values in the score validation loop of
validate_response by checking for bool before the numeric and finite checks. In
examples/rerank/run.py lines 147-150, preserve the existing ValueError for
invalid scores; in examples/rerank/tests/test_example.py lines 38-58, change one
recorded score to True and assert that validate_response raises ValueError.
In `@examples/rerank/tests/test_example.py`:
- Around line 9-11: Update the module-level imports in test_example.py: import
importlib before modifying sys.path, then load the run module via importlib
after sys.path.insert executes, eliminating Ruff E402 while preserving the
existing module resolution.
---
Nitpick comments:
In `@examples/named-entity-extraction/tests/test_example.py`:
- Around line 68-95: Expand failure-path coverage for validate_response beyond
the existing missing-required-anchors case by adding dedicated tests for at
least one or two other validation failures, such as unrequested labels, invalid
or out-of-range offsets, or invalid scores. Each test should construct an
otherwise representative response, assert the expected ValueError and message,
and preserve the existing artifact-based test setup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 70111323-ba00-4b7b-950c-65e59cd6e9a4
⛔ Files ignored due to path filters (41)
.coderabbit.yamlis excluded by none and included by noneexamples/multimodal-search/data/images/black-camera.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/black-handbag.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/blue-running-sneaker.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/green-backpack.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-leather-handbag.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-shoes.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/multimodal-search/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/multimodal-search/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/image-candidate-embeddings.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/text-query-embedding.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/image-candidates.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/text-query.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/named-entity-extraction/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/rerank/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (16)
examples/README.mdexamples/multimodal-search/.gitignoreexamples/multimodal-search/README.mdexamples/multimodal-search/pyproject.tomlexamples/multimodal-search/run.pyexamples/multimodal-search/tests/test_example.pyexamples/named-entity-extraction/.gitignoreexamples/named-entity-extraction/README.mdexamples/named-entity-extraction/pyproject.tomlexamples/named-entity-extraction/run.pyexamples/named-entity-extraction/tests/test_example.pyexamples/rerank/.gitignoreexamples/rerank/README.mdexamples/rerank/pyproject.tomlexamples/rerank/run.pyexamples/rerank/tests/test_example.py
Comment on lines
+9
to
+11
| ROOT = Path(__file__).resolve().parents[1] | ||
| sys.path.insert(0, str(ROOT)) | ||
| import run |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Ruff E402: import not at top of file.
The sys.path.insert + import run pattern triggers Ruff's E402, and this pyproject.toml has no per-file-ignores for tests/. Given the PR's validation explicitly includes a clean Ruff run, this would fail that gate as written.
🔧 Proposed fix
-import run
+import run # noqa: E402Or, alternatively, add a per-file ignore in pyproject.toml:
[tool.ruff.lint.per-file-ignores]
"tests/*" = ["E402"]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT)) | |
| import run | |
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT)) | |
| import run # noqa: E402 |
🧰 Tools
🪛 Ruff (0.15.21)
[error] 11-11: Module level import not at top of file
(E402)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/multimodal-search/tests/test_example.py` around lines 9 - 11, Fix
the Ruff E402 violation in the test module by avoiding the runtime
sys.path.insert followed by a top-level import run; use an import approach that
preserves access to the example’s run module without placing an import after
executable statements. Alternatively, add the targeted tests/* E402 per-file
ignore in pyproject.toml if retaining the current import pattern.
Source: Linters/SAST tools
Comment on lines
+25
to
+93
| class MultimodalSearchExampleTests(unittest.TestCase): | ||
| def test_images_match_the_licensed_source_manifest(self) -> None: | ||
| sources = run.load_and_verify_sources() | ||
| self.assertEqual(len(sources["images"]), 6) | ||
| self.assertTrue( | ||
| all( | ||
| image["license"] and image["license_url"] and image["source"] and image["creator"] | ||
| for image in sources["images"] | ||
| ) | ||
| ) | ||
|
|
||
| def test_recorded_audit_envelopes_match_the_runner(self) -> None: | ||
| sources = run.load_and_verify_sources() | ||
| recorded_query = run.read_json(ROOT / "verified-run" / "requests" / "text-query.json") | ||
| recorded_images = run.read_json(ROOT / "verified-run" / "requests" / "image-candidates.json") | ||
| self.assertEqual(recorded_query, run.build_query_audit_envelope(sources)) | ||
| self.assertEqual(recorded_images, run.build_image_audit_envelope(sources)) | ||
|
|
||
| def test_recorded_ranking_is_recomputed_from_full_vectors(self) -> None: | ||
| sources = run.load_and_verify_sources() | ||
| query = run.read_json(ROOT / "verified-run" / "raw" / "text-query-embedding.json") | ||
| images = run.read_json(ROOT / "verified-run" / "raw" / "image-candidate-embeddings.json") | ||
| observed = run.evaluate(sources, query, images) | ||
| recorded = run.read_json(ROOT / "verified-run" / "evaluation.json") | ||
| self.assertEqual(observed["query"], recorded["query"]) | ||
| self.assertEqual(observed["metric"], recorded["metric"]) | ||
| self.assertEqual(observed["image_count"], recorded["image_count"]) | ||
| for left, right in zip( | ||
| observed["sorted_matches"], | ||
| recorded["sorted_matches"], | ||
| strict=True, | ||
| ): | ||
| self.assertEqual(left["rank"], right["rank"]) | ||
| self.assertEqual(left["file"], right["file"]) | ||
| self.assertEqual(left["sha256"], right["sha256"]) | ||
| self.assertAlmostEqual(left["score"], right["score"], places=15) | ||
|
|
||
| def test_manifest_pins_inputs_and_artifacts(self) -> None: | ||
| manifest = run.read_json(ROOT / "verified-run" / "manifest.json") | ||
| self.assertEqual( | ||
| manifest["inputs"]["sources_sha256"], | ||
| run.sha256_file(run.SOURCES_PATH), | ||
| ) | ||
| expected_images = {f"data/{image['file']}" for image in run.load_and_verify_sources()["images"]} | ||
| self.assertEqual(set(manifest["inputs"]["image_files"]), expected_images) | ||
| for relative, digest in manifest["inputs"]["image_files"].items(): | ||
| self.assertEqual(digest, run.sha256_file(ROOT / relative)) | ||
| expected_artifacts = { | ||
| "verified-run/evaluation.json", | ||
| "verified-run/raw/image-candidate-embeddings.json", | ||
| "verified-run/raw/text-query-embedding.json", | ||
| "verified-run/requests/image-candidates.json", | ||
| "verified-run/requests/text-query.json", | ||
| } | ||
| self.assertEqual(set(manifest["artifacts"]), expected_artifacts) | ||
| for relative, digest in manifest["artifacts"].items(): | ||
| self.assertEqual(digest, run.sha256_file(ROOT / relative)) | ||
|
|
||
| def test_metadata_has_no_temporary_filesystem_paths(self) -> None: | ||
| forbidden = ("/Users/", "/root/", "/tmp/", "reference-batch", "/v4/") | ||
| for directory in (ROOT / "data", ROOT / "verified-run"): | ||
| for path in directory.rglob("*.json"): | ||
| value = json.loads(path.read_text(encoding="utf-8")) | ||
| for text in strings(value): | ||
| self.assertFalse( | ||
| any(marker in text for marker in forbidden), | ||
| f"{path}: {text}", | ||
| ) | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add failure-path tests for the fail-closed guards.
All four tests exercise only success paths. None assert that load_and_verify_sources/evaluate/dense_vector actually raise on tampered checksums, wrong image counts, a synthetic_or_generated_images flag flipped to True, non-finite vectors, or an unexpected top match — the guarantees this example advertises as "offline fail-closed tests."
Example addition:
def test_rejects_synthetic_flag(self) -> None:
sources = run.load_and_verify_sources()
tampered = {**sources, "synthetic_or_generated_images": True}
with self.assertRaises(ValueError):
run._validate_sources_dict(tampered) # or refactor load_and_verify_sources to accept a dictAs per path instructions, "Check that commands match the implementation and that tests cover failure paths."
🧰 Tools
🪛 ast-grep (0.44.1)
[info] 83-83: Do not hardcode temporary file or directory names
Context: "/tmp/"
Note: [CWE-377] Insecure Temporary File.
(hardcoded-tmp-file)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/multimodal-search/tests/test_example.py` around lines 25 - 93, Add
failure-path tests to MultimodalSearchExampleTests covering checksum tampering,
incorrect image counts, a true synthetic_or_generated_images flag, non-finite
vectors passed to dense_vector, and an unexpected top match from evaluate.
Assert each relevant load_and_verify_sources, _validate_sources_dict,
dense_vector, or evaluate call raises ValueError, using the existing fixtures
and validation entry points without weakening success-path assertions.
Source: Path instructions
Comment on lines
+9
to
+11
| ROOT = Path(__file__).resolve().parents[1] | ||
| sys.path.insert(0, str(ROOT)) | ||
| import run |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix Ruff E402 flagged by static analysis.
import run after sys.path.insert triggers Ruff's E402 (confirmed by the static analysis hint), which isn't suppressed anywhere in this file's pyproject.toml. This would fail the "Ruff lint" validation step referenced in the PR description.
🔧 Proposed fix
-import run
+import run # noqa: E402Or, more durably, add a per-file ignore in examples/named-entity-extraction/pyproject.toml:
[tool.ruff]
line-length = 120
target-version = "py312"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*.py" = ["E402"]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT)) | |
| import run | |
| ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(ROOT)) | |
| import run # noqa: E402 |
🧰 Tools
🪛 Ruff (0.15.21)
[error] 11-11: Module level import not at top of file
(E402)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/named-entity-extraction/tests/test_example.py` around lines 9 - 11,
Fix Ruff E402 in the test module by adding a targeted per-file ignore for this
file in the nearest pyproject.toml, rather than changing the runtime path setup
or import order. Keep the existing ROOT, sys.path.insert, and import run
behavior unchanged.
Source: Linters/SAST tools
Comment on lines
+147
to
+150
| for row in scores: | ||
| score = row.get("score") | ||
| if not isinstance(score, (int, float)) or not math.isfinite(score): | ||
| raise ValueError(f"Invalid score for {row.get('item_id')}") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
python - <<'PY'
import math
assert isinstance(True, (int, float))
assert math.isfinite(True)
PYRepository: superlinked/sie
Length of output: 153
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== run.py relevant section =="
sed -n '120,190p' examples/rerank/run.py
echo
echo "== test_example.py relevant section =="
sed -n '1,160p' examples/rerank/tests/test_example.py
echo
echo "== search for validate_response / score tests =="
rg -n "validate_response|Invalid score|math\.isfinite|candidate_count|expected_top_candidate_id|True" examples/rerank -SRepository: superlinked/sie
Length of output: 9657
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '136,156p' examples/rerank/run.py
printf '\n---\n'
sed -n '1,120p' examples/rerank/tests/test_example.py
printf '\n---\n'
rg -n "Invalid score|validate_response|True|False|math\.isfinite|isinstance\(score" examples/rerank -SRepository: superlinked/sie
Length of output: 6214
Reject boolean scores and add a failure-path test.
True and False pass the current isinstance(score, (int, float)) + math.isfinite(score) check in examples/rerank/run.py, so malformed score rows can be accepted as valid.
examples/rerank/run.py: excludeboolbefore the finite check.examples/rerank/tests/test_example.py: flip one recorded score toTrueand assertvalidate_responseraisesValueError.
📍 Affects 2 files
examples/rerank/run.py#L147-L150(this comment)examples/rerank/tests/test_example.py#L38-L58
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/rerank/run.py` around lines 147 - 150, Reject boolean values in the
score validation loop of validate_response by checking for bool before the
numeric and finite checks. In examples/rerank/run.py lines 147-150, preserve the
existing ValueError for invalid scores; in examples/rerank/tests/test_example.py
lines 38-58, change one recorded score to True and assert that validate_response
raises ValueError.
Source: Path instructions
Comment on lines
+9
to
+11
| ROOT = Path(__file__).resolve().parents[1] | ||
| sys.path.insert(0, str(ROOT)) | ||
| import run |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
ruff check examples/rerank/tests/test_example.py --select E402Repository: superlinked/sie
Length of output: 512
🏁 Script executed:
#!/bin/bash
sed -n '1,40p' examples/rerank/tests/test_example.pyRepository: superlinked/sie
Length of output: 1503
Fix the module-level import order. sys.path.insert(...) runs before import run, which triggers Ruff E402; import importlib first and load run after the path update.
🧰 Tools
🪛 Ruff (0.15.21)
[error] 11-11: Module level import not at top of file
(E402)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/rerank/tests/test_example.py` around lines 9 - 11, Update the
module-level imports in test_example.py: import importlib before modifying
sys.path, then load the run module via importlib after sys.path.insert executes,
eliminating Ruff E402 while preserving the existing module resolution.
Source: Linters/SAST tools
Contributor
Author
|
@coderabbitai full review |
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 49 minutes. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/named-entity-extraction/run.py`:
- Around line 173-177: Update the score validation around entity score
extraction to explicitly reject boolean values before numeric validation, while
preserving finite and 0–1 range checks for valid numbers. Add a regression test
in the named-entity extraction example tests confirming boolean scores are
rejected.
In `@examples/rerank/tests/test_example.py`:
- Around line 38-58: Add a failure-path test alongside
test_recorded_responses_pass_fail_closed_checks that mutates a recorded response
to create an invalid validation case, then assert run.validate_response raises
ValueError. Reuse the existing case-loading, artifact lookup, and
response-reading setup, and preserve the current success-path assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 18dc32af-abcb-4276-b7b6-1954fd6befce
⛔ Files ignored due to path filters (41)
.coderabbit.yamlis excluded by none and included by noneexamples/multimodal-search/data/images/black-camera.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/black-handbag.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/blue-running-sneaker.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/green-backpack.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-leather-handbag.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-shoes.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/multimodal-search/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/multimodal-search/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/image-candidate-embeddings.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/text-query-embedding.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/image-candidates.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/text-query.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/named-entity-extraction/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/rerank/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (16)
examples/README.mdexamples/multimodal-search/.gitignoreexamples/multimodal-search/README.mdexamples/multimodal-search/pyproject.tomlexamples/multimodal-search/run.pyexamples/multimodal-search/tests/test_example.pyexamples/named-entity-extraction/.gitignoreexamples/named-entity-extraction/README.mdexamples/named-entity-extraction/pyproject.tomlexamples/named-entity-extraction/run.pyexamples/named-entity-extraction/tests/test_example.pyexamples/rerank/.gitignoreexamples/rerank/README.mdexamples/rerank/pyproject.tomlexamples/rerank/run.pyexamples/rerank/tests/test_example.py
Comment on lines
+173
to
+177
| score = entity.get("score") | ||
| if not isinstance(score, (int, float)) or not math.isfinite(score): | ||
| raise ValueError(f"Invalid score at entity {index}") | ||
| if score < 0 or score > 1: | ||
| raise ValueError(f"Out-of-range score at entity {index}") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '== File map ==\n'
git ls-files 'examples/named-entity-extraction/*' 'examples/rerank/*' | sed -n '1,200p'
printf '\n== Outline: named-entity-extraction/run.py ==\n'
ast-grep outline examples/named-entity-extraction/run.py --view expanded || true
printf '\n== Relevant lines in named-entity-extraction/run.py ==\n'
sed -n '150,190p' examples/named-entity-extraction/run.py
printf '\n== Relevant lines in examples/rerank/run.py ==\n'
sed -n '135,160p' examples/rerank/run.py
printf '\n== Tests in named-entity-extraction ==\n'
fd -a . examples/named-entity-extraction/tests examples/named-entity-extraction | sed -n '1,200p'
if [ -f examples/named-entity-extraction/tests/test_example.py ]; then
sed -n '1,240p' examples/named-entity-extraction/tests/test_example.py
fiRepository: superlinked/sie
Length of output: 13382
Reject boolean scores in examples/named-entity-extraction/run.py:173-177
isinstance(score, (int, float)) accepts True/False, so a malformed score: true passes as 1. Add an explicit bool check here, and cover it with a regression test in examples/named-entity-extraction/tests/test_example.py.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/named-entity-extraction/run.py` around lines 173 - 177, Update the
score validation around entity score extraction to explicitly reject boolean
values before numeric validation, while preserving finite and 0–1 range checks
for valid numbers. Add a regression test in the named-entity extraction example
tests confirming boolean scores are rejected.
Comment on lines
+38
to
+58
| def test_recorded_responses_pass_fail_closed_checks(self) -> None: | ||
| cases, _ = run.load_and_verify_inputs() | ||
| evaluation = run.read_json(ROOT / "verified-run" / "evaluation.json") | ||
| checks = {check["case_id"]: check for check in evaluation["checks"]} | ||
| for case_id, case in cases["cases"].items(): | ||
| name = run.ARTIFACT_NAMES[case_id] | ||
| response = run.read_json(ROOT / "verified-run" / "raw" / f"{name}.json") | ||
| observed = run.validate_response(case_id, case, response) | ||
| self.assertTrue(observed["passed"]) | ||
| self.assertEqual( | ||
| checks[case_id]["expected_top_candidate_id"], | ||
| observed["expected_top_candidate_id"], | ||
| ) | ||
| self.assertEqual( | ||
| checks[case_id]["observed_top_candidate_id"], | ||
| observed["observed_top_candidate_id"], | ||
| ) | ||
| self.assertEqual( | ||
| checks[case_id]["candidate_count"], | ||
| observed["candidate_count"], | ||
| ) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
No failure-path test for validate_response.
This file only exercises the success path (recorded responses passing validation). Unlike examples/named-entity-extraction/tests/test_example.py, which has test_empty_response_fails_required_anchor_check, there's no test that a bad rank, incomplete candidate coverage, invalid score, or wrong top candidate actually raises ValueError in run.validate_response.
Add at least one test that mutates a recorded response (e.g., drop a candidate, corrupt a rank, or swap the top candidate) and asserts run.validate_response raises.
As per path instructions, "Check that commands match the implementation and that tests cover failure paths."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/rerank/tests/test_example.py` around lines 38 - 58, Add a
failure-path test alongside test_recorded_responses_pass_fail_closed_checks that
mutates a recorded response to create an invalid validation case, then assert
run.validate_response raises ValueError. Reuse the existing case-loading,
artifact lookup, and response-reading setup, and preserve the current
success-path assertions.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds runnable examples for reranking, named entity extraction, and multimodal search.\n\nEach example includes primary-source inputs, exact recorded public SIE responses, checksummed evidence manifests, and offline fail-closed tests. The multimodal example includes the licensed source images and full vectors needed to recompute every displayed cosine score.\n\nValidation: 17 unit tests, Ruff, Ty, JSON and artifact hashes, and examples-only scope checks.
Summary by CodeRabbit
.gitignorefiles to exclude common local runtime outputs.