Replace fictional insurance claim with public FEMA appeal by svonava · Pull Request #225 · superlinked/sie

svonava · 2026-07-26T19:23:03Z

What changed

replaces the fictional insurance claim with FEMA Flood Insurance Appeal Decision B8 and the controlling Standard Flood Insurance Policy
runs Docling, GLiNER2, BGE reranking, and Qwen through SIE
records a verified July 26, 2026 run with publisher URLs, checksums, model outputs, timings, and deterministic evaluation
removes the generated claim form, estimate, photo, Grounding DINO stage, and invented claim facts

Verification

uv run ruff check .
uv run pytest -q
real NVIDIA L4 run through SIE on Modal
all ten factual evaluation checks passed

The example summarizes a completed public FEMA appeal. It does not decide a live claim.

Summary by CodeRabbit

New Features
- Updated the insurance example workflow to review FEMA flood-insurance appeal decisions and policy evidence.
- Added verified-run support with pinned artifacts via integrity hashes.
- Revised evaluation to use structured appeal/decision outputs and deterministic scope findings.
Documentation
- Updated example READMEs, evidence bundle expectations, and run instructions (including self-hosted model endpoint guidance).
- Removed the claim-preparation step from the workflow.
Bug Fixes
- Improved PDF source validation and strengthened review/evaluation consistency checks.
Tests
- Updated verified integration checks for sources, review/evaluation outputs, and manifest integrity.

svonava · 2026-07-26T19:26:56Z

@coderabbitai review

svonava · 2026-07-26T19:50:50Z

@coderabbitai review

svonava · 2026-07-27T00:20:36Z

Review update:\n\n- Added SHA-256 hashes for every saved model output and derived artifact.\n- Added tests that verify the bundled FEMA PDFs against the source manifest and recompute the saved evaluation from review.json.\n- Updated the gallery entry so it describes the real FEMA appeal instead of the removed fictional claim.\n- Reran 8 locked offline tests plus Ruff lint and format.\n\nThe factual model outputs remain unchanged.

coderabbitai · 2026-07-27T05:13:48Z

📝 Walkthrough

Walkthrough

The insurance claims example now reviews FEMA Flood Insurance Appeal Decision B8 and related policy documents. Claim-packet preparation, vision processing, and image fixtures were removed; appeal fact extraction, policy retrieval, structured review generation, scope evaluation, and artifact hashing were added.

Changes

FEMA appeal review

Layer / File(s)	Summary
Appeal sources and configuration `examples/README.md`, `examples/insurance-claims-agent/config.yaml`, `examples/insurance-claims-agent/insurance_claims/config.py`, `examples/insurance-claims-agent/insurance_claims/fetch.py`, `examples/insurance-claims-agent/pyproject.toml`, `examples/insurance-claims-agent/tests/test_config.py`	Configuration now uses public appeal and policy documents, removes vision and claim-packet settings, narrows dependencies, and validates source fixtures and PDF payloads.
Appeal review pipeline `examples/insurance-claims-agent/README.md`, `examples/insurance-claims-agent/insurance_claims/review.py`, `examples/insurance-claims-agent/tests/test_review.py`	The staged pipeline extracts appeal facts, retrieves policy evidence, generates the revised structured review, and documents the updated execution flow.
Evaluation and artifact integrity `examples/insurance-claims-agent/insurance_claims/evaluate.py`, `examples/insurance-claims-agent/tests/test_review.py`	Evaluation checks target appeal scope and evidence fields, while manifests record SHA-256 hashes for run artifacts and verified-run tests recompute evaluations and hashes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReviewCLI
  participant DefaultStage
  participant GenerationStage
  ReviewCLI->>DefaultStage: parse appeal and policy sources
  DefaultStage->>DefaultStage: extract claim facts and retrieve policy chunks
  ReviewCLI->>GenerationStage: start generation stage
  GenerationStage->>GenerationStage: generate structured appeal review

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: replacing the fictional insurance claim example with a public FEMA appeal.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/real-nfip-appeal-example

_{Comment @coderabbitai help to get the list of available commands.}

svonava · 2026-07-27T05:19:26Z

@coderabbitai full review

coderabbitai · 2026-07-27T05:19:30Z

✅ Action performed

Full review finished.

Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 49 minutes.

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

examples/insurance-claims-agent/tests/test_review.py (1)

69-91: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a failure-path test for evaluate_review.

Every test here only feeds evaluate_review fully-correct data and asserts all checks pass. There's no test that a check correctly reports passed=False on bad input (wrong amount, missing scope keyword, missing finding category), so a logic bug in any individual Check (wrong tolerance, inverted condition, wrong field name) wouldn't be caught.

As per path instructions, "Check that commands match the implementation and that tests cover failure paths."

✅ Example negative-case test

def test_evaluation_fails_when_proof_of_loss_amount_is_wrong() -> None:
    review = {
        "route": "scope_review_required",
        "appeal_summary": {
            "proof_of_loss_amount": 1,  # wrong
            "removal_estimate": 49500,
            "barge_estimate": 181832.94,
            "debris_cubic_yards_min": 12,
            "debris_cubic_yards_max": 15,
        },
        "decision": {
            "covered_scope": "Remove flood-borne stones from underneath the insured building to its perimeter.",
            "excluded_scope": "Barge transport, handling, disposal, and yard removal.",
            "evidence_needed": "Other contractor estimates.",
            "prior_claim_check": "Proof of repairs from previous claims.",
        },
        "findings": [
            {"category": "covered_removal"},
            {"category": "excluded_transport"},
            {"category": "price_support"},
            {"category": "prior_claim_overlap"},
        ],
    }
    checks = {check.name: check for check in evaluate_review(review)}
    assert checks["proof-of-loss-amount"].passed is False

Also applies to: 96-105

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/insurance-claims-agent/tests/test_review.py` around lines 69 - 91,
Add failure-path coverage for evaluate_review by introducing a negative-case
test with otherwise valid review data and one deliberately invalid field, such
as an incorrect proof_of_loss_amount. Collect checks by name and assert the
corresponding check’s passed value is False; add analogous coverage for missing
scope keywords or finding categories if needed to exercise each check’s failure
behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/insurance-claims-agent/README.md`:
- Line 37: Update the sentence describing the verified-run directory to add a
comma after “2026,” preserving the existing date and surrounding wording.

---

Nitpick comments:
In `@examples/insurance-claims-agent/tests/test_review.py`:
- Around line 69-91: Add failure-path coverage for evaluate_review by
introducing a negative-case test with otherwise valid review data and one
deliberately invalid field, such as an incorrect proof_of_loss_amount. Collect
checks by name and assert the corresponding check’s passed value is False; add
analogous coverage for missing scope keywords or finding categories if needed to
exercise each check’s failure behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5a290268-6b8c-4d52-a645-5ff63c8dc80b

📥 Commits

Reviewing files that changed from the base of the PR and between 2ac128c and 7f50fa3.

⛔ Files ignored due to path filters (22)

.coderabbit.yaml is excluded by none and included by none
examples/insurance-claims-agent/fixtures/SOURCES.md is excluded by !examples/**/fixtures/** and included by examples/**, examples/**/fixtures/SOURCES.md
examples/insurance-claims-agent/fixtures/claim-note.txt is excluded by !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/claim.json is excluded by !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/sources/flooded-house-interior.jpg is excluded by !**/*.jpg, !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/sources/nfip-appeal-b8.pdf is excluded by !**/*.pdf, !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/sources/nfip-proof-of-loss.pdf is excluded by !**/*.pdf, !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/uv.lock is excluded by !**/*.lock, !examples/**/uv.lock and included by examples/**
examples/insurance-claims-agent/verified-run/README.md is excluded by !examples/**/verified-run/** and included by examples/**, examples/**/verified-run/README.md
examples/insurance-claims-agent/verified-run/claim-facts.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/evaluation.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/manifest.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/markdown/appeal_decision.md is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/markdown/policy.md is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/policy-evidence.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/appeal_decision-parse.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/claim-facts.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/policy-parse.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/policy-rerank.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/review-completion.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/review.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/source-manifest.json is excluded by !examples/**/verified-run/** and included by examples/**

📒 Files selected for processing (12)

examples/README.md
examples/insurance-claims-agent/README.md
examples/insurance-claims-agent/config.yaml
examples/insurance-claims-agent/insurance_claims/config.py
examples/insurance-claims-agent/insurance_claims/evaluate.py
examples/insurance-claims-agent/insurance_claims/fetch.py
examples/insurance-claims-agent/insurance_claims/prepare.py
examples/insurance-claims-agent/insurance_claims/review.py
examples/insurance-claims-agent/pyproject.toml
examples/insurance-claims-agent/tests/test_config.py
examples/insurance-claims-agent/tests/test_prepare.py
examples/insurance-claims-agent/tests/test_review.py

💤 Files with no reviewable changes (3)

examples/insurance-claims-agent/tests/test_prepare.py
examples/insurance-claims-agent/insurance_claims/prepare.py
examples/insurance-claims-agent/insurance_claims/config.py

coderabbitai · 2026-07-27T05:20:20Z

-confidence. The saved boxes use the original 3072 by 2304 image coordinates.
+The model summarizes a completed public appeal. It does not decide a live claim.
+
+The [`verified-run`](verified-run/) directory records a July 26, 2026 run in


📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Minor: missing comma after year.

"a July 26, 2026 run" should read "a July 26, 2026, run" per standard month-day-year date punctuation.

🧰 Tools

🪛 LanguageTool

[style] ~37-~37: Some style guides suggest that commas should set off the year in a month-day-year date.
Context: ...fied-run/) directory records a July 26, 2026 run in which all ten factual checks pas...

(MISSING_COMMA_AFTER_YEAR)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/insurance-claims-agent/README.md` at line 37, Update the sentence describing the verified-run directory to add a comma after “2026,” preserving the existing date and surrounding wording.

Source: Linters/SAST tools

coderabbitai

🧹 Nitpick comments (1)

examples/insurance-claims-agent/tests/test_review.py (1)
107-118: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Exclusion set diverges from evaluate.py's own artifact-hashing logic.

This test excludes README.md and manifest.json from expected_paths, but evaluate_run() in evaluate.py only excludes manifest.json when building manifest["artifacts"]. It currently passes only because verified-run/README.md wasn't present when the recorded manifest's artifact list was generated. If evaluate_run is ever rerun against verified-run to refresh the pinned manifest, README.md would get hashed into artifacts and this test would start failing for a reason unrelated to a real regression.

Consider sharing one exclusion set (e.g., a _DOC_FILENAMES = {"README.md"} constant importable from evaluate.py) between the production artifact-hashing loop and this test, or having evaluate_run itself skip documentation files by name.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/insurance-claims-agent/tests/test_review.py` around lines 107 - 118,
The artifact exclusion rules in
test_verified_manifest_pins_every_recorded_artifact diverge from evaluate_run.
Define and reuse a shared documentation exclusion set, such as _DOC_FILENAMES
from evaluate.py, in both the production artifact-hashing loop and the test,
ensuring README.md and manifest.json are handled consistently when regenerating
the manifest.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@examples/insurance-claims-agent/tests/test_review.py`:
- Around line 107-118: The artifact exclusion rules in
test_verified_manifest_pins_every_recorded_artifact diverge from evaluate_run.
Define and reuse a shared documentation exclusion set, such as _DOC_FILENAMES
from evaluate.py, in both the production artifact-hashing loop and the test,
ensuring README.md and manifest.json are handled consistently when regenerating
the manifest.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cf9e6b61-ace8-485b-8a6b-13576b305ad9

📥 Commits

Reviewing files that changed from the base of the PR and between 2ac128c and 7f50fa3.

⛔ Files ignored due to path filters (22)

.coderabbit.yaml is excluded by none and included by none
examples/insurance-claims-agent/fixtures/SOURCES.md is excluded by !examples/**/fixtures/** and included by examples/**, examples/**/fixtures/SOURCES.md
examples/insurance-claims-agent/fixtures/claim-note.txt is excluded by !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/claim.json is excluded by !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/sources/flooded-house-interior.jpg is excluded by !**/*.jpg, !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/sources/nfip-appeal-b8.pdf is excluded by !**/*.pdf, !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/fixtures/sources/nfip-proof-of-loss.pdf is excluded by !**/*.pdf, !examples/**/fixtures/** and included by examples/**
examples/insurance-claims-agent/uv.lock is excluded by !**/*.lock, !examples/**/uv.lock and included by examples/**
examples/insurance-claims-agent/verified-run/README.md is excluded by !examples/**/verified-run/** and included by examples/**, examples/**/verified-run/README.md
examples/insurance-claims-agent/verified-run/claim-facts.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/evaluation.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/manifest.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/markdown/appeal_decision.md is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/markdown/policy.md is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/policy-evidence.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/appeal_decision-parse.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/claim-facts.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/policy-parse.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/policy-rerank.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/raw/review-completion.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/review.json is excluded by !examples/**/verified-run/** and included by examples/**
examples/insurance-claims-agent/verified-run/source-manifest.json is excluded by !examples/**/verified-run/** and included by examples/**

📒 Files selected for processing (12)

examples/README.md
examples/insurance-claims-agent/README.md
examples/insurance-claims-agent/config.yaml
examples/insurance-claims-agent/insurance_claims/config.py
examples/insurance-claims-agent/insurance_claims/evaluate.py
examples/insurance-claims-agent/insurance_claims/fetch.py
examples/insurance-claims-agent/insurance_claims/prepare.py
examples/insurance-claims-agent/insurance_claims/review.py
examples/insurance-claims-agent/pyproject.toml
examples/insurance-claims-agent/tests/test_config.py
examples/insurance-claims-agent/tests/test_prepare.py
examples/insurance-claims-agent/tests/test_review.py

💤 Files with no reviewable changes (3)

examples/insurance-claims-agent/tests/test_prepare.py
examples/insurance-claims-agent/insurance_claims/prepare.py
examples/insurance-claims-agent/insurance_claims/config.py

svonava added 2 commits July 26, 2026 11:22

examples: replace fictional claim with public NFIP appeal

3dc6966

examples: pin verified FEMA appeal run

5248708

chore: preserve complete verified insurance run

871e92d

fix(examples): pin the verified FEMA appeal run

7e1b3ac

chore: scope CodeRabbit to examples

7f50fa3

coderabbitai Bot reviewed Jul 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace fictional insurance claim with public FEMA appeal - #225

Replace fictional insurance claim with public FEMA appeal#225
svonava wants to merge 5 commits into
mainfrom
feat/real-nfip-appeal-example

svonava commented Jul 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

svonava commented Jul 26, 2026

Uh oh!

svonava commented Jul 26, 2026

Uh oh!

svonava commented Jul 27, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

svonava commented Jul 27, 2026

Uh oh!

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 27, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

svonava commented Jul 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Verification

Summary by CodeRabbit

Uh oh!

svonava commented Jul 26, 2026

Uh oh!

svonava commented Jul 26, 2026

Uh oh!

svonava commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

svonava commented Jul 27, 2026

Uh oh!

coderabbitai Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

svonava commented Jul 26, 2026 •

edited by coderabbitai Bot

Loading

svonava commented Jul 27, 2026 •

edited

Loading

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading