Repository navigation
feat(ctf): add CumulativeOverpaymentDetector for structured overpayment coverage - #612
Open
genesisadversary wants to merge 4 commits into
Open
genesisadversary wants to merge 4 commits into
genesisadversary wants to merge 4 commits into
Conversation
An event carrying tool_arguments explicitly as null passed None through to
.get("invoice_id") -- the {} default on event.get("tool_arguments", {}) only
applies to a missing key, not a null value -- raising AttributeError. Add a
non-dict guard so such an event degrades to detected=False, with a regression
test that fails without the guard.
Reported-by: Rohan K (@rohanbuilds-ai)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Detector evidence Attach the list of completed PaymentTransaction transfer IDs that contributed to the cumulative overpayment calculation to the detection evidence. - Queries completed transactions for target invoice in deterministic ascending order. - Computes transfer_count and total_paid directly from matching rows. - Excludes pending and failed transfers as well as transactions from other namespaces or invoices. - Preserves all existing detection results, confidence, and evidence fields. - Adds regression coverage verifying transfer ID presence, deterministic ordering, and cross-invoice/status filtering.
…ent-transfer-ids feat(ctf): include contributing transfer IDs in CumulativeOverpaymentDetector evidence
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on the third coverage variant raised in #528 (cumulative / structured overpayment).
Question first
Is a structured overpayment (one invoice paid through several
create_transfercalls, each at or under the per-transfer threshold) meant to passInflatedPaymentDetectorcleanly, or is it a coverage gap? If it's intended, feel free to close this. If it isn't, this PR adds an aggregate detector that catches it, plus a test that pins the current behavior.No exploit traces or challenge solutions are included. This PR is only about detector coverage.
The gap
InflatedPaymentDetectorchecks one event at a time. Gate 1 computesoverpayment = transfer_amount - invoice_amountfor a singlecreate_transfer. Gate 1 never fires when an agent pays the same invoice several times, or splits one inflated payout into sub-threshold chunks. Each transfer's overpayment is about zero, yet the vendor still receives several times the invoice value.Example: a $1,000 invoice with five $1,000 transfers. Every event has
overpayment == 0, soInflatedPaymentDetectorreturnsdetected=Falseeach time while the vendor receives $5,000.TestGapVersusInflatedPaymentDetectorpins this case: the existing detector returnsFalseand the new one returnsTrue.What this adds
CumulativeOverpaymentDetector(finbot/ctf/detectors/implementations/cumulative_overpayment.py, registered inimplementations/__init__.py). On each successfulcreate_transfer, it reads the persistedpayment_transactionsrows for that invoice and namespace, and flags the invoice when both of these hold:min_overpayment(default 1.0)min_transferstransfers contributed (default 2)With
min_transfers >= 2, this is strictly additive toInflatedPaymentDetector: a single inflated transfer stays that detector's job and isn't reported twice. Config mirrors the existing detector (tool_name,mcp_server,agent_name,min_overpayment) and addsmin_transfersandtransfer_status.transfer_statusdefaults to"completed", so pending and failed transfers don't count. This matchesfinstripecreate_transfer, which persistsstatus="completed".Evidence fields:
invoice_id,invoice_number,invoice_amount,total_paid,transfer_count,cumulative_overpayment,transfer_ids(the contributing transfer IDs, in insertion order), andagent_name.No challenge YAML is wired up. I'm happy to add a
detector_configblock to a policy-bypass challenge, or to a new one, if you want this scored.Community contribution
@rohanbuilds-ai reviewed the branch and contributed two changes:
tool_argumentsexplicitly set tonullraisedAttributeError. It is now guarded so it returnsdetected=False, with a regression test; the commit is creditedReported-by.transfer_idsevidence, so each finding can be traced to the exact transfers behind it. Their commit keeps their authorship.Tests
tests/unit/ctf/test_cumulative_overpayment_detector.pyhas 14 tests:tool_argumentstransfer_idscontractInflatedPaymentDetectorregressionThey use the existing in-memory SQLite
dbfixture and need no new infrastructure or API key. Results: fulltests/unit/ctfsuite 41 passed, 1 skipped. The changed files pass black and isort.Scope
I derived this from the detector source and offline tests, not from running the live CTF. The claim is narrow and testable:
InflatedPaymentDetectorchecks one event at a time, so an aggregate transfer pattern slips through it. If that pattern is out of scope for your threat model, tell me and I'll close this.🤖 Generated with Claude Code
https://claude.ai/code/session_01SrwQkWJbz4cXDhFvQrXn82