docs(email-security): the read budget, and why free text needs a window [HOLD until the gateway ships to prod] - #389
Open
maximelb wants to merge 5 commits into
Open
docs(email-security): the read budget, and why free text needs a window [HOLD until the gateway ships to prod]#389maximelb wants to merge 5 commits into
maximelb wants to merge 5 commits into
Conversation
Two reads recompute rather than serving a cached or seekable answer, and both are now bounded per organization: coverage over an explicit window, and a message search carrying q. Neither was documented as costing anything, and 'Free text over the message's identifying fields' gave no hint that q is matched row by row while every other filter in that table is a lookup. - api-reference: a Read budgets section with the refusal shape, what makes it distinguishable from the EML download's 429 on the same surface, and the two cheapest ways to get a refused read served. - messages: 'Free text needs a window' — what q actually costs, exactly what counts as a bound and what does not (and why an until alone does not), and the 512-character cap. - cli: coverage with no window is memoized and is the right shape for a poller; --window-days recomputes and is budgeted. - automation: the same, where the docs recommend scripted coverage polling. Examples for q are HTTP rather than CLI on purpose: q has no CLI flag today. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lcbill
previously approved these changes
Sep 8, 2026
An independent review of the gateway PR caught that the 429's own advice — 'add a mailbox, sender, campaign or IOC filter' — did not work, because every free-text search was counted. It is true now: a search riding one of those filters is an index lookup and is exempt entirely, and the docs say so rather than describing it as merely cheaper. A bare verdict is deliberately not an exemption, and the reason (the verdict index is keyed verdict-then-time, so benign is most of a tenant's mail) is stated where a reader would otherwise assume symmetry with the other filters. The number moved from 2400 to 7200 after the sizing was re-derived against the console's real 300ms search debounce. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lcbill
previously approved these changes
Sep 8, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lcbill
previously approved these changes
Sep 8, 2026
Contributor
Author
|
HOLD — merge with the gateway prod release. This documents behaviour that does not exist on Reviewed and green (markdownlint, link check, rendered build). Please merge it in the same wave as the lc_api-go production release, not before. |
maximelb
changed the title
Independent review. - The EML admonition was titled 'deliberately the only one that is [rate-limited]' and closed with 'No other Email Security route is rate-limited' — which this same PR makes untrue eighty lines below. It is now about the posture that IS unique to that route (failing closed), and it points at the read budget and says why that one fails OPEN instead. - 'Every other filter in the table above is a lookup' was wrong and contradicted by this same section twenty lines later: state, direction, user_reported, min_score, sender_root_domain, a multi-valued verdict and since/until are all residuals. The six that really are lookups are named. - messages.md offered a single verdict as a way to satisfy the requirement and then listed only the five index filters as exempt from the budget, leaving a reader to assume symmetry. It says now that a single verdict satisfies the rule and does not exempt the search, and why. - api-reference said q is 'subject to the read budget' unconditionally; a q riding an index filter is not counted at all. - 'a once-a-minute poll costs almost nothing' — the memo TTL is exactly 60 seconds and it is per pod, so that poll lands on the expiry most times. The claim that survives is the one the paragraph is actually for: it is not budgeted. Not that it is free. - 'roughly half' was 40%, and the three rate-limit headers are in SECONDS while the body's period is a duration string. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lcbill
previously approved these changes
Sep 8, 2026
This was referenced Sep 8, 2026
…d POST /hunts (#399) The read-budget section documented in documentation#389 covers the two GET reads, which are priced in database work. The rule backtest is priced in something else entirely: it re-reads every stored message in its window — fetch, decrypt, decompress, parse, evaluate — so it carries its own budget, its own rate_bucket and its own refill window. Says what a client actually needs: that mailsec_post_read is NOT mailsec_read and must not be retried on the other's advice, that 6 backtests per 10 minutes is the allowance across every credential in the organization, that it fails open like its sibling, and that no narrower request shape takes a backtest out of the budget the way a filtered search leaves the query budget. Also states why POST /hunts is counted while it still refuses: budgeting it on the day the engine lands would be a change of contract for clients that shipped against it first. Stacked on documentation#389 because it extends that PR's Read budgets section.
lcbill
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the behaviour landing in lc_api-go#944 (security review L-12), board ticket "[Pre-GA] No read-cost governor on the mailsec read surface".
Two reads on the Email Security surface recompute rather than serving a cached or seekable answer, and both are now bounded per organization:
GET /coverageover an explicit window, andGET /messagescarryingq. Neither was documented as costing anything, and the filter table's "Free text over the message's identifying fields" gave no hint thatqis matched row by row while every other filter in that table is a lookup.## Read budgetssection: which two reads, the 2,400/hour figure, the429body,rate_bucket: mailsec_readas what distinguishes it from the raw-download refusal on the same surface,Retry-After, and the two cheapest ways to get a refused read served. TheGET /coverageandGET /messagesrows now point at it.### Free text needs a window: whatqactually costs, exactly what counts as a bound (since, the five lookup filters, a single verdict) and what does not (untilalone, and the residuals), and the 512-character cap.coveragewith no window is memoized and is the right shape for a poller;--window-daysrecomputes and is budgeted.The
qexamples are HTTP rather than CLI on purpose:qhas no CLI flag today (a parity gap filed separately).Anchors checked by hand:
api-reference.md#read-budgets(new),#free-text-needs-a-window(new, depth 3 so withintoc_depth), and#get-messagesmsg_uuideml, which detections.md already links to.Public repo — not merging this myself. Ready for review whenever it suits.