Skip to content

feat(platform): hold chat titles, Improve with AI and later tool rounds - #4520

Merged
yannickmonney merged 74 commits into
mainfrom
feat/direct-call-holds
Oct 9, 2026
Merged

yannickmonney merged 74 commits into
mainfrom
feat/direct-call-holds

Conversation

@yannickmonney

@yannickmonney yannickmonney commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Reconciled with main (2026-10-09). Main's 0ee5fd9 added its own reservations for automation llm steps (automation-llm ops) and voice output (tts_audio_chunks holds, migration 0164). This PR now keeps those and drops its own llm-step lease, voice holds and migration 0157. What remains here: the direct-call lease (0156, direct-calls.ts) for chat titles and Improve with AI, a reply's later tool rounds, and the erasure pass for direct-call rows (it now runs after the model-request pass, as main's prerequisite order requires).

What

Work that was spending against budgets without being held or checked now is:

  • Improve with AI (Inbox) is checked against the writer's limits before it calls the model. At a limit it answers 429 BUDGET_EXCEEDED, and the composer shows Usage limit reached with the limit's sentence (en/de/fr). It used to book its tokens without ever being checked.
  • A chat's title is checked and held under the thread's member, with the API key that sent the message and the thread's project. A refused title is not an error: the chat is named from its first message, with no model call. A send refused for its sender's budget therefore no longer pays for a title.
  • An automation's llm step now holds the prompt plus its whole output cap while the call runs (feat(platform): hold automation llm steps to the run's budgets #4507 only checked before the call). It is refused when a limit has too little room for that.
  • Voice output: a chunk's admission takes the organization's budget-admission lock, counts every hold in flight, and holds its own estimate while it is made. It used to check booked usage only, hold nothing, and refuse one request early. Audio the provider was paid for is now booked even when its blob can't be stored or a newer attempt owns the chunk.
  • A chat reply's later tool rounds: each further round adds its worst case to the reply's hold before it runs. The opening held only the first round. A reply admitted at its start still runs to its end.
  • Model access: titles and Improve pick only models the member may use under the organization's model access rules, as chat turns do. When those rules close every served model, Improve now says so (IMPROVE_NO_MODEL_ACCESS, 403) instead of telling the writer to connect a provider.
  • Improve's limit toast is localized (en/de/fr), as chat's is, instead of showing the server's English sentence with a raw UTC timestamp.
  • A refused first message: the user row a refusal writes now carries the sending API key, so the naming call is held and booked under the key too. A message an input guardrail blocked is never sent to a model to name its thread; the thread is named from the message's own words.
  • Erasure: direct-call op rows of an erased person are deleted once booked, or pseudonymised while running, before the ledger pass, so a call that ends after the erasure books under the pseudonym (ERASE-R5).

How

  • governance/direct-calls.ts is a reservation for any call the platform makes straight to a provider, with no gateway key in between.
    • openDirectCall (reserveTurnBudget, new kind direct-call) measures the worst case whole against every cap on the subject, counting holds in flight. It records the hold on an op row stamped with subject, key and projects.
    • settleDirectCall books the cost under that stamp in the hold's place; releaseDirectCall closes a hold that spent nothing.
    • openTokenCall and settleTokenCall price a text model at the catalog rate, the same price the booking uses.
    • Nothing is held while no budget binds the organization, but the call is still recorded on its row (with no budget_cents, so no admission counts it). Settle always books from the row, so a replayed settle books nothing twice and erasure can reach a running call.
    • The sandbox watchdog releases a hold past its deadline and deletes settled rows after a day; migration 0156 adds partial indexes for both passes. A call that ends after its hold lapsed is still booked, exactly once.
  • automations/llm-metering.ts and core/automations/llm_call.ts: open, then settle or release, through three ctx-shim mutations that replace feat(platform): hold automation llm steps to the run's budgets #4507's check and record.
  • core/chat/generate_title.ts takes a TitleMeter that the chat.generate_title job builds (domains/chat/title-meter.ts). The job payload now carries the sending key: appendMessageRow gets it from the turn open, and from a failed REST send.
  • pickDirectModel(…, userId) filters candidates through resolveModelGovernanceInternal, in one read.
  • Voice output: reserved_cost_cents on tts_audio_chunks (migration 0157, with a partial index on pending rows). readInFlightReservations counts pending chunks within TTS_PENDING_STALE_MS, toward the requester and the thread's project.
  • Chat: TurnStore.holdNextRound (optional). The Postgres store raises the generation row's hold under the admission lock, only while a budget binds.
  • A refusal whose cap still has room, but too little for the work's worst case, now says so instead of "is used up" (budgetRefusalMessage).
  • Docs (en/de/fr): usage attribution (Improve row, holds, refused titles) and policies and limits. Spec GOV-R5 names the holds; the two Not yet entries this fixes are removed. Governance README lanes, rules 6 and 9, and guards. Manual box GOV-F56.

Verification

  • Unit tests for the direct-call module, the llm door and metering, the title generator, meter and job, Improve, voice output (lock order, the hold on the row, refusal from holds, booking unkept audio), the chat store and the turn (hold raise), and the reachability gate.
  • Platform server + PII suites: 1,099 files, 82,891 passed. The one failure was the reachability list's old handler names, fixed in the same commit as the handlers. Also: conversations UI tests (273), i18n guards, tsc, type-aware oxlint, knip, lint:manual, lint:links, docs tests (685), and the CLI transit test.
  • After the review fixes: the affected server suites (705 tests), the editor's jsdom tests (localized toasts), i18n guards, tsc, type-aware oxlint, knip, lint:manual, lint:links; real Postgres checkAutomations,checkProjectBudgets 33/33 (adds: with no budget bound, a call's row holds nothing and is booked once) and checkFiles,checkErasure 30/30 (the new erasure pass on the real schema).
  • Real Postgres before the review fixes (ITEST_LANES=checkAutomations,checkProjectBudgets, 32/32):
    • an llm call's hold is visible in both of its projects while it runs, and gone once it is booked;
    • a pending voice chunk holds 3¢ in the project until it is ready;
    • a direct call past its deadline held 5¢, was released by the sweep, and was still booked (1.5¢) when it ended;
    • a reply's next round raised its hold by 4¢;
    • a live automation run over HTTP books its llm call, and a capped run is refused before the provider.

Rolling deploy

Two migrations, both additive: indexes, and a nullable column the previous image never reads. During the roll, the previous image's calls hold nothing, as before. A chat.generate_title job with nameWithoutModel that an old worker picks up is named by a model, as before (its schema ignores the field).

Stacked on #4507.

DatePicker takes minDate/maxDate (days outside are disabled and read
"Not available" in the UI language) and forwards id, aria-labelledby
and aria-describedby to its trigger, which dropped them before.
The create dialog offered 7/30/90 days, a year or never, and the date a
key stopped working was shown nowhere. Expiration now adds Custom date,
a calendar bounded to tomorrow through a year from today (the api-key
plugin's window), and names the day the key expires under the field.
The keys table gets an Expires column: the date, Never, or Expired for a
key that ran out while the list was open.
Nothing outside the expiry helpers reads the 365-day ceiling, and Knip refuses an export no module imports.
Owners and Admins can now make an API key for another member, a team, a
project or the organization itself. A member's key acts as that member, in
this organization only; the member is notified, sees it and can revoke it.
A team's, a project's or the organization's key is not a person: it acts as
its own identity with the role chosen under Acts as (never above the
maker's), keeps working when its maker leaves, and works in its one
organization. A team's key sees what the team sees; a project's key reaches
its project alone, and any other REST route answers 403
API_KEY_SCOPE_FORBIDDEN.

Such a key spends under its own name: no personal, role or default limit
binds it, the organization's limits and the key's own rules do, and a
team's key counts toward and is held to its team's caps. Usage analytics
shows it as its own row, never an active user. A key ends with what it
belongs to: deleting the team or the project, removing the member or
deleting the organization revokes it, audited as the system.

The binding lives in app.api_key_owners (0154). The REST door reads it on
every keyed request, GET /me answers key.owner (contract 3.18.0), and the
api-key plugin's update and delete refuse a key an organization manages.
The keys table gains a Belongs to column.
API keys, policies and limits, usage attribution and the API reference say who a key made for someone else acts as, what it reaches and whose limits its spend counts toward, in English, German and French. The Create API key screenshot shows Belongs to, and a second one shows the organization's own key acting as a Developer.
A run a team's, project's or organization's key started through a REST comment books under the key's identity without naming the key, so the key's own cap never saw it. The key's bucket now counts the identity's bookings and holds too. The budget gate also reads a revoked binding: the work a key started before its revocation keeps spending as the key instead of being measured as a person with the default cap.
The capability dispatch behind POST /api/v1/mcp read the member table directly, so a team's or the organization's own key passed the door and was refused there. It now reads the acting member, and refuses a project's key, which reaches its project alone.
A review found ways a key made for someone else could reach further than
its owner meant:

- A project's key could read the organization's contacts, products,
  websites and inbox through the chat assistant, and hub document titles
  through its file listing. The chat tools now give it its project's files
  and tasks alone.
- A key made for a member authenticated as the member, so an image that
  does not read the binding (mid-roll or after a rollback) let it act as
  the member in every organization. Every bound key now authenticates as an
  identity of its own, and only the binding says whom it acts as.
- A member's key outlived its maker's authority: an admin could demote a
  peer, make a key for them and promote them back. The key now acts only
  while its maker is an Owner or Admin above the member, and the keys a
  member made end when they leave.
- An expired key's identity kept acting in runs it had started, a key
  identity could be made a member, a member's key carried the member's
  standing in other organizations into the browser-session gate, and a
  key could be bound to a target removed while it was minted. Each is
  closed: acting needs a live key row, the members door refuses key
  identities, the gate reads the bound membership alone, and the binding
  transaction checks its target again.
The team's delete confirmation now counts the API keys that belong to the
team, which stop working with it.

The key dialogs also say what they could not before: a picker for whom a
key belongs to names why Create stays off when nothing is picked, a search
that matches nobody says so instead of claiming there is no one to pick,
a key made for someone else is not "your" key on the success view, a key
of a deleted team or project reads as such, and a failed revocation is
reported once, with the refusal's own words.
Owners and Admins see and revoke the keys made here for others, never a member's own keys. A member's key stops when its maker no longer outranks the member; deleting a team or a project, or removing a member, revokes keys, while deleting the organization deletes them. A project's key reads none of the organization's contacts, products, websites or inbox through the chat assistant, and keys made for others have their own rate-limit budget. The team and member pages, usage analytics and the API reference say so, and the manual boxes and error-code rows match.
A budget rule can now cap one project: everything spent in it, whoever
spends it — its chats (titles, read-aloud answers and the assistant's
tool calls included), its agents' and automations' turns with their
images, and its own API keys at the model endpoints. A project's cap
binds work in the project on top of the spender's own caps, a run a
schedule started included, and nothing outside it.

Bookings that name a project land in the project's own period buckets,
app.project_usage (0155), beside the ledger row; holds in flight count
toward the project through their thread or the project their op stamped.
Project caps live in the budgets file's projectRules, so an image from
before them still parses the file and keeps every other cap. Retention
ages the buckets with the ledger. A REST 429 BUDGET_EXCEEDED names such
a cap as data.scope project (contract 3.19.0).
…chat

The budget editor offers a Project scope with a picker of the active
projects; a rule keeps naming its project after the project is archived,
and is saved into the file's projectRules. A chat send a project's cap
refuses says so — the project's limit, which Settings > Usage does not
list — instead of pointing the member at their own usage. The arena
door names the refusing cap as the send door does.
Policies and limits, usage attribution and API keys (en/de/fr) say what
a project's budget caps and how it combines with the other limits; the
governance suite gains GOV-F54 and the automation reference names the
specs that own the rest.
A project's cap warns no one, so the threshold the dialog offered for it limited nothing. The field is gone for the Project scope, and switching a rule to it drops its threshold.
A new capture of the Add budget rule dialog with a project chosen illustrates the project budget on Policies and limits (en/de/fr), and the rate-limit and editor guides list a project's cap among those a 429 BUDGET_EXCEEDED can name.
…hanges

Switching a rule's scope now drops the previous scope's target, which used to save a project cap naming a user or a role and matching nothing. The table keeps the order the saved file reads back, so a row's Edit and Delete keep naming the same rule when the save lands. A rule's project shows the Archived badge when it is archived and reads Deleted project, with a hint in the dialog, once it is gone.
A parked send the worker fires, an arena column refused after the pair was admitted, and a REST-queued turn now store whose cap stopped them beside the budget_exceeded code, so their transcript rows read as the project's limit rather than pointing at Settings > Usage, which never lists it. The toast no longer promises that chats outside the project work: another cap may be spent too.
Runs a schedule started are held to a project's cap when they run in one; Settings > Usage does not list a project's limit; GOV-F54 now expects the toast the app shows and the reply plus title as the project's two requests; the REST send and BUDGET_EXCEEDED scope descriptions read plainly.
…on image holds

Moving a conversation to another project now moves its hidden edit and regenerate branches and its arena column too: the app runs later turns on those rows, so a turn on a branch kept spending in, and being capped by, the project the conversation had left — a way past a project's cap. An image's hold on an agent turn whose op its reservation did not stamp (a subscription turn's) now records the turn's project, so concurrent images in a project count against its cap together.
A project's budget counts the agent steps of automations run in it, not their model steps, which no limit admits or books yet (spec Not yet). Chat replies and agent turns hold against it while they run; voice output is checked but holds nothing; a refused chat message says the project's limit was reached.
…oject cap

A run a schedule or an event starts for an automation installed in more than one project runs in none of them today, so no project's budget binds it; the governance spec records which project such a run should spend in as an open question.
…e lineage test

The toast and the transcript chose a reached cap's hint in two ternaries, which the i18n usage guard could not trace to the chat catalog; budgetExceededI18nKey in lib/shared/chat-errors.ts now owns the choice. The refiling test's fake row takes a branch's root id.
The project-budget lane files a conversation with a hidden branch out of its project and back over the real chat door, and checks the branch follows both moves on the real schema.
An automation run belongs to the project it names; one that names none belongs to every project its automation is bound to — the set its language context, skills and session reach already act in — and an automation bound to none spends in no project. Such a run now counts toward each of its projects and must fit each one's cap, as a member's spend counts toward each of their teams.

A budget subject, a booking and an op's hold carry a set of projects (projectIds); sandbox_session_ops stamps project_ids (0155, not yet on main); holds count toward every project they name, and each project's buckets are booked once per call.
An automation counts toward the project it runs in; a run that names none counts toward every project its automation is installed in and must fit each budget; an automation installed in none counts toward the organization alone (en/de/fr).
Agent turns and their images each built the budget subject for a run's
attributed work in their own copy: a person through loadBudgetSubject,
or an impersonal subject for a trigger's run or no run at all.
loadAttributedBudgetSubject (governance/attributed-subject.ts) is now
the one builder both use.

An automation run's billing subject is mapped in one place too:
resolveAutomationRunAttribution reads it by the run's id, and the
workflow-agent op resolves its session's run and asks the same
resolver. No behaviour changes.
An automation's llm step was admitted against no budget and booked to
no ledger. Each live call is now its run's spend, under the subject its
agent steps use: the starter, or __automation__ for a trigger's run,
under the automation's name, with the run's key and its projects (the
run's own, else every project its automation is bound to).

Before the call, checkLlmStepBudget measures that subject against every
cap that binds it, counting the work in flight; a reached cap fails the
step with budget_exceeded, naming the limit. After it,
recordLlmStepUsage books the reported tokens through the priced ledger
writer, an empty reply included; a failed booking is logged and keeps
the reply. The call holds nothing while it runs.

Spec GOV-R14 counts the llm steps; the Not yet entry keeps the missing
hold. Manual box GOV-F55; the project-budget lane proves the check and
the booking in two projects on real Postgres.
An automation's llm step is now checked against the budget limits
before each model call and counted after it. The usage-attribution and
policies pages list the llm steps beside the agent steps a project's
limit measures, and say that a refused call fails the run with
budget_exceeded and that the step holds nothing while it runs. The
automation concepts page and the API reference's failure table name the
same refusal. English, German and French.
checkAutomations now asserts, through the HTTP start door, the job
worker and the stepper, that a live run's llm step books the
provider's 12+6 tokens once under the run's starter and the
automation's name, and that with an organization request cap reached
the next run fails with budget_exceeded before its provider is called,
its detail naming the cap. The lane restores the org's budgets file
afterwards. GOV-F55 quotes the detail as a run shows it: the node id,
the refusal and its hint.
# Conflicts:
#	services/platform/tests/manual/readme.md
# Conflicts:
#	docs/de/platform/admin/api-keys.md
#	docs/en/platform/admin/api-keys.md
#	docs/fr/platform/admin/api-keys.md
#	services/platform/app/features/settings/api-keys/components/api-key-create-dialog.test.tsx
#	services/platform/app/features/settings/api-keys/components/api-key-create-dialog.tsx
#	services/platform/app/features/settings/api-keys/hooks/use-api-keys-table-config.test.tsx
#	services/platform/app/features/settings/api-keys/hooks/use-api-keys-table-config.tsx
#	services/platform/app/features/settings/api-keys/lib/expiry.test.ts
#	services/platform/messages/de/settings.yml
#	services/platform/messages/en/settings.yml
#	services/platform/messages/fr/settings.yml
#	services/platform/tests/manual/readme.md
#	services/platform/tests/manual/suites/settings.md
# Conflicts:
#	services/platform/backend/domains/chat/deferred-sends.ts
Base automatically changed from feat/automation-llm-budgets to main October 9, 2026 00:30
# Conflicts:
#	docs/de/platform/admin/governance/policies-and-limits.md
#	docs/de/platform/admin/governance/usage-attribution.md
#	docs/en/platform/admin/governance/policies-and-limits.md
#	docs/en/platform/admin/governance/usage-attribution.md
#	docs/fr/platform/admin/governance/policies-and-limits.md
#	docs/fr/platform/admin/governance/usage-attribution.md
#	services/platform/backend/core/automations/llm_call.test.ts
#	services/platform/backend/core/automations/llm_call.ts
#	services/platform/backend/core/automations/stepper.ts
#	services/platform/backend/domains/automations/llm-metering.test.ts
#	services/platform/backend/domains/automations/llm-metering.ts
#	services/platform/backend/domains/automations/shim.test.ts
#	services/platform/backend/domains/automations/shim.ts
#	services/platform/backend/domains/governance/README.md
#	services/platform/backend/domains/governance/budget-reservations.ts
#	services/platform/backend/domains/governance/project-budget.integration.ts
#	services/platform/backend/domains/governance/spec.md
#	services/platform/backend/domains/sandbox/turn-budget.ts
#	services/platform/backend/domains/tts/service.project-budget.test.ts
#	services/platform/backend/domains/tts/service.ts
#	services/platform/backend/jobs/task-list.generate-title.test.ts
#	services/platform/backend/jobs/task-list.ts
#	services/platform/tests/manual/suites/governance.md
@yannickmonney
yannickmonney enabled auto-merge (squash) October 9, 2026 01:46
@yannickmonney
yannickmonney disabled auto-merge October 9, 2026 02:55
# Conflicts:
#	docs/de/platform/admin/governance/policies-and-limits.md
#	docs/de/platform/admin/governance/usage-attribution.md
#	docs/en/platform/admin/governance/policies-and-limits.md
#	docs/en/platform/admin/governance/usage-attribution.md
#	docs/fr/platform/admin/governance/policies-and-limits.md
#	docs/fr/platform/admin/governance/usage-attribution.md
#	services/platform/backend/core/automations/llm_call.test.ts
#	services/platform/backend/core/automations/llm_call.ts
#	services/platform/backend/core/lib/handler_names.ts
#	services/platform/backend/domains/automations/llm-metering.test.ts
#	services/platform/backend/domains/automations/llm-metering.ts
#	services/platform/backend/domains/automations/shim.test.ts
#	services/platform/backend/domains/automations/shim.ts
#	services/platform/backend/domains/erasure/service.lifecycle.test.ts
#	services/platform/backend/domains/erasure/spec.md
#	services/platform/backend/domains/governance/README.md
#	services/platform/backend/domains/governance/budget-reservations.ts
#	services/platform/backend/domains/governance/project-budget.integration.ts
#	services/platform/backend/domains/governance/spec.md
#	services/platform/backend/domains/sandbox/turn-budget.ts
#	services/platform/backend/domains/tts/service.project-budget.test.ts
#	services/platform/backend/domains/tts/service.ts
@yannickmonney yannickmonney changed the title feat(platform): hold every direct model call, voice chunk and tool round feat(platform): hold chat titles, Improve with AI and later tool rounds Oct 9, 2026
@yannickmonney
yannickmonney merged commit 8bf6bd7 into main Oct 9, 2026
1 of 2 checks passed
@yannickmonney
yannickmonney deleted the feat/direct-call-holds branch October 9, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant