Skip to content

Fixes from running a real multi-agent app through AgentFox - #77

Merged
architsharm merged 1 commit into
mainfrom
claude/live-agent-findings
Oct 9, 2026
Merged

architsharm merged 1 commit into
mainfrom
claude/live-agent-findings

Conversation

@architsharm

Copy link
Copy Markdown
Owner

Ran openai/openai-cs-agents-demo (5 agents with handoffs, 11 tools incl. cancel/compensation, its own guardrail agents) live on GPT models through the gateway, governed by the OpenAI Agents adapter, and configured it from the dashboard like a customer.

Bugs it found (fixed here)

  • Proxy dropped request fields: tools, tool_choice, response_format, parallel_tool_calls were never forwarded, and temperature was always sent. Tool-using agents could not work through the gateway. Now passed through per protocol; streaming forwards tool-call deltas.
  • Row lock held across the model call: on Postgres (5 s lock_timeout) concurrent requests for the same agent failed when the model took longer than 5 s. Preflight now commits before the call; the non-streaming route no longer blocks the event loop.
  • System prompts scanned as user input: at High sensitivity every request was blocked as injection because the app's guardrail prompt talks about jailbreaks. App-authored roles are skipped unless marked untrusted via X-AgentFox-Trust.
  • Provider outage stopped guard checks for teams calling their own model.
  • "Allow" didn't allow: granted but undeclared tools were held by tool.not_declared. Granting now registers the tool with a guessed risk, editable on the Access tab.
  • Approval spam / no continuation: identical held calls reuse the pending approval; the Agents adapter redeems an approved call on retry. Verified live: held → approved on the Approvals page → agent completed the cancellation once.

Dashboard

Editable tool risk; approval limits offer the agent's tools as answers; approval cards show the id; Protect wizard start levels.

Tests

Full backend 3462 passed; dashboard 67; ruff/format/import-linter clean; wheels rebuilt.

🤖 Generated with Claude Code

Ran openai/openai-cs-agents-demo (5 agents, handoffs, 11 tools, its own
guardrail agents) on GPT models through the gateway with the OpenAI Agents
adapter, then configured it from the dashboard as a customer would.

Gateway
- The OpenAI/Anthropic proxy rebuilt requests and dropped tools,
  tool_choice, response_format and parallel_tool_calls, and always sent
  temperature: client fields now pass through to a provider speaking the
  same protocol. Streaming forwards and records tool-call deltas.
- The proxy held the request transaction (agent row lock) across the model
  call; preflight now commits first. The non-streaming route ran the model
  call on the event loop; it now runs in a worker thread.
- System, developer and earlier assistant messages were scanned as user
  input, so a guardrail prompt about jailbreaks blocked every request at
  High sensitivity. App-authored roles are skipped unless marked untrusted.
- A model-provider outage no longer stops /v1/guard checks.
- Granting a tool registers it (risk guessed from its name, editable), so
  Allow no longer leaves every call held by tool.not_declared.
- An identical held call reuses its pending approval; the Agents adapter
  redeems an approved call when the agent retries it.

Dashboard
- Risk is editable per tool on the agent's Access tab.
- Approval limits offer the agent's own tools as one-click answers.
- Approval cards show the approval id the agent quoted.
- Protect wizard starts protections at their inherited level and leaves
  model-backed ones off when the model is not installed.

Co-Authored-By: Claude <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
guardrails-api Building Building Preview Oct 9, 2026 2:21am UTC
guardrails-dashboard Building Building Preview Oct 9, 2026 2:21am UTC
guardrails-redteam-lang Building Building Preview Oct 9, 2026 2:21am UTC

This branch was successfully deployed

3 active deployments
Preview – guardrails-redteam-lang — ba491730 Deployed Oct 9, 2026 by vercel[bot]
Preview – guardrails-api — ba491730 Deployed Oct 9, 2026 by vercel[bot]
Preview – guardrails-dashboard — ba491730 Deployed Oct 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant