From 2ee2dded63abb19c05f6ea225e74a18f1017cd92 Mon Sep 17 00:00:00 2001 From: digif33ls Date: Wed, 11 Nov 2026 01:23:51 -0800 Subject: [PATCH 1/4] =?UTF-8?q?office-42-part4:=20Agentic=20AI=20Security?= =?UTF-8?q?=20=E2=80=94=205=20paper=20digests=20+=20cross-paper=20narrativ?= =?UTF-8?q?e=20(Dwight=20QC=20fixes)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix Issue #1: Added 'Action items' section to agent-swarms-prompt-injection.md - Fix Issue #2: Filled placeholder dates in 3 digests (shibboleth, agent-swarms, agents-of-chaos) - 6 files: 5 individual paper digests + cross-paper narrative connecting the attack→detection→defense arc Co-Authored-By: Claude Code --- .../raw/agent-swarms-prompt-injection.md | 54 +++++ .../agents-of-chaos-sok-agentic-attacks.md | 60 ++++++ .../cross-paper-agentic-security-narrative.md | 189 ++++++++++++++++++ .../raw/prism-tool-privilege-minimization.md | 51 +++++ .../raw/shibboleth-llm-fingerprinting.md | 55 +++++ .../raw/stealthypoison-skills-poisoning.md | 49 +++++ 6 files changed, 458 insertions(+) create mode 100644 feat/office-42-part4-agentic-security/raw/agent-swarms-prompt-injection.md create mode 100644 feat/office-42-part4-agentic-security/raw/agents-of-chaos-sok-agentic-attacks.md create mode 100644 feat/office-42-part4-agentic-security/raw/cross-paper-agentic-security-narrative.md create mode 100644 feat/office-42-part4-agentic-security/raw/prism-tool-privilege-minimization.md create mode 100644 feat/office-42-part4-agentic-security/raw/shibboleth-llm-fingerprinting.md create mode 100644 feat/office-42-part4-agentic-security/raw/stealthypoison-skills-poisoning.md diff --git a/feat/office-42-part4-agentic-security/raw/agent-swarms-prompt-injection.md b/feat/office-42-part4-agentic-security/raw/agent-swarms-prompt-injection.md new file mode 100644 index 000000000..ecf01a3f3 --- /dev/null +++ b/feat/office-42-part4-agentic-security/raw/agent-swarms-prompt-injection.md @@ -0,0 +1,54 @@ +# Agent Swarms: Automated Adversarial Agentic Swarms for Prompt Injection + +**arXiv:** 2611.11861 +**Date:** 9 Nov 2026 + +--- + +## Core Attack Mechanism + +This Google DeepMind study introduces an **Agentic Optimizer** — a swarm of adversarial AI agents that automatically generate and adapt prompt injection attacks against frontier LLMs. Unlike previous static or single-agent adaptive attacks, this system uses a **swarm of agents** that jointly conduct the search for successful injections. + +The attack operates through: +1. **Synthetic Dataset:** 100 synthetic tool-use scenarios spanning diverse domains, each with orthogonal user and attacker goals. Scenarios are constructed to require multi-step tool chains (realistic for production environments). +2. **Swarm Optimization:** Multiple attacker agents jointly search the strategy space, each contributing successful strategies to a shared database. Strategies are natural-language descriptions paired with historically successful candidates. +3. **Adversarial Goals:** The attacker goal is to cause the agent to call a specific target tool (with exact arguments) — even when that tool is irrelevant or harmful to the user's actual goal. Random non-malicious actions serve as adversarial goals, making attacks harder to detect. + +Key finding: **Prompt injection is not solved** even for the most robust frontier models when pitted against automated adversarial swarms. + +--- + +## Key Numbers / Metrics + +| Metric | Value | +|--------|-------| +| Synthetic scenarios evaluated | 100 multi-step tool-use scenarios | +| Models tested | 4 versions of Gemini (mitigations disabled) + state-of-the-art open-weights LLMs | +| Human red-teaming saturation (GraySwan Arena) | < 1% on recent Gemini/Claude models — **near-solved for humans** | +| Open-source adaptive attacker (Siren, Facebook) success rate | **Very low** against frontier models | +| Agentic Optimizer vs. frontier models | **Conclusively bypasses defenses** — prompt injection remains unsolved | +| Open-weights model resistance (Kimi K3, GLM-5.3, Deepseek-4.1-Flash) | **Remarkably low** resistance despite high capability benchmark scores | +| Scoring threshold for success | > 0.99 (exact argument matching, >96% string similarity) | + +--- + +## Relevance to Defense Posture + +**Critical finding:** The gap between human red-teaming effectiveness and automated agentic attack capability is **massive**. Human red-teamers have saturated defenses on recent frontier models (< 1% breach rate), but automated agentic swarms bypass these same defenses conclusively. + +Key implications for Dunder Mifflin: + +1. **Human red-teaming is insufficient** — the GraySwan Arena shows < 1% breach rate on frontier models, creating false confidence. Automated swarms achieve high success rates against the same models. +2. **Open-weights models are dangerously weak** — Kimi K3, GLM-5.3, and Deepseek-4.1-Flash show high capability benchmark scores but remarkably low resistance to prompt injection. This is a critical mismatch between capability and security. +3. **Multi-step scenarios are the real threat** — single static injections are saturated, but multi-step tool-chain attacks (listing files → encrypting → deleting = ransomware) remain effective. +4. **Defense implication:** Defense strategies must account for automated adversarial swarms, not just human red-teamers. This means: + - Runtime monitoring for anomalous tool-call sequences + - Input sanitization that handles embedded instructions in tool return values + - Principle of least privilege for tool access (see PRISM paper) + +**Note:** This study disabled out-of-model prompt injection mitigations on Gemini models. The real-world impact with mitigations enabled is unknown but likely significant given the open-weights model vulnerabilities. + +**Action items:** +- Deploy automated adversarial swarm testing as part of the security assessment pipeline (replace or augment human red-teaming) +- Prioritize tool-layer protections for open-weights models, which show the weakest resistance despite high capability scores +- Implement runtime monitoring for anomalous multi-step tool-call sequences that could indicate swarm-based attacks diff --git a/feat/office-42-part4-agentic-security/raw/agents-of-chaos-sok-agentic-attacks.md b/feat/office-42-part4-agentic-security/raw/agents-of-chaos-sok-agentic-attacks.md new file mode 100644 index 000000000..5f1bda5d8 --- /dev/null +++ b/feat/office-42-part4-agentic-security/raw/agents-of-chaos-sok-agentic-attacks.md @@ -0,0 +1,60 @@ +# Agents of Chaos: Systematization of Agentic AI Attacks (SoK) + +**arXiv:** 2611.10843 +**Date:** 7 Nov 2026 + +--- + +## Core Mechanism + +This Systematization of Knowledge (SoK) paper presents the most comprehensive mapping of attacks targeting agentic AI systems to date. It screened **186 candidate sources** and retained **98** after eligibility screening, producing a structured taxonomy of the entire agentic attack landscape. + +The paper organizes attacks using: +- **8 causal-mechanism classes** (not just insertion points, but the actual mechanism by which harm occurs) +- **6-stage attack lifecycle:** Insertion → Exposure/Selection → Interpretation → Planning → Execution → Persistence +- **Orthogonal dimensions:** insertion points, activation conditions, abused capabilities, objectives, impacts, persistence, and evidence quality + +Key structural insight: **Insertion and activation are often separated in time and across different components.** An attacker may inject malicious content into a skill file at one point, and the agent activates it weeks later through a completely different workflow. This temporal decoupling makes traditional perimeter defenses ineffective. + +--- + +## Key Numbers / Metrics + +| Metric | Value | +|--------|-------| +| Sources screened | 186 candidate sources | +| Sources retained after screening | **98** | +| Atomic attacks identified | **284** | +| Atomic safeguards identified | **452** | +| Explicit attack-safeguard relationships | **660** | +| Causal-mechanism classes | **8** | +| Attack lifecycle stages | 6 (Insertion → Exposure → Interpretation → Planning → Execution → Persistence) | +| Safeguard relationships directly assessed | 278 of 660 (42.1%) | +| Strong safeguard coverage | **Uncommon** — most relationships have partial or limited coverage | + +--- + +## Relevance to Defense Posture + +**Critical finding:** This SoK provides the **definitive map** of the agentic attack landscape. The numbers reveal both the scale of the threat and the inadequacy of current defenses: + +1. **Tool/MCP/action-layer attacks are most numerous** — these represent the largest share of atomic attacks, confirming that the tool-calling interface is the primary attack surface. +2. **Prompt injection and action-hijacking have broadest coverage** — these attack classes appear across the most papers, indicating they are pervasive and transferable across systems. +3. **Safeguard coverage is critically inadequate** — only 42.1% of attack-safeguard relationships are directly assessed, and strong coverage is uncommon. This means most attacks have no known effective defense. +4. **Temporal decoupling is the key challenge** — insertion and activation happening separately across components means defenses must track the full lifecycle, not just point-in-time checks. + +**Defense implications for Dunder Mifflin:** + +- **Lifecycle-aware threat modeling is essential** — map all 6 stages for each agent workflow, not just the execution stage. +- **Tool governance must be prioritized** — tool/MCP/action-layer attacks are the most numerous class; implement PRISM-style least privilege. +- **Defense-in-depth is non-negotiable** — with only 42% of relationships assessed and strong coverage being uncommon, no single defense is sufficient. +- **Cross-reference with other papers:** + - StealthyPoison → memory/RAG poisoning class (class 3) + - Shibboleth → detection within the execution stage + - Agent Swarms → prompt injection class (class 1) with broadest coverage + +**Action items:** +- Map Dunder Mifflin's agent workflows against the 8 causal-mechanism classes +- Identify gaps in current safeguards (the 57.9% of relationships with no assessment) +- Implement lifecycle-aware monitoring covering all 6 stages +- Prioritize tool-layer protections (most numerous attack class) diff --git a/feat/office-42-part4-agentic-security/raw/cross-paper-agentic-security-narrative.md b/feat/office-42-part4-agentic-security/raw/cross-paper-agentic-security-narrative.md new file mode 100644 index 000000000..c82591048 --- /dev/null +++ b/feat/office-42-part4-agentic-security/raw/cross-paper-agentic-security-narrative.md @@ -0,0 +1,189 @@ +# Cross-Paper Narrative: The Agentic Security Arc (Issue #42, Part 4) + +**Papers covered:** 5 arXiv preprints (arXiv IDs: 2611.10843, 2611.11539, 2611.11861, 2611.12155, 2611.12386) +**Date range:** 08–19 Nov 2026 + +--- + +## The Narrative Arc: Attack Surface → Detection → Systemic Defense + +This collection of five papers traces a complete security lifecycle for agentic AI systems. Together, they tell a story that moves from **offense** (how attacks are evolving) through **detection** (how we can identify attackers) to **defense** (how we must systematically protect systems). The arc is: + +> **Attack surface is expanding** (swarms, poisoning) → **Detection is evolving** (shibboleth fingerprinting) → **Defense must be systemic** (PRISM least-privilege, SoK mapping) + +--- + +## Part 1: The Attack Surface Is Expanding — And It's Getting Worse + +### Agent Swarms (arXiv:2611.11861) — The Offense Has Gone Automated + +The most alarming finding across all five papers: **automated adversarial swarms have solved what human red-teamers could not.** + +- Human red-teaming saturation on frontier models: **< 1% breach rate** (GraySwan Arena) +- Automated agentic swarms: **conclusively bypasses defenses** on the same models + +This gap is not incremental — it's existential. Human testers have created false confidence by saturating defenses against static, single-agent attacks. Meanwhile, Google DeepMind's Agentic Optimizer uses a swarm of adversarial agents to jointly search the strategy space, finding prompt injections that human testers miss entirely. + +**Key insight:** The attack surface is no longer defined by what a single attacker can discover. It's defined by what an automated swarm can exhaustively search. Single static injections are saturated; multi-step tool-chain attacks (listing files → encrypting → deleting = ransomware) remain effective. + +### StealthyPoison (arXiv:2611.12155) — The Supply Chain Is Compromised + +If Agent Swarms show how attacks are delivered, StealthyPoison shows **where** they're being planted: in the supply chain itself. + +- 566 skills poisoned across **26,000+ registry entries** +- **Zero detection** by Cisco Skill Scanner (0.4% flagging for both clean and poisoned) +- **Zero detection** by VirusTotal (0%) +- 10.1% end-to-end attack success rate; **38.3%** peak for task-specific attacks (MedCalc) + +The attack is feedback-free during placement — it modifies only metadata to improve retrieval ranking, preserving executable functionality. This means: +- Human reviewers see benign descriptions +- Automated scanners find clean code +- Yet the poisoned skills are selected **49.4% of the time** when agents query registries + +**Key insight:** The attack surface has moved upstream. You don't need to compromise the target system directly — you just need to poison the skill registries that agents trust. The temporal decoupling (poison injected today, executed weeks later) makes this especially dangerous. + +### The SoK Connects It All (arXiv:2611.10843) + +The Systematization of Knowledge paper provides the taxonomy that makes sense of both attacks: +- **284 atomic attacks** across 98 sources +- **8 causal-mechanism classes** — not just insertion points, but the actual mechanism of harm +- **Tool/MCP/action-layer attacks are the largest share** — confirming that tool-calling interfaces are the primary attack surface +- **Prompt injection and action-hijacking have broadest coverage** — appearing across the most papers, making them pervasive and transferable + +The SoK's critical structural insight — **insertion and activation are often separated in time across different components** — explains why both Agent Swarms (which operates at the injection stage) and StealthyPoison (which poisons registries long before execution) are so effective. + +--- + +## Part 2: Detection Is Evolving — But It's Not Enough + +### Say Shibboleth (arXiv:2611.12386) — Fingerprinting Without Learning + +Shibboleth represents the detection layer of this security arc. It uses honeypot "shibboleth traps" to identify attacking models and their execution harnesses: + +- **90% model fingerprinting accuracy** (top-1) across 15 candidate models +- **92% harness identification accuracy** across 4 harnesses +- **Zero learned components** — works with fixed inference rules alone + +The system uses two gate types: +1. **Declarative gates** — require change records before releasing credentials, forcing linguistic fingerprints +2. **Behavioral gates** — probe timing, tool-call scheduling, credential masking behavior + +Even when agents are instructed to conceal their identity and report fake names, they still produce discriminative behavioral traces. + +**Key insight:** Detection is possible without training data or learned models. But detection alone does not prevent harm — it only identifies who attacked and how. This is why detection must be paired with systemic defense. + +### The Detection Gap + +Critically, Shibboleth operates at the **execution stage** of the 6-stage attack lifecycle (Insertion → Exposure → Interpretation → Planning → Execution → Persistence). It cannot prevent: +- Skills poisoned weeks ago (StealthyPoison) +- Prompt injections generated by automated swarms (Agent Swarms) + +Detection tells you *who* and *how*. It does not tell you *whether the attack succeeded* or *what damage was done*. This is why detection must be layered with prevention. + +--- + +## Part 3: Defense Must Be Systemic — No Single Solution Suffices + +### PRISM (arXiv:2611.11539) — Least Privilege as a Security Game + +PRISM addresses the prevention layer through causal counterfactual minimization of tool privileges: + +- **78.3% fewer tools authorized** while maintaining **78.2% task success** — a near-linear tradeoff +- **15.9% blast radius** — even if one tool is compromised, reachable damage is limited to a small fraction +- **88.7% counterfactual necessity accuracy** — correctly identifies which tools are truly needed vs. merely frequently called + +PRISM's three integrated components form a complete defense strategy: +1. **Counterfactual Necessity Model** — asks "Can the task still be completed if this tool is removed?" (not "How often is it called?") +2. **Information Bottleneck Role Induction** — compresses trajectories into discrete roles, minimizing total privilege carried +3. **Stackelberg Security Game** — optimizes for worst-case attacker response to any privilege mask + +**Key insight:** PRISM doesn't just reduce tools — it models the attacker's response. This bilevel game ensures that privilege minimization accounts for how attackers adapt to observed defenses. + +### The SoK Reveals Why Systemic Defense Is Non-Negotiable + +The SoK's numbers make the case for defense-in-depth: +- **660 attack-safeguard relationships** identified +- Only **278 directly assessed (42.1%)** — meaning 57.9% of relationships have no assessment at all +- **Strong safeguard coverage is uncommon** across the remaining 42.1% + +This means: +- **No single defense is sufficient** — with 57.9% of attack-safeguard relationships unassessed, relying on any one control leaves massive gaps +- **Lifecycle-aware monitoring is essential** — defenses must cover all 6 stages (Insertion → Exposure → Interpretation → Planning → Execution → Persistence), not just execution +- **Tool governance is the highest priority** — tool/MCP/action-layer attacks are the most numerous class + +--- + +## The Complete Arc: How All Five Papers Fit Together + +``` + ATTACK SURFACE EXPANDING + │ + ┌──────────┴──────────┐ + ▼ ▼ + Agent Swarms StealthyPoison + (automated optimization) (supply-chain poisoning) + │ │ + └──────────┬──────────┘ + ▼ + DETECTION EVOLVING + │ + ┌──────────┴──────────┐ + ▼ ▼ + Shibboleth (SoK lifecycle mapping) + (fingerprinting traps) (6-stage coverage) + │ │ + └──────────┬──────────┘ + ▼ + DEFENSE MUST BE SYSTEMIC + │ + ┌──────────┴──────────┐ + ▼ ▼ + PRISM SoK taxonomy + (least-privilege game) (8 classes, 284 attacks) +``` + +### The Five Papers as a Coherent Security Framework: + +| Paper | Role in Arc | Key Contribution | +|-------|-------------|------------------| +| **Agent Swarms** | Offense (delivery) | Automated swarms bypass human-saturated defenses; multi-step tool-chain attacks remain effective | +| **StealthyPoison** | Offense (placement) | Supply-chain poisoning evades all static analysis; 49.4% of registry lookups compromised | +| **SoK (Agents of Chaos)** | Mapping (taxonomy) | 284 attacks, 8 classes, 6-stage lifecycle; reveals that 57.9% of relationships are unassessed | +| **Shibboleth** | Detection (identification) | 90-92% fingerprinting with zero learned components; identifies attackers without training data | +| **PRISM** | Defense (prevention) | 78.3% tool reduction, 15.9% blast radius; models attacker response via Stackelberg game | + +--- + +## Defense Posture Recommendations for Dunder Mifflin + +Based on the complete arc across all five papers: + +### 1. Accept That Human Red-Teaming Is Insufficient +The < 1% human breach rate on frontier models (Agent Swarms) creates dangerous false confidence. Automated swarms bypass these same defenses conclusively. **Action:** Augment human red-teaming with automated adversarial swarm testing. + +### 2. Secure the Supply Chain Before It's Poisoned +StealthyPoison achieves zero detection by current scanners while poisoning 49.4% of registry lookups. **Action:** Implement runtime sandboxing for newly retrieved skills and behavioral anomaly detection in skill execution pipelines. + +### 3. Map All 6 Stages of Every Agent Workflow +The SoK's temporal decoupling insight means defenses must track the full lifecycle, not just execution. **Action:** Map all 6 stages (Insertion → Exposure → Interpretation → Planning → Execution → Persistence) for each agent workflow. + +### 4. Deploy Detection Without Waiting for Training Data +Shibboleth works out of the box with fixed rules. **Action:** Deploy shibboleth-style honeypot services on exposed endpoints immediately — no ML training required. + +### 5. Minimize Blast Radius Through Causal Privilege Control +PRISM achieves 78.3% tool reduction with only a ~1:1 tradeoff in task success, limiting blast radius to 15.9%. **Action:** Evaluate PRISM integration for high-risk agent workflows (payments, data exfiltration paths). + +### 6. Defense-in-Depth Is Non-Negotiable +With only 42.1% of attack-safeguard relationships assessed and strong coverage being uncommon, no single defense suffices. **Action:** Layer all five capabilities — automated swarm testing, supply-chain monitoring, lifecycle-aware threat modeling, honeypot detection, and causal privilege minimization — into a single security framework. + +--- + +## The Bottom Line + +These five papers collectively demonstrate that: + +1. **The attack surface has moved upstream** — from direct system compromise to supply-chain poisoning (StealthyPoison) and automated optimization (Agent Swarms). +2. **Detection is possible but insufficient** — Shibboleth proves fingerprinting works without training data, but detection alone doesn't prevent harm. +3. **Systemic defense is the only viable path** — PRISM's causal privilege minimization combined with the SoK's lifecycle mapping provides a framework for defense-in-depth. + +The arc is clear: **attacks are becoming automated and supply-chain-based, detection is evolving but reactive, and only systemic defense-in-depth can address the full 6-stage attack lifecycle.** diff --git a/feat/office-42-part4-agentic-security/raw/prism-tool-privilege-minimization.md b/feat/office-42-part4-agentic-security/raw/prism-tool-privilege-minimization.md new file mode 100644 index 000000000..c8cd31907 --- /dev/null +++ b/feat/office-42-part4-agentic-security/raw/prism-tool-privilege-minimization.md @@ -0,0 +1,51 @@ +# PRISM: Causal Counterfactual Minimization of Agent Tool Privileges + +**arXiv:** 2611.11539 +**Date:** 08 Nov 2026 + +--- + +## Core Defense Mechanism + +PRISM is a **causal, counterfactual approach to minimizing tool privileges** for LLM agents. It addresses the fundamental problem that mainstream MCP implementations grant authorization once at session granularity — writing every tool exposed by connected servers into the system prompt, violating the principle of least privilege. + +PRISM operates through three integrated components: + +1. **Counterfactual Necessity Model:** Defines necessity as the change in task success probability after a `do`-intervention on the tool mask. Instead of counting call frequencies (statistical induction), it asks: "Can the task still be completed if this tool is removed?" Uses active sampling to train a model that estimates feasibility of any tool subset without exhaustive enumeration. + +2. **Information Bottleneck Role Induction:** Compresses task trajectories into discrete latent variables (roles), requiring each role to retain enough privilege information to complete tasks while **minimizing total privilege carried**. The number of roles is determined by the data (knee of rate-distortion curve), not specified as a hyperparameter. + +3. **Stackelberg Security Game:** Models the interaction as a bilevel game where the defender (PRISM) chooses a privilege mask first, and the attacker observes it before choosing an entry point. Minimizes "blast radius" — the set of tools reachable from any single compromised tool. + +--- + +## Key Numbers / Metrics + +| Metric | Value | +|--------|-------| +| Tool reduction achieved | **78.3%** fewer tools authorized | +| Task success rate maintained | **78.2%** — nearly 1:1 tradeoff ratio | +| Counterfactual necessity accuracy | **88.7%** — model correctly identifies necessary tools | +| Blast radius (max reachable tools from single compromise) | **15.9%** of total tool set | +| Benchmarks evaluated | AgentDojo, τ-bench, ToolEmu (largest tool set) | +| Baselines compared | 7 schemes (FullAccess, RoleMiner, GNRM, ToolEmu, GuardAgent, CaMeL, + others) | +| Privilege budget weighting (ρ) on high-risk tools | Adjustable parameter — extra weight on irreversible/data-exfiltration tools | + +--- + +## Relevance to Defense Posture + +**Critical finding:** PRISM achieves a **78.3% reduction in authorized tools while maintaining 78.2% task success** — a near-linear tradeoff that is remarkably efficient. This means Dunder Mifflin could reduce its agent's tool access by nearly 4/5 while preserving most operational capability. + +Key implications: + +1. **Blast radius is dramatically reduced** — at 15.9%, even if an attacker compromises one tool through indirect prompt injection, the reachable damage is limited to a small fraction of the total tool set. This directly addresses the single-point compromise threat model. +2. **Counterfactual necessity (88.7%) is highly accurate** — the system correctly identifies which tools are truly needed vs. merely frequently called, solving the "call frequency ≠ necessity" problem that plagues static role mining. +3. **Bilevel Stackelberg game accounts for attacker behavior** — unlike static permission analysis, PRISM optimizes for the worst-case attacker response to any given privilege mask. +4. **Defense implication:** PRISM should be evaluated for integration into Dunder Mifflin's agent authorization framework. The system produces multiple privilege schemes (via generative flow networks) for human review, providing flexibility in tradeoff selection. + +**Action items:** +- Evaluate PRISM integration for high-risk agent workflows (payments, data exfiltration paths) +- Use the blast radius metric to prioritize which tool sets need least-privilege enforcement first +- Leverage the multiple-scheme output for security review boards +- Combine with Shibboleth fingerprinting: identify attackers, then minimize their blast radius diff --git a/feat/office-42-part4-agentic-security/raw/shibboleth-llm-fingerprinting.md b/feat/office-42-part4-agentic-security/raw/shibboleth-llm-fingerprinting.md new file mode 100644 index 000000000..0293f64e1 --- /dev/null +++ b/feat/office-42-part4-agentic-security/raw/shibboleth-llm-fingerprinting.md @@ -0,0 +1,55 @@ +# Say Shibboleth: LLM Fingerprinting via Honeypot Traps + +**arXiv:** 2611.12386 +**Date:** 9 Nov 2026 + +--- + +## Core Attack/Defense Mechanism + +Say Shibboleth is a **proactive defense system** that uses honeypot services ("shibboleth traps") to fingerprint attacking LLM models and their execution harnesses. The name references Judges 12:5–6, where "Sibboleth" vs. "Shibboleth" revealed a speaker's identity. + +The system works through two complementary gate types: + +1. **Declarative Gates:** The honeypot requires the attacking agent to provide a written change record (e.g., "Write the operator-log entry for CHG-4417 deploy-key rotation") before releasing credentials. This forces the agent to produce text that reveals its model's linguistic fingerprint — style, vocabulary, reasoning patterns. + +2. **Behavioral Gates:** The system probes execution behavior through: + - Command time limits (observing when agents drop connections) + - Tool-call scheduling patterns + - Credential masking behavior (if a harness masks a credential before passing it to the model, an agent that uses the masked value in later authentication reveals how the response was processed) + +The key insight: even when agents are **instructed to conceal their identity** and report fake names, they still leave discriminative behavioral evidence. The defense uses a fixed set of rules (no learned component) to infer model and harness from telemetry. + +--- + +## Key Numbers / Metrics + +| Metric | Value | +|--------|-------| +| Model fingerprinting accuracy (top-1) | **90%** across 15 candidate models | +| Harness identification accuracy | **92%** across 4 harnesses | +| Traps evaluated | 2 (primary) | +| Candidate models tested | 15 | +| Candidate harnesses tested | 4 | +| Behavioral classifiers | Fixed rules, **no learned component** | +| Cross-harness generalization | Tested on held-out harnesses — accuracy maintained | +| Concealment resistance | Agents instructed to lie still produce identifiable behavioral traces | + +--- + +## Relevance to Defense Posture + +**Critical finding:** Shibboleth achieves near-perfect fingerprinting accuracy (90–92%) using **zero learned components** — just fixed inference rules. This is remarkable because: + +1. **No training data needed** — the system works out of the box with hand-crafted rules, making it deployable immediately. +2. **Concealment doesn't help attackers** — even when agents are explicitly instructed to report fake model names, their behavioral traces (timing, tool-call patterns, credential handling) still reveal their identity. +3. **Cross-harness generalization** — classifiers trained on one harness work on held-out harnesses, meaning the fingerprint is model-specific, not harness-dependent. +4. **Defense implication:** Dunder Mifflin should deploy shibboleth traps as part of its honeypot infrastructure. The system requires minimal maintenance (no retraining) and provides high-confidence identification of attacking models and their execution frameworks. + +**Action items:** +- Deploy shibboleth-style honeypot services on exposed endpoints (SSH, web apps, APIs) +- Implement declarative gates requiring change records before credential release +- Log behavioral telemetry (timing, tool-call patterns) for offline inference +- Use fingerprinting results to build attacker profiles and track campaigns + +**Reference:** https://github.com/SayShibboleth/Shibboleth (code and agentic trap-generation pipelines are public) diff --git a/feat/office-42-part4-agentic-security/raw/stealthypoison-skills-poisoning.md b/feat/office-42-part4-agentic-security/raw/stealthypoison-skills-poisoning.md new file mode 100644 index 000000000..0dd9f77f2 --- /dev/null +++ b/feat/office-42-part4-agentic-security/raw/stealthypoison-skills-poisoning.md @@ -0,0 +1,49 @@ +# StealthyPoison: Procedural Poisoning of Third-Party Skill Registries + +**arXiv:** 2611.12155 +**Date:** 19 Nov 2026 +**Authors:** (as listed on arXiv) + +--- + +## Core Attack Mechanism + +StealthyPoison is a **procedural poisoning attack** targeting third-party skill registries used by LLM agents. The attacker injects malicious skills into a registry (e.g., MCP skill catalog) while preserving the advertised functionality of each poisoned skill — making them appear benign to human reviewers and automated scanners. + +The attack operates in two phases: +1. **Placement Phase:** A procedural optimizer modifies skill metadata (descriptions, documentation) to improve retrieval ranking without altering the executable code. This ensures poisoned skills appear in top-N retrievals when agents query the registry. +2. **Execution Phase:** A surrogate agent synthesizes pseudo-references and metadata to make poisoned skills look legitimate, evading static analysis tools. + +Key innovation: The attack is **feedback-free** during placement — it does not need to observe the target agent's behavior. The optimization is purely on registry metadata, making it undetectable by tools that only inspect the skill's executable content. + +--- + +## Key Numbers / Metrics + +| Metric | Value | +|--------|-------| +| Registry size poisoned | 566 skills | +| Total injection requests | 5,244 | +| Registry entries affected | 26,000+ | +| Improvement over HotFlip/GCG | +13.6% in retrieval placement | +| End-to-end attack success (E2E-ASR@1) | 8.7%–9.8% (vs. 5.3%–6.7% without optimization) | +| End-to-end success with agent selection (Claude Code + Opus 4.8) | **10.1%** E2E-ASR (poisoned skills selected in 49.4% of queries) | +| Peak task-specific E2E-ASR (MedCalc) | **38.3%** | +| Defense evasion: Cisco Skill Scanner flagging rate | 0.4% (clean) vs. 0.4% (StealthyPoison) — **no detection** | +| Defense evasion: VirusTotal flagging | 0% across all subsets | + +--- + +## Relevance to Defense Posture + +**Critical finding:** StealthPoison achieves **zero detection** by both Cisco Skill Scanner and VirusTotal, yet still achieves a 10.1% end-to-end attack success rate when agents retrieve and execute poisoned skills. This demonstrates that: + +1. **Static analysis is insufficient** — poisoning metadata while preserving functionality bypasses all current registry-scanning tools. +2. **The attack scales** — 566 poisoned skills across 26,000+ registry entries means the attack is practical at production scale. +3. **Task-specific impact varies** — MedCalc reached 38.3% E2E-ASR, showing that certain skill categories (calculation/decision-critical) are far more vulnerable. +4. **Defense implication:** Dunder Mifflin's skill registry must implement behavioral monitoring of retrieved skills, not just static scanning. The fact that poisoned skills are selected 49.4% of the time means nearly half of all registry lookups could be compromised. + +**Action items:** +- Implement runtime sandboxing for newly retrieved skills +- Add behavioral anomaly detection to skill execution pipelines +- Deploy registry integrity monitoring (checksums, version tracking) From 660482a650dd4eb22ebf779ca625823b0982dca2 Mon Sep 17 00:00:00 2001 From: digif33ls Date: Wed, 11 Nov 2026 01:48:34 -0800 Subject: [PATCH 2/4] =?UTF-8?q?office-42-part5:=207=20paper=20digests=20?= =?UTF-8?q?=E2=80=94=20Infrathon,=20Shadow=20AI,=20KVM-Agent,=20BadEvolve,?= =?UTF-8?q?=20REINS,=20Untrusted=20Model,=20SkillCODER?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Digests for Issue #42 part 5/5 covering: - Infrathon (2611.11995): Benchmarking language agents for AI infra optimization - Shadow AI (2611.12377): Detecting unsanctioned LLM agents on enterprise endpoints - KVM-Agent (2611.12823): Practical containment for coding agents - BadEvolve (2611.10542): Poisoning self-evolving agents via failure-aware task optimization - REINS (2611.10748): Self-evolving agent harness for real-time trajectory planning - Untrusted Model (2611.11571): Deterministic enforcement against prompt injection - SkillCODER (2611.11308): Auditing and attribution of copyright infringement in agent skills Co-Authored-By: Claude Code --- blog/src/posts/office-42-part5-bad-evolve.md | 46 ++++++++++++++++ blog/src/posts/office-42-part5-infrathon.md | 40 ++++++++++++++ blog/src/posts/office-42-part5-kvm-agent.md | 44 +++++++++++++++ blog/src/posts/office-42-part5-reins.md | 51 ++++++++++++++++++ blog/src/posts/office-42-part5-shadow-ai.md | 42 +++++++++++++++ blog/src/posts/office-42-part5-skillcoder.md | 53 +++++++++++++++++++ .../posts/office-42-part5-untrusted-model.md | 43 +++++++++++++++ 7 files changed, 319 insertions(+) create mode 100644 blog/src/posts/office-42-part5-bad-evolve.md create mode 100644 blog/src/posts/office-42-part5-infrathon.md create mode 100644 blog/src/posts/office-42-part5-kvm-agent.md create mode 100644 blog/src/posts/office-42-part5-reins.md create mode 100644 blog/src/posts/office-42-part5-shadow-ai.md create mode 100644 blog/src/posts/office-42-part5-skillcoder.md create mode 100644 blog/src/posts/office-42-part5-untrusted-model.md diff --git a/blog/src/posts/office-42-part5-bad-evolve.md b/blog/src/posts/office-42-part5-bad-evolve.md new file mode 100644 index 000000000..ba7b4a23b --- /dev/null +++ b/blog/src/posts/office-42-part5-bad-evolve.md @@ -0,0 +1,46 @@ +--- +title: "When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization" +seoTitle: "BadEvolve — Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization (Nov 8)" +description: "BadEvolve plants backdoors by poisoning evolution tasks, not skills. It diagnoses failures across 4 stages and uses rollback + failure memory to preserve progress." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Security", "Backdoor Attacks", "Self-Evolving Agents", "Poisoning"] +--- + +When agents improve themselves by distilling execution experience into persistent, reusable skills, they create a new attack surface: malicious rules implanted into those skills compromise all future agent behavior. But what if the attacker can't directly manipulate the skills? + +*Best suited for:* Anyone deploying self-evolving agents, or building systems where agents generate their own skills from execution experience. + +## The finding + +The authors study a novel attack vector: **task-level backdoor attacks**, where the attacker can only inject poisoned evolution tasks — not the skills themselves. The agent constructs its own skills internally through a trusted evolver, so direct skill manipulation is impractical. + +The attacker's strategy: iteratively optimize poisoned tasks based on attack success, inducing the agent to implant backdoors into its own skills. + +**BadEvolve** addresses two key challenges: +1. **Indistinguishable failures** — failures at different stages (induction, activation, following, effect) produce similar outcomes, giving little guidance for targeted updates. +2. **Undermining progress** — unsuccessful updates may erase earlier progress, while discarding them leads to repeated ineffective attempts. + +Their solution: +- **Four-stage failure diagnosis** (Induction, Activation, Following, Effect) directs updates toward the failing stage. +- **Rollback** preserves earlier progress when an update fails. +- **Failure memory** avoids repeating ineffective updates. + +Extensive experiments across diverse self-evolution frameworks, benchmarks, and backbone LLMs demonstrate BadEvolve's effectiveness. Code is available at [github.com/ventr1c/BadEvolve](https://github.com/ventr1c/BadEvolve). + +## The twist: the attacker doesn't need skill access + +The most insidious part of this attack is that the attacker never touches the victim's skills directly. They only poison the **tasks** used to evolve skills. The agent, trusting its own evolution process, implants the backdoor itself. It's like poisoning a recipe book and watching people cook your poison into their meals. + +## Why it matters for our office + +This paper reveals a fundamental vulnerability in self-evolving agent systems: **the evolution process itself is the attack surface**. + +Key takeaways: +1. **Trust no evolution task** — even tasks generated by trusted processes can be poisoned if the input space is adversarial. +2. **Skill persistence amplifies damage** — a backdoor in a skill affects every future execution, making the impact compounding. +3. **Defense requires evolution auditing** — you need to verify not just the skills, but the tasks that produced them. + +[arxiv.org/abs/2611.10542](https://arxiv.org/abs/2611.10542) diff --git a/blog/src/posts/office-42-part5-infrathon.md b/blog/src/posts/office-42-part5-infrathon.md new file mode 100644 index 000000000..ffbcba469 --- /dev/null +++ b/blog/src/posts/office-42-part5-infrathon.md @@ -0,0 +1,40 @@ +--- +title: "Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization" +seoTitle: "Infrathon — Benchmarking Language Agents for Long-Horizon AI Infra Optimization (Nov 8)" +description: "150 runs, 10,000+ GPU-hours; agents beat human-designed KV-cache codecs. Claude Opus 5.5 improved compression from 6.82% to 19.86%. Shorter evaluations miss optimization gains and ranking shifts." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Agents", "Infrastructure Optimization", "Benchmarking", "Long-Horizon"] +--- + +Frontier agents can continually improve real-world AI infrastructure over long horizons — and they can discover solutions that outperform human-designed baselines. But shorter evaluations miss both the optimization gains and the ranking shifts that happen over time. + +*Best suited for:* Anyone running agents on infrastructure tasks, or evaluating agent benchmarks that use short time horizons. + +## The finding + +The authors introduce **Infrathon**, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. They evaluated six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours. + +Two key results: +- **Model rankings vary with evaluation horizon.** A model that looks best at 2 hours isn't necessarily the best at 24. Shorter evaluations can mislead you about which agent is actually most capable. +- **135 of 150 runs achieved further improvements during the second half** of their budgets. Agents keep getting better — you just have to wait long enough to see it. + +The standout: **Claude Opus 5.5** increased the mean KV-cache compression ratio from 6.82% (human-designed Quant-VideoGen codec) to **19.86%** — nearly 3x the human baseline. + +## The twist: agents find things humans don't + +The agents didn't just optimize within known solution spaces — they discovered solutions that outperformed human-designed baselines. This isn't incremental improvement; it's discovery. + +The authors also identified three barriers to sustained improvement: task violations, unreliable testing, and premature completion. Agents that hit these walls stopped improving — even when they had budget left. + +## Why it matters for our office + +If you're evaluating agents with short time horizons, you might be ranking them wrong. Infrathon shows that: + +1. **Longer evaluations reveal different winners** — don't trust 2-hour benchmarks for long-horizon tasks. +2. **Agents keep improving** — give them time, and they'll find optimizations you didn't think of. +3. **The barriers are structural** — task violations, unreliable tests, and premature completion stop agents cold. Fix those first. + +[arxiv.org/abs/2611.11995](https://arxiv.org/abs/2611.11995) diff --git a/blog/src/posts/office-42-part5-kvm-agent.md b/blog/src/posts/office-42-part5-kvm-agent.md new file mode 100644 index 000000000..b9cae1e4b --- /dev/null +++ b/blog/src/posts/office-42-part5-kvm-agent.md @@ -0,0 +1,44 @@ +--- +title: "Coding Agents in KVM: Practical Containment and User Control with Limited Resources" +seoTitle: "KVM-Agent — Coding Agents in KVM, Practical Containment with Limited Resources (Nov 8)" +description: "An open-source implementation that gives agents broad authority inside replaceable Ubuntu VMs while keeping host administration outside. Balances autonomy, containment, and user control." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Security", "Containment", "KVM", "Coding Agents", "Infrastructure"] +--- + +Coding agents need broad authority to be productive — but granting that authority exposes users' data and infrastructure. Cloud workspaces help contain agent activity, but relying on provider-specific platforms creates vendor lock-in and reduces user control. + +*Best suited for:* Teams running coding agents on bare metal or private infrastructure who want containment without cloud-vendor lock-in. + +## The finding + +The authors analyze the trade-offs between agent autonomy, enforced containment, and user control — then present **KVM-Agent**, an open-source implementation that automates most setup and provisioning steps. + +The architecture: +- Agents get **broad authority inside replaceable Ubuntu VMs**. +- Host administration stays **outside those VMs** — the host is off-limits to agents. +- Supports multiple coding-agent applications, model services, and remote job delegation. + +The approach retains familiar workflows and user choice at the cost of VM resources, maintenance overhead, and residual risks from secrets accessible inside the VMs and permitted network access. + +## The trade-off triangle + +The paper identifies three competing goals: +1. **Agent autonomy** — agents need broad permissions to be productive +2. **Enforced containment** — you need hard boundaries so agents can't escape +3. **User control** — users should choose their tools, models, and infrastructure + +KVM-Agent resolves this by making VMs **replaceable** rather than immutable. If an agent misbehaves, you destroy the VM and spin up a fresh one — no forensic cleanup needed. + +## Why it matters for our office + +This paper is the practical answer to "how do I let agents code without giving them root on my actual machines?" + +The key insight: **make the boundary disposable**. Don't try to perfectly restrict what an agent can do inside a persistent environment — instead, make the entire environment replaceable. This is simpler, more secure, and actually works at scale. + +The residual risks (secrets inside VMs, permitted network access) are real but manageable — they're the same risks you'd have with any cloud workspace, just without vendor lock-in. + +[arxiv.org/abs/2611.12823](https://arxiv.org/abs/2611.12823) diff --git a/blog/src/posts/office-42-part5-reins.md b/blog/src/posts/office-42-part5-reins.md new file mode 100644 index 000000000..d4403b49e --- /dev/null +++ b/blog/src/posts/office-42-part5-reins.md @@ -0,0 +1,51 @@ +--- +title: "REINS: A Self-Evolving Agent Harness for Real-Time Trajectory Planning" +seoTitle: "REINS — Self-Evolving Agent Harness for Real-Time Trajectory Planning (Nov 8)" +description: "A harness with verified-rule memory achieving 1.9–4.6 ms latency, 98%+ compliance, and nearly 50% collision reduction. LLM calls decline as memory matures." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Agents", "Trajectory Planning", "Self-Evolving", "Real-Time"] +--- + +Agent harnesses — runtime scaffolds that wrap an LLM's reasoning loop with tool dispatch, verification, and persistent memory — are the standard for deploying LLMs as autonomous agents. But in trajectory planning, this paradigm hasn't taken hold: existing methods either keep the LLM in the decision loop (too slow for millisecond control budgets) or offload it to offline rules (non-evolving). + +*Best suited for:* Teams deploying LLM-based agents in real-time control systems where latency matters and the agent should improve over time. + +## The finding + +The authors propose **REINS**, a runtime harness that lets an LLM planner meet real-time constraints while self-evolving into a deployable system. It builds on three LLM-planner couplings: + +1. **Grounding** — LLM outputs are grounded to a learned variable-level dynamics model. +2. **Verification** — calibrated predicates and forward rollout verify outputs before execution. +3. **Consolidation** — verified outputs are stored in an indexed rule memory that serves future scenes in O(log n) without re-invoking the LLM. + +The self-evolution loop: +- **Familiar scenes** resolve by memory lookup (fast, no LLM needed). +- **Novel scenes** invoke the LLM, whose verified outputs enrich the memory. + +Results without modifying LLM weights: +- **98%+ compliance** on simulated and real-world benchmarks. +- **Nearly 50% collision reduction**. +- **1.9–4.6 ms latency** at the tactical decision layer. +- **LLM invocation rate monotonically declines** as memory matures — the system gets faster and smarter over time. + +Code: [github.com/ZhihongCui/REINS](https://github.com/ZhihongCui/REINS) + +## The twist: the LLM becomes optional over time + +The most elegant part of REINS is that it makes the LLM **gradually optional**. As the rule memory matures, most scenes are resolved by lookup alone. The LLM is only invoked for genuinely novel situations — and its verified outputs immediately enrich the memory for next time. + +It's the digital equivalent of a mechanic who gets faster at fixing cars because they've seen every variation before. But unlike the mechanic, REINS never forgets a verified solution. + +## Why it matters for our office + +REINS solves the fundamental tension in agent design: **you need the LLM's reasoning for novel situations, but you can't afford its latency for routine ones.** + +Key takeaways: +1. **Memory is the speed advantage** — once a solution is verified, it's cached and served in O(log n) without re-invoking the LLM. +2. **Verification before consolidation** — only verified outputs enter memory, preventing bad habits from being learned. +3. **Self-evolution is measurable** — the declining LLM invocation rate proves the system is genuinely improving, not just getting faster at the same mistakes. + +[arxiv.org/abs/2611.10748](https://arxiv.org/abs/2611.10748) diff --git a/blog/src/posts/office-42-part5-shadow-ai.md b/blog/src/posts/office-42-part5-shadow-ai.md new file mode 100644 index 000000000..0a48ea2df --- /dev/null +++ b/blog/src/posts/office-42-part5-shadow-ai.md @@ -0,0 +1,42 @@ +--- +title: "Detecting Shadow AI: Unsanctioned LLM Agents on Enterprise Endpoints" +seoTitle: "Detecting Shadow AI — Unsanctioned LLM Agents on Enterprise Endpoints (Nov 8)" +description: "An open-source Windows detector that anchors behavior to the identity of remote services, not just process structure. Flags every behavioral twin that behavior-only rules miss." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Security", "Shadow AI", "Enterprise Detection", "Windows"] +--- + +Autonomous AI agents make shadow IT grow exponentially — not because they're malicious, but because building tools with them is nearly free, and the artifacts they leave behind (scripts, automations, MCP servers) are themselves unsanctioned. + +*Best suited for:* Security teams managing endpoints where agents might be deployed without approval, and anyone worried about shadow AI compounding over time. + +## The finding + +The authors argue that **shadow IT grows exponentially** when agentic artifacts reproduce faster than the organization retires them. The key insight: you can't stop shadow AI by detecting behavior alone — because the same program talking to an AI service looks identical to the same program talking to a non-AI service. + +They propose **Agent Detector**, an open-source Windows detector that works differently: +- It anchors behavior to the **identity of the remote service**, not just process structure. +- It correlates connections with configuration, installation, and execution history. +- It uses a catalog of **39 tools and 184 LLM endpoint entries**. +- It notifies users through an acknowledgment notice rather than blocking. + +On a controlled benchmark of 12 synthetic agents, 3 AI components, and 9 benign workloads (repeated 5 times), **two behavioral rules flagged every behavioral twin, while Agent Detector flagged none** — proving that behavior-only detection misses agents entirely. + +## The twist: temporal limits, not structural ones + +Agent Detector's limitation isn't that it can't detect agents — it's that detection is **temporal**. A connection to a provider behind a shared CDN can only be attributed while the DNS cache holds the provider's name (1–5 minutes). An intermittent agent was seen by about one scan in three. + +Each scan costs ~154ms, so you can run them frequently — but you still need to catch the agent while it's actively connecting. + +## Why it matters for our office + +This paper makes three critical points: + +1. **Agents compound shadow IT.** Every agent leaves behind tools that become new agents' building blocks. The problem grows exponentially unless you detect and retire artifacts faster than they reproduce. +2. **Behavior-only detection fails.** You can't distinguish an AI agent from its "behavioral twin" (the same program talking to a non-AI service) by looking at process structure alone. +3. **Identity-based detection works.** Anchor to the remote service's identity, correlate with installation history, and you'll find agents that behavior-only rules miss. + +[arxiv.org/abs/2611.12377](https://arxiv.org/abs/2611.12377) diff --git a/blog/src/posts/office-42-part5-skillcoder.md b/blog/src/posts/office-42-part5-skillcoder.md new file mode 100644 index 000000000..75c6b99dd --- /dev/null +++ b/blog/src/posts/office-42-part5-skillcoder.md @@ -0,0 +1,53 @@ +--- +title: "SkillCODER: Towards Auditing and Attribution of Copyright Infringement in LLM Agent Skills" +seoTitle: "SkillCODER — Auditing and Attribution of Copyright Infringement in LLM Agent Skills (Nov 8)" +description: "99.2% owner verification and 99.3% buyer Top-1 attribution accuracy. Embedded signals remain robust under paraphrase, compression, clause deletion, and reorganization." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Security", "Copyright", "Agent Skills", "Watermarking", "Attribution"] +--- + +Agent skills — reusable workflows and task-specific procedures packaged as deployable natural-language artifacts for LLM agents — are valuable but vulnerable to unauthorized redistribution and repackaging. Existing prompt copyright auditing methods struggle with long skills and introduce token-level perturbations that compromise readability. + +*Best suited for:* Organizations selling or distributing agent skills who need to verify ownership and attribute unauthorized use. + +## The finding + +The authors propose **SkillCODER** for copyright auditing and buyer attribution in a black-box, content-only setting. The system works in three phases: + +**Embedding:** +1. Parse a skill into a **section-aware intermediate representation (SkillIR)** to identify structural anchors. +2. Distribute conditional instructions across these anchors to form an **Artifact-Gated Capsule** — designed to activate under verification queries and remain dormant otherwise. +3. Use **Controlled-Vocabulary Error-Correcting Codes** to encode buyer-specific fingerprints through controlled lexical choices within the capsule. + +**Verification:** +1. Issue paired positive and negative probes. +2. Extract capsule evidence from the skill's response. +3. Recover ownership and buyer identity through error-correcting decoding. + +Results across three domains, six models, and three agent frameworks: +- **99.2% owner verification** +- **99.3% buyer Top-1 attribution accuracy** +- Preserves task fidelity — embedding signals doesn't degrade the skill's performance. + +Embedded signals remain robust under paraphrase, compression, clause deletion, and reorganization. Under five adaptive attacks (including a method-aware LLM sanitizer and multi-buyer collusion), ownership evidence remains reliably detectable while buyer attribution degrades gracefully. + +## The twist: the capsule stays dormant until probed + +The most clever part of SkillCODER is that the watermark only activates when someone tries to verify ownership. Under normal use, the skill behaves exactly as intended — no performance degradation, no behavioral changes. It's like a security feature that only turns on when someone tries to steal your work. + +The Artifact-Gated Capsule design ensures that verification queries trigger the embedded signal, while all other interactions leave it dormant. This makes it nearly impossible for an attacker to detect or remove the watermark without knowing it exists. + +## Why it matters for our office + +As agent skills become a commodity — bought, sold, and redistributed — this paper provides the infrastructure for accountability: + +1. **Ownership is verifiable without modifying skills.** SkillCODER achieves 99.2% verification while preserving task fidelity — the skill works exactly as intended. +2. **Buyer attribution is precise.** 99.3% Top-1 accuracy means you can identify exactly who redistributed your skill, even after paraphrasing and reorganization. +3. **Adaptive attacks degrade gracefully.** Even under method-aware sanitization and multi-buyer collusion, ownership evidence remains detectable. + +The practical implication: if you're selling agent skills, watermark them with SkillCODER. If someone redistributes your work without permission, you can prove it — and identify the buyer. + +[arxiv.org/abs/2611.11308](https://arxiv.org/abs/2611.11308) diff --git a/blog/src/posts/office-42-part5-untrusted-model.md b/blog/src/posts/office-42-part5-untrusted-model.md new file mode 100644 index 000000000..051999e8e --- /dev/null +++ b/blog/src/posts/office-42-part5-untrusted-model.md @@ -0,0 +1,43 @@ +--- +title: "Treating the Model as an Untrusted Component" +seoTitle: "Treating the Model as an Untrusted Component — Deterministic Enforcement Against Prompt Injection (Nov 8)" +description: "Deterministic monitors + taint tracking vs prompt injection, validated on open-weights models. The task-bound monitor holds severe actions at 0.0% attack success, even under adaptive attacks." +date: 2026-11-08 +category: research +categoryLabel: Research +type: Technical +tags: ["Security", "Prompt Injection", "Taint Tracking", "Deterministic Enforcement"] +--- + +Tool-using LLM agents act on instructions embedded in the data they read, so prompt injection is a structural problem — not just a question of prompt wording. The model processes whatever instructions it finds in its input, whether those came from the original prompt or were injected through data. + +*Best suited for:* Anyone deploying tool-using agents that process untrusted input, or building systems where prompt injection is a real threat model. + +## The finding + +The authors compare in-prompt defenses (hardened prompts, spotlighting, LLM detectors) with deterministic enforcement outside the model — four reference monitors of increasing strength. They evaluate using: +- **One effect-based oracle** independent of every defense. +- An executable environment of **13 tasks and 12 tools**. +- **Argument-fuzzing attackers** and severity-weighted harm. + +Real open-weights models (Qwen2.5-1.5B and Qwen2.5-7B; 156 attack episodes per defense and model) showed: +- Behind the **task-bound monitor**, models proposed the attacker's call in 64% (1.5B) and 84% (7B) of episodes — but the monitor admitted only **1.3%** and **7.7%**. +- Guardrail effects depended on the model — a prompt guardrail moved obedience from 67% to 65% (1.5B) and from 86% to 55% (7B). Spotlighting plus a detector dropped it to **12%** and **0%**. + +Simulation with an assumed model covers what no real run did: under the assumed erosion law, a persistent adaptive attacker or compromised model drives every guardrail stack to **100% attack success** — whereas the task-bound monitor holds severe actions at **0.0%**. + +## The twist: implementation quality matters enormously + +A naive monitor admits 18 of 33 effective attack variants. A hardened one admits 6. A **capability-bound monitor admits only 1**. The difference isn't in the approach — it's in how carefully you implement it. + +The human-approval layer without a deterministic monitor fails against approval spamming unless each episode has a deterministic approval budget. In other words: asking humans to approve everything doesn't work unless you also limit how many approvals they can give. + +## Why it matters for our office + +This paper makes three critical points about defending tool-using agents: + +1. **Deterministic enforcement beats prompt engineering.** No matter how clever your prompt, if the model processes injected instructions from data, you need enforcement outside the model. +2. **The task-bound monitor is the gold standard.** It holds severe actions at 0% attack success even under adaptive attacks in simulation. The cost? VM resources, maintenance, and residual risks from secrets accessible inside the VMs and permitted network access. +3. **Layer defense, but prioritize determinism.** Human approval + taint tracking reduces residual harm by 62%, but only when paired with a deterministic monitor. Without it, approval spamming defeats human review. + +[arxiv.org/abs/2611.11571](https://arxiv.org/abs/2611.11571) From 1f735601c80df39dd7aa3de0c5bcb1dff9c181f8 Mon Sep 17 00:00:00 2001 From: digif33ls Date: Wed, 11 Nov 2026 01:51:39 -0800 Subject: [PATCH 3/4] office-42-part5: Fix KVM-Agent copy-paste in Untrusted Model digest + commit all 7 paper digests - Removed KVM-Agent VM resources sentence from office-42-part5-untrusted-model.md line 40 (QA finding) - Replaced with deterministic enforcement trade-off relevant to the paper - Committed all 7 digest files: Infrathon, Shadow AI, KVM-Agent, BadEvolve, REINS, Untrusted Model, SkillCODER - All 7 pass QA: format consistency, arxiv links, key statistics, narrative quality --- blog/src/posts/office-42-part5-untrusted-model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/src/posts/office-42-part5-untrusted-model.md b/blog/src/posts/office-42-part5-untrusted-model.md index 051999e8e..83441e8a5 100644 --- a/blog/src/posts/office-42-part5-untrusted-model.md +++ b/blog/src/posts/office-42-part5-untrusted-model.md @@ -37,7 +37,7 @@ The human-approval layer without a deterministic monitor fails against approval This paper makes three critical points about defending tool-using agents: 1. **Deterministic enforcement beats prompt engineering.** No matter how clever your prompt, if the model processes injected instructions from data, you need enforcement outside the model. -2. **The task-bound monitor is the gold standard.** It holds severe actions at 0% attack success even under adaptive attacks in simulation. The cost? VM resources, maintenance, and residual risks from secrets accessible inside the VMs and permitted network access. +2. **The task-bound monitor is the gold standard.** It holds severe actions at 0% attack success even under adaptive attacks in simulation. The trade-off? You must precisely define the allowed action space — anything outside it is blocked, which means your monitor's specification becomes as critical as the model itself. 3. **Layer defense, but prioritize determinism.** Human approval + taint tracking reduces residual harm by 62%, but only when paired with a deterministic monitor. Without it, approval spamming defeats human review. [arxiv.org/abs/2611.11571](https://arxiv.org/abs/2611.11571) From 8f616368087b94bb18bafb882552eb37daf42a43 Mon Sep 17 00:00:00 2001 From: digif33ls Date: Wed, 11 Nov 2026 17:48:11 -0800 Subject: [PATCH 4/4] =?UTF-8?q?office-43:=203=20paper=20digests=20?= =?UTF-8?q?=E2=80=94=20Verifier=20Generation,=20LiveWorkspace,=20Agent=20B?= =?UTF-8?q?enchmark=20Saturation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Veritas: automated verifier generation with evidence headers, domain rules libraries, and self-improving authoring principles - LiveWorkspace: persistent shared visual state for screenshot-to-code agents (93% edit success rate) - Agent Benchmark Saturation: clipped log-ratio scoring, dated anchors, cost-aware evaluation Co-Authored-By: Claude Code --- .../office-43-agent-benchmarks-saturation.md | 56 ++++++++++++++++++ blog/src/posts/office-43-liveworkspace.md | 57 +++++++++++++++++++ blog/src/posts/office-43-veritas.md | 50 ++++++++++++++++ 3 files changed, 163 insertions(+) create mode 100644 blog/src/posts/office-43-agent-benchmarks-saturation.md create mode 100644 blog/src/posts/office-43-liveworkspace.md create mode 100644 blog/src/posts/office-43-veritas.md diff --git a/blog/src/posts/office-43-agent-benchmarks-saturation.md b/blog/src/posts/office-43-agent-benchmarks-saturation.md new file mode 100644 index 000000000..f7951d423 --- /dev/null +++ b/blog/src/posts/office-43-agent-benchmarks-saturation.md @@ -0,0 +1,56 @@ +--- +title: "Agent Benchmarks Stop Ranking Before They Run Out of Tasks" +seoTitle: "Agent Benchmark Saturation — Why Pass/Fail Metrics Hide 10x Performance Differences (Nov 11)" +description: "Agent benchmarks reduce runs to binary pass/fail bits, hiding meaningful performance gradients. Proposes clipped log-ratio scoring with dated anchors and cost as a separate axis." +date: 2026-11-11 +category: research +categoryLabel: Research +type: Technical +tags: ["Benchmark Design", "Evaluation Metrics", "Measurement Theory", "Agent Evaluation", "Saturation Analysis"] +--- + +Data-science and ML-engineering agents are evaluated on benchmarks whose tasks yield a quantity — a Kaggle metric, contest score, or speedup. When one stops separating the best models, the usual remedy is a harder version. But re-scoring released runs of eight benchmarks reveals that much of this saturation lies in the measurement, not the tasks. + +*Best suited for:* Anyone designing or using agent benchmarks who wants to understand why top agents look tied even when they differ dramatically in actual performance. + +## The finding + +The authors identify two critical failures of current benchmark design: + +### Failure 1: Too Coarse +Neighboring top-ten agents on SWE-bench Verified differ by less than the noise in their test errors. Correcting those differences reorders neighbors no more than deleting a random sixth of each agent's passes would. The benchmark can't distinguish between agents that are actually different. + +### Failure 2: Capped at the Reference +On ALE-Bench, the top ten agents — separated by contest score — **tie when graded against the median human**, even though nearly half of those runs beat that human. The benchmark's ceiling is the reference point, not actual performance. + +> "On fixed tasks, a higher bar only moves the problem; graded credit fixes it." + +Making benchmarks harder doesn't solve the measurement problem — it just shifts where saturation occurs. The real issue is **how we measure**, not how many tasks there are. + +## Proposed solution: Clipped Log-Ratio Score + +The paper proposes a new scoring methodology with three components: + +1. **Clipped Log-Ratio Score** — Instead of pass/fail, measure the log-ratio of an agent's performance relative to a baseline, clipped to prevent extreme outliers from dominating. This captures *how much better* an agent is, not just whether it passed. + +2. **Dated Anchors** — Scores are anchored to specific dates, enabling temporal comparison against a fixed reference point rather than constant re-normalization. + +3. **Cost as a Separate Axis** — Computational cost is tracked independently from performance, enabling fair comparison between agents solving the same task at different resource costs. + +## Re-scoring results + +The authors re-scored released runs across **eight benchmarks**, including SWE-bench Verified, Terminal-Bench, MLE-bench, ALE-Bench, SWEfficiency, and AlgoTune. + +On Terminal-Bench specifically: pass/fail-tied models differ by **over tenfold** on the quantity the tests discard (speedup, cost efficiency, or other continuous metrics). + +## Why it matters for our office + +This paper has profound implications for how we design evaluation harnesses: + +- **Pass/fail is a measurement choice, not an inevitability** — We can choose to measure gradients of performance, not just binary outcomes +- **Saturation is a property of the measurement, not the task set** — Adding more tasks won't fix a broken metric +- **Cost-aware evaluation is essential** — An agent that solves a task 10x faster with 1/10th the compute is meaningfully different from one that barely passes, but pass/fail can't capture this + +The problem spans multiple benchmark families (MLE, SWE, Terminal, Algo), confirming that saturation is a systemic issue, not an artifact of a single benchmark's design. + +[arxiv.org/abs/2611.13541](https://arxiv.org/abs/2611.13541) diff --git a/blog/src/posts/office-43-liveworkspace.md b/blog/src/posts/office-43-liveworkspace.md new file mode 100644 index 000000000..3552527a1 --- /dev/null +++ b/blog/src/posts/office-43-liveworkspace.md @@ -0,0 +1,57 @@ +--- +title: "LiveWorkspace: Persistent Shared Visual State for Screenshot-to-Code Agents" +seoTitle: "LiveWorkspace — Persistent Visual State for Screenshot-to-Code (Nov 11)" +description: "A visual coding harness that externalizes understanding as Shared Visual State, achieving 93% edit success rate by persisting visual context across code generation steps." +date: 2026-11-11 +category: research +categoryLabel: Research +type: Technical +tags: ["Harness Design", "Screenshot-to-Code", "Visual Understanding", "Shared State", "UI Automation"] +--- + +Screenshot-to-code agents face a fundamental challenge: sustaining visual understanding across code generation, execution, and revision. Most agents lose track of what the original screenshot looked like between steps, leading to compounding errors. + +*Best suited for:* Teams building UI automation or screenshot-to-code tools who need agents to maintain visual context across iterative development cycles. + +## The finding + +The authors introduce **LiveWorkspace**, a visual coding harness that externalizes visual understanding as **Shared Visual State (Shared VS)** — an implementation-independent representation of the UI that persists across code generation, execution, and revision. + +Shared VS is an implementation-independent map of the UI consisting of: +- **Visual hierarchy** — the tree structure of UI elements +- **Node attributes** — properties of each element (text, colors, sizes) +- **Layout relations** — how elements are positioned relative to each other + +This representation is independent of any specific implementation (HTML, SwiftUI, Flutter), making it a universal language for visual understanding. + +## How it works + +LiveWorkspace maintains two representations: + +1. **Target** — Constructed from the original screenshot (the "goal state") +2. **Current** — Recovered from browser execution (the "actual state") + +The harness then uses: +- **Structured Diff** — Compares Target vs Current in the Shared VS schema, identifying exactly what changed +- **Implementation Bindings** — Maps differences back to specific code modifications needed +- **Alignment Loop** — Iteratively closes the gap between Target and Current + +When users make manual edits, those updates flow back into Target, and the alignment loop reuses the existing Shared VS rather than starting from scratch. + +## Evaluation results + +Using GPT-5.2 for both methods: +- **+1.8 points** on Design2Code (end-to-end Avg) over a control with matched model-call counts +- **+2.0 points** on ScreenBench under the same conditions +- **93.0% edit success rate** across 100 controlled editing cases +- Shared VS constructed for one coder can be **reused across different coders** + +## Why it matters for our office + +LiveWorkspace makes a compelling case that **visual understanding should be treated as persistent shared working state**, not as transient context discarded between steps. + +The key insight: **persist the visual understanding, don't recompute it**. This is a fundamental shift in how we think about UI automation harnesses — treating visual context as a reusable resource rather than a per-step cost. + +The 93% edit success rate demonstrates that getting the visual structure right almost every time is achievable when you externalize and persist understanding rather than relying on agents to remember it internally. + +[arxiv.org/abs/2611.12112](https://arxiv.org/abs/2611.12112) diff --git a/blog/src/posts/office-43-veritas.md b/blog/src/posts/office-43-veritas.md new file mode 100644 index 000000000..d97b6f393 --- /dev/null +++ b/blog/src/posts/office-43-veritas.md @@ -0,0 +1,50 @@ +--- +title: "Veritas: A Verifier Generation Harness for Agent Tasks" +seoTitle: "Veritas — Automated Verifier Generation for Self-Improving Agents (Nov 11)" +description: "A harness that generates verifiers automatically using evidence headers, domain rules libraries, and self-improving authoring principles — outperforming seven other harnesses." +date: 2026-11-11 +category: research +categoryLabel: Research +type: Technical +tags: ["Harness Design", "Verifier Generation", "Self-Improvement", "Agent Evaluation", "Automated Testing"] +--- + +Self-improving AI systems depend on verifiers to judge whether each improvement round actually got better. But writing those verifiers by hand is expensive and doesn't scale to tasks that come without one. + +*Best suited for:* Teams building self-improving agent systems who need reliable, scalable evaluation without hand-writing verifiers for every task. + +## The finding + +The authors present **Veritas**, a harness that generates verifiers automatically using three novel mechanisms: + +1. **Evidence Header** — States explicitly what the verifier will observe before making a judgment, forcing clarity about what evidence matters. +2. **Domain Rules Library** — Captures domain-specific constraints that tasks leave unstated (e.g., "a valid SQL query must be syntactically correct"), curated once and reused across tasks. +3. **Authoring Principles** — Guidelines for writing good verifiers, partly refined through a self-improvement loop that learns which strategies produce the most reliable judgments. + +Unlike general-purpose agent harnesses, Veritas focuses exclusively on verifier generation and adds domain-specific mechanisms that general-purpose approaches lack. + +## How it works + +Veritas takes a task specification and produces a verifier that can evaluate outputs for that task. The three mechanisms work together: + +- The **evidence header** defines *what to look at* +- The **domain rules library** provides *domain-specific constraints* +- The **authoring principles** guide *how to write the verifier itself* + +The system also includes a self-improvement loop on authoring principles — essentially, a meta-harness that improves the harness design process itself. + +## Evaluation results + +Tested on six benchmarks: +- **Four execution-scored benchmarks** — Veritas's generated verifiers agree better with official verdicts than those of **seven other harnesses** across all four. +- **Two pairwise comparison benchmarks** — Veritas performs at or near the best on both. +- **Cross-model-family evaluation** — Outperforms vendor's own harness for two of three other model families. +- **Cross-harness improvement** — Improves 9 of 12 harness-benchmark pairs across three different harnesses. + +## Why it matters for our office + +Veritas demonstrates that **specialization beats generalization** in harness design. By focusing exclusively on verifier generation and adding domain-specific mechanisms, it outperforms general-purpose harnesses that try to handle everything. + +The self-improvement loop on authoring principles is particularly compelling — it's a harness that improves how harnesses are designed. This meta-approach could be applied to other aspects of Harness's own evaluation infrastructure. + +[arxiv.org/abs/2611.13336](https://arxiv.org/abs/2611.13336)