Harness Issue #43: Verifier Generation, LiveWorkspace & Benchmark Saturation

blog/src/posts/office-42-part5-bad-evolve.md

@@ -0,0 +1,46 @@
+---
+title: "When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization"
+seoTitle: "BadEvolve — Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization (Nov 8)"
+description: "BadEvolve plants backdoors by poisoning evolution tasks, not skills. It diagnoses failures across 4 stages and uses rollback + failure memory to preserve progress."
+date: 2026-11-08
+category: research
+categoryLabel: Research
+type: Technical
+tags: ["Security", "Backdoor Attacks", "Self-Evolving Agents", "Poisoning"]
+---
+
+When agents improve themselves by distilling execution experience into persistent, reusable skills, they create a new attack surface: malicious rules implanted into those skills compromise all future agent behavior. But what if the attacker can't directly manipulate the skills?
+
+*Best suited for:* Anyone deploying self-evolving agents, or building systems where agents generate their own skills from execution experience.
+
+## The finding
+
+The authors study a novel attack vector: **task-level backdoor attacks**, where the attacker can only inject poisoned evolution tasks — not the skills themselves. The agent constructs its own skills internally through a trusted evolver, so direct skill manipulation is impractical.
+
+The attacker's strategy: iteratively optimize poisoned tasks based on attack success, inducing the agent to implant backdoors into its own skills.
+
+**BadEvolve** addresses two key challenges:
+1. **Indistinguishable failures** — failures at different stages (induction, activation, following, effect) produce similar outcomes, giving little guidance for targeted updates.
+2. **Undermining progress** — unsuccessful updates may erase earlier progress, while discarding them leads to repeated ineffective attempts.
+
+Their solution:
+- **Four-stage failure diagnosis** (Induction, Activation, Following, Effect) directs updates toward the failing stage.
+- **Rollback** preserves earlier progress when an update fails.
+- **Failure memory** avoids repeating ineffective updates.
+
+Extensive experiments across diverse self-evolution frameworks, benchmarks, and backbone LLMs demonstrate BadEvolve's effectiveness. Code is available at [github.com/ventr1c/BadEvolve](https://github.com/ventr1c/BadEvolve).
+
+## The twist: the attacker doesn't need skill access
+
+The most insidious part of this attack is that the attacker never touches the victim's skills directly. They only poison the **tasks** used to evolve skills. The agent, trusting its own evolution process, implants the backdoor itself. It's like poisoning a recipe book and watching people cook your poison into their meals.
+
+## Why it matters for our office
+
+This paper reveals a fundamental vulnerability in self-evolving agent systems: **the evolution process itself is the attack surface**.
+
+Key takeaways:
+1. **Trust no evolution task** — even tasks generated by trusted processes can be poisoned if the input space is adversarial.
+2. **Skill persistence amplifies damage** — a backdoor in a skill affects every future execution, making the impact compounding.
+3. **Defense requires evolution auditing** — you need to verify not just the skills, but the tasks that produced them.
+
+[arxiv.org/abs/2611.10542](https://arxiv.org/abs/2611.10542)

blog/src/posts/office-42-part5-infrathon.md

@@ -0,0 +1,40 @@
+---
+title: "Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization"
+seoTitle: "Infrathon — Benchmarking Language Agents for Long-Horizon AI Infra Optimization (Nov 8)"
+description: "150 runs, 10,000+ GPU-hours; agents beat human-designed KV-cache codecs. Claude Opus 5.5 improved compression from 6.82% to 19.86%. Shorter evaluations miss optimization gains and ranking shifts."
+date: 2026-11-08
+category: research
+categoryLabel: Research
+type: Technical
+tags: ["Agents", "Infrastructure Optimization", "Benchmarking", "Long-Horizon"]
+---
+
+Frontier agents can continually improve real-world AI infrastructure over long horizons — and they can discover solutions that outperform human-designed baselines. But shorter evaluations miss both the optimization gains and the ranking shifts that happen over time.
+
+*Best suited for:* Anyone running agents on infrastructure tasks, or evaluating agent benchmarks that use short time horizons.
+
+## The finding
+
+The authors introduce **Infrathon**, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. They evaluated six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours.
+
+Two key results:
+- **Model rankings vary with evaluation horizon.** A model that looks best at 2 hours isn't necessarily the best at 24. Shorter evaluations can mislead you about which agent is actually most capable.
+- **135 of 150 runs achieved further improvements during the second half** of their budgets. Agents keep getting better — you just have to wait long enough to see it.
+
+The standout: **Claude Opus 5.5** increased the mean KV-cache compression ratio from 6.82% (human-designed Quant-VideoGen codec) to **19.86%** — nearly 3x the human baseline.
+
+## The twist: agents find things humans don't
+
+The agents didn't just optimize within known solution spaces — they discovered solutions that outperformed human-designed baselines. This isn't incremental improvement; it's discovery.
+
+The authors also identified three barriers to sustained improvement: task violations, unreliable testing, and premature completion. Agents that hit these walls stopped improving — even when they had budget left.
+
+## Why it matters for our office
+
+If you're evaluating agents with short time horizons, you might be ranking them wrong. Infrathon shows that:
+
+1. **Longer evaluations reveal different winners** — don't trust 2-hour benchmarks for long-horizon tasks.
+2. **Agents keep improving** — give them time, and they'll find optimizations you didn't think of.
+3. **The barriers are structural** — task violations, unreliable tests, and premature completion stop agents cold. Fix those first.
+
+[arxiv.org/abs/2611.11995](https://arxiv.org/abs/2611.11995)

blog/src/posts/office-42-part5-kvm-agent.md

@@ -0,0 +1,44 @@
+---
+title: "Coding Agents in KVM: Practical Containment and User Control with Limited Resources"
+seoTitle: "KVM-Agent — Coding Agents in KVM, Practical Containment with Limited Resources (Nov 8)"
+description: "An open-source implementation that gives agents broad authority inside replaceable Ubuntu VMs while keeping host administration outside. Balances autonomy, containment, and user control."
+date: 2026-11-08
+category: research
+categoryLabel: Research
+type: Technical
+tags: ["Security", "Containment", "KVM", "Coding Agents", "Infrastructure"]
+---
+
+Coding agents need broad authority to be productive — but granting that authority exposes users' data and infrastructure. Cloud workspaces help contain agent activity, but relying on provider-specific platforms creates vendor lock-in and reduces user control.
+
+*Best suited for:* Teams running coding agents on bare metal or private infrastructure who want containment without cloud-vendor lock-in.
+
+## The finding
+
+The authors analyze the trade-offs between agent autonomy, enforced containment, and user control — then present **KVM-Agent**, an open-source implementation that automates most setup and provisioning steps.
+
+The architecture:
+- Agents get **broad authority inside replaceable Ubuntu VMs**.
+- Host administration stays **outside those VMs** — the host is off-limits to agents.
+- Supports multiple coding-agent applications, model services, and remote job delegation.
+
+The approach retains familiar workflows and user choice at the cost of VM resources, maintenance overhead, and residual risks from secrets accessible inside the VMs and permitted network access.
+
+## The trade-off triangle
+
+The paper identifies three competing goals:
+1. **Agent autonomy** — agents need broad permissions to be productive
+2. **Enforced containment** — you need hard boundaries so agents can't escape
+3. **User control** — users should choose their tools, models, and infrastructure
+
+KVM-Agent resolves this by making VMs **replaceable** rather than immutable. If an agent misbehaves, you destroy the VM and spin up a fresh one — no forensic cleanup needed.
+
+## Why it matters for our office
+
+This paper is the practical answer to "how do I let agents code without giving them root on my actual machines?"
+
+The key insight: **make the boundary disposable**. Don't try to perfectly restrict what an agent can do inside a persistent environment — instead, make the entire environment replaceable. This is simpler, more secure, and actually works at scale.
+
+The residual risks (secrets inside VMs, permitted network access) are real but manageable — they're the same risks you'd have with any cloud workspace, just without vendor lock-in.
+
+[arxiv.org/abs/2611.12823](https://arxiv.org/abs/2611.12823)

blog/src/posts/office-42-part5-reins.md

@@ -0,0 +1,51 @@
+---
+title: "REINS: A Self-Evolving Agent Harness for Real-Time Trajectory Planning"
+seoTitle: "REINS — Self-Evolving Agent Harness for Real-Time Trajectory Planning (Nov 8)"
+description: "A harness with verified-rule memory achieving 1.9–4.6 ms latency, 98%+ compliance, and nearly 50% collision reduction. LLM calls decline as memory matures."
+date: 2026-11-08
+category: research
+categoryLabel: Research
+type: Technical
+tags: ["Agents", "Trajectory Planning", "Self-Evolving", "Real-Time"]
+---
+
+Agent harnesses — runtime scaffolds that wrap an LLM's reasoning loop with tool dispatch, verification, and persistent memory — are the standard for deploying LLMs as autonomous agents. But in trajectory planning, this paradigm hasn't taken hold: existing methods either keep the LLM in the decision loop (too slow for millisecond control budgets) or offload it to offline rules (non-evolving).
+
+*Best suited for:* Teams deploying LLM-based agents in real-time control systems where latency matters and the agent should improve over time.
+
+## The finding
+
+The authors propose **REINS**, a runtime harness that lets an LLM planner meet real-time constraints while self-evolving into a deployable system. It builds on three LLM-planner couplings:
+
+1. **Grounding** — LLM outputs are grounded to a learned variable-level dynamics model.
+2. **Verification** — calibrated predicates and forward rollout verify outputs before execution.
+3. **Consolidation** — verified outputs are stored in an indexed rule memory that serves future scenes in O(log n) without re-invoking the LLM.
+
+The self-evolution loop:
+- **Familiar scenes** resolve by memory lookup (fast, no LLM needed).
+- **Novel scenes** invoke the LLM, whose verified outputs enrich the memory.
+
+Results without modifying LLM weights:
+- **98%+ compliance** on simulated and real-world benchmarks.
+- **Nearly 50% collision reduction**.
+- **1.9–4.6 ms latency** at the tactical decision layer.
+- **LLM invocation rate monotonically declines** as memory matures — the system gets faster and smarter over time.
+
+Code: [github.com/ZhihongCui/REINS](https://github.com/ZhihongCui/REINS)
+
+## The twist: the LLM becomes optional over time
+
+The most elegant part of REINS is that it makes the LLM **gradually optional**. As the rule memory matures, most scenes are resolved by lookup alone. The LLM is only invoked for genuinely novel situations — and its verified outputs immediately enrich the memory for next time.
+
+It's the digital equivalent of a mechanic who gets faster at fixing cars because they've seen every variation before. But unlike the mechanic, REINS never forgets a verified solution.
+
+## Why it matters for our office
+
+REINS solves the fundamental tension in agent design: **you need the LLM's reasoning for novel situations, but you can't afford its latency for routine ones.**
+
+Key takeaways:
+1. **Memory is the speed advantage** — once a solution is verified, it's cached and served in O(log n) without re-invoking the LLM.
+2. **Verification before consolidation** — only verified outputs enter memory, preventing bad habits from being learned.
+3. **Self-evolution is measurable** — the declining LLM invocation rate proves the system is genuinely improving, not just getting faster at the same mistakes.
+
+[arxiv.org/abs/2611.10748](https://arxiv.org/abs/2611.10748)

blog/src/posts/office-42-part5-shadow-ai.md

@@ -0,0 +1,42 @@
+---
+title: "Detecting Shadow AI: Unsanctioned LLM Agents on Enterprise Endpoints"
+seoTitle: "Detecting Shadow AI — Unsanctioned LLM Agents on Enterprise Endpoints (Nov 8)"
+description: "An open-source Windows detector that anchors behavior to the identity of remote services, not just process structure. Flags every behavioral twin that behavior-only rules miss."
+date: 2026-11-08
+category: research
+categoryLabel: Research
+type: Technical
+tags: ["Security", "Shadow AI", "Enterprise Detection", "Windows"]
+---
+
+Autonomous AI agents make shadow IT grow exponentially — not because they're malicious, but because building tools with them is nearly free, and the artifacts they leave behind (scripts, automations, MCP servers) are themselves unsanctioned.
+
+*Best suited for:* Security teams managing endpoints where agents might be deployed without approval, and anyone worried about shadow AI compounding over time.
+
+## The finding
+
+The authors argue that **shadow IT grows exponentially** when agentic artifacts reproduce faster than the organization retires them. The key insight: you can't stop shadow AI by detecting behavior alone — because the same program talking to an AI service looks identical to the same program talking to a non-AI service.
+
+They propose **Agent Detector**, an open-source Windows detector that works differently:
+- It anchors behavior to the **identity of the remote service**, not just process structure.
+- It correlates connections with configuration, installation, and execution history.
+- It uses a catalog of **39 tools and 184 LLM endpoint entries**.
+- It notifies users through an acknowledgment notice rather than blocking.
+
+On a controlled benchmark of 12 synthetic agents, 3 AI components, and 9 benign workloads (repeated 5 times), **two behavioral rules flagged every behavioral twin, while Agent Detector flagged none** — proving that behavior-only detection misses agents entirely.
+
+## The twist: temporal limits, not structural ones
+
+Agent Detector's limitation isn't that it can't detect agents — it's that detection is **temporal**. A connection to a provider behind a shared CDN can only be attributed while the DNS cache holds the provider's name (1–5 minutes). An intermittent agent was seen by about one scan in three.
+
+Each scan costs ~154ms, so you can run them frequently — but you still need to catch the agent while it's actively connecting.
+
+## Why it matters for our office
+
+This paper makes three critical points:
+
+1. **Agents compound shadow IT.** Every agent leaves behind tools that become new agents' building blocks. The problem grows exponentially unless you detect and retire artifacts faster than they reproduce.
+2. **Behavior-only detection fails.** You can't distinguish an AI agent from its "behavioral twin" (the same program talking to a non-AI service) by looking at process structure alone.
+3. **Identity-based detection works.** Anchor to the remote service's identity, correlate with installation history, and you'll find agents that behavior-only rules miss.
+
+[arxiv.org/abs/2611.12377](https://arxiv.org/abs/2611.12377)

blog/src/posts/office-42-part5-skillcoder.md

@@ -0,0 +1,53 @@
+---
+title: "SkillCODER: Towards Auditing and Attribution of Copyright Infringement in LLM Agent Skills"
+seoTitle: "SkillCODER — Auditing and Attribution of Copyright Infringement in LLM Agent Skills (Nov 8)"
+description: "99.2% owner verification and 99.3% buyer Top-1 attribution accuracy. Embedded signals remain robust under paraphrase, compression, clause deletion, and reorganization."
+date: 2026-11-08
+category: research
+categoryLabel: Research
+type: Technical
+tags: ["Security", "Copyright", "Agent Skills", "Watermarking", "Attribution"]
+---
+
+Agent skills — reusable workflows and task-specific procedures packaged as deployable natural-language artifacts for LLM agents — are valuable but vulnerable to unauthorized redistribution and repackaging. Existing prompt copyright auditing methods struggle with long skills and introduce token-level perturbations that compromise readability.
+
+*Best suited for:* Organizations selling or distributing agent skills who need to verify ownership and attribute unauthorized use.
+
+## The finding
+
+The authors propose **SkillCODER** for copyright auditing and buyer attribution in a black-box, content-only setting. The system works in three phases:
+
+**Embedding:**
+1. Parse a skill into a **section-aware intermediate representation (SkillIR)** to identify structural anchors.
+2. Distribute conditional instructions across these anchors to form an **Artifact-Gated Capsule** — designed to activate under verification queries and remain dormant otherwise.
+3. Use **Controlled-Vocabulary Error-Correcting Codes** to encode buyer-specific fingerprints through controlled lexical choices within the capsule.
+
+**Verification:**
+1. Issue paired positive and negative probes.
+2. Extract capsule evidence from the skill's response.
+3. Recover ownership and buyer identity through error-correcting decoding.
+
+Results across three domains, six models, and three agent frameworks:
+- **99.2% owner verification**
+- **99.3% buyer Top-1 attribution accuracy**
+- Preserves task fidelity — embedding signals doesn't degrade the skill's performance.
+
+Embedded signals remain robust under paraphrase, compression, clause deletion, and reorganization. Under five adaptive attacks (including a method-aware LLM sanitizer and multi-buyer collusion), ownership evidence remains reliably detectable while buyer attribution degrades gracefully.
+
+## The twist: the capsule stays dormant until probed
+
+The most clever part of SkillCODER is that the watermark only activates when someone tries to verify ownership. Under normal use, the skill behaves exactly as intended — no performance degradation, no behavioral changes. It's like a security feature that only turns on when someone tries to steal your work.
+
+The Artifact-Gated Capsule design ensures that verification queries trigger the embedded signal, while all other interactions leave it dormant. This makes it nearly impossible for an attacker to detect or remove the watermark without knowing it exists.
+
+## Why it matters for our office
+
+As agent skills become a commodity — bought, sold, and redistributed — this paper provides the infrastructure for accountability:
+
+1. **Ownership is verifiable without modifying skills.** SkillCODER achieves 99.2% verification while preserving task fidelity — the skill works exactly as intended.
+2. **Buyer attribution is precise.** 99.3% Top-1 accuracy means you can identify exactly who redistributed your skill, even after paraphrasing and reorganization.
+3. **Adaptive attacks degrade gracefully.** Even under method-aware sanitization and multi-buyer collusion, ownership evidence remains detectable.
+
+The practical implication: if you're selling agent skills, watermark them with SkillCODER. If someone redistributes your work without permission, you can prove it — and identify the buyer.
+
+[arxiv.org/abs/2611.11308](https://arxiv.org/abs/2611.11308)