Signal

An open-source router that switches between AI models mid-task can preserve task quality while cutting inference costs by up to 90%.

The Problem

AI agents don't make a single model call — they execute multi-step workflows managed by a harness that handles observation, context, tool use, verification, and recovery [§1]. Meanwhile, frontier models are becoming structurally specialized: one model excels at code editing, another at long-context retrieval, another at mathematical reasoning, and yet another at fast, cheap responses [§1]. No single model dominates every axis.

This creates a routing problem. Existing model-routing systems mostly optimize single-turn cost-quality trade-offs — they look at a user's prompt and decide whether to send it to a cheap or expensive model [§2]. But agent execution is multi-step and stateful. A cheap model at step five might cause an expensive correction at step eight. A verification failure at one step changes what model should handle the next step. The router needs to see the full execution state, not just the original question [§1].

The paper frames this gap precisely: prior routers "do not directly model tool failures, context pressure, verifier feedback, recovery attempts, or the accumulated trajectory that determines whether a cheap model remains sufficient at the current step" [§2].

What They Did

The authors propose what they call harness-native agentic routing: at each decision point during agent execution, a router selects which model (or set of models) should act next, based on the full harness state — including the current observation, tool history, recovery status, and verification signals [§3.1].

The system operates in two user-selected modes [§1, Figure 1]. In cost-effective mode, the router picks a single best-fit model for each step — like choosing between a sports car and a minivan depending on whether you're on a highway or hauling groceries. In high-accuracy mode, it selects multiple complementary models whose outputs are combined through ensemble techniques such as voting, verification, or fallback policies.

The concrete implementation, called OpenSquilla, uses a four-layer routing stack [§1]. First, a token-complexity filter catches trivial requests and sends them to cheap models — think of it as a triage nurse directing minor cases away from the ER. Second, a task-type classifier separates requests into coding, reasoning, chat, and tool-use categories. Third, a context-aware refiner incorporates the current harness state. Fourth, a LightGBM ranker — a gradient-boosted decision tree model, open-sourced for cold-start use — selects the cheapest capable model from the remaining candidates.

The paper's central architectural insight is that routing is also a data-producing operation [§1]. Every step-level decision generates a structured record: the query, harness state, model choice, execution trace, outcome, and realized cost. Critically, the labels come from the environment — did the step succeed? what did it cost? — rather than from human annotators. This creates what the authors call a "harness-native data flywheel": better routing lowers cost, lower cost allows more tasks under the same budget, more tasks produce more traces, and more traces train better routers [§1].

The Results

The evaluation covers both routing modes on two agentic benchmarks: PinchBench and DRACO.

For singleton routing, the open LightGBM router preserves end-to-end task quality while cutting realized cost by roughly 90% on PinchBench and about 43% on DRACO, relative to a fixed strong-model baseline [§1]. For multi-model routing, selected complementary proposer sets with aggregation push the DRACO quality frontier upward while often lowering cost [§1].

These are aggregate deployed-policy frontier points rather than per-step breakdowns [§1]. The paper notes that logged arena records provide the basis for route-level decomposition of routing granularity, harness-state signals, and model-pool effects, but the published evaluation stays at the aggregate level [§1].

Several limitations deserve concrete attention. The evaluation covers only two agentic benchmarks — PinchBench and DRACO [§1]. Production agent deployments involve far more diverse task distributions, unpredictable tool environments, and mixed human-AI workflows. The gap between controlled benchmark routing and reliable production routing likely requires substantial additional validation, probably 1-2 years of deployment iteration for teams building on this approach. The paper also does not report the size of the model pool used or name specific models routed between, making it difficult to assess how the results generalize to different model combinations. Additionally, the data flywheel — the paper's central conceptual contribution — is described as a staged path rather than empirically validated end-to-end: the paper argues that traces train better routers but does not report iterative improvement metrics across flywheel cycles [§1].

The open-source boundary is also deliberate and worth noting: the LightGBM cold-start ranker is released, but accumulated arena records and later high-accuracy policy generations are served through the OpenSquilla API [§1]. This means the flywheel's most valuable output — the accumulated routing data — remains proprietary.

Why It Matters

For teams running multi-step AI agents at meaningful scale, inference cost is often the binding constraint on how many tasks can be attempted. A 43-90% cost reduction while preserving quality, if it holds beyond benchmarks, changes the economics of agent deployment directly [§1].

The deeper implication is architectural. The paper argues that routing should not be a thin serving layer but a "meta-decision across harness responsibilities" [§1] — one that also produces structured training data as a byproduct. This positions agentic routing as "not merely cost control, but a data engine for agent-native training" [Abstract]. This is a lab proof-of-concept with promising benchmark numbers, not a production-validated system. But for teams designing agent architectures today, the question of where routing logic lives — and whether it should be harness-aware and step-level rather than query-level — is worth answering now, before the architecture calcifies.

Pruning and regrowing the expert pool in Mixture-of-Experts models before fine-tuning consistently boosts domain performance — up to 6 points on coding benchmarks — without changing model size or inference cost.

The Problem

Mixture-of-Experts (MoE) architectures power many frontier language models, including Qwen3.5, DeepSeek-V4, and Kimi-K2 [§1]. They work by maintaining a large pool of expert sub-networks but activating only a small subset for each token, scaling capacity without proportional compute cost [§3.1]. The catch: when you want to specialize one of these models for a particular domain — math, code, science — you inherit an expert pool shaped by broad pre-training.

That pool isn't homogeneous. Pre-training on diverse data encourages experts to develop different functional specializations, making their relevance uneven under any single target distribution [§1]. Some experts have high relevance to your domain; others contribute little but still receive non-trivial routing. Standard supervised fine-tuning (SFT) updates these inherited experts in place but never asks whether the expert slots themselves should be reallocated [§3.1]. The result: part of the model's fixed capacity remains poorly aligned with the target domain.

The question the authors pose is direct: can you reorganize the expert pool into a more domain-aligned starting point before fine-tuning, so that more expert slots become useful? [§1]

What They Did

UMoE (Unlocking MoE Capacity) is a three-step pipeline that runs before standard fine-tuning [§3.1].

**Step 1: Prune.** Score every expert's relevance to the target domain using a small calibration subset of the training data, then drop the bottom half per layer. The scoring method, called REAP saliency, measures how large an expert's output is when it's actually selected, weighted by how strongly the router favors it [§3.2]. Think of it as measuring not just how often an expert gets called on, but how much useful work it does when it shows up. Experts that produce small outputs even when selected score low and get cut.

**Step 2: Expand.** Regrow the pool back to its original size from the surviving experts. Each freed slot gets initialized as a copy of a surviving expert, assigned round-robin. Then — and this is critical — both the original expert and its copy receive independent random perturbations to their weights [§3.3]. This symmetric perturbation prevents a problem the authors identified in ablations: if you only perturb the copy, the unchanged original acts as a "privileged anchor" that absorbs most routing during subsequent training, and performance degrades [§5.1]. The perturbation is applied to both the expert's internal weights (the MLP) and its router entry, providing complementary sources of functional and routing diversity [§3.3].

**Step 3: Fine-tune.** Apply standard SFT on the domain data. The reorganized model has the exact same expert count, parameter count, and inference cost as the original — nothing changes at deployment [§3.1].

The entire pipeline uses a single frozen recipe with noise scale σ = 0.05 and 50% pruning ratio, with no per-domain hyperparameter tuning [§3.3, Abstract].

The Results

UMoE was tested on Qwen3-30B-A3B across five domains — math, code, science, tool-use, and agentic coding — spanning 12 benchmarks [Abstract].

The headline numbers: UMoE improves math average accuracy by 3.4 points and SWE-bench Verified (agentic coding) by 6.0 points over direct SFT [Abstract]. On a strong in-house math corpus where direct SFT already surpasses Qwen3-30B-A3B-Thinking (82.81 vs. 81.06), UMoE pushes the average to 84.17 — an additional 1.36 points on top of an already strong baseline [Abstract]. This matters because it shows the technique isn't just helping weak baselines; it adds value even when the training data is already high-quality.

Data-scaling experiments show the gain persists as training data grows, rather than washing out with more data [Abstract]. Architecture transfer was validated on Qwen3.5-35B-A3B in the math domain [§1].

Analysis reveals why this works: in the direct-SFT model, substantial routed-expert compute goes to a low-saliency subset that can be removed post hoc with little average degradation [Abstract]. These experts are essentially dead weight that still consumes routing slots. UMoE converts this redundant capacity into useful domain capacity, achieving lower training loss with gains spanning all difficulty levels [Abstract].

Ablations confirm design choices matter. REAP saliency and activation frequency perform comparably for pruning, while unweighted activation norms are weaker [§5.1]. For expansion, perturbing both sides (parent and copy) outperforms one-sided perturbation, and perturbing both router and MLP weights gives the best average result [§5.1].

**Limitations worth noting:** The evaluation covers two MoE architectures from the same model family (Qwen3 and Qwen3.5) [Abstract]. Whether the fixed recipe transfers to architecturally different MoE models — say, DeepSeek's multi-head latent attention variants — remains untested. For teams considering adoption, this means the technique is promising but should be validated on your specific architecture before committing to production pipelines. The fixed 50% pruning ratio also hasn't been explored across a wide range; different domains or architectures might benefit from different ratios, which would complicate the "no tuning needed" claim.

Why It Matters

This is a lab-validated technique at the "pattern emerging" stage of maturity. The core insight — that MoE expert pools carry domain-irrelevant dead weight that standard fine-tuning never addresses — is demonstrated concretely across multiple domains and benchmarks [Abstract].

For practitioners fine-tuning MoE models on specific domains, UMoE offers a pre-processing step that costs nothing at inference time and requires no architectural changes. The frozen recipe (50% prune, symmetric perturbation at σ = 0.05) means it can be dropped into existing SFT pipelines with minimal engineering overhead [§3.3]. If your organization is building domain-specialized models from MoE base models, this technique is worth benchmarking against your current direct-SFT approach — particularly given that gains persist even with strong training data [Abstract].

An autonomous AI agent exploited all 104 web applications in a standardized security benchmark — including challenges that stumped experienced human pentesters.

The Problem

Autonomous exploitation — an AI agent that finds and weaponizes a vulnerability without step-by-step guidance — is one of the hardest tests of AI tool use. As the authors put it: "Detection is easy to fake; exploitation is not: either the flag comes out of the running application, or it does not" [§I-C].

The XBOW validation-benchmarks provide 104 self-contained web applications, each seeded with a single hidden flag that can only be retrieved by actually exploiting a specific bug [§II-A]. The vulnerabilities span 26 classes — SQL injection, server-side template injection, insecure deserialization, request smuggling, padding oracles, race conditions, and more — across three difficulty tiers [§II-B]. Each application generates a cryptographically fresh, random flag on every build, so an agent cannot pass by memorizing answers [§I-B].

Prior AI systems had gotten close: the best published result before this work was 103/104 from KinoSec, while a combined team of five human pentesters managed 91/104, and a single staff-level pentester solved roughly 61/104 [Table II]. The question was whether full coverage was achievable, and what architecture could get there.

What They Did

Mako is built around what the authors call a Self-Evolving Agentic Operating System (SE-AOS) — an agent architecture where the set of exploit tools isn't fixed at deployment but grows during operation [§III]. Think of it like an operating system that can write and install its own device drivers: when Mako encounters a vulnerability it can't exploit with existing tools, it diagnoses the failure, writes a new exploit primitive, validates it against the live target, and loads it into its toolkit for future use [§III].

The core agent runs a bounded loop of perceive → reason → act, with a default budget of 25 turns per target [§III-A]. It uses Google's Gemini 2.5 Flash as its primary reasoning model, escalating to Gemini 3.1 Pro Preview only when it has burned through 70% of its turn budget without a confirmed finding [Table I].

The system maintains roughly 180 registered tools, with about 50 exploit and detection tools newly built or enhanced during this campaign [Table I]. The self-evolution loop follows a closed cycle: diagnose failure → author new capability → validate against a live target → chain into exploit sequences → re-verify [§I-C].

Critically, verification never trusts the model's self-report. A fresh random flag is injected into each application at build time via a Docker build-arg. A solve counts only when that exact flag string appears in a genuine application response captured by a tool — not in the agent's own output [§I-B]. The authors call this "fabrication-proof evaluation" [§I-A].

The Results

Mako solved 104 out of 104 benchmarks — full-suite coverage across all 26 vulnerability classes and all three difficulty tiers [Table I]. For context, the next-best AI system (KinoSec) achieved 103/104, BoxPwnr reached 101/104, and a combined team of five human pentesters solved 91/104 [Table II]. A single experienced human pentester with 20 years of experience, given 40 hours, solved approximately 85 [Table II].

The turn economics are striking. The median solve took 7 agent turns. The hardest difficulty tier (L3) had a median of just 2 turns — fewer than the easy tier's 7.5 [Table III]. The authors argue this inversion is the central finding: "well-described tools collapse the hardest challenges into one-call solves" [§I-C]. Once Mako had the right exploit primitive available, even complex vulnerabilities fell to a single tool invocation.

Total API cost for the entire 104-benchmark campaign was $478.99, approximately $4.61 per benchmark, processing roughly 1.03 billion tokens [Table I].

Several limitations deserve attention. The benchmark consists of containerized CTF-style applications with single injected vulnerabilities — production web applications involve layered defenses, complex authentication, and interacting components that make exploitation substantially harder. The timeline to reliable autonomous exploitation of real-world production systems remains unclear. The authors also note that per-turn model selection (Flash vs. Pro) was not separately logged, so the escalation model's exact contribution is unconfirmed [Table I]. And because the authors withheld all payloads, exploit chains, tool source code, and operational details citing dual-use concerns [§I-B], independent reproduction is currently impossible — reviewers must trust the verification methodology rather than verify results directly.

Why It Matters

The authors frame their central finding as a "law of autonomous exploitation: once a capability exists and is discoverable, difficulty collapses; capability, not reasoning, is what is scarce" [Abstract]. The L3 inversion — hard problems solved in fewer turns than easy ones — provides concrete evidence for this claim [Table III].

This is a lab proof-of-concept on standardized targets, not a production-ready offensive tool. But the implications are directional. At under $5 per target and median 7 turns, the economics of comprehensive vulnerability scanning shift dramatically if these results generalize beyond CTF environments. For security teams, the finding that a $479 AI campaign outperformed a team of five human pentesters on the same benchmark [Table II] suggests that continuous AI-driven offensive testing is approaching practical viability for known vulnerability classes. Organizations building security programs should be tracking this capability trajectory — not because it's deployable today against real infrastructure, but because the gap between controlled benchmarks and production targets is narrowing faster than most defensive roadmaps assume.

A harmful payload can be split across multiple AI agents so that every per-message safety check passes — and no stronger local classifier can fix this.

The Problem

Modern AI systems increasingly involve multiple LLM agents working together: a planner assigns subtasks, specialized agents fetch data or write content, and a final agent assembles the result [§1]. The standard safety approach is a runtime monitor that checks each message, tool call, or step individually — asking whether that one object looks malicious [§1].

But what if the attack isn't in any single message? A distributed backdoor splits a harmful payload across agents. Each fragment looks like ordinary text — "data retrieval complete; 247 entries logged" — but when assembled, the fragments reconstruct executable code that exfiltrates credentials [Figure 1]. The monitor checks each fragment before the pieces combine, so at check time there is no attack to see [§1].

The question isn't whether splitting defeats monitors (sometimes split fragments still leak suspicious tokens). The harder question is: when each fragment is genuinely benign-looking and statistically indistinguishable from normal traffic, can any local monitor catch the attack? [§1]

What They Did

The researchers formalized this as an **observability boundary** — a monitor catches only what its view can tell apart from benign traffic [§2]. They define two conditions that make a local monitor blind. First, *local benignness*: each fragment scores as safe, while the assembled object is harmful [Definition 1]. Second, *local indistinguishability*: the statistical distribution of attack fragments, as seen through the monitor's view, is within some small distance ε of benign traffic [Definition 2].

Think of it like a jigsaw puzzle where each piece looks like a plain blue sky tile. No single tile is suspicious. But assembled, they form a picture of something dangerous. If the monitor only ever sees one tile at a time, and that tile is statistically identical to a real sky tile, no amount of squinting helps.

They prove this formally: when attack and benign fragments are within ε in total variation distance (a standard measure of how different two distributions are), any detector seeing only one fragment has a detection advantage — true positive rate minus false positive rate — of at most ε [Proposition 1]. The fix isn't a better classifier; it's changing what the monitor sees.

To test this, they built a testbed with a single dial: how much of the attack remains visible in a single fragment. At one end, fragments still carry clues like a data path from source to sink. At the other end, each fragment reads like ordinary text [§1]. They tested across four evaluation beds: R-Judge, Collaborative Shadows (their main testbed), AgentDojo (an external benchmark with its own trained prompt-injection detector), and DemonAgent [§2].

They compared six monitor types along two axes: whether the monitor sees the assembled object, and whether it sees that object decoded. These range from per-step monitors that see one fragment, to a decoded-view gate that sees the assembled output after decoding the encoding scheme [Table 1].

The Results

The pattern is consistent across all four evaluation beds. As fragments become locally benign, per-step monitors drop to chance-level detection — 0.5 AUROC, meaning they literally cannot distinguish attack fragments from benign ones [Abstract, §2]. This happens regardless of whether the monitor is a statistical detector, a trained prompt-injection classifier, or a taint-tracking system that follows data paths [§1].

Detection returns only when the monitor's observation window expands to include the assembled object. A monitor trained only on benign traffic recovers the attack's code structure across held-out encodings with 0.874 mean AUROC [Abstract]. The decoded-view gate — which sees the assembled output after decoding — blocks every tested attack when calibrated at a 5% false positive rate on benign traffic [Abstract, §2].

But a wider view alone isn't sufficient. Full-trace monitors and decoders still fail unless they reach the specific representation where the payload is exposed [§1]. A codec-blind monitor that sees the raw assembled text but doesn't know the encoding scheme misses attacks that use unfamiliar encodings. The decoded-view gate works only when given the encoding family [Table 1].

The technical limitation is clear: the testbed focuses on structured code-payload assembly — fragments that become executable programs [§2]. Whether the same boundary applies to non-code harms (social engineering, misinformation assembled from innocuous claims) is an open question. For practitioners, this means the formal guarantees hold for code-execution risks today, but extending monitoring to other compositional harms likely requires solving what the authors call "representation discovery" — figuring out what assembled form to check [§1]. That problem has no current solution.

The evaluation also uses controlled conditions with known encoding families. Production multi-agent systems involve arbitrary tool outputs, mixed human-AI content, and encoding schemes the defender may not anticipate — a significantly harder setting where even the decoded-view gate's guarantees would not directly apply.

Why It Matters

This work is a lab proof-of-concept with formal backing, not a production tool. But the core finding is structural, not empirical: per-message monitoring has a provable ceiling against compositional attacks, and no amount of classifier improvement can overcome it [Proposition 1].

For teams building or evaluating runtime safety for multi-agent LLM systems, the practical takeaway is specific: monitor the assembled output, not just the fragments. And monitor it in the right representation — raw text isn't enough if the payload is encoded. The decoded-view gate demonstrates this is feasible when the encoding family is known, but the open problem is detecting the encoding when you don't know what to decode [§1]. Until that's solved, the gap between local safety and global safety in multi-agent systems remains real and formally bounded.

AI agents sharing the same tool server no longer need to independently rediscover how each tool works — a provider-side memory graph lets them learn once and reuse everywhere, improving task completion by up to 21.61% [Abstract].

The Problem

Modern AI agents increasingly call external tools through shared servers, particularly under the Model Context Protocol (MCP), where providers like GitHub and Stripe expose capabilities that many different agents consume [§1]. The problem: when Agent A figures out that a filesystem tool silently truncates files over 10MB, that hard-won knowledge stays locked inside Agent A's memory. Agent B has to discover the same limitation from scratch.

This happens because existing approaches store tool-use knowledge on the agent side — in the agent's parameters, prompts, or memory logs [§1]. The knowledge gets entangled with each agent's specific workflow, action format, and reasoning style, making it impossible to share. The authors call this the "Generalization Dilemma" [§1, Figure 1(A)].

There's a second problem they call "Capability Blindness": agent-side memory only records tool behaviors that past tasks happened to exercise. If no task ever tested what happens when you pass a negative offset to a pagination tool, that boundary condition remains unknown. The memory reflects the task distribution, not the tool's actual capability frontier [§1, Figure 1(B)].

What They Did

ToolAtlas flips the ownership model. Instead of each agent maintaining its own tool knowledge, the tool provider maintains a shared memory graph that any agent can query [§1]. The framework has three stages: bootstrapping, exploration, and inference-time traversal [§3].

**Bootstrapping** starts by generating seed tasks for each tool exposed by an MCP server. Since tool specifications are often too terse to support diverse task generation, the system also pulls real-world usage snippets via web search [§3.1]. Each seed task runs multiple times, and the results — both successes and failures — get distilled into a graph. Critically, the distillation strips out agent-specific details (retry patterns, action syntax, reasoning quirks) and keeps only tool-level knowledge: which tool was called, why, and what happened [§3.1].

The graph has three interlinked layers. A Tool-Trace Graph stores condensed execution traces as sequences of (tool, rationale) pairs. A Tool-Capability Graph attaches "affordance" entries (what a tool can do) and "boundary" entries (where it fails) to each tool, plus edges connecting tools that get used together. A Tool-Strategy Graph captures recurring orchestration patterns across many tasks [§1, §3.1].

**Capability Exploration** is what addresses the blindness problem. Rather than passively waiting for agent trajectories, an LLM examines the current graph for gaps and generates targeted probes — pushing outward to test untested boundary conditions and filling inward to verify unrecorded affordances and tool compositions [§3]. Each probe runs against the live tool, and verified outcomes fold back into the graph, sharpening the next round of exploration.

At inference time, a lightweight navigator traverses the graph adaptively, expanding relevant regions step by step rather than retrieving a fixed top-k set of entries, then compresses the gathered information into task-conditioned guidance for the requesting agent [§3].

The Results

Across two MCP-based benchmarks (MCPMark and MCP-Universe) spanning eight services, ToolAtlas outperforms existing tool-side optimization and agent-side memory baselines by up to 21.61% in pass@1 and 18.61% in pass@4 [Abstract].

The transferability results are where the provider-side argument gets concrete. The same tool memory graph, built once, transfers across different environment instances with up to 24.16% relative gain in pass@1 and 16.22% in pass@4 [Abstract]. It also transfers across entirely different agent frameworks — without retraining or any task-time exploration — yielding up to 17.49% and 14.27% relative gains in pass@1 and pass@4, respectively [Abstract].

Ablation studies confirm that the gains come from combining two specific design choices: organizing memory around tools rather than tasks, and using capability-guided execution probing rather than passive trajectory collection [Abstract]. Removing either component degrades performance.

The evaluation covers eight MCP services, which is a meaningful but bounded scope. The benchmarks test controlled task scenarios rather than messy production codebases where multiple tools interact in unpredictable ways — reliable deployment in complex real-world MCP ecosystems likely requires further validation, probably 1-2 years of iteration. The framework also depends on an LLM to generate exploration probes and distill traces, meaning the quality of the tool memory is bounded by the capability of the LLM used during construction — a dependency the paper does not extensively stress-test across weaker models.

Why It Matters

This is a lab proof-of-concept, but it validates a structural argument: in shared-server deployments, tool knowledge should travel with the tool, not with each agent that calls it. The MCP ecosystem is growing — GitHub and Stripe already host official MCP servers [§1] — and the number of downstream agents per server will only increase.

If you're building or hosting MCP tool servers, the practical takeaway is that investing in provider-side tool memory could eliminate redundant per-agent exploration. The transferability results [Abstract] suggest this isn't just a performance trick for one agent — the same graph helps different agents in different environments. Code is public at the linked repository [Abstract], making it possible to evaluate against your own tool servers today.