Signal

GPT-5 completes only ~44% of tasks when LLM agents face realistic noise, missing tools, and ambiguous instructions — conditions that existing benchmarks conveniently avoid.

The Problem

Most agent benchmarks hand the model everything it needs on a silver platter. They provide pre-selected tools, clean inputs, and fully specified instructions [§1]. But real deployments look nothing like this: user requests are vague, data is messy, and the right tool might not be in the agent's starter kit — it has to go find it [§1].

This creates a dangerous gap. An agent that scores 90% on a benchmark with curated tools and clean data might crater when a user gives it an ambiguous request and a CSV full of duplicates. Prior benchmarks like AgentBench and AgentGym use simulated environments with pre-specified tools and no injected noise [Table 1]. None of them simultaneously test tool discovery, end-to-end data analysis with dirty data, and robustness to distractors [Table 1].

The result: organizations deploying agents based on benchmark performance are flying partially blind about how those agents will handle the mess of production.

What They Did

AgentGym2 is a benchmark framework that strips away the safety nets. Instead of giving agents a curated set of task-specific tools, it provides a basic, general-purpose toolbox covering five categories — web browsing, information retrieval, file processing, multimodal understanding, and code execution — supporting over 27 actions [§1, §3.1]. Agents must figure out which tools to combine and, critically, discover additional tools and services through exploration during the task [Table 1].

The benchmark spans three scenarios: complex tool use, data analysis, and deep search, across over 27 domains ranging from law and economics to entertainment and life sciences [§1, Figure 2]. Task instances are grounded in real user demands curated from platforms like GitHub, Reddit, and Kaggle, then transformed into evaluation instances through automated synthesis, query augmentation, and manual annotation [§1].

What makes this "de-idealized" is threefold. First, tool discovery: necessary tools aren't provided upfront and must be found during interaction — like a developer who needs to locate and install the right library mid-task rather than having it pre-imported [Table 1]. Second, end-to-end data analysis: tasks include the full pipeline from data cleaning (deduplication, normalization) through modeling and prediction, not just the analysis step [Figure 3]. Third, injected distractors: tasks deliberately include noisy information in search queries and dirty or irrelevant content in data files [Table 1].

All tasks go through a cascading verification pipeline combining model-based checks with human review to validate task validity, clarity, and expected outcomes [§1]. The architecture uses environment isolation and parallel tool invocation for reproducible evaluation [§1].

The Results

The headline numbers are sobering. Across 15 proprietary and open-source models, GPT-5 achieved approximately 44% and Claude Sonnet 4.5 approximately 37% [§1, Figure 1]. These are the best-performing models tested — and they fail on more than half the tasks.

For open-source models, post-training makes a measurable difference. Nex-N1-32B and Nex-N1-671B outperformed their respective base models by roughly 9.16% and 10.53% [§1]. This suggests that targeted training on agentic tasks can partially close the gap, though the absolute performance levels remain low.

Scaling model parameters and computational cost produced consistent gains [§1, Figure 1]. But an interesting negative result: increasing the number of interaction turns did not produce a scaling effect [§1]. More rounds of tool use didn't help — agents that couldn't solve a task in a reasonable number of steps generally didn't solve it with more attempts either [Figure 1b].

Fine-grained analysis revealed recurring failure patterns: insufficient exploration (agents not proactively seeking out needed tools) and hallucinations [§1]. These aren't edge cases — they're systematic behaviors that current architectures exhibit under realistic conditions.

The benchmark covers only three scenario types (complex tool use, data analysis, deep search), and while the 27+ domains are broad, production environments involve even more heterogeneous workflows with cross-domain dependencies. Extending coverage to scenarios like multi-agent coordination or long-horizon planning would likely reveal additional failure modes — reliable benchmarking of fully unconstrained agent deployments is probably years away.

Why It Matters

AgentGym2 quantifies something practitioners have suspected: the gap between benchmark performance and production readiness is large. When you remove pre-packaged tools, clean inputs, and fully specified instructions, even the best models fail on the majority of tasks [§1].

This is a lab proof-of-concept benchmark, not a production testing suite — but the findings are directly actionable. If your organization is deploying LLM agents for data analysis, research, or complex tool use, the ~44% ceiling for GPT-5 on realistic tasks [§1] should calibrate expectations. The finding that more interaction rounds don't help [§1] suggests that simply giving agents more time or retries won't compensate for fundamental capability gaps in tool discovery and noise robustness.

The dataset and code are publicly available [§1], making it possible to test your own agent pipelines against these de-idealized conditions before committing to production deployment.

Giving every sub-block of a LoRA adapter its own independent expert router matches the performance of a single global coordinator — while protecting against gradient interference across tasks.

The Problem

LoRA fine-tuning works by adding small, trainable low-rank matrices to frozen model weights. It's efficient, but it has a structural vulnerability: all input channels share a single bottleneck. When inputs have skewed activation profiles — where a few channels carry disproportionate signal — those outlier channels can hijack the gradient updates for the entire adapter. GraLoRA [10] isolated this as "channel dominance," where sparse outlier signals inject destructive gradient entanglement across unrelated input-output pathways [§2.1].

This problem gets worse in multi-task settings. When a model processes interleaved tasks with different statistical profiles, competing gradient trajectories backpropagate through shared sub-blocks and trigger what the authors call "optimization deadlock" [§2.1]. The adapter weights collapse into unspecialized averages [Abstract].

Prior work attacked this from two angles that never met. Spatial partitioning methods like Localized LoRA [2] and GraLoRA [10] split the weight matrix into isolated sub-blocks, limiting how far outlier gradients can propagate — but these layouts are fixed and can't adapt when the task distribution shifts [§1]. Mixture-of-Experts (MoE) approaches like LoRAMoE [5, 6] add conditional routing to select among multiple expert adapters — but treat the underlying weight matrix as a monolith, blind to its internal spatial structure [§1]. No existing framework combines both [§1].

What They Did

The authors propose Localized LoRA-MoE, which fuses spatial block partitioning with dynamic expert routing in two configurations [Abstract].

**Block-Wise LoRA-MoE (Centralized Macro-Routing)** maintains multiple complete block-grid experts, each containing its own set of sub-block adapter pairs across the full spatial grid. A single gating network observes the entire input vector and produces a softmax distribution over experts, weighting their contributions [§2.2]. Think of it as a dispatcher that looks at the whole input and decides which complete "template" of sub-block weights to emphasize.

**Cell-Wise LoRA-MoE (Decentralized Micro-Routing)** eliminates the central dispatcher entirely. Instead, every individual cell (i, j) in the block grid gets its own tiny router — a small gating network that sees only its local slice of the input and independently selects among its own pool of low-rank experts [§2.3]. Each sub-block makes its own routing decision based purely on local features, with no coordination across cells.

The key design constraint: both architectures operate under strict parameter parity with static baselines [§1]. The increased architectural flexibility doesn't come from adding more parameters — it comes from redistributing them across experts and routing networks.

The input and output dimensions are partitioned into Kx and Ky contiguous segments respectively, creating a Ky × Kx grid of coordinate cells. Each cell (i, j) maps input segment j to output segment i through low-rank pairs. The centralized router uses a weight matrix in R^(E×d_in) to gate over E experts globally [Eq. 5], while each decentralized micro-router uses a much smaller matrix in R^(E×(d_in/Kx)) conditioned only on its local input slice [Eq. 7].

The Results

The authors tested across three settings: high-dimensional SVD matrix simulations, real-world tabular transformations, and spatial vision perception under sensor degradation [§3].

The central empirical finding: decentralized cell-level routing achieves "complete statistical parity" with the centralized global router across all benchmarks [Abstract]. This means a collection of tiny, independent, locally-informed routers — each seeing only a fraction of the input — matches a single omniscient coordinator that sees everything. Both configurations outperform static block-partitioned baselines that lack dynamic routing [Abstract].

The authors frame the decentralized architecture as providing a "gradient firewall" — when one region of the input space is corrupted (e.g., by sensor failure in the vision task), the routing in unaffected cells continues operating normally, protecting surviving pathways from fault-propagated corruption [Abstract].

However, the limitations are significant. The benchmarks span synthetic matrix recovery, tabular data, and a vision tracking task — none involve actual LLM fine-tuning [§3]. The paper does not report experiments on language model benchmarks like GLUE, MMLU, or instruction-following tasks. For practitioners building LLM adaptation pipelines, this means the core claims about resolving gradient warfare in LoRA remain validated only in proxy settings. Demonstrating these benefits on transformer-scale language tasks is likely a necessary next step before this architecture could inform production PEFT decisions — plausibly 1-2 years out.

Additionally, the cell-wise architecture multiplies the number of routing networks by the number of grid cells (Ky × Kx routers instead of one), which, while parameter-matched for the adapter weights themselves, introduces inference-time overhead from computing many independent softmax gates that the paper does not quantify.

Why It Matters

The parity result between decentralized and centralized routing is the most interesting finding here. It suggests that for spatially partitioned adapters, global coordination may be unnecessary — local information suffices for effective expert selection [Abstract]. If this holds on actual LLM architectures, it would mean multi-task adapter systems could be built without single-point routing bottlenecks, improving fault tolerance and modularity.

This is a lab proof-of-concept. The architectural pattern — combining spatial isolation with conditional routing at the sub-block level — is well-motivated by documented gradient pathologies in LoRA [10], but its practical value for LLM fine-tuning remains unvalidated. Teams designing multi-task PEFT systems should note the design pattern; teams shipping products should wait for transformer-scale evidence.

AI coding agents internally represent whether their code is correct — and can anticipate the outcome of edits up to 25 steps before making them.

The Problem

When a coding agent tackles a software engineering task, it spends dozens of steps reading files, reasoning, editing code, running tests, and revising. A typical trajectory in this study had a median of 52 steps [§1]. Throughout all of this, the underlying language model maintains internal hidden states that shift with every action — but nobody has systematically asked what those states actually encode about the program being built.

Prior work has shown that language models trained on board games develop internal representations of the game state [§1]. In the code domain, researchers have shown that a model's hidden states can reveal whether a single generated function is correct [§1]. But those studies assume a one-shot setting: the full program sits in context and never changes. A coding agent is different — it iteratively edits a real codebase it can only partially observe, over many steps. Whether agents maintain and update a coherent internal picture of the evolving program has remained an open question [§1].

This matters because if agents do carry such internal representations, those signals could be extracted and used — for example, to detect early that an agent is heading down a dead-end path, without waiting for expensive test execution.

What They Did

The researchers ran two open-weight language models through a coding agent framework (mini-swe-agent) on two established benchmarks: SWE-Bench-Verified and SWE-Bench-Pro [§1]. Each run produced a multi-step trajectory of the agent reading, editing, and testing code.

At each step, they extracted the model's residual stream — think of it as the running internal scratchpad that every transformer layer reads from and writes back to, carrying information forward through the model's computation [§2]. This is the standard target for probing what a model internally represents.

They then trained linear probes — simple logistic regression classifiers — on these hidden states to predict four binary properties of the program at each point in the trajectory [§3.3, Figure 2]:

- **Well-formedness**: Does the code parse or compile? - **Full correctness**: Does it pass the entire test suite? - **Partial correctness**: Does it reduce the number of failing tests? - **Regression**: Does the edit introduce new test failures?

Ground-truth labels were computed by actually checking out each program version and evaluating it [§3.2]. The key design choice: probes are linear, meaning they can only recover information that's already encoded in a straightforwardly extractable way in the hidden states. If a linear probe succeeds, the model isn't just "knowing" something in a deeply entangled way — it's representing it along a readable direction in its internal space.

The critical twist is the "programming horizon" experiment [§2, Definition 3]. Instead of only probing what the model knows about the current program, they trained probes to predict properties of the program k steps into the future — before those edits are written to disk. This tests whether the agent is planning ahead.

The Results

Probes decoding the current program state achieved AUC up to 0.83 for full correctness, well above a shuffled-label control baseline [Abstract]. The signal was strongest in the model's intermediate layers and persisted throughout the trajectory [§1].

The more striking result: probes trained to predict future edit outcomes succeeded above chance up to roughly k=25 steps in advance [Abstract, §1]. This means the model's current hidden state already carries information about programs it hasn't written yet — what the authors call the "latent programming horizon" [Definition 3]. The authors frame this as the coding analogue of learned look-ahead observed in game-playing networks [§1].

As a validation check, probes trained on one benchmark transferred to the other without retraining and still performed above chance [Abstract]. This cross-benchmark transfer suggests the representations capture genuine program properties rather than benchmark-specific artifacts.

Several limitations deserve attention. The study uses two open-weight models on two benchmarks with trajectories of median 52 steps [§1] — this is a controlled academic setting. Real-world coding agents operate on larger, messier codebases with longer trajectories and more diverse tool usage. The probes are also trained and evaluated on the same agent framework (mini-swe-agent), so it's unclear how well findings generalize to different agent architectures. For practitioners, this means the core finding is solid as a research result but extracting these signals in production pipelines would require substantial additional engineering and validation — likely several years out.

The AUC of 0.83 for correctness, while meaningfully above chance, also means the probes are far from perfect. A system relying on these signals for real decisions (like terminating an agent run early) would face non-trivial false positive and false negative rates.

Why It Matters

This work sits firmly at the lab proof-of-concept stage, but it opens a concrete direction. If coding agents already internally track whether their code works — and have some foresight about where their edits are heading — those signals are potentially actionable. Teams building AI coding pipelines could eventually use probe-like mechanisms to monitor agent confidence in real time, flag trajectories that are unlikely to succeed, or allocate compute more efficiently by stopping early.

The cross-benchmark transfer result [Abstract] is particularly encouraging: it suggests these aren't fragile, dataset-specific patterns but something more fundamental about how code models represent programs during iterative editing. For anyone working on AI agent reliability or interpretability, this paper provides a concrete experimental protocol and evidence that there's meaningful signal to extract from the internals of coding agents.

A single misleading document in your RAG pipeline can override five accurate ones — but cross-referencing claims across sources before generation can catch it.

The Problem

Retrieval-Augmented Generation improves LLM factuality by grounding answers in external documents. But real-world retrieval corpora are messy. They contain not just irrelevant documents, but what the authors call "polluted evidence" — passages that are semantically relevant and plausible-sounding but contain subtle fabrications, entity swaps, or incorrect temporal attributions [§1].

The core vulnerability: LLMs exhibit a "sycophancy" bias toward retrieved context, often propagating these errors into final answers with high confidence [§1]. A single high-scoring polluted document can steer the model away from established facts [Figure 1]. Standard RAG defenses mostly filter *irrelevant* context, but polluted passages are relevant — they just happen to be wrong [§2].

This matters for any system retrieving from uncontrolled sources. Web indices, enterprise knowledge bases with user-contributed content, and news aggregators all contain plausible misinformation that passes relevance filters.

What They Did

MIRAGE operates as a wrapper around any existing retrieve-then-generate pipeline, requiring no model retraining [§3.1]. It works in two stages: build a claim graph, then decide whether to trust retrieval.

**Claim extraction and graph construction.** For each retrieved passage, MIRAGE splits text into sentence-level claims and tracks which source document and domain (e.g., wikipedia.org, nytimes.com) each claim came from [§3.2]. It then pairs claims from *different* sources — never within the same document — and runs a natural language inference (NLI) model on each pair. Think of NLI as a fact-checker that takes two sentences and outputs whether one supports, contradicts, or is unrelated to the other. The result is a graph where claims are nodes and edges represent support or contradiction relationships [§3.2, Equations 2a-2b].

The system also probes for missing counter-evidence by retrieving additional passages targeted at top-ranked claims and checking for contradictions [§3.2, Appendix J.3].

**Pruning and gating.** Each claim gets a score based primarily on how many cross-source claims support it, with domain trust and retrieval rank as secondary signals [§3.3, Equation 3]. MIRAGE then selects a mutually consistent subset by approximately maximizing a function that rewards individually reliable claims and penalizes internal contradictions [§3.3, Equation 4].

The key decision point is the "Defended-Claims Gate." MIRAGE computes two statistics on the surviving claim set: the ratio of contradictory to supportive edges, and how many distinct domains are represented [§3.4, Equations 5a-5b]. If contradictions are too high or source diversity too low, MIRAGE blocks retrieval entirely and tells the LLM to answer from its own parametric knowledge [§3.4, Equation 6]. If the gate passes, the LLM receives only the verified claim subset as context [Figure 2].

The underlying intuition is straightforward: correct information tends to be corroborated across independent sources, while misinformation creates structural inconsistency in the graph [§1, Figure 3]. Gate thresholds are set once from held-out data and applied globally across all datasets and models [§3.4].

The Results

The authors evaluate across four long-form QA benchmarks using a pollution protocol with four perturbation families — Unambiguous, Conflicting, Misleading, and Fabricated — under clean, mixed (some polluted passages), and fully polluted retrieval conditions [§1, §4].

Retrieval pollution severely degrades vanilla RAG [§1]. MIRAGE consistently restores factuality under both mixed and fully polluted settings and outperforms prior robust-RAG methods including AstuteRAG, TrustRAG, and RobustRAG [§1, §6]. Under mixed pollution, MIRAGE retains mutually supported clean claims while filtering inconsistent evidence. Under full pollution, it detects unreliable retrieval and falls back to parametric generation [§1]. The gate passes most clean queries, degrades under mixed retrieval, and blocks nearly all fully polluted queries [§3.4, Table 3].

Critically, MIRAGE preserves performance on clean retrieval — it doesn't sacrifice accuracy when evidence is trustworthy [§1].

**Limitations worth noting.** The pollution protocol uses controlled, minimal-edit corruptions [§4]. Real-world misinformation is messier — coordinated disinformation campaigns might plant consistent false narratives across multiple sources, which would survive MIRAGE's cross-source consistency check. For production systems facing adversarial actors rather than incidental misinformation, this defense likely needs additional layers, and robustness against coordinated attacks remains an open research problem.

The token-overlap filter used for efficient pair selection before NLI [§3.2, Equation 1] means paraphrased misinformation — same false claim, different wording — could evade detection. The authors acknowledge this as an "efficiency-oriented approximation" with a paraphrase-robustness trade-off [§3.2, Appendix J.2]. For production systems processing adversarially crafted content, this is a meaningful gap.

MIRAGE also adds computational overhead: NLI inference on claim pairs and counter-evidence retrieval happen before generation. The authors cap pairs per query for efficiency [§3.2], but latency-sensitive applications would need to benchmark this carefully.

Why It Matters

This is a lab proof-of-concept with practical architecture. Every component — sentence segmentation, NLI models, dense retrievers — is off-the-shelf, and the system requires no training or model modification [§3.1]. The code and datasets are public [Abstract].

For teams building RAG systems over uncontrolled corpora, the core design pattern is worth studying now: treat retrieved evidence as testimony from multiple witnesses, cross-examine it for consistency, and have a fallback when the testimony doesn't hold up. If your RAG pipeline retrieves from the open web, mixed enterprise sources, or any corpus you don't fully control, pre-generation evidence adjudication is moving from optional to necessary.