Signal

Open-source language models that score well on standard benchmarks perform near random chance when asked to identify which piece of context actually supports their answer.

The Problem

LLMs increasingly operate over rich, complex inputs: long agent trajectories with tool calls, dense images with subtle visual details. Getting the right answer requires more than reasoning ability — it requires grounding decisions in specific, often sparse evidence buried in the context [§1]. A model editing code might ignore a variable definition earlier in the file. A vision-language model might misread a value on a chart despite the information being plainly visible [Figure 1].

The authors call this failure mode "context unawareness": the relevant information is available, but the model's prediction isn't grounded in it [§1]. To measure it, they built a diagnostic probe: 200 contrastive pairs from agentic trajectories and 200 from visual QA images. Each example gives the model a question, an answer, and two nearly identical contexts — only one of which supports that answer. The model must pick the right context [§1].

The results are stark. GPT-5.4 and Claude Opus 4.7 score 98–99% on the agentic version and 83–84% on the multimodal version. Qwen3-VL 8B and Qwen3.5 9B — competitive open-source models — score 52.5–58%, barely above the 50% random baseline [Figure 2]. Strong benchmark performance, it turns out, can coexist with near-total inability to identify supporting evidence.

What They Did

ContextRL adds a single auxiliary training objective to standard GRPO (Group Relative Policy Optimization, a common RL post-training method). Instead of only rewarding correct final answers, it also rewards the model for correctly identifying which context supports a given answer [§2].

Concretely, each training example contains a question, an answer, and two contexts that are superficially almost identical but support different answers. The model sees all three elements and must select the supporting context. Think of it like a reading comprehension test where both passages look nearly the same, and you have to figure out which one actually justifies the conclusion you've been given. This is implemented as a logit-level contrastive loss — the model's confidence score on the correct context option must exceed its score on the incorrect one [§2.3].

Building these contrastive pairs required domain-specific pipelines. For coding agents, the team mined 66k trajectories from SWE-smith, applying cascading filters: same repository, same commit, same file, same function, but different issues. Patch contents were masked to prevent shortcut cues. Only 1k pairs (1.5% of candidates) survived automated verification by GPT-5.4 plus manual review of uncertain cases [§2.1].

For multimodal tasks, they used two strategies across five visual domains (charts, geometry, math, science diagrams, natural images). Natural images were edited using a generative model to change the answer-relevant region while preserving global scene structure, with a ~65% rejection rate during quality verification. Structured images like charts and diagrams were paired via embedding similarity search, requiring cosine similarity ≥ 0.85 and different answers. This produced 7k contrastive image pairs total [§2.2].

The context-awareness loss is trained jointly with the standard GRPO objective on original task data, requiring no architectural changes [§2.3].

The Results

On long-horizon tasks, ContextRL improved over standard GRPO by +3.2% on average across five agentic and long-context benchmarks using Klear-AgentForge-8B, and +1.5% using Qwen3-8B [§1]. On multimodal tasks, it improved by +2.0% on Qwen2.5-VL-7B and +1.6% on Qwen3-VL-8B across 12 visual QA benchmarks [§1].

The most telling result is what happened with the baselines designed to isolate the source of improvement. The researchers took the exact same contrastive data and fed it to models through conventional training: supervised fine-tuning (SFT) or standard outcome-based RL where the contrastive contexts were repurposed as regular query-context-answer examples. These baselines collapsed. Supervised augmentation drove the long-horizon agent's resolve rate to as low as 0%. Outcome-based augmentation added essentially nothing [§1, §2]. Only the context-selection objective — the specific way the signal was integrated — converted the data into gains.

The improvements, while consistent, are modest in absolute terms. The contrastive data is limited: 1k pairs for coding, 7k for multimodal [§1]. The agentic experiments use only two base models (both ~8B parameters), and the multimodal experiments use two more [§1]. Whether these gains hold at larger model scales, with more diverse contrastive data, or on tasks requiring different types of grounding remains untested — scaling studies would likely take another research cycle.

Why It Matters

This work makes two contributions worth tracking separately. First, the diagnostic probe itself is useful: a simple, cheap test that exposes whether a model can identify supporting evidence, independent of whether it gets the final answer right. The 40-point gap between proprietary and open-source models on this probe [Figure 2] suggests that context grounding is a meaningful axis of model quality that standard benchmarks miss.

Second, the training method demonstrates that you can improve context grounding without changing model architecture or collecting large-scale human annotations — just by adding the right auxiliary objective to existing RL pipelines [§1]. The fact that the same data fails when used through conventional training channels makes the case that the objective design, not the data volume, is doing the work [§1].

This is a lab proof-of-concept. The gains are consistent but small, the evaluation covers a limited set of model sizes, and the contrastive data construction pipelines rely on GPT-5.4 for verification. For teams already running GRPO post-training on agentic or multimodal models, the method is lightweight enough to test. For everyone else, the diagnostic probe is the immediately actionable piece: a way to check whether your model actually reads what you give it.

A multi-model search framework automatically constructs reusable skill libraries that transfer across different LLM backbones — addressing a key bottleneck in building reliable AI agents for complex, multi-step tasks.

The Problem

LLM agents operating in complex environments like OpenClaw — where they must coordinate files, tools, web pages, and execution feedback over multiple steps [§1] — need reusable procedural strategies ("skills") for recurring subtasks like tool use, verification, and error recovery [§1]. The problem: building these skills is expensive. Current skills are "largely handcrafted and require substantial manual design and maintenance, making large-scale skill construction costly, time-consuming, and labor-intensive" [§1].

Recent work on automatic skill construction has made progress, but the authors identify three persistent problems [§1, Figure 1]. First, **skill fragmentation**: methods produce isolated procedures for individual subtasks without mechanisms to orchestrate sequences or handle long-term dependencies. Second, **limited diversity**: skills typically come from a single model's trajectories, inheriting that model's biases and blind spots. Third, **limited transferability**: skills that work well with one LLM backbone often degrade when applied to a different one.

These limitations matter most for long-horizon tasks in real-world interactive environments, where an agent might need to chain dozens of steps together reliably [§1].

What They Did

The researchers propose Collective Skill Tree Search (CSTS), a framework that organizes skills into tree structures where each layer corresponds to a subtask and each node represents a candidate skill. A path through the tree represents a complete strategy for solving a complex task — think of it as a decision tree where each branch point offers different procedural approaches to the next step [§3.1].

CSTS operates in three stages. First, **Complex Task Decomposition** breaks a task into an ordered sequence of subtasks [§3.1]. Then two phases iterate:

**Collective Skill Node Generation (CSN-Gen)** pools candidate skills from multiple different LLMs rather than relying on a single model [§3.1]. By aggregating suggestions from heterogeneous models, the framework captures diverse problem-solving strategies — one model might favor a verification-first approach while another prefers error-recovery patterns. This directly attacks the diversity problem: "CSN-Gen explicitly alleviates single-model bias and enriches the skill space with diverse and complementary behavioral patterns" [§1].

**Collective Skill Node Assessment (CSN-Assess)** uses multiple models as judges to evaluate candidate skills through two scoring mechanisms [§1, §3.1]. The first, collective quality scoring, aggregates independent evaluations from multiple judges into a robust effectiveness estimate — similar to how peer review uses multiple reviewers to reduce individual bias. The second, collective transferability scoring, explicitly tests whether a skill generated by one model actually helps other models perform better. Skills that only work for their originating model get filtered out [§1].

The framework then constructs skill-augmented training data from these curated skill trees. Models are first trained via standard supervised fine-tuning on this data [§1]. The authors then introduce **Collective Skill Reinforcement Learning (CSRL)**, which selects multiple relevant skills from the tree for each task during RL training rather than conditioning on just one. This "substantially broadens the exploration of the solution space, thereby preventing the model from being trapped in a single skill and its resulted homogeneous or suboptimal solutions" [§1]. The model learns to adaptively fit the most appropriate skill among candidates.

The resulting trained models are called OpenClaw-Skill [§1].

The Results

The authors report that OpenClaw-Skill "exhibits outstanding agentic capabilities in long-horizon planning, tool use and generalization over challenging benchmarks" [Abstract]. The models demonstrate "superior performance on multiple challenging benchmarks" with capabilities spanning long-horizon planning, tool use, error recovery, and cross-task generalization [§1].

This paper reports no specific benchmark numbers in the available text — the full experimental sections were not included in the reviewed portion. The contributions are primarily methodological: the tree-search framework for skill construction, the collective generation and assessment mechanisms, and the multi-skill reinforcement learning approach [§1].

Key limitations deserve attention. The framework requires access to multiple LLMs during skill construction — both for generation and assessment — which increases computational cost compared to single-model approaches. For teams with limited compute budgets, this means the skill construction phase is more expensive, though the resulting skills are meant to be reused. The transferability scoring mechanism tests generalization across the specific models used during construction; whether skills transfer to entirely unseen architectures remains an open question — meaning production deployments that frequently swap backbone models may need to re-run the assessment phase, adding maintenance overhead.

Why It Matters

This work sits at the lab proof-of-concept stage. The core insight — that pooling procedural knowledge from multiple models and explicitly testing for cross-model transferability produces better skills than single-model extraction — is architecturally significant for anyone building agent frameworks.

For practitioners building OpenClaw-style systems where agents must chain tool use, file manipulation, and reasoning over many steps, the tree-structured skill organization offers a concrete alternative to flat skill banks. The CSRL training approach, which exposes models to multiple skill options rather than a single prescribed procedure, addresses a real failure mode in current agent training where models overfit to one solution strategy [§1]. If your team is designing agent training pipelines, the multi-skill conditioning approach is worth evaluating even independently of the full CSTS framework.

A fully open-source robot navigation system now matches closed-source GPT-4o accuracy on the hardest spatial reasoning tasks — using binary search.

The Problem

Service robots that traverse long routes build up a memory of what they've seen and where. When a user asks "where did I see the dry cleaner this morning?" or "where is the convenience store on the route from my office to the subway station?", the robot needs to return a specific coordinate that its navigation system can act on [§1].

The best existing systems for this — called Spatial Question Answering (SQA) — rely on closed-source models like GPT-4o to orchestrate retrieval, reasoning, and visual verification across multiple steps [§1]. That's a problem for real deployments. A robot guiding a human can't assume reliable internet connectivity, can't tolerate the latency of round-tripping every retrieval call to a cloud API, and can't absorb the ongoing cost of sending video frames and queries to a paid model [§1].

The obvious fix — swapping in open-source models of comparable size — causes accuracy to "drop sharply," particularly on the hardest query type: global queries that require reasoning along a route between two landmarks [§1]. But the authors argue this performance gap isn't really about model capability. It's about the algorithm. Prior frameworks lean on GPT-4o to coordinate multi-step graph reasoning at query time. Open-source models struggle with that coordination, not with the underlying spatial problem [§1].

What They Did

BinTrack exploits a structural property that prior systems ignored: a robot's trajectory is already a temporally ordered sequence of spatially indexed observations [§1]. When a query asks "where is Z on the route from X to Y?", the two anchor landmarks (X and Y) define a contiguous slice of that sequence. Finding Z within that slice is a one-dimensional search problem [§4.2].

The core algorithm is Binary Tracking — essentially binary search applied to trajectory segments. It identifies the segment indices for the two anchor landmarks, then recursively halves the interval between them. At each step, it ranks candidates within each half by semantic similarity to the query target and descends into the half with stronger evidence [§4.2]. Think of it like searching for a specific shop on a long street: instead of checking every storefront, you jump to the middle, decide which direction looks more promising, and repeat.

This replaces the graph-based path search used by prior systems, which reconstructed the robot's already-available trajectory as a waypoint graph and then relied on a strong model to reason over it across multiple steps [§4.2].

The second contribution is a multi-view memory representation. Previous work summarized each trajectory segment with a single caption from a 2×2 grid of four frames. With smaller open-source vision-language models, this produces generic descriptions like "a street with shops" instead of useful details like "a coffee shop named BANAPRESSO on the right" [§4.1]. BinTrack instead generates three captions per segment from the same open-source model using different prompts: a full scene view, a center view describing what's directly ahead, and a detail view capturing storefronts, signs, and readable text [§4.1]. All three are embedded and stored in a local vector database for joint retrieval.

The entire pipeline — captioning, embedding, retrieval, verification — uses only open-source models [Abstract].

The Results

BinTrack improves overall accuracy by up to 22.8% over other open-source implementations on the SpaceLocQA benchmark [Abstract]. On global queries — the most challenging category, requiring multi-step reasoning along a route between two landmarks — it matches the reported closed-source GPT-4o result [Abstract]. A prediction counts as correct if it falls within 15 meters of the ground-truth coordinate [§3].

BinTrack also achieves more than a 1.5× inference speedup over previous baseline approaches [Abstract], which matters for onboard deployment where compute is constrained.

The team also releases GangnamLoop, a multi-trip outdoor benchmark collected by deploying a real quadruped robot on public streets in Seoul's Gangnam district [§1]. It covers the same locations under different outdoor conditions and pairs the robot's low viewpoint with the human owner's perspective, addressing limitations of existing SQA benchmarks [Abstract].

Limitations are worth noting. All benchmark evaluations use the SpaceLocQA dataset and the new GangnamLoop dataset — both structured research benchmarks, not messy production environments [§3]. The 15-meter accuracy threshold is generous for some indoor applications but reasonable for outdoor urban navigation [§3]. Production codebases with mixed indoor/outdoor environments, dynamic obstacles, and degraded sensor data represent a harder problem. Teams should treat this as a validated algorithmic approach, not a deployment-ready system.

Why It Matters

The practical takeaway is that algorithmic structure can compensate for model capability gaps. Rather than waiting for open-source models to match GPT-4o's multi-step reasoning, BinTrack sidesteps the problem by reducing complex spatial queries to a search primitive that smaller models can execute reliably [§1].

This sits at the boundary between lab proof-of-concept and early production pattern. The algorithm is validated on benchmarks and real robot data, code and datasets are public, and the approach runs entirely on open-source components [Abstract]. For teams building service robots that need spatial memory and can't depend on cloud connectivity — delivery robots, warehouse systems, assistive platforms — this provides a concrete architecture to evaluate. The key constraint is that it assumes a single continuous trajectory with temporal ordering [§4.2]; environments requiring multi-robot fusion or non-sequential exploration would need additional work.

Asking an LLM to "think step by step" increases anti-Muslim violent associations by 12–34% compared to simple prompt completion.

The Problem

In 2021, researchers found that GPT-3 completed the phrase "Two Muslims walked into a..." with violent content roughly 66% of the time [§1]. Five years and several model generations later, the bias persists across LLM families and frontier systems [§1]. But the way we measure it hasn't kept pace with how these models are actually used.

Nearly all published anti-Muslim bias evaluations use single-turn prompt completion — type something in, see what comes out [§1]. That no longer reflects reality. Modern LLMs operate in three distinct regimes: chain-of-thought reasoning (where models are asked to work through problems step by step), agentic decision-making (where models issue consequential judgments in workflows like hiring or lending), and retrieval-augmented generation (where models pull in external documents before responding) [§1]. Each regime creates a different pathway for latent bias to become real-world harm, and none had been systematically audited for anti-Muslim bias.

The most effective prompt-engineering mitigations reduce bias by at most 87.7%, and simpler interventions perform substantially worse [§1]. The question MIRAGE asks is whether even those partial fixes hold up when models reason, decide, and retrieve.

What They Did

The researchers built MIRAGE (Muslim-Identity Reasoning and Agentic Generation Evaluation), a benchmark of 1,200 base prompts organized into six template families [§3.2]. Each prompt exists in a Muslim-identifying variant and four matched controls — Christian, Jewish, Hindu, and secular — differing by a single word swap with all other content held fixed [§3.1]. This minimal-edit design means any output difference is attributable to the identity signal, not to surrounding text [§3.1]. The total corpus spans 6,000 individual prompts before condition sampling [§3.2].

The benchmark tests three conditions. C1 (direct completion) replicates the classic 2021 setup for backward compatibility [§3.3]. C2 (chain-of-thought) asks models to reason through their responses step by step. C3 (agentic decision-making) places models in simulated high-stakes roles: content moderation, lending triage, refugee claim summarization, and hiring screens [Abstract]. Each condition uses identical underlying evidence — only the inference regime changes.

A retrieval-augmented layer adds time-coupled context. Models receive news passages drawn from four pools — neutral, historical, and recent-conflict corpora — to test whether bias tracks the news cycle [Figure 1]. This operationalizes an observation from the original 2021 work: anti-Muslim bias rises alongside terrorism-related media coverage [§1].

Five independent human annotators reviewed all prompts for naturalness and label accuracy, with violent-content labels adjudicated by majority vote at an inter-annotator agreement of κ = 0.81 [§3.2]. A 400-prompt subset was translated into Modern Standard Arabic and three regional dialects (Egyptian, Levantine, Maghrebi) by bilingual speakers, then independently post-edited [§3.1].

Six frontier models were audited: GPT-4o, Claude Opus 4, Llama-3.3-70B, Qwen2.5-72B, DeepSeek-V3, and Mistral Large [Figure 1].

The Results

The headline finding inverts a common assumption. Chain-of-thought reasoning — widely adopted because it improves accuracy — amplifies Muslim-violence associations by 12–34% relative to direct completion [Abstract]. Rather than helping models "think past" stereotypes, step-by-step reasoning appears to surface and reinforce learned associations. Prior work hypothesized this mechanism [§2], and MIRAGE provides the first religious-bias-specific confirmation across six models.

In agentic decision tasks, Muslim-identified cases received systematically worse outcomes than matched non-Muslim cases on identical evidence, with asymmetries of 9–22 percentage points [Abstract]. The most severe gap appeared in refugee claim summarization — the highest-stakes task in the evaluation suite [§1]. This means a model reviewing two identical asylum applications would produce materially different summaries depending solely on whether the applicant's name signals Muslim identity.

Bias is also sharply time-coupled. When models retrieved context from recent-conflict news corpora, bias increased 18–27% compared to neutral retrieval contexts [Abstract]. This confirms that retrieval-augmented generation doesn't just passively reflect corpus content — it actively modulates the strength of identity-linked associations at inference time.

Perhaps most concerning for practitioners: existing prompt-based mitigations (cultural prompting, self-debiasing, multi-step pipelines) transfer poorly across conditions [Abstract]. They suppress direct-completion bias — the regime they were designed for — while leaving agentic decision asymmetry largely intact [Abstract]. The largest amplification under chain-of-thought was observed in open models lacking dedicated religious-bias alignment [§1].

The benchmark currently covers six models and four non-Muslim control groups, with Arabic dialect coverage limited to three regional variants [§3.1]. Production codebases involve mixed model usage, multi-turn agent orchestration, and dynamic retrieval corpora — conditions more complex than MIRAGE's controlled simulations. Reliable auditing tools for those environments are likely years away. The cross-lingual evaluation, while more comprehensive than prior work, does not yet cover non-Arabic languages with large Muslim populations (Urdu, Turkish, Bahasa).

Why It Matters

MIRAGE demonstrates that measuring bias in the way models were used in 2021 misses the harm surface of 2026 deployments. The finding that chain-of-thought amplifies bias has immediate implications: teams using reasoning-enhanced prompting in sensitive domains cannot assume that "thinking harder" produces fairer outputs [Abstract]. The agentic asymmetry finding is more urgent still — a 9–22 point gap in lending, hiring, or refugee triage is not a measurement artifact; it's a discrimination vector [Abstract].

This work sits at the lab proof-of-concept stage: it identifies and quantifies the problem but does not solve it. The mitigation audit explicitly shows that current defenses don't generalize across deployment conditions [Abstract]. For organizations deploying LLMs in consequential decision pipelines, the practical takeaway is that bias testing confined to direct completion provides false assurance. MIRAGE and its evaluation harness are publicly released to support targeted mitigation research [Abstract].

Neural operator models adapted to real-world experimental data can now retain their core physics knowledge — reducing low-frequency prediction errors by up to 32% compared to standard fine-tuning.

The Problem

Neural operators — models that learn to approximate PDE solutions from simulation data — are fast and increasingly accurate. But when you deploy them on actual experimental measurements, performance often degrades substantially [§1]. This sim-to-real gap arises because simulations rest on idealized assumptions, while real data carries measurement noise, unmodeled effects, and systematic domain shift [§1].

The obvious fix is fine-tuning on a small set of real experimental data. But unconstrained fine-tuning creates its own problem: the model can overwrite the low-frequency physical structures it learned during pretraining — things like vortex streets and mean flow profiles — while chasing high-frequency noise in the sparse real data [§1]. These large-scale coherent patterns often capture the core physics of the system.

Techniques for preventing this kind of knowledge destruction exist in vision and language AI (elastic weight consolidation, gradient projection, LoRA), but neural operators are architecturally different. They rely on spectral convolutions, branch-trunk factorizations, and physics-aware attention rather than the semantic feature hierarchies of image classifiers or language models [§1]. Whether knowledge-preservation methods transfer to this setting had not been systematically tested [§2].

What They Did

PhysGuard starts from a specific insight: the physical knowledge in a pretrained neural operator concentrates along a small number of directions in parameter space [§1, §3.2]. If you can identify those directions, you can block fine-tuning updates from touching them while leaving everything else free for adaptation.

To find these directions, PhysGuard computes the empirical Fisher Information Matrix (FIM) on simulation data [§3.2]. Think of the FIM as a sensitivity map over all model parameters — it measures how much the model's predictions would change if you nudged each parameter. Directions where predictions are highly sensitive correspond to the physics the model has learned; directions where predictions barely change are safe to modify.

Concretely, the method takes gradient vectors from the pretrained model evaluated on simulation samples, stacks them into a matrix, and performs a matrix decomposition to extract the top eigenvectors — the most physics-sensitive directions [§3.2]. Since neural operators can have millions of parameters, directly decomposing the full FIM would be computationally prohibitive. PhysGuard sidesteps this by working with a much smaller Gram matrix (N×N instead of d×d, where N is the number of samples and d is the number of parameters), which shares the same non-zero eigenvalues [§3.2].

An adaptive threshold automatically determines how many directions to protect: the method keeps adding eigenvectors until they collectively account for a target fraction τ of the total Fisher information [§3.2]. During fine-tuning on real data, each gradient update is projected onto the null space of these protected directions — geometrically, the update is allowed to move freely in any direction that doesn't interfere with the physics-critical subspace [§3.3, Figure 2].

The approach requires no auxiliary loss terms, no architecture-specific modifications, and applies to any differentiable neural operator [§1].

The Results

PhysGuard ranks first on 38 of 48 metric–architecture–scenario combinations on RealPDEBench, tested across four neural operator architectures (FNO, CNO, DeepONet, Transolver) and three physical systems [Abstract]. The benefits are most pronounced under severe domain shift, where it reduces low-frequency error by up to 32% compared to standard fine-tuning [Abstract].

A spectral probe experiment provides mechanistic evidence for why this works. The authors found that the FIM spectrum of neural operators is low-rank — a small number of eigenvectors capture most of the Fisher information — and these top eigenvectors are strongly associated with low-frequency output structures [§4.2, Abstract]. This confirms the design premise: protecting the top Fisher directions specifically preserves the large-scale physical patterns.

PhysGuard does not dominate every single metric. It ranks first on 38 of 48 combinations, meaning 10 combinations are won by other methods [Abstract]. The paper also notes that the method "maintains adaptability" alongside physics preservation [Abstract], but the tradeoff between preservation and adaptation flexibility is scenario-dependent.

The benchmark itself has scope limitations. RealPDEBench provides paired simulation and experimental data for three physical scenarios [§1]. Real production deployments often involve more complex multi-physics systems, mixed model usage, and noisier data pipelines — the method's robustness under those conditions is untested. For teams considering deployment, this means PhysGuard is validated in controlled benchmark settings but would need further testing on domain-specific experimental data before production use.

The Gram-matrix formulation makes the method tractable for million-parameter models [§3.2], but the paper uses 10% of simulation samples for FIM estimation [§3.2 footnote], and how sensitive the results are to this fraction across different data regimes is not extensively characterized.

Why It Matters

This is a lab proof-of-concept with public code, not a production-ready tool. But it addresses a genuine bottleneck in scientific machine learning: the moment you try to move a simulation-trained model into the real world with limited experimental data.

The spectral probe finding — that Fisher-critical directions in neural operators specifically encode low-frequency physics [§4.2] — is independently useful. It suggests a diagnostic: before fine-tuning any neural operator, you could inspect its FIM spectrum to understand what knowledge is at risk.

For practitioners building sim-to-real pipelines for PDE surrogates, PhysGuard offers an architecture-agnostic method that requires no physics-specific loss engineering. If your workflow involves pretraining on simulation data and adapting to sparse experimental measurements, this is worth benchmarking against your current fine-tuning approach — especially if you've observed degradation in large-scale physical structures after adaptation.