Signal

Current benchmarks for AI web coding miss most of what makes a website work — visual fidelity, interactive behavior, and the ability to fix bugs — and a new 1,526-task evaluation exposes how badly models struggle with aesthetics.

The Problem

Evaluating whether an AI model can build a good website is fundamentally different from evaluating whether it can solve an algorithm problem. Success depends on visual fidelity, interaction behavior, responsiveness, and user experience — none of which are captured by standard metrics like pass@k on HumanEval or unit-test pass rates on SWE-Bench [§1].

Existing web coding benchmarks each cover only a narrow slice of the problem. Some test generation from text prompts, others from screenshots, but none jointly evaluate generation, editing, and repair across text, image, and video inputs [Table 1]. More critically, most rely on static correctness checks rather than actually running the generated website and testing its interactive behavior [§1].

This matters because real web development is an iterative cycle: you generate code, edit it based on feedback, and repair bugs. A benchmark that only tests generation from text descriptions tells you very little about whether a model can handle the full workflow.

What They Did

WebCompass is a 1,526-task benchmark organized into seven task categories that combine three input modalities (text, image, video) with three task types (generation, editing, repair) [§2.1]. The tasks cover 15 generation domains, 16 editing operation types, and 11 repair defect types, each annotated at Easy, Medium, and Hard difficulty levels [Abstract].

The benchmark construction used a multi-stage, human-in-the-loop pipeline [§2.2]. For text-guided generation, the team collected queries from multiple sources, deduplicated them using embedding-based clustering, then had an LLM expand underspecified requests into structured design documents covering page content, interaction behaviors, and visual appearance [§2.2.1]. This addresses a real problem: vague prompts like "make me a dashboard" produce wildly different outputs across models, making automated comparison nearly impossible.

For editing and repair tasks, the team built on existing codebases. Repair tasks use a reverse-engineering approach: they start with working code, inject specific bugs, and provide exact search/replace annotations mapping buggy code to clean targets [Table 1]. This ensures deterministic ground truth — you know exactly what the correct fix should be.

The evaluation design splits by task type. For editing and repair, WebCompass uses a checklist-guided LLM-as-a-Judge protocol — essentially giving a language model a rubric and having it score before/after screenshots and code diffs [§1]. For generation, the team built something more ambitious: an Agent-as-a-Judge protocol where an autonomous agent launches the generated website in a real browser, explores it through the Model Context Protocol (MCP), synthesizes targeted test cases, and scores the result based on actual execution [Abstract]. Think of it as an automated QA tester that clicks through your website, checks if buttons work, and verifies that visual elements render correctly.

The Results

The performance gap between closed-source and open-source models is stark. Looking at the radar chart [Figure 1], the top closed-source models score between roughly 55 and 87 across the seven task types. The best open-source model, Qwen2.5-VL-72B-Instruct, drops to a range of approximately 23 to 69, with other open-source models performing even worse [Figure 1].

The task-type breakdown reveals interesting patterns. Repair tasks that involve restoring interactivity (Repair-RCT) show the highest scores overall — top models reach scores in the high 70s to mid-80s [Figure 1]. But generation tasks are harder: Vision-Guided Generation (Gen-SPI) shows significantly lower performance across all models, with even the best performers dropping into the 50s [Figure 1].

Aesthetics emerges as the most persistent bottleneck, especially for open-source models [Abstract]. Models can often get the structure and functionality roughly right but produce visually unappealing results. Framework choice also matters materially: Vue consistently proves more challenging for models, while React and Vanilla/HTML perform more strongly depending on task type [Abstract].

Editing tasks show a distinct difficulty profile from repair. Text-Guided Editing (Edit-ITG) generally shows lower scores than the structurally similar Diagnostic Repair (Repair-RCT) tasks [Figure 1]. Repair preserves interactivity better but remains execution-challenging [Abstract].

The benchmark's scope of 1,526 tasks [Abstract] represents a significant expansion over existing web coding evaluations, though as with any benchmark, it represents a sampling of the full space of web development challenges. The Agent-as-a-Judge evaluation, while more realistic than static checks, depends on the capability of the judge model itself, introducing a moving target as models improve. For teams building internal evaluation pipelines, this means the framework is useful for comparative model selection today but will need recalibration as both tested and judge models evolve.

Why It Matters

For practitioners choosing models for web development workflows, the findings are concrete and actionable. If you're building with Vue, expect worse model performance than with React or vanilla HTML [Abstract]. If visual polish matters — and for customer-facing work it always does — closed-source models remain substantially ahead [Abstract, Figure 1].

This is a lab-stage evaluation framework. The Agent-as-a-Judge protocol is a meaningful step toward automated acceptance testing for AI-generated websites, but it approximates rather than replaces human judgment [Abstract]. The benchmark's public availability (data, code, and project page) [Abstract] means teams can run these evaluations on their own model candidates today, which is its most immediate practical value. The larger contribution is establishing that web coding evaluation requires testing across the full lifecycle — generation, editing, and repair — not just one-shot code generation from a text prompt.

You can now add new capabilities to a post-trained language model without retraining it from scratch — and without degrading what it already knows.

The Problem

When you want to extend a language model with a new capability — say, better math reasoning or tool use — you face an ugly tradeoff. Retraining from scratch on all your data is expensive and requires full access to the original pipeline. But continuing to train the existing model on new data risks catastrophic forgetting: the model gets better at math but worse at code, or gains tool-use ability but loses safety alignment [§1].

This problem compounds as you add more domains. Each new capability means reprocessing everything, and the authors claim the cost scales quadratically with the number of domains [Abstract]. Worse, the multi-stage nature of modern training pipelines — mid-training, then supervised finetuning (SFT), then reinforcement learning (RL) — creates ordering conflicts. Late-stage RL for one domain can degrade capabilities learned during earlier stages that lack RL data entirely [§2].

Prior modular approaches like Branch-Train-MiX (BTX) and FlexOlmo addressed this for pre-training but fail when applied to post-training. Freezing shared parameters during expert training — the standard recipe for pre-training — significantly degrades performance in the post-training setting [§1, §3.3].

What They Did

The researchers developed BAR (Branch-Adapt-Route), a three-stage process for building modular domain experts and composing them into a single model.

**Stage 1: Train experts independently.** Starting from a base model, BAR creates a separate two-expert Mixture-of-Experts (MoE) for each target domain. Think of MoE as a model with multiple parallel "specialist" modules and a traffic controller that routes each input token to the right specialist. One expert in each pair is an "anchor" — a frozen copy of the original model's feed-forward network (FFN) weights that preserves existing capabilities. The other is the domain expert, initialized from the pre-trained (not post-trained) base to give it a clean starting point [§3.2]. Each domain expert goes through whatever training stages its domain needs: math and code use the full pipeline (mid-training → SFT → RL), while tool use and safety use SFT only [Figure 1].

The key technical insight is how shared parameters — attention layers, embeddings, normalization — are handled across stages. During mid-training, shared layers stay frozen (matching prior work). During SFT, embeddings and the output head are unfrozen to support new tokens like `<thinking>`. During RL, all shared parameters are unfrozen to allow the behavioral shifts RL requires [§3.3]. This progressive unfreezing is critical; ablations show that freezing shared layers throughout post-training significantly hurts performance [§5.3].

**Stage 2: Merge experts.** The independently trained domain experts are combined with the anchor into a single (k+1)-expert MoE. Where shared parameters diverged during training (e.g., attention layers unfrozen during RL), BAR simply averages them. This averaging introduces little to no measurable performance loss on domain-specific evaluations compared to any single expert [§3.3].

**Stage 3: Train the router.** A small linear layer at each transformer block learns to route tokens to the right experts. This uses only 5% of the full SFT dataset, making it computationally cheap [§3.3].

The Results

At 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1 averaged across 7 evaluation categories. This exceeds the retraining baseline that uses post-training only (47.8), BTX (46.7), continual post-training (45.3), and model merging approaches (36.9 and 6.5). It approaches — but does not match — full retraining with mid-training from scratch, which scores 50.5 [§1, Abstract].

The structural advantage is where BAR pulls ahead of monolithic approaches. Because each domain's pipeline is isolated, late-stage RL for math cannot degrade code capabilities or safety alignment. In monolithic training, this cross-domain interference from sequential RL is a documented failure mode [§1]. BAR also enables upgrading a single expert (say, swapping in a better math expert) without retraining or affecting other experts — the cost of an update is constant rather than scaling with the number of domains [Figure 1, §3.1].

The limitations are concrete. This work demonstrates results at 7B scale with four domains — math, code, tool use, and safety [§4]. Whether the approach holds at larger scales or with more diverse, overlapping domains remains untested. The merged model uses more parameters than a dense model of equivalent active compute (each token activates only a subset of experts, but all expert weights must be stored), which has memory implications for deployment. And the 1.4-point gap to full retraining with mid-training (49.1 vs. 50.5) suggests some performance cost to modularity — for teams where that margin matters and retraining is affordable, the monolithic approach still wins on raw quality. Production systems with dozens of domains and continuous update cycles are a harder test, likely requiring further work on router robustness and expert interaction.

Why It Matters

This is a lab proof-of-concept, not production tooling. But the structural argument is compelling: if you can train domain experts independently and compose them without degradation, you fundamentally change the economics of multi-domain LM development. Adding a fifth domain costs the same as adding the first, rather than requiring reprocessing of all existing data [§3.1].

For teams managing post-training pipelines across multiple capabilities — particularly those hitting catastrophic forgetting when adding RL for new domains — BAR demonstrates that decoupled training is viable at the 7B scale. The progressive unfreezing strategy [§3.3] and the finding that naive shared-layer freezing fails for post-training [§3.3] are immediately useful design insights, even outside the full BAR framework. If your organization is planning multi-domain post-training pipelines, this work suggests that modular expert architectures deserve serious evaluation alongside monolithic retraining.

An agentic workflow combining web search and retrieval reduced explicit racial bias in LLM-generated differential diagnoses — but not uniformly across all metrics.

The Problem

LLMs are increasingly used in healthcare for tasks like generating clinical notes, summarizing cases, and suggesting diagnoses [§2.1]. But racial bias in these systems is a documented concern. African-American women in the United States are approximately three times more likely to die from pregnancy-related causes than white women, even with similar health conditions [§1]. When LLMs trained on historically biased data enter clinical workflows, they risk inheriting or amplifying these disparities.

Prior work has shown that GPT-4 can associate race with medical outcomes in subtle ways — predicting higher costs and longer hospital stays for white patients, or using different language patterns when generating HIV-related guidance [§2.2]. Explicit bias shows up too: prior research has shown that GPT-4 can prioritize different diseases or offer more optimistic treatment predictions for white patients [§2.2]. But most studies focus on a single model and give less attention to whether anything can actually reduce the bias [Abstract].

The EU AI Act adds regulatory urgency. Under Article 6, AI systems used as safety components of medical devices or falling within high-risk use cases may face requirements for risk management, data governance, transparency, and bias mitigation throughout their lifecycle [§2.5]. This makes systematic bias evaluation not just a research question but a governance one.

What They Did

The researchers tested five LLMs — GPT-4.1, Llama 3.3, DeepSeek V3, Grok 3, and Gemini 2.5 Pro — on two tasks designed to capture different types of racial bias [§4.1].

The first task measured implicit bias through synthetic patient-case generation. For each of ten diseases (including COVID-19, tuberculosis, HIV, lupus, and prostate cancer), the researchers used structured prompt templates to generate 100 synthetic patient cases per disease per model — 5,000 cases total across all five models (1,000 per model across the 10 selected diseases) [§3.2.2]. They then compared the racial distribution of the generated cases against real epidemiological prevalence data for the United States, compiled from public health sources [§3.1.1]. Think of it as asking: if an LLM invents a patient with lupus, does the racial makeup of those invented patients match who actually gets lupus in the real world?

The second task measured explicit bias through differential diagnosis ranking. The researchers took 10 real emergency department cases from the NEJM Healer benchmark, each with expert-generated ranked diagnosis lists [§3.2.1]. They swapped in four racial identifiers (Black, White, Hispanic, Asian) while keeping everything else identical, then asked each model to produce a ranked list of possible diagnoses. Each combination was repeated 10 times, yielding 400 outputs per model and 2,000 total [§3.2.2]. The question: does changing only the patient's race change what diagnoses the model suggests?

Based on the results from these two tasks, the best-performing model was selected for a second experiment. That model — DeepSeek V3 — was embedded in an agentic workflow built on Flowise, an open-source platform for constructing LLM pipelines [§4.1]. The workflow combined two components: a Search Agent connected to a web search API (Brave Search) and a RAG Agent — a retrieval system that pulls relevant information from a patient-case knowledge base before the model generates its answer, rather than relying solely on what the model memorized during training [§4.1]. The differential diagnosis task was then re-run through this workflow to see if retrieval-augmented reasoning reduced racial bias.

The Results

In the synthetic case generation task, all five models deviated from observed racial distributions in the United States. GPT-4.1 showed the smallest overall deviation [Abstract]. No model accurately reproduced epidemiological reality — every one over- or under-represented certain racial groups for certain diseases.

In the differential diagnosis task, DeepSeek V3 produced the strongest overall results across the reported metrics [Abstract]. When embedded in the agentic workflow, DeepSeek V3 showed an improvement of 0.0348 in mean p-value, 0.1166 in median p-value, and 0.0949 in mean difference relative to the standalone model [Abstract]. Higher p-values here indicate that the diagnosis rankings across racial groups were less statistically distinguishable from each other — meaning the model treated races more similarly. But the authors are explicit: improvement was not uniform across every metric [Abstract].

The technical limitations are significant. The study covers five models, four racial categories, and ten diseases/cases — a controlled benchmark, not a reflection of real clinical complexity where patients present with ambiguous symptoms, comorbidities, and mixed demographic profiles [§3.1.1]. The agentic workflow was tested only on the differential diagnosis task (Sub-experiment 2), since the RAG component relies on an existing patient-case knowledge base rather than generating new synthetic cases like in Sub-experiment 1 [§4.1]. For organizations considering deployment, this means the mitigation approach has been validated in a narrow, structured setting — scaling to messy real-world clinical workflows would require substantially more testing.

Why It Matters

This work establishes two practical points. First, no single metric captures racial bias in medical LLMs — the authors explicitly advocate for multi-metric evaluation [Abstract]. A model that looks fair on one measure can fail on another. Second, retrieval-based agentic workflows represent a concrete, testable mitigation strategy, not just a theoretical one. The improvement is real but partial.

This sits firmly at the lab proof-of-concept stage. The agentic workflow reduced some forms of explicit bias in benchmarked diagnostic tasks [Abstract], but the inconsistency across metrics means no one should treat this as a solved problem. For teams building or deploying clinical AI under frameworks like the EU AI Act, the takeaway is structural: bias evaluation needs multiple lenses, and mitigation via agentic architectures is a direction worth investing in — but not yet a guarantee. Prompt templates, datasets, and code pipelines are publicly available on the authors' GitHub [Abstract].

A 7-billion-parameter model trained only on Python-to-other-language tasks now outperforms DeepSeek-R1 (671B) at code translation — without ever seeing parallel code pairs between non-Python languages.

The Problem

Code translation — converting source code from one programming language to another — matters for legacy system migration, security improvements, and supporting developers who work in less popular languages. But it's hard. Rule-based approaches require painstaking manual engineering for every language pair, and they break when language semantics don't align [§1].

LLM-based approaches have shown promise, but training-based methods face a combinatorial data problem. Supporting N languages requires N×(N−1) directed translation pairs — that's O(N²) language pairs [§1]. For 10 languages, that's 90 pairs. For low-resource languages like Perl, Haskell, or Ruby, high-quality parallel data barely exists. As the paper documents, a model's coding proficiency correlates strongly with how much training data exists for that language [Figure 1], meaning low-resource languages get the worst translation quality precisely when they need it most.

Previous reinforcement learning approaches to this problem also hit a wall: they relied on either heavyweight compiler feedback or sparse rewards based on how many test cases a translated program passes — neither provides enough signal for the model to learn effectively [§1].

What They Did

CodePivot's core insight is simple: instead of training on all 90 language pairs, train only on Python-to-X translations and let Python serve as a shared intermediate representation that implicitly connects all languages [§1]. The idea is that if a model deeply understands how to translate Python to Rust and Python to Java, it can generalize to Rust-to-Java — even though it never saw that pair during training.

Why Python? Prior work used low-level compiler representations like LLVM IR as a bridge, but that only works for languages that compile to LLVM (C, C++, Rust, etc.) [§1]. Python is the most data-rich programming language available, its syntax reads close to natural language (which aligns well with how LLMs process text), and it sidesteps the need to model language-specific features like Rust's ownership rules or pointer semantics [§1].

The team built a dataset called DPY-TRANS containing approximately 1 million Python-to-Others translation tasks spanning 9 target languages: C++, C#, Java, JavaScript, Golang, Perl, Ruby, Rust, and Haskell [§1]. Each task includes a Python source program, language-agnostic test cases, and metadata. The source programs cover algorithms, logic puzzles, math, scientific computation, and system modeling, including external API calls like file I/O [§1].

For training, they used fewer than 30K tasks from this dataset — a small fraction — to fine-tune Qwen2.5-Coder-7B-Instruct [§1].

The second contribution is a new RL reward called Aggressive-Partial-Functional reward. Standard rewards give credit based on the raw number of test cases passed, which creates sparse feedback — most attempts either pass everything or nothing. The new reward specifically emphasizes outputs that pass *some* test cases but not all. Think of it as coaching: instead of only rewarding perfect translations, you reward the model for getting closer — turning completely broken outputs into partially working ones [§1]. This provides denser learning signal without requiring heavyweight compiler analysis.

The Results

The 7B model trained using CodePivot's approach outperforms DeepSeek-R1 (671B parameters) by 5.34% on Python-to-Others transpilation tasks [§1]. On the broader Others-to-All benchmark — where the model must translate between non-Python language pairs it never trained on — it outperforms Qwen3-235B-A22B-Instruct-2507 by 15.19% [§1]. On low-resource language tasks specifically, the improvement over Qwen3-235B is 10.67% [§1].

Perhaps most telling: the Python-pivot model outperforms a counterpart trained directly on Any-to-Any translation tasks by 5.83% on general transpilation [§1]. Training on one direction (Python-to-X) transfers better than training on all directions with the same compute budget.

Limitations are real. The training set uses fewer than 30K tasks "due to computational limitations" [§1] — it's unclear how performance scales with more data. The benchmarks, while covering 10 languages and 8.1K unique tasks in Others-to-All [§1], are constructed from the same dataset pipeline rather than drawn from real-world migration projects. For teams considering this for production legacy migration (say, COBOL-to-Java), the gap between curated benchmark programs and messy real codebases with decades of accumulated complexity is substantial — likely requiring significant additional work on domain adaptation and validation infrastructure.

The evaluation also relies on test-case pass rates, which measure functional correctness but not code quality, idiomatic style, or maintainability — all critical for actual migration projects.

Why It Matters

This is a lab proof-of-concept, but the pivot-language training strategy addresses a genuine bottleneck. If you need to support 10 languages, you need 9 translation directions instead of 90 — a 10× reduction in data requirements [§1]. For low-resource languages where parallel corpora are scarce, this could be the difference between feasible and infeasible.

The practical implication for teams building code migration tools: you may not need parallel corpora for every language pair. A well-chosen intermediate language combined with targeted RL can bootstrap multilingual capability from a single translation direction. The code and data are publicly available [Abstract], making this reproducible. But the distance from benchmark results to production-grade migration tooling — handling partial files, mixed-language codebases, framework-specific idioms — remains considerable.

A two-level memory system that lets an LLM reason about which past conversations matter — rather than retrieving everything that looks similar — improves adversarial question answering F1 from 0.54 to 0.78 compared to A-Mem while pulling 10x fewer dialogue turns.

The Problem

Conversational agents that persist across many sessions need to remember what happened before. The standard approach is vector similarity: encode every past dialogue turn as a numerical fingerprint, then retrieve the ones whose fingerprints most closely match the current query. This works well initially, but as conversation histories grow into hundreds of turns, it produces bloated evidence sets. "Once the most relevant memories have been recalled, adding more semantically similar memories often brings diminishing retrieval precision, inflating context cost, and making evidence harder to inspect and manage" [§2].

The core issue is that vector similarity cannot judge whether a memory is actually *worth reading* for a specific question. It lacks "a mechanism for reasoning over different levels of abstraction to actively assess which fine-grained details actually contribute to answering a query" [§1]. You end up feeding the answer-generation model a pile of vaguely relevant dialogue turns, which wastes tokens and can degrade response quality.

This matters practically because long-term agents — customer support bots, personal assistants, therapeutic chatbots — need to handle conversations spanning hundreds of turns across multiple sessions. The retrieval system has to be both thorough (high recall) and selective (high precision).

What They Did

HiGMem organizes memory into two layers. The bottom layer stores individual dialogue turns enriched with LLM-generated metadata — keywords, tags, timestamps, and contextual notes. The top layer groups related turns into "Event nodes," each containing a short summary and a structured fact sheet with explicit links back to the specific turns it covers [§3.1]. Think of it like a book's table of contents: each chapter heading (event) points to specific pages (turns).

When a new dialogue turn arrives, the system does two things simultaneously. It creates a Turn node with metadata extracted by the LLM using a sliding window of the five most recent turns for context [§3.2]. It also determines which existing Event nodes the turn belongs to, using vector similarity to find candidates and then an LLM to make the final affiliation decision [Eq. 4]. Events with fewer than 10 linked turns get their summaries fully refreshed; larger events just get the new entry appended [§3.2].

The retrieval strategy is where HiGMem diverges most from standard approaches. Given a query, the system retrieves the 10 most similar Turn nodes *and* the 10 most similar Event nodes via vector search. But instead of stopping there, it uses the Event nodes as "semantic anchors" — the LLM reads each event summary and then reasons about which of that event's linked turns might actually help answer the query [§3.3, Eq. 6]. This is like scanning chapter summaries to decide which specific paragraphs to read, rather than searching every paragraph individually.

The turns identified through both paths (direct vector retrieval and event-guided prediction) are merged, then an LLM-based filter produces the final evidence set [Eq. 7-8]. This adds LLM inference cost at retrieval time, but the system constrains it by only reasoning over turns linked to the top-10 events, not the entire memory store [§3].

All experiments use GPT-4o-mini for LLM operations and the same all-MiniLM-L6-v2 encoder for vector retrieval across comparable systems [§4.1].

The Results

On the LoCoMo10 benchmark — 10 long conversations averaging 587 turns each, with five question categories — HiGMem achieves the best F1 on four of five question categories [Abstract]. The most striking result is on adversarial questions (designed to test whether the system can correctly identify information *not* present in the conversation history): HiGMem improves F1 from 0.54 to 0.78 over A-Mem [Abstract]. This suggests the LLM-guided filtering is particularly effective at *not* retrieving irrelevant evidence — a critical capability for avoiding hallucinated answers.

HiGMem achieves this while retrieving "an order of magnitude fewer turns" than the strongest baseline [§1]. Fewer retrieved turns means lower token cost at the answer-generation stage and a more inspectable evidence set.

The limitations are significant. The evaluation covers a single benchmark with 10 conversations [§4.1]. The conversations in LoCoMo10 are synthetic multi-session dialogues, not production chat logs with messy, overlapping topics and mixed human-AI authorship. The system also adds multiple LLM calls per query (keyword extraction, event-guided prediction, filtering), which increases latency and API cost compared to pure vector retrieval — a tradeoff the paper acknowledges but does not quantify in terms of wall-clock time or dollar cost. For production systems handling thousands of queries per minute, this latency overhead could require significant engineering optimization to be viable at production scale.

The comparison also excludes H-MEM, a concurrent hierarchical system, because the authors "could not identify an official public implementation" with sufficient detail for fair reproduction [§4.1].

Why It Matters

This work demonstrates a concrete pattern: using LLM reasoning at retrieval time — constrained to a manageable scope via hierarchical structure — can dramatically improve precision without sacrificing recall. The adversarial F1 jump from 0.54 to 0.78 [Abstract] is particularly relevant for any application where confidently saying "I don't know" matters as much as finding the right answer.

This is a lab proof-of-concept, not production-ready infrastructure. But the architectural insight — let the LLM scan summaries first, then selectively drill into details — is immediately applicable as a design pattern for teams building persistent-memory agents. The code is publicly available, making it a reproducible starting point [Abstract]. If your agent framework is struggling with context bloat as conversation histories grow, this retrieval pattern is worth prototyping against your own data.