Signal

LLM-based recommender systems fail not because they can't rank well, but because the right item never reaches them — standard retrievers surface the correct cold-start item only 4.6–22.9% of the time.

The Problem

Most LLM recommendation systems follow a two-stage pipeline: a fast retriever proposes a few hundred candidate items, then an LLM reranks them using item descriptions and user history. The pitch is that LLMs' semantic understanding should shine precisely where traditional systems struggle — new items, new users, and long-tail content where interaction data is sparse [§1].

But a reranker can only reorder items it receives. If the correct item never enters the candidate pool, no amount of LLM scaling or prompt engineering can recover it [§1]. Most prior evaluations sidestep this by injecting the gold item into the candidate pool, which tests reranking in isolation but says nothing about whether the pipeline would have surfaced that item in practice [§2].

The question this paper asks: how much of the cold-start problem is actually a retrieval problem, and how much is a reranking problem?

What They Did

The researchers built a benchmark across five domains — Amazon Arts, Amazon Video Games, MIND news, MovieLens-20M, and Yelp restaurants — with a protocol that explicitly separates the two stages [§3.1].

They define three evaluation regimes [Figure 1]. In the "positive-controlled" regime, the correct item is guaranteed to be in the candidate pool, isolating reranking quality. In the "retrieval-realistic" regime, retrievers search the full catalogue without injection, measuring whether the correct item appears at all. In the end-to-end regime, recall decomposes cleanly: end-to-end performance equals coverage (did retrieval find it?) multiplied by conditional accuracy (did the ranker pick it given it was there?) [§1, Eq. 1].

Interactions are split chronologically into 70/10/20 train/validation/test, and cold-start labels are computed from training data only [§3.1]. This temporal split is critical: under it, 32–91% of test targets are brand-new items with zero training interactions [Table 1]. MIND news is the extreme case, with 90.6% new items.

The retriever pool includes popularity, item-kNN, TF-IDF, BPR, LightGCN, SASRec, dense retrieval with SBERT/BGE, and a two-tower model. Rerankers include a cross-encoder and LLM rerankers (Qwen3-8B, Qwen3-32B, Llama-3.3-70B), scored via calibrated log-probability of a "Yes" answer [§3.2].

To address the retrieval bottleneck, they introduce LHF (Learned Hybrid Fusion), a validation-trained layer that learns to combine scores from multiple retrievers into a single ranked list — essentially a lightweight model that decides how much to trust each retriever for each query, trained on the validation split [§6].

The Results

In controlled pools where the gold item is present, LLM rerankers do not win any domain. LightGCN achieves .490 Recall@10 on MovieLens-20M versus .219 for the 8B LLM; graph/content fusion wins Arts (.220 vs .145) and Yelp (.159 vs .085) [Table 2]. Scaling from Qwen3-8B to 32B narrows the gap on four domains but still loses to the best collaborative or fusion baseline on four of five [Table 3]. The one exception is MIND news, where interaction signal is weak and text is rich — the 32B model reaches .169 versus .093 for the best baseline [Table 3].

The retrieval-realistic regime reveals the deeper problem. Standard single retrievers cover the gold item in only 4.6–22.9% of 200-item pools [Table 4]. Strong dense encoders max out at 10.6% coverage, and a two-tower retriever doesn't surpass the strongest collaborative retriever on collaboratively strong domains [§5]. The reason is structural: items with zero training interactions have no embeddings for collaborative methods to use, and text-based retrieval over large catalogues produces weak user-to-item matches [§5].

LHF is the only combiner tested that beats every single retriever on all five domains [Figure 2]. It recovers 17–61% of oracle coverage headroom on content-rich domains (Arts, Video Games, MIND) but only 5–7% on collaboratively strong domains (MovieLens, Yelp) [Abstract]. Even so, end-to-end experiments show a mismatch: a lightweight learned ranker (gradient-boosted) exploits the improved LHF pool, while prompt-level LLM reranking often degrades it — demoting correct items that simpler rankers would have surfaced [§1, Abstract].

Negative sampling choices also matter. With popularity-weighted negatives on Video Games, the best collaborative method drops from .157 to .084 while the LLM improves from .143 to .171 [§4.2]. Evaluation design can flip conclusions about LLM reranking quality.

The benchmark covers five domains but uses capped test sets of 2,592–5,000 users [Table 1]. The LHF fusion layer is validation-trained per domain, not a general-purpose retriever — production systems with hundreds of domains and mixed retriever stacks would need substantially more engineering. The temporal split, while realistic, creates an adversarial setting that may overstate the cold-start fraction relative to systems with faster retraining cycles.

Why It Matters

This work reframes where practitioners should invest when building LLM-augmented recommendation. The binding constraint for cold-start performance is retrieval coverage, not reranking sophistication [Abstract]. Scaling LLMs from 8B to 70B parameters does not solve a problem that lives upstream in the pipeline.

LLMs do show genuine semantic advantage in one specific configuration: text-rich domains where the correct item is already in the pool [§4.1, Table 3]. But under current retrieve-then-rerank architectures, that advantage is "largely unreachable" because retrieval rarely surfaces the item in the first place [Abstract]. This is a lab-stage diagnostic — it identifies the bottleneck but doesn't yet offer a production-ready fix. Teams building cold-start recommendation should measure their retrieval coverage before upgrading their reranker.

The first large-scale trace of real coding-agent workloads reveals that prefix-cache reads — not fresh computation — dominate LLM serving costs.

The Problem

Coding agents like Claude Code and Codex are among the fastest-growing LLM applications. GitHub Copilot has over 20 million users; Claude Code hit a $1 billion annualized revenue run-rate within six months of launch [§2.1]. But serving these agents efficiently requires understanding what their traffic actually looks like — and until now, nobody had the data.

Existing LLM serving traces from systems like Mooncake and Splitwise capture traditional chat and completion workloads, not the multi-step, tool-calling loops that define coding agents [§2.2]. Benchmarks like SWE-bench measure whether an agent can solve a task, but they test isolated problems and miss the context accumulation, tool diversity, and human idle time that shape real serving costs [§2.2]. Neither source captures coding agents as persistent, tool-using systems with long sessions, repeated tool calls, and human-paced gaps between requests [§2.2].

What They Did

Researchers at the University of Washington built a pipeline to extract, normalize, and anonymize the session logs that Claude Code and Codex already produce by default [§3.1]. They collected roughly 4,300 sessions from 43 developers over about eight months, spanning more than 20 model versions across both providers [§1]. The result: approximately 350,000 LLM steps and 430,000 tool calls from real day-to-day coding work — building systems, running evaluations, analyzing results [§2.2].

The key engineering challenge was normalization. Claude Code and Codex use different event structures, token accounting, and tool-timing metadata [§3.2]. The team unified both into a step-level schema where each row represents one LLM invocation plus any tool calls it triggers. For each step, they decomposed total input tokens into prefix tokens (history retrieved from cache) and append tokens (genuinely new input from the user or tool results) [§3.2].

The analysis then characterized the workload along four dimensions: session-level autonomy, per-step token distributions, tool-call patterns, and prefix-cache behavior [§1].

The Results

**Sessions are long and autonomous.** On average, completing a single user request requires 8.8 LLM calls and 10.8 tool invocations, taking 4.3 minutes with a p90 exceeding 6.4 minutes [§1]. This is not a quick question-and-answer exchange — it's a sustained loop where the model reasons, calls tools, reads results, and repeats.

**Steps have massive cached contexts but tiny new inputs.** The median step involves about 119,000 prefix tokens (cached history), only 875 append tokens (new input), and 214 output tokens [§1]. Despite long contexts, decode speeds are reasonable: a median of 46.8 normalized tokens per second for Claude and 33.9 for Codex, both with coefficient of variation exceeding 50% [§1].

**Tool usage is heavily skewed.** More than 80 distinct tools appear in the traces, but the top 3 tools for each provider account for over 80% of all calls — Bash, Read, and Edit for Claude; exec_command, write_stdin, and apply_patch for Codex [§1]. Tool latencies span milliseconds to hours. Calls longer than one minute represent only 4% of invocations but consume 85% of total tool-call time [§1].

**Prefix caching helps enormously but breaks at human boundaries.** The global prefix cache hit rate is 95.7%, but most misses cluster at the boundary between one request's final output and the next request's user input [§1]. The delay while a human reads, thinks, and types often exceeds the cache eviction timeout. These misses cause 3.8× more tokens to be prefilled than the truly unique input tokens would require [§1].

The dataset has clear scope limitations. It covers two commercial providers (Claude Code and Codex) used by 43 developers — all researchers at one university doing systems work [§1]. Open-source agents, non-coding agentic workloads, and enterprise-scale fleet behavior are not represented. For infrastructure teams, this means the specific distributions (tool popularity, session length, idle gaps) may shift significantly in other populations. The traces capture real usage diversity but not the full breadth of production coding-agent deployments — generalizing these patterns to fleet-level serving decisions will require broader data collection, likely a multi-year effort.

Why It Matters

The core finding is structural: coding-agent traffic is dominated by reading cached context, not generating new tokens. Prefix-cache reads drive the majority of API cost under current pricing [§1]. This inverts the optimization target compared to traditional chat workloads, where prefill of fresh prompts is the bottleneck.

The authors identify five concrete serving-system research directions grounded in their data: fused or batched tool invocations to reduce the overhead of frequent tool-LLM switching; append-length-aware prefill routing since new input varies wildly in size; semantic-aware tool-latency prediction for smarter cache eviction (because a "bash" call running a unit test has very different latency than one running a full build); sparse attention to reduce decoding cost over long prefixes; and better KV-cache management around human idle gaps [§1].

This is a pattern-emerging contribution, not a production solution. No new serving system is proposed. But for teams building or tuning LLM infrastructure for agentic workloads, the traces provide the first empirical foundation for understanding what they're actually serving. The dataset, collection pipeline, and analysis code are all public [Abstract].

Textual instructions that guide AI agents can now be compiled directly into model parameters — eliminating the need to stuff them into context windows at all.

The Problem

Agentic LLM systems increasingly rely on "skills" — structured text documents that encode problem-solving procedures, like step-by-step guides for debugging a CI pipeline or auditing user access. These skills get retrieved from a library and injected into the model's context window so it knows what to do [§1]. The approach works, but it has a ceiling.

The first bottleneck is instruction-following capacity. Open-source models of varying scales "largely fail to fully exploit skills through in-context learning, especially in long-context, complex task-completion scenarios where vital instructions are difficult to locate and follow" [§1]. Prior work has shown that in-context learning consistently underperforms fine-tuning, and the gap widens as model size decreases [§1].

The second problem is that optimizing skills in text space is decoupled from the model itself. Existing skill evolution methods iteratively rewrite skill documents based on failure analysis, but "the models themselves remain unoptimized" [§1]. The skill gets better on paper; the model doesn't change.

What They Did

ParametricSkills converts textual skills into LoRA adapters — small, plug-and-play weight patches that modify a frozen base model's behavior without retraining it [§2.1]. The conversion happens through a hypernetwork: a secondary neural network whose job is to produce the weights of another network. Given a skill document as input, the hypernetwork outputs LoRA parameters in a single forward pass — no gradient updates on the base model, no skill-specific data collection, and no context-window consumption at deployment [§2.2].

Think of it as a compiler for procedural knowledge: you feed in a text recipe, and out comes a small weight patch that makes the model behave as if it had been fine-tuned on that skill.

Training the hypernetwork happens in three stages [§3.1]. First, a pretraining phase uses three self-supervised objectives to teach the hypernetwork to faithfully encode skill content. In the simplest variant, the hypernetwork receives a full skill document and must produce adapters that let the base model reconstruct that document. Two harder variants — prefix completion (given only the beginning, predict the rest) and section-level cloze (a random section is removed, predict it from surrounding context) — force deeper structural understanding [§3.1.1].

Second, a fine-tuning phase trains the hypernetwork on actual skill exploitation trajectories — paired examples of a skill plus the correct multi-step behavior an agent should exhibit when applying it. The researchers synthesized these trajectories using the OpenCode sandbox across single-turn and multi-turn formats [§3.1.2].

To build the training data, the team constructed a library of 45,800 skills spanning 13 domains (software, AI/LLM, security, etc.), sourced from web crawls and summarized from real-world agentic trajectories [§1, Abstract].

The Results

On six software engineering subtasks, ParametricSkills outperforms in-context learning by an average of 6.44 points as judged by DeepSeek-V4-Flash, with +1.17 points in BERT Score and +5.53% in F1 score [Abstract]. Notably, SHINE — a prior text-to-LoRA baseline that compresses context into adapters — fails to surpass in-context skill performance at all [§1], suggesting that naive context compression isn't sufficient; the training pipeline matters.

Beyond raw performance, the paper demonstrates two additional capabilities. First, parametric skills integrate with existing text-based skill evolution algorithms: you can iteratively improve a skill in text space, then re-compile it into parameters, making "skill evolution directly equivalent to improving the model" [§1]. Second, parametric skills from multiple tasks can be accumulated into a global adapter via exponential moving average merging, offering what the authors describe as "a preliminary yet promising avenue toward test-time continual learning" [Abstract, §1].

The limitations are significant. Evaluation covers only software engineering tasks — six subtasks within a single domain [Abstract]. The skill library, while large at 45.8k entries, spans domains that may not represent the full diversity of production agentic workloads. The continual learning results are described as "preliminary" by the authors themselves [Abstract]. For practitioners, this means the approach is validated in a controlled SWE setting but unproven for general-purpose agent deployment — expect at least 1-2 years before this could be reliably integrated into production agentic frameworks across diverse domains.

Additionally, the evaluation relies on LLM-as-judge scoring (DeepSeek-V4-Flash) alongside automated metrics, which introduces its own biases. The paper does not report results on established agentic benchmarks like SWE-bench, making direct comparison with other agent improvement methods difficult.

Why It Matters

This work sits at lab proof-of-concept stage, but it addresses a real pain point. As agentic systems scale their skill libraries, context windows fill up, and smaller models struggle to follow complex procedural instructions buried in long prompts. Converting skills to parameters sidesteps this entirely — the model doesn't need to "read" the skill because the skill is already in its weights [§1, §2.2].

For teams building agentic systems with skill retrieval pipelines, the practical implication is architectural: if parametric skill generation matures, the skill retrieval step could output weight patches instead of text chunks, eliminating a class of instruction-following failures. The accumulation property — merging task-specific adapters into a persistent global skill — also suggests a mechanism for agents to genuinely learn from deployment experience without full retraining [§1]. Both capabilities are demonstrated only in controlled settings, but they point toward a design pattern worth monitoring.

A lightweight calibration layer lets RAG systems decide per-query whether to skip retrieval, fetch one passage, or fetch five — cutting passage budgets without sacrificing accuracy.

The Problem

Standard RAG pipelines retrieve a fixed number of passages for every query, regardless of difficulty. This creates two problems. First, many queries are already answerable from the model's parametric memory — retrieval for these wastes latency, bandwidth, and context tokens [§1]. Second, retrieval isn't monotonic: additional passages can introduce distractors that flip a correct closed-book answer to an incorrect one [§1].

Prior adaptive retrieval methods like Self-RAG, Adaptive-RAG, and TARG have explored when to trigger retrieval, but they typically produce raw uncertainty scores that are hard to compare across datasets, models, or actions [§1, §2]. A score that ranks queries well isn't the same as a probability you can use to set deployment thresholds or trade accuracy against cost.

What They Did

The core idea is simple: turn a raw confidence signal into a calibrated probability, then use that probability to pick how much retrieval each query gets.

For each query, the system runs the frozen LLM once without any retrieved context, generates a short greedy answer, and computes its length-normalized sequence log-probability — essentially, how confident the model's own token predictions were. A logistic regression model (fit on held-out data) maps this single number to a probability that the closed-book answer is correct [§3.2]. Think of it as converting a model's internal "gut feeling" into a percentage you can actually set thresholds on.

With that calibrated probability in hand, a graded policy routes each query to one of three retrieval budgets: skip retrieval entirely (k=0) if confidence is high, fetch one passage (k=1) at medium confidence, or fetch five passages (k=5) at low confidence. A fourth option — abstain — kicks in when even the retrieved answer's confidence is too low [§3.3, §3.4].

The same calibration framework also works on other uncertainty signals. The authors apply it to TARG-style prefix entropy and margin scores (which measure how uncertain the model is about its very first output tokens), showing that calibration improves these signals too [§3.2, Table 1].

Critically, the authors separate three cost axes that are often conflated: retrieval calls (did you hit the retriever at all?), full-context usage (did you prepend all five passages?), and passage budget (total passages consumed, normalized). This accounting matters because fetching one passage still costs a retrieval call [§3.3].

The Results

Calibration dramatically improves probability quality. On Qwen3-8B, raw sequence log-probability has an expected calibration error (ECE) of 0.275 on TriviaQA, 0.643 on Natural Questions, and 0.711 on MS MARCO. After out-of-fold logistic calibration, those drop to 0.062, 0.009, and 0.031 respectively [Table 1]. Ranking performance (AUROC) stays essentially unchanged — calibration fixes the probability scale without hurting discrimination.

Graded allocation improves the right cost axes. On TriviaQA with Qwen3-8B, binary gating achieves a call-AUC of 0.731. Graded allocation raises full-context AUC to 0.773 and passage-budget AUC to 0.760, but call-AUC stays at 0.732 [Table 2]. The same pattern holds for TARG entropy: binary call-AUC of 0.736 becomes full-context AUC of 0.775 [Table 2]. The takeaway: graded routing saves passages and context length, not retrieval calls.

Scaling matters. On TriviaQA, closed-book accuracy rises from 27.3% at 1.7B parameters to 69.3% at 32B, and the retrieval rate needed to match always-RAG drops from 99.5% to 46.0% [§5.3]. The method is most useful when the model already knows a lot.

Deployable thresholds behave sensibly. On TriviaQA, validation-selected thresholds skip roughly one-third of queries while coming within 1.5 percentage points of always-RAG accuracy. On NQ and MS MARCO, the policy correctly chooses near-always retrieval because closed-book headroom is low [Table 3, §5.4].

Latency results are sobering. The closed-book probe is an extra generation pass, so gating is faster only when enough queries are skipped. At matched-accuracy operating points, gating increases latency by about 27% on Qwen3-8B but saves about 8% on Qwen3-32B [Abstract, §3.5]. The break-even depends on model size and skip rate.

Limitations are real. The method is evaluated on three QA benchmarks with single-hop factoid questions — multi-hop reasoning, long-form generation, and mixed human-AI query streams are untested. MS MARCO's noisier answer semantics already reduce the method's headroom [§4]. Thresholds must be recalibrated per model [§5.3], and the cost model assumes a two-stage pipeline where the probe runs to completion before retrieval starts — streaming or speculative architectures would change the math. For production systems handling diverse query types, reliable tooling is likely 1-2 years out.

Why It Matters

This work reframes adaptive RAG as a calibration problem rather than a gating problem. The practical implication: if you're building a RAG system where context tokens or passage volume drive cost — think per-token API billing or limited context windows — calibrated routing can reduce that cost on queries the model already handles well. But if your bottleneck is retrieval latency and your model's closed-book accuracy is modest, the extra inference pass may not pay for itself [§5.5, Abstract].

This sits at the lab proof-of-concept stage with deployable threshold experiments. The framework is model-agnostic (confirmed across Qwen and Llama families [§5.3]) and the code is public. Teams running high-volume RAG with measurable skip-rate headroom have the clearest path to testing it.