Signal

A three-axis uncertainty estimator tailored to code properties improves average error detection AUROC from 0.696 to 0.776 across five LLMs — and a single-pass token entropy signal alone can match multi-pass baselines while being over 3× cheaper.

The Problem

When an LLM generates code that's silently wrong — compiles fine, looks reasonable, but returns incorrect results — the consequences compound. Bad code slips past review, gets integrated into pipelines, and breaks things downstream. Uncertainty estimation (UE) is supposed to catch this: score how confident the model is in its output, then flag or defer low-confidence generations for human review [§1].

The problem is that current UE methods for code are borrowed directly from natural language generation. They compute things like average token likelihood or semantic similarity between sampled outputs — techniques designed for prose, where a wrong word rarely ruins the whole passage [§1]. Code is different. A single flipped operator can break an entire program. Two syntactically different implementations can encode the same algorithm. And unlike natural language, code can actually be *run* [§1].

These three properties — which the authors call token fragility, the intent–code gap, and executability — mean that NL-derived uncertainty methods are systematically ignoring information that's available and useful for code [§1].

What They Did

The core idea is to decompose code uncertainty into three independent axes, each targeting one code-specific property [§2].

**Lexical uncertainty (token fragility).** Instead of averaging entropy across all generated tokens — the NL default — this signal focuses on the *most uncertain* positions. The method computes per-token entropy from the model's output distribution, then averages only the K highest-entropy tokens [§2.2]. Think of it as checking the model's confidence only at the few critical decision points (which operator? which index? which function name?) rather than diluting that signal across hundreds of boilerplate tokens. This requires only a single forward pass — no resampling.

**Algorithmic uncertainty (intent–code gap).** Two correct quicksort implementations can look completely different syntactically, and two near-identical code snippets can encode different algorithms. Comparing raw code is unreliable. Instead, the method prompts the LLM to generate N natural-language solution plans — numbered reasoning steps, no code allowed — and measures how much those plans agree with each other using a step-aware text similarity metric [§2.3]. Low agreement across plans means the model is uncertain about *which algorithm to use*, independent of implementation details. Conditioning Qwen3-14B on ground-truth pseudo-code lifts MBPP pass@1 from 54% to 98%, confirming that algorithmic-level information carries substantial correctness signal [§2.3].

**Functional uncertainty (executability).** The model generates test cases for the problem, then the candidate program is executed against them in a sandbox. The uncertainty score is simply the fraction of self-generated tests that the program fails [§2.4]. This is a behavioral signal unavailable in any NL setting. Importantly, this is used as a *calibration* signal — scoring confidence — not as a selection mechanism like CodeT-style reranking [§2.4].

The three scores are combined via a rank-normalized weighted sum. The ensemble exists to demonstrate that the axes are complementary; the individual signals are the primary contribution [§1].

The Results

Evaluated across four benchmarks (APPS-Intro, APPS-Interview, HumanEval, MBPP) and five code-capable LLMs, the three-axis ensemble achieves 0.776 average AUROC, compared to 0.696 for the strongest NL-derived baseline — an improvement of 8.1 points [Abstract]. On Qwen3-14B specifically, the ensemble reaches 0.800 average AUROC [§1].

The single-pass Top-K token entropy signal is surprisingly strong on its own. On Qwen3-14B, it matches the strongest multi-pass NL baseline while costing over 3× less compute, since it requires no resampling [Abstract]. Across all five models, it remains competitive as a low-cost signal [§1].

A revealing diagnostic: splitting token entropy over code tokens versus comment tokens shows that code-only entropy achieves 0.716 AUROC, while comment-only entropy scores 0.375 — worse than random [§1]. Full-sequence averaging, the NL default, masks this asymmetry entirely.

The limitations are concrete. The evaluation uses four benchmarks with clean problem specifications and official test suites [§2.1] — production codebases involve ambiguous specs, mixed human-AI authorship, and partial test coverage, which would stress all three axes differently. The functional signal depends on self-generated tests, which can be wrong, biased toward easy cases, or miss edge conditions [§2.4]. The algorithmic signal requires N additional LLM calls to generate solution plans [§2.3], adding latency that may be impractical for real-time autocomplete. For teams needing production-grade uncertainty tooling for messy, real-world codebases, reliable solutions are likely still a few years out.

Why It Matters

This work demonstrates that treating code as "just another token sequence" for uncertainty estimation leaves measurable performance on the table. Each of the three code-specific axes independently matches or outperforms the strongest NL baseline on the property it targets [§1], and combining them yields clear gains.

For practitioners, the immediate takeaway is practical: if you're using likelihood-based or semantic-consistency UE in a code generation pipeline, Top-K token entropy is a drop-in, single-pass alternative that's cheaper and potentially more informative [§1]. The three-axis ensemble is better but requires execution infrastructure and multiple LLM calls.

This is a lab proof-of-concept — four benchmarks, five models, controlled conditions. But the finding that code-specific properties carry orthogonal uncertainty information is well-supported by the results and suggests that code UE methods should be designed for code, not inherited from NL.

A 32B open-weight model now performs cloud console documentation verification within 1.82 percentage points of the best frontier model — at 92% lower inference cost.

The Problem

Cloud platforms like Alibaba Cloud ship hundreds of products with rapid UI changes, but their documentation is maintained manually. The result is documentation drift: step-by-step guides that no longer match the actual console interface [§1]. Detecting these mismatches at scale requires an estimated 4 million recurring inspections annually, each taking roughly 1.6 hours of manual effort, yet current manual coverage sits below 1% [§1, Abstract].

The team's approach treats execution as verification: an AI agent follows each documented procedure end-to-end in the real console, and any step that can't be completed flags a potential documentation defect [§1]. An initial production deployment using frontier proprietary models confirmed the approach works — auditing 54,000+ procedures and surfacing 4,399 confirmed defects accepted by product teams [§1]. But scaling further hits two walls: the per-call cost of frontier models is prohibitive for millions of inspections, and sending cloud metadata containing sensitive information to external APIs creates compliance risk [§1].

What They Did

The team trained a 32B vision-language model (AliyunConsoleAgent-32B, built on Qwen3-VL architecture) through two stages: supervised fine-tuning on trajectories distilled from frontier models, then reinforcement learning in real cloud environments [§3, Abstract].

**Stage 1: Learning from frontier models.** The team used capable proprietary models to generate successful task trajectories — complete recordings of an agent navigating the console step by step. These trajectories were filtered for quality and used to fine-tune the smaller model, essentially teaching it to mimic expert behavior [§2, Abstract]. The agent operates in a loop: at each step it sees a full-page screenshot with numbered labels overlaid on interactive elements (a technique called Set-of-Mark prompting), reads extracted page text, reasons about what to do next, then clicks, types, or takes another action [§3.2].

**Stage 2: Reinforcement learning in real environments.** The harder contribution is making RL work in a live cloud console. Cloud tasks have deep resource dependencies — you can't test "modify a database whitelist" if no database exists in the account. Without proper setup, the agent fails before it can even attempt the task, and the training process can't distinguish environmental failures from genuine mistakes. This pollutes the reward signal and prevents the model from learning [§4].

To solve this, the team built a four-layer rollout environment [§4]. A pool of isolated test accounts prevents cross-task interference. For each task, a "Resource META" template — containing dependency descriptions, configuration attributes, and executable Terraform scripts — pre-provisions the exact resources needed [§4, Table 1]. When pre-provisioning isn't sufficient, an on-demand system called ResourceCoder (an LLM-based coding agent) writes and executes infrastructure scripts mid-task [§3.2]. All provisioned resources are created from scratch with randomized naming to ensure portability across accounts [§4].

For reward signals, the team uses a dual-channel approach: rule-based verification against ActionTrail (Alibaba Cloud's API audit logs) checks whether the agent actually performed the correct backend operations, combined with LLM-based outcome evaluation [§3.1, Abstract]. Grounding rewards in audit logs rather than visual checks makes the signal objective and resistant to reward hacking — the agent can't game the reward by making the screen look right without actually completing the operation [Abstract].

The Results

On a 278-task benchmark where even the best frontier model (Gemini 3 Pro Preview) achieves only 65.34%, AliyunConsoleAgent-32B reaches 63.52% — a 20.24 percentage-point improvement over the base model [Abstract]. The gap to the frontier model is 1.82 percentage points, and the bootstrap 95% confidence interval ([−1.27, 7.39]) includes zero, suggesting the difference may not be statistically significant [Abstract]. This performance comes at 92% lower inference cost [Abstract, Figure 1].

In production, the frontier-model-based system has already audited over 54,000 documented procedures and identified 4,399 confirmed defects accepted by product teams [§1].

Several limitations deserve attention. The benchmark contains 278 tasks specific to Alibaba Cloud's console — generalization to other cloud platforms (AWS, Azure, GCP) with different UI patterns and API structures is untested. For teams considering cross-platform deployment, significant domain adaptation work would be needed, likely requiring new training data and environment infrastructure for each platform. The approach also depends on audit log availability for reward signals; platforms without comprehensive API logging would need alternative verification mechanisms. The rollout environment, while effective, requires substantial engineering investment — Terraform templates, account pool management, and containerized execution — making this a heavy infrastructure commitment rather than a lightweight training recipe.

Why It Matters

This work demonstrates that the SFT-then-RL training paradigm can close most of the gap between a privately deployable 32B model and frontier proprietary models on complex, real-world web automation tasks [Abstract]. The key engineering insight is that RL in production environments requires explicit isolation of environmental noise from the training signal — without the resource provisioning infrastructure, the agent can't learn effectively because most failures aren't its fault [§4].

This sits at the boundary between lab proof-of-concept and early production: the frontier-model version is already deployed and finding real defects, and the trained 32B model has demonstrated sufficient performance to serve as a replacement [§1]. For organizations running large-scale cloud operations or enterprise web automation, the pattern is clear: distill from expensive models, then refine with RL in controlled real environments. The team has open-sourced their evaluation benchmark, training data, rollout infrastructure, and model training code [§1].

Compressing the procedural documents that guide AI agents can backfire — removing the wrong details cuts prompt costs but can inflate downstream execution costs by as much as 14% in the authors' evaluation.

The Problem

LLM agents don't just run on model weights — they depend on "skills," which are reusable procedural documents that encode workflows, tool usage patterns, code snippets, validation checks, and domain rules [§1]. Think of a skill as an internal runbook: it tells the agent how to call a specific API, what order to run checks in, or what formula to apply.

As these skill documents grow, a natural instinct is to compress them. Shorter prompts mean fewer tokens, which means lower inference cost. But skills aren't ordinary prompts. Their value often lives in what the authors call "sparse operational anchors" — an API constructor signature, a command-line flag, a validation threshold, a recovery rule [§1]. Remove the surrounding explanation and you save tokens. Remove the wrong anchor and the agent starts exploring, debugging, retrying tool calls, and burning far more tokens downstream than you saved upfront.

In the authors' own evaluation, a fixed workflow-oriented rewrite shortened skill documents but pushed downstream agent-token usage to 1.14× the original baseline [§1]. The question isn't "how short can we make this?" — it's "which information should a skill remember?"

What They Did

The researchers built a controlled framework with three stages: profile the skill, rewrite it using a specific preservation strategy, then evaluate the rewrite by running the agent and measuring both quality and cost [§3.1].

**Profiling.** For each task-skill pair, they compute a structural profile capturing features like code-token ratio, API usage density, number of validation rules, formulas, and task domain [§3.2]. This profile acts as a fingerprint describing what kind of operational knowledge the skill contains.

**Rewriting strategies.** Rather than one-size-fits-all compression, they define a menu of information-preservation strategies. Each strategy specifies which class of operational anchors to keep prominent after rewriting [§3.2, Table 1]. For example, an API/code anchoring strategy preserves import statements, object constructors, and code snippets. A workflow-guarding strategy preserves ordered procedural steps and failure-recovery cues. A rule/formula anchoring strategy preserves thresholds, schemas, and tie-breaking conventions. Rewrites are audited with lightweight checks — token ratio, code-term coverage, code-block retention, missing-anchor detection — and repaired when protected anchors are dropped [§3.2].

**Policy selection.** A lightweight task-conditioned policy learns to pick the best preservation strategy for each task based on its structural profile [§1, §3]. This is a selector, not a generator — it chooses among the predefined strategies rather than writing new skills from scratch. The policy is trained on profiling runs and then frozen for evaluation.

Critically, throughout all experiments, the task instructions, execution environments, and verifiers remain fixed. Only the skill documents change, isolating skill wording and structure as the experimental variable [§3.1].

The Results

The core finding: no single rewrite strategy dominates across all tasks. API/code anchoring, workflow guarding, and rule/formula anchoring each benefit different task families [Abstract]. This confirms the authors' hypothesis that effective rewriting depends on task and skill structure.

The learned policy, which selects strategies per task, reduced total execution cost by 7.0% and downstream agent-token cost by 6.0% on the primary held-out evaluation while maintaining or slightly improving quality [Abstract, §1]. When the same frozen policy was transferred to additional agent stacks without retraining, the reductions averaged 14.7% for total cost and 13.7% for agent-token cost, while verifier-measured quality was maintained or slightly improved [Abstract, §1].

The economic metrics the authors introduce separate five dimensions: task quality, direct skill-token cost, downstream agent-token cost, total execution cost, and execution overrun [§1]. This decomposition matters because it reveals cases where compression helps on one axis but hurts on another — the workflow rewrite that shortened skills but inflated agent tokens to 1.14× baseline being the clearest example [§1].

**Limitations.** All experiments run on SkillsBench [§1], a benchmark with structured procedural tasks and deterministic verifiers. Real production agent deployments involve messier conditions: mixed human-and-AI-authored skills, ambiguous task boundaries, non-deterministic evaluation, and skills that evolve over time. The cross-model transfer results are encouraging but cover a limited set of agent stacks — scaling this to the diversity of production configurations is an open problem, likely requiring substantially more profiling data and strategy coverage.

Why It Matters

This work reframes skill optimization from a text-compression problem to a knowledge-engineering problem [Abstract]. The practical implication is concrete: if you're managing a library of agent skills and your optimization strategy is "make them shorter," you may be increasing total cost.

The framework sits at the lab proof-of-concept stage. The benchmark is controlled, the strategy menu is manually defined, and the policy is lightweight by design. But the core insight — that different task types need different information preserved — is immediately useful for teams building agent systems today. Even without adopting this specific framework, tracking downstream agent-token cost alongside prompt length would surface cases where compression is counterproductive.

For organizations running LLM agents at scale, the 7–15% cost reduction range demonstrated here [Abstract] is meaningful, especially since it comes without quality degradation. The authors release their framework as SkillEE [Abstract], making the profiling and evaluation pipeline available for teams to test on their own skill libraries.

A handful of learned prompt tokens can replicate the safety behavior of a full guard model at less than 1% additional memory cost.

The Problem

The standard way to make an LLM safe in production is to pair it with a guard model — a second LLM that inspects every response and blocks anything harmful before the user sees it [§1]. This works well, but it means running two full models: the LLM generates a response, then the guard classifies the prompt-response pair as safe or unsafe [§3]. If unsafe, a canned refusal replaces the output [Equation 1].

On a phone or other edge device, that's a dealbreaker. Two full forward passes through LLM-scale models significantly increases memory and compute requirements [§1]. It also degrades time-to-first-token, since the guard has to wait for the complete LLM output before it can classify anything [§1]. The question the authors set out to answer: which combination of lightweight adaptation method and training objective can best replicate the dual-model safety system's behavior without actually running two models [§1]?

What They Did

The core idea is distillation — training a tiny set of extra parameters to mimic the input-output behavior of the full dual-model system, then throwing away the guard model at inference time [§3.1].

The authors tested three families of adaptation methods: **soft prompts** (a short sequence of learned continuous vectors prepended to the user's input embeddings), **LoRA adapters** (low-rank weight updates injected into model layers), and **steering vectors** (directional nudges applied to internal representations) [§1, §2]. They crossed these with multiple training objectives: their proposed total variation (TV) distillation, KL divergence distillation, perplexity-based training, and policy gradient methods [§3.1, §3.2].

The distillation target is the full safe LLM system's output distribution. Think of it this way: the dual-model system either passes through the LLM's original response (if the guard says it's safe) or substitutes a refusal message (if unsafe). The distilled model needs to learn to do both of these things on its own, without the guard [§3.1, Figure 2].

For the TV distillation objective specifically, the training loss has two terms. The first preserves the LLM's original response when the guard would have deemed it safe. The second teaches the model to produce the refusal message when the guard would have flagged the response [Equation 4]. Crucially, this requires no labeled safety dataset — the guard model itself provides the safe/unsafe signal during training [§3.1].

The TV objective comes with a formal guarantee: for any bounded evaluation function, the performance gap between the distilled model and the full dual-model system is at most twice the total variation distance between their output distributions [Theorem 3.1]. The KL divergence objective provides a similar but looser bound via Pinsker's inequality [§3.1].

The soft prompts themselves are small — around 100 learned vectors amounting to a few thousand parameters [§3.1]. At inference, they're simply prepended to the user's prompt embeddings, requiring a single forward pass through the LLM.

The Results

Across four LLM architectures and multiple safety benchmarks, soft prompts trained via distillation (both TV and KL variants) consistently achieved the best safety-usefulness trade-offs compared to LoRA adapters, steering vectors, and non-distillation training objectives [§1, Abstract].

The practical efficiency gains are stark: soft prompt distillation adds less than 1% memory overhead and less than 10% compute overhead at inference, compared to the approximately 2× cost of running the full dual-model system [§1]. The authors validated these numbers with on-device measurements on smartphones powered by Qualcomm Snapdragon hardware [§1].

Both TV and KL distillation proved effective empirically [§3.1], though the TV objective provides tighter theoretical guarantees on downstream safety performance [Theorem 3.1]. The authors found that even a modestly sized set of 100 soft prompt vectors was sufficient to reduce the total variation distance to a small value while maintaining the LLM's fluency and the guard model's safety properties [§3.1].

There are important caveats. The evaluation spans four LLM architectures and standard safety benchmarks, but production codebases face adversarial users, distribution shift, and prompt injection attacks that these benchmarks may not capture — meaning real-world robustness is unproven and likely requires additional hardening work over the next 1-2 years. The method also inherits the guard model's limitations: if the guard model itself is vulnerable to adversarial perturbations — which recent work has shown is possible [§1, citing Mangaokar et al., 2024] — the distilled soft prompts will likely replicate those blind spots. Additionally, the distillation requires access to the guard model during training, so organizations need the full dual-model system available offline even if they don't deploy it [§3.1].

Why It Matters

For teams shipping LLMs on phones, wearables, or other edge hardware, this work offers a concrete path to safety without doubling resource requirements. The approach is a lab proof-of-concept validated on real hardware, not yet battle-tested against sophisticated attacks or diverse production workloads.

The practical takeaway: if you're currently planning to deploy a guard model alongside your LLM on-device, soft prompt distillation is worth evaluating as an alternative that could recover most of the safety benefit at a fraction of the cost [Abstract]. The training pipeline needs only unlabeled prompts and access to the guard model offline [§3.1], making it relatively straightforward to integrate into existing safety workflows. The formal guarantees on downstream performance deviation [Theorem 3.1] provide a principled basis for safety auditing — something ad hoc fine-tuning approaches lack.