SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

The paper addresses the challenge of evaluating LLM agents that execute long-horizon tasks via tool use and environment interaction. As evaluation shifts from final-response scoring to full-execution verification, skill-augmented agents require judges to understand the procedural knowledge encoded in task-time skills—this indicates what evidence to inspect and which failures are critical. Existing judge benchmarks mostly expose final responses or static trajectories, rarely combining task-time skills with directly inspectable artifacts and environments.

To fill this gap, the authors introduce SkillTV-Bench, a benchmark of 681 real agent trajectories from 50 tasks across eleven domains, designed to evaluate skill-aware trajectory verification for both LLM-as-a-Judge and Agent-as-a-Judge methods. They also propose SkillTV-Evolve, which externalizes verification knowledge as a reusable JudgeSkill. This skill guides an agent judge to plan targeted inspections and issue evidence-grounded verdicts. An automated evolution loop refines the JudgeSkill using misjudged cases from a disjoint development pool.

On SkillTV-Bench, the refined skill improves the same agent judge's accuracy by 14.8 percentage points. In offline rollout-pool selection, it increases selected-trajectory success from 22.9% with one rollout to 45.5% with ten rollouts. Code and data are publicly available at the provided GitHub link.