Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation

The paper addresses a known weakness of LLM-as-a-Judge: scoring bias, where an evaluator LLM assigns the same scores regardless of the text being evaluated. To counter this, the authors instruct the LLM to first randomly generate a number token as part of the evaluation process, introducing variability that shifts scores away from default patterns. This lightweight approach does not require additional training or external data, enabling easy integration into existing evaluation pipelines. If successful, it could improve the reliability of automatic evaluation across tasks like summarization and dialogue quality, where LLM judges are increasingly used over traditional metrics.