RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

The paper identifies a mismatch between generative reward models, which naturally compare responses, and the scalar scoring paradigm used in existing reinforcement learning (RL) algorithms. This mismatch limits the effectiveness of generative reward models in RL fine-tuning. To address this, the authors propose Ranking-based Reward Construction (RRC), which converts relative preference rankings from generative reward models into rewards suitable for RL.

RRC consists of two complementary strategies: self-competitive ranking, which leverages comparisons among sampled responses from the model itself, and anchor-guided ranking, which scales reward construction using a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks show that RRC consistently outperforms existing reward construction approaches when combined with generative reward models. Code is publicly available at https://github.com/wangclnlp/RRC.