Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

Evolution Strategy (ES) is an attractive alternative to gradient-based fine-tuning for LLM reasoning when compute is limited, but directly applying ES to billion-parameter models fails because random perturbations are nearly orthogonal to useful update directions in high-dimensional spaces. Hyper-ES addresses this by first running a few inexpensive gradient-based fine-tuning runs to obtain descent directions, then using CMA-ES to optimize layer-wise DARE-TIES merging coefficients within the subspace spanned by those directions. This lets ES combine meaningful descent directions instead of searching arbitrary full-model perturbations, exploiting ES's strength in low-dimensional optimization while avoiding its weakness in full-parameter search.

Evaluated on three Qwen2.5-Instruct and DeepSeek-R1-Distill backbones across six mathematical reasoning datasets, Hyper-ES consistently outperforms GRPO-LoRA by about 1% while requiring 10% fewer space-consuming gradient updates. The method is open-sourced at the provided GitHub repository.