LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

LangChoiceBench, introduced in arXiv:2608.06041, is a project-level code-generation benchmark designed to systematically measure Python preference, recommendation-implementation consistency, and language diversity in large language models. It covers 28 projects across seven software areas where Python is often a poor default, providing a controlled test of whether models consider project requirements or simply fall back on Python. The authors evaluate 25 diverse LLMs and find that Python is still heavily over-selected overall, that recommendation-implementation consistency (whether the chosen language matches the language recommended in reasoning) is low, and that smaller open-weight models tend to show stronger Python preference and lower language diversity. They also analyze 9,826 reasoning traces, revealing that most Python choices are either automatic or driven primarily by ease rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python—a failure mode the authors call 'phantom evidence'—or produce code that contradicts the language selected in their own reasoning. These findings highlight a persistent bias in LLM code generation and provide a benchmark for tracking progress toward more context-aware language selection.