Why don't machine learning research agents overfit?

Machine learning is fundamentally about generalization rather than memorization: a model must perform well on unseen examples, and if it does well on training data but poorly on fresh data, it has overfit and only fooled itself into thinking it learned something.

The standard safeguard is to hold out data. A validation set is consulted repeatedly while building a model — to compare candidates, tune hyperparameters, and decide what to try next — while a final test set is meant to be touched only once. Because the training procedure never saw the test set, strong performance there is supposed to be a correct proxy for new examples encountered in the wild.

That guarantee depends on the held-out set staying genuinely unseen. If you check performance on it, tweak the training procedure in response, recheck, and iterate toward better numbers, the set is no longer unseen; it becomes part of your training procedure. Done enough times, you can overfit it just as you might overfit the training set, losing the proxy for unseen data. This applies to any reused held-out set, including a validation set, which is reused by design.

Real machine learning research resembles exactly this iterative improvement loop. The community relies on a handful of benchmark datasets that go unrevised for years, repeatedly evaluating models, revising training procedures, re-evaluating, publishing, and letting the next group eke out slightly more improvement. By the textbook account, this hill-climbing against held-out benchmarks should produce rampant overfitting and saturate leaderboards with models that look great on the benchmark but mediocre everywhere else.

Yet studies that build entirely fresh test sets for old, heavily reused benchmarks have found that improvements largely transfer: on the new data, models demonstrate the same gains they did on the old benchmark. Benchmark-driven machine learning has therefore produced rapid and largely real progress, contrary to the textbook prediction. The excerpt says there is no shortage of hypotheses for why, but they are hard to test empirically because the “subject” of the experiment would be the entire human research community/process — the text cuts off mid-word at “human r.”