Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers

The paper starts from the observation that retrieval-augmented generation (RAG) is commonly used to supply external evidence when a model's parametric knowledge may be outdated, but retrieval only helps if the retrieved evidence is still valid. It identifies a temporal alignment failure called stale-document poisoning, in which outdated evidence causes a model to answer incorrectly even though the model answered correctly without retrieval.

To study this, the authors construct a benchmark of 317 verified knowledge reversals across medicine, law, software, and platform policy, grounded in dated official sources. Across 12 models, recent medical reversals are harder than long-established ones. Outdated retrieval flips 30% of Llama answers and 37% of Qwen answers even without instructions to trust the document; when explicit follow instructions are added, those flip rates rise to 66% and 75%. Across four open models and four domains, poisoning ranges from 17% to 91%, whereas matched up-to-date evidence is followed in 97–100% of trials.

To isolate temporal applicability, the authors keep the historical evidence unchanged across 50 reversals and vary only the evaluation date. Dates alone produce only modest adaptation, but when models are explicitly told when the old evidence stops applying, the larger models switch to the appropriate answer almost perfectly. Causal interventions confirm that this validity information directly shapes the final decision. The same internal components also support broader comparison tasks, suggesting that temporal applicability can recruit a general reasoning mechanism used for other comparisons.

Finally, a fixed recency-aware hybrid re-ranker reduces poisoning by 4.6–10.0 points when dates are accurate, though gains depend on reliable temporal metadata. The authors conclude that reliable RAG requires selective trust: models must determine not only what retrieved evidence says but whether it still applies.