Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation

Marketers increasingly use large language models as "synthetic personas" to predict how an audience will react to a piece of copy before it ships, encouraged by evidence that profile-conditioned LLMs mimic human samples. This study asks whether that prediction is actually valid against real behaviour, and whether the persona machinery helps at all.

The work is a sim-to-real validity study using the Upworthy Research Archive — thousands of headline A/B tests on shared real traffic, with measured click-through — as held-out ground truth. It compares a ten-persona panel grounded in the real audience's demographics against a no-persona zero-shot baseline that simply asks the model how likely a typical reader is to click.

Two findings stand out. First, ground-truth reliability is the binding constraint: most A/B tests have no statistically distinguishable winner, so validity can only be measured on the reliable subset (n = 399). Second, counter to the persona-simulation premise, persona conditioning degrades predictive validity: the no-persona baseline ranks variants markedly better (Kendall τ = 0.361, a medium effect; top-1 accuracy 49.2%) than the persona panel (τ = 0.084; top-1 34.6%), with non-overlapping confidence intervals. Asking the model directly appears to tap an accurate population-level prior, while forcing it to role-play specific personas injects bias and noise.

The result replicates across three independent Upworthy splits, holds in direction on a different-domain news dataset, and is robust to seed, prompt phrasing, and model choice — across three Gemini tiers and a different model family (OpenAI gpt-4.1, significant paired gap). The takeaway: for predicting aggregate engagement, a plain LLM ranker beats persona simulation — synthetic personas are not merely a weak predictor, they are worse than not using them. All numbers regenerate from a public, artifact-first replication package.