The LLMentalist Effect (2023)
Writing after a year researching language and diffusion models in software businesses, the author is perplexed that many people are convinced chat-based LLMs are intelligent. He argues there is no mechanism in LLMs that would enable this: LLMs are not brains and do not share the biological mechanisms animals or people use to reason, and they are fundamentally a mathematical model of language tokens that returns a mathematically plausible response to input text. There is no reason to believe they think or reason, and AI researchers and vendors have repeatedly emphasized that these models do not think. He lays out two possibilities: either the tech industry accidentally invented the initial stages of a completely new kind of mind based on unknown principles and processes with no biological parallel, or the intelligence illusion is in the mind of the user, not the LLM. The author, along with many AI critics, firmly holds the second view, which is why his book on generative AI risks is titled The Intelligence Illusion.
After months of work, he proposes a mechanism for this illusion: it resembles a psychic's con, commonly called cold reading, and looks like an accidental automation of that tactic. Both chatbot and psychic use validation statements, such as sentences that exploit the Forer effect, to give the impression of extremely specific answers that are in fact statistically generic. The psychic appears to read minds or hear secrets of the dead; the chatbot appears to be an intelligence specifically engaging with the user and their work, but that impression is just a statistical trick.
The idea took shape when he reviewed statements people made about the reasoning of these 'AI' systems. He initially dismissed them as classic tech-bubble enthusiasm, but concluded that AI has attracted a different crowd whose believers sound unlike those of prior bubbles. He quotes examples: 'This is real. It’s a bit worrying, but it’s real'; 'There really is something there. Not sure what to think of it, but I’ve experienced it myself'; and 'You need to keep your mind open to...'. He now believes there is even less intelligence and reasoning in these LLMs than he previously thought, and that many proposed use cases look like borderline fraudulent pseudoscience.