These startups are chasing the next big thing in LLMs

MIT Technology Review's What's Next series examines the future of large language models, noting that the transformer architecture introduced in 2017's "Attention Is All You Need" underpins every major LLM today. However, transformers are showing their age: recent advances such as reasoning models and handling large inputs are workarounds that patch over fundamental flaws rather than natural extensions of the core technology.

The core problem lies in dense attention, which compares every token with every other token via multiplication. This leads to quadratic scaling: a 10,000-word document may require 50 million multiplications, making LLMs extremely power-hungry. Costs are massive—OpenAI is projected to spend $50 billion on computing this year, and the IEA predicts data center electricity consumption will double by 2030. Transformers also struggle with very large context windows and with keeping track of information, limiting their ability to reason over entire codebases or libraries.

Because of these constraints, a growing number of startups—such as Subquadratic, led by CEO Justin Dangel—are chasing alternative architectures for what MIT calls 'LLMs+.' While many will likely fail, these newcomers have less to lose than the industry giants and are betting on new approaches to surpass the limits of transformer-based models.