Rich Sutton, author of ‘The Bitter Lesson’ and founder of Oak Lab, has publicly cautioned against over-relying on continuous parameter and text scaling for large language models. Sutton argued that raw compute alone will not achieve general intelligence without novel algorithmic approaches, criticizing synthetic data reliance and advocating for agents that learn continuously from real-world interaction.
The commentary has reignited debate across the AI research ecosystem. Former OpenAI researcher Jerry Tworek noted that while computation remains essential, research has become overly focused on transformer pretraining, leaving alternative scaling paths unexplored. Conversely, OpenAI CEO Sam Altman disputed claims that scaling has hit a wall, pointing to recent mathematical conjecture proofs driven by standard model scaling.
The debate coincides with growing venture backing for non-standard architectures. Emerging research labs such as Tworek’s Core Automation and Yann LeCun’s Advanced Machines Intelligence are raising funds specifically to explore continuous learning systems, spatial reasoning, and world models beyond standard pretraining paradigms.
Why it matters
Signals a growing research shift away from pure LLM pretraining toward continuous learning architectures.
Highlights strategic divergence between lab founders pursuing alternative paradigms and incumbents scaling standard transformers.
Points to rising investor interest in world models and post-transformer intelligence approaches.
Source: the-decoder.com



