In a recent interview on Sequoia Capital's podcast Training Data, Rich Sutton, a pioneer of reinforcement learning, criticized synthetic data as a huge mistake and argued that large language models only achieve a quarter of intelligence. He also announced the founding of Oak Lab to advance continual learning.
SynthePulse Insight · AI deep readingMembers
Version 1 · 1 source
Reinforcement learning pioneer Rich Sutton said on a Sequoia Capital podcast that synthetic data is a huge mistake and that large models have only achieved a quarter of intelligence. Through his new company, Oak Lab, he is betting on continual learning to let AI update its knowledge in real time after deployment.
In an interview on the Sequoia Capital podcast 'Training Data' on August 18, Sutton pointed out that the capabilities of current large models are mostly formed before deployment, with parameters essentially fixed. Even if they encounter vast amounts of new information daily, they do not convert it into long-term abilities like humans do. He argues that this static paradigm confuses the boundary between knowledge and learning.
Sutton emphasized that context only temporarily changes the information the model can currently see; the model's internal knowledge structure is not updated, and once the conversation ends or the context window is cleared, the information may disappear. Although the product may give the impression of 'learning,' if the model weights do not change, it is merely using current information for reasoning without forming new long-term knowledge.
This article is primarily based on an interview on the Sequoia Capital podcast 'Training Data,' reported by DeepTech. The views of Sutton and Khurram Javed are directly quoted, but some technical details (such as the 'continual backpropagation' algorithm) come from the team's statements and have not been independently verified. Oak Lab's goals and Moore's law estimates are company statements or inferences and should be treated with caution.
Primary report
Primary source