Back to feed
News Story
BStandard67
The Gradient
1 sources

AGI Is Not Multimodal

Original

The article argues that achieving AGI requires more than multimodal capabilities; it emphasizes the importance of tacit embodied understanding, which current AI models lack.

SynthePulse Insight · AI deep reading

AGI Is Not Multimodal: Why Stitching Senses Together Won't Lead to True General Intelligence

Version 1 · 1 source

Multimodal large models appear to approach general intelligence by stitching together vision, language, and other modules, but this article argues that without embodied physical understanding, this strategy is doomed in the near future. True AGI must solve physical-world problems like car repair and cooking, which requires intelligence embedded in a physical world model, not a stack of symbolic operations.

  • True AGI must solve physical-world problems (car repair, untying knots, meal preparation, etc.), which cannot be reduced to symbolic operations.
  • What LLMs learn through next-token prediction is more likely a vast set of heuristics rather than a true world model.
  • Cases like OthelloGPT show that models can achieve high scores on sequence prediction without learning the underlying world model that generated the data.
  • Multimodal approaches patch together different sensory modules but lack embodied interaction and physical causal understanding, failing to achieve sensorimotor reasoning, motion planning, and social coordination.
  • Priority should be given to developing intelligence centered on embodiment and environmental interaction, treating modality processing as an emergent phenomenon.
Open section navigationPhysical-World Problems Cannot Be Symbolized

Physical-World Problems Cannot Be Symbolized

AGI must solve problems rooted in physical reality, such as car repair, untying knots, and meal preparation. These tasks require some concept of the physical world that far exceeds what humans can describe in language.

The game of Othello is inherently symbolic and can be played entirely with pen and paper; but sweeping, washing dishes, and driving cannot be done with just pen and paper. Physical-world problems cannot be fully represented as symbolic systems and solved through symbolic manipulation.

LLMs' 'World Model' Is an Illusion

Some theories suggest that LLMs induce a world model through next-token prediction, with evidence including SOTA model performance on benchmarks, convergence of internal representations in large models, and the philosophical metaphor that 'language mirrors the structure of reality.' However, the author warns that this metaphor is taken too literally.

OthelloGPT research shows that board states can be predicted from hidden states. But this result is difficult to generalize to natural language: legal move sequences in Othello can provably deduce the full board state, whereas a complete picture of the physical world cannot be inferred from language descriptions.

A paper by Melanie Mitchell et al. points out that generative models can achieve very high scores on sequence prediction tasks yet fail to learn the world model that generated the sequence data. For example, OthelloGPT learned sequence prediction rules that do not apply to all Othello games (e.g., 'if B4 is not marked before A4 appears, then B4 is empty'). The next-token prediction objective only seeks to retain information relevant to predicting the next token; if something easier than a world model can accomplish this, the model will take shortcuts.

Multimodal Patching Cannot Achieve Embodied Intelligence

Multimodal approaches optimize large modular networks for multiple modalities, appearing general, but the author argues that this strategy is doomed in the near future and cannot reach human-level AGI.

True AGI requires the ability to perform sensorimotor reasoning, motion planning, and social coordination—capabilities that depend on embodied and environmental interaction, not simple stitching of modalities.

Priority should be given to developing intelligence methods centered on embodiment and environmental interaction, treating modality processing as an emergent phenomenon rather than a starting point.

Credibility boundary

This is an opinion piece; the author explicitly rejects the multimodal path and advocates for embodied intelligence. The article cites studies like OthelloGPT as supporting evidence, but the author also acknowledges counterevidence supporting the LLM world model hypothesis (e.g., convergence of internal representations). The core claims are the author's inferences and require reader evaluation.

Insight takeaway

Multimodal large models appear to approach AGI by stitching together sensory modules, but lacking embodied understanding of the physical world, they cannot solve real physical problems like car repair or cooking. True general intelligence should start from embodied interaction, not modal patching.

Sources for this version

  1. AGI Is Not MultimodalThe Gradient

Primary report

The Gradient

Primary source