This article explores the potential of large language models (LLMs) like Car-GPT in autonomous driving. It discusses whether LLMs can be trusted for self-driving cars and highlights key challenges such as safety, reliability, and decision-making. The piece examines the current state and future possibilities of integrating AI language models into vehicle control systems.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
As traditional modular approaches hit a bottleneck and end-to-end learning faces the black-box dilemma, large language models (LLMs) are being explored as an unexpected answer for autonomous driving. This article reviews research progress in perception, planning, generation, and other tasks, while examining their limitations and uncertainties.
In the 2010s, most autonomous vehicles adopted a modular approach, dividing software into independent modules such as perception, localization, planning, and control. Perception is responsible for 'seeing' the world, localization for determining the vehicle's position, planning for generating trajectories (the car's 'brain'), and control for outputting commands like 'turn right 20°'. However, this mature method has not completely solved the autonomous driving problem.
Subsequently, end-to-end learning emerged, with the core idea of using a single neural network to directly predict steering and acceleration, replacing all modules. But this introduced a black-box problem—the internal decision-making process is difficult to interpret.
The core of LLMs is tokenization, Transformer architecture, and next-token prediction. Input text is converted into tokens (numbers), processed through the Transformer's attention mechanism, and finally predicts the next token.
Applying LLMs to autonomous driving requires modifying inputs and outputs: inputs can be images, LiDAR point clouds, radar point clouds, or even algorithmic data (lane lines, objects, etc.), all of which can be 'tokenized' via Vision Transformer or Video Vision Transformer. The Transformer model itself only operates on tokens, regardless of input type. Outputs depend on the task, ranging from describing the scene to direct driving commands (e.g., change lanes).
In perception, LLMs can perform three core tasks: detection, prediction, and tracking. For example, the GPT-4 Vision model can return object descriptions from images, similar to an object detector. Other models like HiLM-D and MTD-GPT can also perform similar tasks, with some supporting video input.
The PromptTrack model combines the DETR object detector with an LLM, enabling it to assign unique IDs to objects (e.g., 'front vehicle ID #3'), achieving functionality similar to 4D perception models.
In planning tasks, LLMs can describe the driving behavior to be taken (e.g., continue driving, yield, etc.) based on images, bird's-eye views, or perception outputs. This essentially replaces traditional trajectory planning with language generation.
In generation tasks, LLMs can use diffusion models to generate training data, alternative scenarios, etc., helping to expand autonomous driving training datasets. Additionally, LLMs can be used in question-answering systems to create chat interfaces, allowing users to inquire about scene information through natural language.
Although LLMs show potential in autonomous driving, current research is still in its early stages. The article points out that not all autonomous driving tasks are suitable for 'GPT-ification', and the black-box nature of LLMs may exacerbate the interpretability issues already present in end-to-end learning.
Furthermore, the real-time performance, safety, and reliability of LLMs have not been fully validated in real driving environments. The article does not provide any benchmark test results or actual road test data, so these capabilities remain at the level of research hypotheses.
This article is based on an opinion piece from The Gradient, written by a technical writer who is not an LLM expert. The content is primarily conceptual introduction and research review, without providing original experimental data or third-party verification. All descriptions of model capabilities come from studies cited in the article (e.g., GPT-4 Vision, HiLM-D, PromptTrack), but lack independent validation.
LLMs offer a new approach for autonomous driving, especially in the linguistic representation of perception and planning, but there remains a huge gap before actual deployment. Current evidence mainly comes from academic exploration, not engineering validation.
Primary report
Primary source