Back to feed
News Story
量子位
1 sources

Video Large Model Three-Party Feedback Paradigm Launched for Real-Time Live Streaming Decision-Making

Fudan University's Institute of Trustworthy Embodied AI, Bytedance International Live Content AI team, and Shanghai Innovation Institute jointly released Live Assistant, a three-party feedback paradigm based on content understanding for real live streaming scenarios. The paradigm models live streaming as a heterogeneous, long-duration, multi-party interactive, mixed-driven multimodal signal stream, using a content-understanding-driven state machine to decide whether to output, to whom, and what type of output, while treating silence as a legitimate action. The technical design covers Omni-native multimodal signals, multi-turn SFT and RL, a long-horizon reasoning framework, and training-inference consistency.

SynthePulse Insight · AI deep readingMembers

The New Challenge for Live-Streaming Large Models: Not Speaking Better, but Knowing When to Shut Up, Remember, and Whom to Address

Version 1 · 1 source

Fudan University's Institute of Trustworthy Embodied Intelligence, the Bytedance International Live Content AI team, and Shanghai Innovation Institute jointly released Live Assistant, defining "whether to speak, to whom, and what type of content" as the model's entire action grammar. This tripartite feedback paradigm attempts to answer what is harder than "what was seen" in live-streaming scenarios.

  • Live Assistant was jointly released by Fudan University's Institute of Trustworthy Embodied Intelligence, the Bytedance International Live Content AI team, and Shanghai Innovation Institute, proposing a content-understanding-based tripartite feedback paradigm for real live-streaming scenarios.
  • This paradigm defines the three questions of "whether to speak, to whom, and what type of content" as the model's entire action grammar, rather than merely improving commentary capability.
  • The team points out three hurdles in existing live-streaming understanding work: simplifying live streams to "video frames + text," a single feedback target, and not treating "staying silent" as a legitimate action.
Open section navigationA Default Premise: The Model's Task Is to Keep Talking

A Default Premise: The Model's Task Is to Keep Talking

Most existing video large models remain at the stage of "commentary while watching" or "you ask, I answer," implying a premise: the model's task is to keep talking. The team believes that in real live-streaming rooms, the hardest thing may not be "what was seen," but knowing when to shut up, when to remember, and whom to address.

This judgment points to a more fundamental problem definition: a real live stream is essentially a heterogeneous, long-duration, multi-party interactive, mixed-driven multimodal signal stream. What the model needs to model is not a single response, but continuous state decisions throughout the live-streaming process. Live Assistant is developed around precisely this proposition.

Three Hurdles: Live Streams Simplified, Single Feedback Target, Silence Not Treated as an Action

The team summarizes the limitations of existing live-stream/online video understanding work into three hurdles. First, live streams are simplified into "video frames + text": many works still treat live streams as "image frames + text questions," or first convert audio into ASR text before feeding it to the model. The team believes that the streamer's tone, background sound, danmaku density, gift events, changes in online viewer count, room meta information, and platform rules are not ancillary information, but native signals that jointly change the model's judgment.

Second, the feedback target is too single: existing systems either answer user questions or generate commentary for viewers, but a live-streaming assistant truly serves multiple parties—viewers need explanations and recaps, streamers need comment-section questions, pacing changes, and key event reminders, and the platform/moderation side needs risk clues and segments pending review.

Third, "staying silent" is not treated as an action: many online models are still driven by user questions or fixed time slices, outputting a passage for each segment. But in a real live stream, silence itself is an important and legitimate action. The model must, based on content understanding, judge whether to continue observing, write into memory, or proactively provide feedback when key events, highlights, comment explosions, pacing anomalies, or safety risks appear.

Free for now

Read the full analysis

5 more sections of analysis, plus the full takeaway

Loading

Credibility boundary

The factual basis of this article comes from QbitAI's report on Live Assistant, which is a contributed piece, with core information provided by the project team. The joint release parties, the tripartite feedback paradigm definition, the six-layer technical design, the multi-turn SFT and RL mechanisms, the two-level memory and train-inference consistency design, etc., are all source_claim from the team's own statements, not independently verified by third parties. Conclusions such as base model metrics being 0 and progressive training bringing stable improvements lack specific numerical values and reproducible evaluations, and should be regarded as team claims rather than confirmed facts.

Sources for this version

  1. 视频大模型三方反馈范式上线，面向真实直播建模持续决策量子位

Primary report

量子位

Primary source