Ant Group and Zhejiang University Propose TRACE Framework for Streaming Emotion Understanding

Most existing emotion recognition research uses offline settings: the model first receives the complete video, audio, or dialogue text, then outputs an emotion label. But real human-computer interaction does not work this way—in real-time video conversations, embodied interaction, and remote meetings, information arrives continuously, and the system must form judgments while observations are still incomplete, adjusting as new evidence comes in.

The paper summarizes this gap into three key challenges: information is not yet complete but timely judgment is needed; high model confidence does not mean the judgment is reliable; multimodal reasoning is effective but calls incur costs. In other words, streaming emotion understanding must answer not only 'what emotion is this,' but also 'is the current judgment reliable, is it worth continuing to wait, and is more evidence needed.'

TRACE is designed around these three questions. It consists of three interconnected modules—online emotion judgment, trajectory reliability assessment, and context re-judgment—aiming to balance recognition performance and response efficiency.

TRACE's core hypothesis is that whether a judgment is reliable depends not only on how confident the model is at the moment, but also on how that judgment was formed. A model that judges 'sad' multiple times in a row versus one that switches between 'happy,' 'surprised,' and 'sad' may have different reliability even with the same final confidence.

The paper's validation subset analysis provides data support for this hypothesis: when the predicted category does not switch, online judgment accuracy is 72.6%; when one switch occurs, it drops to 38.3%; when three switches occur, it is 24.1%. Fluctuations in the prediction trajectory indeed provide information beyond single-shot confidence.

More critically, samples with unstable trajectories often benefit more from context re-judgment. Among samples with one category switch, context re-judgment brought 74 effective corrections, while cases where originally correct judgments were changed to incorrect ones numbered 19. This provides a basis for 'using complex reasoning where it matters most.'

The online emotion judgment module is based on Qwen2.5-Omni-3B, continuously processing audio prefixes and using cues such as intonation, pauses, rhythm, and energy changes to form an emotion probability distribution updated over time. During training, the team supervised audio prefixes of different lengths and assigned weights according to observation progress, enabling the model to learn to form preliminary judgments when evidence is still incomplete.

The trajectory reliability assessment module comprehensively analyzes information such as confidence, uncertainty of the prediction distribution, category switches, and distribution stability, while estimating two questions: how great is the risk that the current judgment is wrong? And if context is introduced, how likely is it to correct the current error? Based on these two estimates, the system dynamically chooses among three actions—output directly, continue waiting, or request re-judgment.

When the system requests re-judgment, the context re-judgment module based on Qwen2.5-Omni-7B re-analyzes by combining the current utterance's multimodal information, preceding context, historical summaries, and the existing prediction trajectory. This module not only outputs an emotion label but also generates the basis for the judgment and updates the dialogue summary to provide background information for subsequent utterances. The online setting uses only information that has already arrived; subsequent utterances are used only for offline comparative analysis.

To study emotion understanding under incomplete observations, the team built the StreamMER dataset based on the first two seasons of Friends. It contains 8,926 utterance samples, including 8,111 training samples and 815 test samples, with a total video duration of approximately 10.33 hours for the annotated utterances.

The dataset organizes data into coherent scenes of about 1.5 minutes, preserving the preceding context needed to understand the current emotion, and covers major emotion categories such as joy, anger, neutral, anxiety, surprise, sadness, and embarrassment.

The annotation process combines model assistance with human verification: Gemini-3.1-Pro is used to generate initial annotations, which are then independently checked and voted on by three annotators, with disputed samples discussed. About 30% of the initial emotion labels were modified during human verification.

In the evaluation design, the online module observes only the audio prefix that has already arrived, calls context re-judgment on demand, and incorporates additional latency and inference cost into the evaluation. This allows researchers to simultaneously examine whether the model 'recognizes accurately,' 'when it makes a judgment,' and 'how much computational cost it pays for the judgment.'

On StreamMER, the paper reports that TRACE improves the accuracy of online audio-only judgment from 59.87% to 69.47%, a 9.60 percentage point improvement over online-only judgment, while avoiding 44.42% of context re-judgment calls.

On MELD and MER2024, the team conducted supplementary validation using an offline multimodal emotion understanding setting, and TRACE reduced re-judgment calls by 45.37% and 41.79%, respectively.

In terms of efficiency, the paper reports TRACE's real-time factor (the ratio of inference time to input duration) is 0.091, lower than the 0.127 of full context re-judgment, a reduction of about 28.3%. In the streaming decision strategy experiment, TRACE's average decision time is 3.10 seconds, compared to 4.05 seconds for the full re-judgment strategy.

The paper also presents a typical case: as audio is gradually input, the model's prediction shifts from 'happy' to 'surprised,' and the trajectory is unstable; when TRACE triggers context re-judgment, the model combines the accusatory tone, direct gaze, and confrontational dialogue content to revise the emotion to 'angry.'

First, it must be made clear that all the above performance numbers—accuracy, re-judgment reduction ratio, real-time factor, decision time—are self-reported by the paper's authors and lack third-party independent replication and verification. They reflect results under the paper's experimental settings, not universally confirmed conclusions.

Second, StreamMER is built on the first two seasons of Friends, and the distribution difference between its emotional expressions and real human-computer interaction scenarios is not yet clear. About 30% of the initial labels were modified during human verification, and the degree to which annotation quality and consistency affect the conclusions has not been quantified.

The paper itself also points out that TRACE's ability to understand more complex and subtle emotional expressions still needs further exploration; more precise call strategies and richer real interaction data remain future directions not covered by the current approach. In addition, the consistency between the arXiv number linked in the paper and the 2026 timeline cannot be verified from the source.