HaluMem: First Operation-Level Hallucination Benchmark for Agent Memory Systems Released

When an AI assistant gets a user's new job title wrong, the intuition is to attribute it to the model 'making things up' at the moment of answering. But the error may not occur at that moment: it might have misrecorded the job title while organizing the conversation, or it might have kept an old record without completing the update, or it might have retrieved the correct memory but swapped in another title when answering. All three failures present the same wrong answer to the user, yet the directions for fixing them are completely different.

This is where HaluMem comes in. Existing memory evaluations mainly use end-to-end question answering: input a series of dialogues, ask a question, and compare the answer with the ground truth. This can measure how well the system ultimately answers, but it is hard to explain why it answered incorrectly. HaluMem also provides checkable ground truth for intermediate operations, pushing evaluation into three stages—memory extraction, memory updating, and memory question answering—to trace where errors arise and how they affect subsequent answers.

The work was proposed by researchers from China Telecom Research Institute and Shanghai MemTensor, among others. The paper is titled 'HaluMem: Evaluating Hallucinations in Memory Systems of Agents,' with authors including Ding Chen, Simin Niu, Kehang Li, Peng Liu, Xiangping Zheng, Bo Tang, Xinchi Li, Feiyu Xiong, and Zhiyu Li. The paper has been accepted to the EMNLP 2026 main conference, and both the project and the arXiv paper have been made public. It should be noted that the claim of being 'the first operation-level hallucination evaluation benchmark for agent memory systems' comes from the paper's authors and those reporting on it, and has not yet been independently confirmed by a third party.

HaluMem examines three questions separately. Memory extraction: is the information that should be remembered recorded completely and accurately, and are fabricated details written in as well? Memory updating: when new information appears, is the old memory correctly modified, and are there missed changes, wrong changes, or leftover conflicts? Memory question answering: can the system ultimately use memory to give a correct answer, and what proportion of wrong answers versus failures due to missing information?

These checks unfold in the chronological order of the dialogue and are triggered immediately after the relevant session is processed. As a result, the evaluation no longer yields just a single total score but also reveals specific problems at different operation stages. Two examples from the report illustrate this distinction: a user says, 'I've recently started liking parrots; I really didn't like them before,' but the system extracts 'the user does not like parrots'—the error occurs at the very first step of writing to memory. A user clearly states that their health has worsened, but the system, based on the assistant's incorrect recollection, updates the health status to 'good'; even if subsequent retrieval is correct, the answer is already built on a wrong record.

There is an even more insidious case: details fabricated by the assistant itself are treated by the memory system as user facts. In a conversation, the AI might casually mention a friend or experience that does not exist; the user does not explicitly confirm it, yet the system stores it in long-term memory. A single unverified expression can become the 'basis' for the next answer.