Recommender Systems Guess Your Likes but Don't Listen? CAS Team Proposes Multi-Agent Black-Box Controllability Evaluation Framework

Traditional recommender system evaluation metrics such as Recall, Hit Rate, and NDCG primarily measure how accurately a model predicts the next interaction based on history, answering 'how well the system guesses.' However, they struggle to assess whether external users can guide the system to change its recommendations. The CtrlBench-Rec framework, proposed by the Institute of Information Engineering, Chinese Academy of Sciences, extends evaluation from static prediction to dynamic external guidance, focusing on how many interaction rounds a user needs to steer the system, which content can be surfaced, and which interests can be shaped.

The framework treats the recommender system as a completely opaque black box, accessing no model parameters, gradients, retrieval logic, or internal user representations, and measuring only through external behavior interfaces such as searches and clicks. This makes it applicable to online recommender systems where internal information is unavailable, serving as a foundational tool for algorithm auditing and user control mechanism design.

CtrlBench-Rec designs three progressive tasks: target content discovery (assessing whether the system can be made to consistently expose specified content), interest profile shaping (determining whether the interest structure reflected in recommendations can be altered), and popularity bias mitigation (testing whether the system can be pushed to reduce reliance on popular items and explore long-tail regions). These three tasks correspond to explicit targets, implicit interest representation, and system-inherent biases, forming a layered measurement.

A single agent struggles to fully explore a black-box system within a limited budget. Therefore, the framework employs a multi-agent mechanism: first, a set of initial agents with different preferences is constructed based on real user attributes and historical behaviors, each entering the black-box environment to explore. Then, agent profiles and behavior trajectories are encoded and clustered, grouping agents with similar strategies. Within each group, experiences are exchanged, and a user profile fusion module distills a more focused control strategy, producing a small number of Super Probes. These Super Probes do not modify the recommendation model but serve as optimized external probes to continue testing the system's responsiveness.

Experiments cover two datasets, MovieLens-1M and Amazon Toys & Games, constructing five types of black-box recommendation environments: NARM, SASRec, TwHIN-BERT, BGE, and Qwen3.5-4B. At round 20, CtrlBench-Rec achieves higher coverage than random probes of the same scale across all five architectures: NARM 3.72%, SASRec 8.95%, TwHIN-BERT 12.07%, BGE 27.41%, and Qwen3.5-4B 13.93%. External controllability does not increase monotonically with model parameter scale; it is more likely dependent on the recommendation architecture, item representation, and the match between target semantics and retrieval mechanisms.

External controllability varies significantly across content categories. In MovieLens-1M, Film-Noir accounts for only about 1.1% of all items but achieves a final coverage of 6.8%; Romance has more items yet only 6.4% coverage, lower than the smaller War category. The paper suggests that some moderately popular categories have overly dispersed internal semantics, lacking a clear representation center, making them equally difficult to guide consistently. This indicates that external controllability is influenced not only by popularity but also by the structure of the target category within the system's representation space.

Extending the interaction budget to 200 rounds, CtrlBench-Rec's cumulative coverage gradually saturates around 55.7%, without approaching 100%. This demonstrates that external controllability has clear boundaries: some target content may be nearly unreachable under the current behavior interface due to missing retrieval, sparse representation, or insufficient collaborative signals. More interactions can help probes approach this boundary faster but cannot guarantee breaking through the structural limitations inherent in the recommender system.

In interest shaping experiments, under a simple scenario where the agent has a short initial history, target similarity rises from approximately 0.53 to 0.61 within the first ten rounds; in a multi-interest balancing scenario, final target similarity exceeds 0.72; in an interest migration scenario, the agent has a strong existing interest opposite to the target direction, and the system exhibits significant historical inertia early on, but starting around round 9, the original interest similarity continuously declines, eventually dropping to about 0.34. This indicates that sustained, coherent external behavior can shape interest structure to some extent, but stronger existing histories incur higher switching costs.

In popularity bias mitigation experiments, CtrlBench-Rec can discover new long-tail content faster in early stages, but as interactions continue, the absolute amount of long-tail discovery remains low for all methods, and the gap with random methods narrows. Better external probes can improve long-tail exploration efficiency but cannot eliminate the popularity bias inherent in the recommender system solely through user-side behavior. When long-tail items lack exposure, semantic associations, and collaborative signals, external guidance can alleviate the problem but struggles to fully overcome the underlying retrieval and representation limitations of the system.