Researchers at Carnegie Mellon University developed Social Gym, a platform where 7 AI models compete in 21 games to test strategic, deceptive, and theory-of-mind abilities. Results show GPT-5-mini leads in several capabilities, while Qwen3-32B performs worst in Werewolf. The study aims to quantify AI's social reasoning and reveals issues like parroting in smaller models.
SynthePulse Insight · AI deep readingMembers
Version 1 · 1 source
Carnegie Mellon University tested 7 AI models' social abilities with 21 games, finding GPT-5-mini the most deceptive, but AI's 'social savvy' is not a unified trait, and self-reflection doesn't always lead to improvement.
Researchers at Carnegie Mellon University argue that while AI excels at tasks like math and programming, real-world AI agents must interact with people and other agents, facing information asymmetry, disagreements, and other complex situations. Thus, they need to test AI's 'social savvy'—its social reasoning and theory of mind.
Traditional tests struggle to quantify abilities like negotiation and persuasion. Social Gym uses games with built-in win/lose rules, such as Prisoner's Dilemma, Chicken Game, and Werewolf, to put AI in environments where social savvy is essential for winning, bypassing subjective evaluation.
This article is based on DeepTech's report on Carnegie Mellon University's research; all data comes from that report and has not been independently verified against the original paper. Some conclusions, like 'GPT-5-mini is the most deceptive,' are based on the study's results, but experimental limitations may affect generalizability.
Primary report
Primary source