OpenAI has developed GPT-Red, an LLM designed to act as a super-hacker, to improve the security of its models by simulating cyberattacks. The latest GPT-5.6 was trained against GPT-Red, resulting in its most robust release yet.
SynthePulse Insight · AI deep reading
Version 1 · 3 sources
OpenAI has unveiled GPT-Red, an internal automated red-teaming system that uses adversarial self-play to discover prompt injection vulnerabilities, slashing the failure rate of its latest model, GPT-5.6 Sol, to one-sixth of what it was four months ago. This breakthrough method has uncovered a previously unknown 'fake chain of thought' attack, sparking debate about a paradigm shift in AI safety testing.
OpenAI researchers built GPT-Red using a training method called 'self-play.' They started with an LLM untrained in hacking and placed it in a simulated environment with multiple other models. GPT-Red's goal was to attack these models, while the others tried to defend. After multiple rounds of adversarial interaction, GPT-Red's attack capabilities improved, and the defense capabilities of the attacked models also strengthened. This training took place in a simulated environment called 'The Dojo,' which mimics various real-world deployment scenarios for LLMs, including browsing the web, reading emails or calendar apps, and editing code.
Dylan Hunn, co-creator of GPT-Red, said that compared to human red teamers, GPT-Red is very good at finding the most effective attacks and will persistently dig deeper into discovered attacks. When GPT-Red finds a new type of attack, it explores multiple variants to find the most effective one for a specific scenario.
GPT-Red's most notable discovery is a previously unknown prompt injection attack that researchers call 'fake chain of thought.' Chain of thought is an LLM's 'diary' of intermediate steps when processing a problem. GPT-Red found a way to insert false entries into another model's chain of thought, tricking the model into acting on fabricated information. Research scientist Chris Choquette-Choo explained: 'It's like I tell you 1+1=3, and you've already verified it. The model thinks, "Oh, okay, sure," and directly outputs 3.'
When replicating a 2025 human red-teaming experiment, GPT-Red was asked to find weaknesses in an earlier version of GPT-5. It was more successful than humans at finding effective attacks. Additionally, OpenAI tested GPT-Red against Vendy, an automated vending machine agent developed by Andon Labs. GPT-Red successfully hacked Vendy, changing product prices and canceling customer orders. When OpenAI applied the strongest attacks discovered by GPT-Red to its own models, over 90% of attacks were effective against GPT-5 (released August 2025), while less than 23% were effective against the newly released GPT-5.6 Sol.
Despite GPT-Red's impressive performance, it is not perfect. It struggles with attacks involving multi-turn conversations, where human attackers excel. It also performs poorly at prompt injection via images. OpenAI emphasizes that GPT-Red complements human red-teaming efforts, not replaces them. Humans can find attacks that GPT-Red misses, and vice versa. One approach being explored is to give GPT-Red attacks discovered by humans and have it find all variants.
Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology (CSET), believes the self-play approach is promising but human expertise remains crucial. She noted: 'Being able to distinguish where human testing is most needed would be very useful.' OpenAI says it will not release GPT-Red publicly, arguing that its 'super hacker' capabilities are difficult to replicate because training requires over a year and enormous computational resources.
The launch of GPT-Red marks a significant step in AI safety testing, shifting from fully manual to automated, scalable methods. As LLMs become more complex and are used for broader tasks—especially as agents that can interact with files, websites, third-party code, and other agents—human teams struggle to keep up with all potential attack types. As research scientist Nikhil Kandpal put it: 'The risk surface is expanding, and the blast radius is also expanding.' GPT-Red aims to future-proof the safety testing process, ensuring that as more powerful models emerge, the system can automatically discover new attack patterns.
However, GPT-Red's proprietary and non-replicable nature raises questions about transparency in AI safety. If only a few companies possess such powerful automated red-teaming tools, could industry-wide safety standards become unbalanced? OpenAI's response is that building GPT-Red requires substantial resources and time, making it not easily replicable. Whether this claim will alleviate external concerns remains to be seen.
This article primarily draws from OpenAI's official blog and X platform announcements (primary sources), as well as an exclusive report from MIT Technology Review (secondary source). As a stakeholder, OpenAI's claims about GPT-Red's performance (e.g., '6 times fewer failures') are source claims and have not been independently verified by third parties. The MIT Technology Review report provides additional details and third-party expert commentary, enhancing credibility.
GPT-Red demonstrates the immense potential of automated red-teaming systems in discovering AI model vulnerabilities, especially through self-play to uncover novel attacks. However, its limitations (e.g., struggles with multi-turn conversation attacks) and proprietary nature suggest that human-machine collaboration remains the optimal approach for AI safety testing. Going forward, balancing automation with human oversight, and proprietary systems with industry transparency, will be key issues in AI security.
Primary report
Primary source
Also covered by 2 sources