On the CyberGym benchmark, with up to five CodeMender invocations of 3.5 Flash Cyber, the overall agent achieved performance competitive with significantly larger models. Note: Competitor results are from provider self-reports.
In evaluations independently built by the Big Sleep team, focusing on critical vulnerabilities in complex codebases like Chrome and Safari, 3.5 Flash Cyber significantly outperformed the baseline 3.5 Flash and 3.6 Flash.
In evaluations on the Google Chrome production commit scanning pipeline, with vulnerabilities undisclosed to avoid contamination, results showed significant improvements of 3.5 Flash Cyber over 3.5 Flash. Newer competitor models (e.g., after Opus 4.6) are not shown because they refused to execute tasks due to built-in safety guardrails.