The OSWorld 2.0 benchmark increases task complexity by an order of magnitude: median time spent increased from 2 minutes in v1.0 to 1.6 hours. The 108 long-horizon tasks cover document processing, software development, financial analysis, sales support, etc., requiring collaboration across multiple applications (e.g., Slack, LinkedIn, REAPER, AWS). The current best model, Claude Opus 4.8, achieves only 20.6% accuracy in maximum thinking mode, with performance declining sharply as task duration increases, especially when needing to restore hidden state, track multiple items, or resolve conflicting information.
However, the history of v1.0 provides an important warning: the best model in July 2025 had about 30% accuracy, which rose to about 75% within a year. If this improvement pace continues, OSWorld 2.0 scores may surpass human levels within the next 12–18 months. This means that even though current AI still appears clumsy on complex multi-step tasks, its learning curve is catching up exponentially.