Can AI Really Use Scientific Software? Tsinghua Team Launches OSWorld-Science, a Hard Test for Agents

Getting AI to explain a scientific question, list analysis steps, or even write analysis code is no longer difficult. The hard part is taking the next step: opening scientific software, importing data, configuring parameters, running the analysis, and saving and exporting the results. OSWorld-Science turns this step into an evaluation target.

This benchmark is led by a research team at Tsinghua University and jointly launched by researchers from multiple well-known universities and research institutions at home and abroad, oriented toward real scientific research scenarios. It contains 146 expert-designed tasks, covers 7 major scientific domains and 14 scientific software configurations, and was given to 12 vision-language models for evaluation. In the current evaluation, the average task score of the 12 models is only 34.3%.

It should be emphasized that the score here measures the degree of task completion and is not equivalent to a success rate. It is more like a ruler that measures how far a model has advanced a scientific task, rather than simply judging whether it was "done" or "not done."

The tasks in OSWorld-Science were transformed from real research workflows by domain experts, and validation criteria were set for each task. The agent needs to operate software in a virtual machine via keyboard and mouse or terminal, and after each action decide the next step based on the new interface state.

The final evaluation is based on the file artifacts and application states after the task ends. In other words, the model's description of what it completed is not accepted; actual results must prove it. This design directly determines the nature of the failure cases that follow: not "answered incorrectly," but "left nothing deliverable."

To make results from different models comparable, the team adapted the multimodal PromptAgent based on OSWorld for scientific software, providing a unified prompt and interaction workflow, and adopting a consistent task step budget and basic response length limit. Requests, responses, token consumption, and execution logs are also recorded, so researchers can see both what the model delivered and return to the process to see where it encountered difficulties.

On the 146 tasks, Claude Fable 5.1 leads with an average task score of 73.7%. Claude Opus 5, GPT-6 Astra, and GPT-5.6 sol score 57.9%, 57.4%, and 50.1%, respectively.

But when the overall leaderboard is broken down by discipline, the picture changes. In statistics tasks, all models perform relatively well; in chemistry tasks, only a few models, such as Claude Fable 5.1 and Claude Opus 5, can score above 20%. GPT-6 Astra reaches a 69% score on medical imaging tasks but performs poorly on chemistry tasks.

The same model entering different scientific domains may exhibit different capability boundaries. This makes model selection in scientific research scenarios more specific: the overall leaderboard can provide a reference, but what really needs to be compared is whether a model can complete the actual work in the corresponding scientific domain.

This evaluation shows that the highest-scoring model is not the most expensive one, indicating that the relationship between model performance and invocation cost is not simply linear. At the same time, although low-cost models can complete some scientific tasks, there is still a clear gap compared with high-performance models in task completion, execution stability, and final result quality.

Another result comes from differences in output volume. Claude Fable 5.1 outputs about 21,000 tokens per task on average and scores 73.7%; Gemini 3.1 Pro outputs about 157,000 tokens on average and scores 17.2%.

Caution is needed here: output volume includes reasoning tokens and cannot by itself represent cost; input token volume, model pricing, and repeated interactions all affect the final cost. But this comparison still raises a question worth pursuing—did more output actually lead to higher-quality task completion? The article raises this question and does not give a definitive answer.

Some difficulties can only be seen clearly by returning to the operation process. In one run of the biology software Mnova, the model needed to process a Bruker one-dimensional H spectrum and assign each hydrogen of 6-bromoindole to a spectral peak. Due to a coordinate mapping deviation, Claude Sonnet 5 repeatedly clicked the same wrong position 39 times, eventually exhausting its budget and failing the task.

The first wrong click may just be a positioning deviation. But after repeated wrong clicks, the agent needs to recognize the error pattern and reflect on changing its behavior: determine whether the interface has changed as expected, whether the current operation may be ineffective, and whether a different method is needed next. Without this step, behavior after a failed operation may become more repeated failures.

In the analysis of published trajectories, the team found that 61% of failed runs left no scorable artifact while exhausting their step or time budget. This makes the meaning of "completing the task" clearer: even if part of the analysis has been done, if it is not saved, exported, and verified in time, the work may still remain in an undeliverable state.

Software running successfully does not mean the scientific task is over. In the ANSYS Fluent cylinder flow adjoint task, Kimi K3 discovered that the drag reading was zero. This anomaly prompted it to recheck the settings, and it then corrected the problem of a missed wall selection, ultimately receiving a full score. Here, the result is not just read out and delivered directly; it also becomes a basis for the model to check whether its previous operations were correct.

But checking itself can also go wrong. A chemistry task case shows that the agent performed multiple format conversions and internal consistency checks but still repeated the same stereochemistry error. The reason is that if the checks always revolve around the same erroneous representation, consistent results may simply mean the same error has been preserved.

Scientific validation requires independent evidence and allows that evidence to overturn prior judgments. Raw data, physical constraints, and precision requirements can all serve as such a basis. What the agent needs to do is put the results back into the task requirements and relevant scientific laws to test their accuracy and reasonableness.

Looking at tasks from different disciplines together, OSWorld-Science focuses on a capability that runs throughout: enabling AI to translate professional knowledge into scientific software operations and automate specific scientific tasks into verifiable scientific outputs. Interface positioning determines whether operations land in the correct place, environment adaptation determines whether the workflow can continue, progress checking helps the agent escape ineffective loops, and scientific validation determines whether the final result can be trusted.

This research brings together researchers from institutions including Tsinghua University, the University of California, Los Angeles, Yale University, Carnegie Mellon University, the University of California, Berkeley, Stanford University, Zhejiang University, the University of Hong Kong, and the University of Tokyo, and received support from the foundation model departments of companies such as Kimi, Z.ai, and Alibaba.

The research team also welcomes more researchers to join in co-building tasks: on the one hand, it plans to add more real problems in scientific domains, including but not limited to inorganic chemistry, life sciences, mechanics, materials science, earth science, and social sciences; on the other hand, it aims to design more comprehensive cross-software AI workflows, such as a multi-step process chaining PyMOL, AutoDock Vina, and Python in protein structure analysis. The 146 tasks are only a starting point, and the current benchmark's coverage will continue to change.