OSWorld-Science: Computer-Use Agents on Scientific Software
A benchmark that tests whether AI agents can operate real scientific software on a virtual desktop. I built its chemistry domain, the largest and hardest part of the benchmark.
What it measures
OSWorld-Science gives agents real scientific desktop software and checks the files they produce, rather than their text answers. Tasks span domains such as chemistry, pathology image analysis, statistics and physical simulation.
My part
I designed the chemistry domain: tasks like drawing molecules and planning syntheses in real chemistry software, each graded automatically by comparing the agent’s output molecule with RDKit (canonical SMILES, InChIKey and stereochemistry). I also wired the tasks into the OSWorld harness and ran the model evaluations.