Logo PhyX

Does Your Model Have the "Wits" for Physical Reasoning?

Hui Shen1, 2, Taiqiang Wu1, Qi Han3, Yunta Hsieh2, Jizhou Wang4, Yuyue Zhang3, Yuxin Cheng1,
Zijian Hao3, Yuansheng Ni5, Xin Wang6, Zhongwei Wan6, Kai Zhang6, Wendong Xu1, Jing Xiong1,
Ping Luo1, Wenhu Chen5, Chaofan Tao1†, Z. Morley Mao2, Ngai Wong1†,
1The University of Hong Kong, 2University of Michigan, 3Independent,
4University of Toronto, 5University of Waterloo, 6The Ohio State University
†Corresponding author.
arXiv Code 🤗 Dataset
Teaser image.

Data examples of the PhyX dataset. The dataset contains 3,000 human-annotated physics problems with visual context.

Introduction

We introduce PhyX: the first large-scale benchmark designed to assess models capacity for physics-grounded reasoning in visual scenarios. PhyX includes 3K meticulously curated multimodal questions spanning 6 reasoning types across 25 sub-domains and 6 core physics domains: thermodynamics, electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In our comprehensive evaluation, even state-of-the-art models struggle significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and GPT-o4-mini achieve only 32.5%, 42.2%, and 45.8% accuracy respectively-performance gaps exceeding 29\% compared to human experts.

Logo PhyX Benchmark

Comparisons with Existing Benchmarks

To further distinguish the difference between PhyX and other existing ones, we elaborate the benchmark details in Figure. From the realistic perspective, the prior benchmarks are heavily focused on abstract lining. In contrast, our benchmark are realistic and delete text redundancy.

comparison with existing benchmarks comparison with existing benchmarks

Comparison with existing physics benchmarks.

Statistics

sampled PhyX

Sampled PhyX examples from each discipline.

Key statistics

Key statistics of the MMMU benchmark

Distribution of question domains

Distribution of question domains in the PhyX dataset

Logo

Experiment Results

Data

You can directly download our data from Huggingface datasets. For guidance on how to access and utilize the data, please consult our instructions on Github.

Leaderboard

We evaluate various models including LLMs and LMMs. In each type, we consider both closed- and open-source models. Our evaluation is conducted under a zero-shot setting to assess the capability of models to generate accurate answers without fine-tuning or few-shot demonstrations on our benchmark.


Human Expert Open-Source Proprietary

Click on PhyX(OE), PhyX(MC) or PhyX(domain) to expand detailed results.

Reset PhyX(OE) PhyX(MC) PhyX(domain)
Name Size Date Text-DeRedundancy Text-DeRedundancy Overall

Overall results of different models on the MMMU leaderboard. The best-performing model in each category is in-bold, and the second best is underlined. *: results provided by the authors.

BibTeX

@misc{shen2025phyxdoesmodelwits,
        title={PhyX: Does Your Model Have the "Wits" for Physical Reasoning?}, 
        author={Hui Shen and Taiqiang Wu and Qi Han and Yunta Hsieh and Jizhou Wang and Yuyue Zhang and Yuxin Cheng and Zijian Hao and Yuansheng Ni and Xin Wang and Zhongwei Wan and Kai Zhang and Wendong Xu and Jing Xiong and Ping Luo and Wenhu Chen and Chaofan Tao and Zhuoqing Mao and Ngai Wong},
        year={2025},
        eprint={2505.15929},
        archivePrefix={arXiv},
        primaryClass={cs.AI},
        url={https://arxiv.org/abs/2505.15929}, 
  }