MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence
Learn how the benchmark is built and why current models still fall short.
Evaluating AI’s grasp of speech, sound effects, and music has been fragmented, so the authors assembled a single test covering every auditory domain. MMAU-Pro gathers 5,305 instances, each pairing one or more real‑world audio files with expert‑written question‑answer sets. The benchmark spans 49 skills, from long‑form narrative comprehension to spatial audio reasoning, demanding multi‑hop inference. By pulling audio directly from the wild, it avoids the bias of curated datasets.
Each instance presents a question that forces the model to link multiple cues within or across clips, then answer either by selecting from choices or generating free text. The questions were authored by domain experts to ensure they cannot be solved by surface pattern matching. Multi‑audio scenarios combine speech, music, and environmental sounds, requiring the system to track temporal and spatial relationships. The design mirrors human auditory tests that probe both low‑level perception and high‑level semantic reasoning.
The benchmark’s construction follows a strict curation pipeline: raw recordings are filtered for quality, then annotated with a hierarchy of skill tags. Human annotators verify that every answer requires at least two reasoning steps, preventing shortcuts. Open‑ended items are scored with exact‑match or tolerant metrics, while multiple‑choice items use standard accuracy. This systematic approach yields a reproducible, fine‑grained evaluation across all 49 abilities.
When 22 leading multimodal models were run through MMAU-Pro, the best performer, Gemini 2.5 Flash, reached only 57.33 % accuracy, and Audio Flamingo 3 lagged at 45.9 %. In many skill categories the scores hovered near random chance, exposing gaps in spatial and multi‑audio reasoning. The results show that even state‑of‑the‑art systems struggle with holistic audio understanding despite strong performance on narrower tasks.
For developers, the benchmark signals that current pipelines need deeper audio representation learning and better cross‑modal integration. The open‑ended failure modes suggest research into multi‑hop audio reasoning and robust handling of uncontrolled sound sources. Until models improve on MMAU-Pro’s diverse challenges, claims of audio general intelligence remain premature.
TakeawayCurrent AI models still lack holistic audio reasoning across real‑world sounds.