Scale Labs Benchmark finds 25 multimodal models lag behind humans in common-sense understanding

Article

25 models collectively lose to humans:AI Still unable to understand much common sense in daily life

Beating AI Flash reports that Scale AI's research institute Scale Labs jointly released a new benchmark, Humanity’s Sixth Sense (HSS), with Elorian, specifically testing whether AI can identify implicit information in images and videos. 522 open-ended questions cover 288 images and 234 video clips. For example: whether cars can pass each other, why a woman suddenly slows down while chasing a bus, and who has more influence in a given setting.

The research team tested 25 multimodal models and had 20 human participants answer the questions. Humans achieved an accuracy rate of 93.1%, while the top-ranked GPT-6 Astra achieved only 53.6%。GPT-6.1 Sol and Claude Opus 5.5 scored 46.6%and 44.6%, respectively, even at maximum reasoning effort; the median score across all models was only 30.9%.

Researchers analyzed 8573 model failures and found that 94%were related to missing key clues, misidentifying objects, or being unable to infer implicit relationships in the scene. Only about 5%were classified as logical reasoning errors. Social understanding was particularly difficult: of the 25 models, 21 performed worst on this type of question. Video questions were also generally more difficult than image questions.

Increasing reasoning effort was not necessarily effective. Models consumed an average of approximately 4000 reasoning tokens per question; on some questions, the longer they reasoned, the worse they performed. The research team also tried having an Agent zoom in, crop, and repeatedly inspect the images. Scores improved, but a clear gap with humans remained.

Original link https://m.theblockbeats.info/flash/370817

Key points

Scale Labs and Elorian released a benchmark called Humanity’s Sixth Sense (HSS).
The benchmark includes 522 open-ended questions, 288 images, and 234 video clips.
The research team tested 25 multimodal models and had 20 human participants answer the questions.
The reported human accuracy rate was 93.1%，GPT-6 Astra was 53.6%，GPT-6.1 Sol was 46.6%，Claude Opus 5.5 was 44.6%。
The median score across all models was 30.9%。
Researchers analyzed 8573 model failures, of which 94%were related to missing key clues, misidentifying objects, or being unable to infer implicit relationships in the scene.
The report states that, of the 25 models, 21 performed worst on social-understanding questions, and video questions were generally more difficult than image questions.
Models consumed an average of approximately 4000 reasoning tokens per question; on some questions, performance was worse when reasoning took longer.
Using Agents to zoom in, crop, and repeatedly inspect images improved scores, but a clear gap with humans remained.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

This evaluation primarily examines whether models can discover clues that are not stated directly in images and videos, such as determining whether a vehicle can pass through, understanding the reasons behind people's actions, and identifying who holds speaking authority in a situation. Reports indicate that the model's overall score was lower than that of humans, and that increasing the amount of reasoning does not necessarily improve results.

Why it matters to readers

It suggests that multimodal models may still be unreliable in social understanding, video information, and judgments of implicit relationships.

A model's ability to recognize objects in an image does not mean that it can correctly understand the situation, causes, and relationships between people.

Risks and unknowns

Not provided in the original text HSS The original research report, question set, scoring methodology, or reproducible experimental materials.
The original text does not specify the exact versions and configurations of the models evaluated.
The current information comes from reports; the specific evaluation conclusions and scores still require verification against the original materials.

Scale Labs Benchmark finds 25 multimodal models lag behind humans in common-sense understanding

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center