SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for Multimodal Large Language Models (MLLMs) have been lacking due to the rich nature of social interaction. |
| Approach: | They propose a video benchmark to evaluate MLLMs' capabilities across social scene understanding, social state reasoning, and social dynamics prediction. |
| Outcome: | The proposed benchmarks evaluate MLLMs' capabilities across social scene understanding, social state reasoning, and social dynamics prediction tasks. |
Similar Papers
MM-SOC: Benchmarking Multimodal Large Language Models in Social Media Platforms (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. |
| Approach: | They propose a benchmark to evaluate MLLMs' understanding of multimodal social media content and a large-scale YouTube tagging dataset to evaluate their performance. |
| Outcome: | The proposed model performs better in a zero-shot setting, suggesting potential improvements. |
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues (2026.findings-acl)
Copied to clipboard
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Moore Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Shihao Li, Yanghai Wang, Tianhao Peng, Jiaheng Liu
| Challenge: | Existing evaluation benchmarks for Multimodal Large Language Models (MLLMs) focus on single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. |
| Approach: | They propose a video understanding benchmark for MLLMs in multi-turn dialogues that assesses six core competencies that focus on perceptivity and interactivity. |
| Outcome: | The MT-Video-Bench evaluates 1,000 multi-turn dialogues from diverse domains and reveals significant performance discrepancies and limitations in handling multi-turned video dialogues. |
GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs (2026.acl-long)
Copied to clipboard
Weidong Tang, Jierui Li, Yueling Hou, Zihan Mei, Can Zhang, Xinyan Wan, Zhiyuan Liang, Pengfei Zhou, Yang You, Wangbo Zhao
| Challenge: | Existing models for general intelligence fail to model how mental states interact and crystallize into group-level outcomes. |
| Approach: | They propose a multimodal benchmark for group-level Theory of Mind (ToM) to probe nonlinear collective behavior. |
| Outcome: | The proposed model performs significantly below human levels, exposing blind spots in modeling social structures and nonlinear collective behavior. |
MoMentS: A Comprehensive Multimodal Benchmark for Theory of Mind (2025.findings-emnlp)
Copied to clipboard
Emilio Villa-Cueva, S M Masrur Ahmed, Rendi Chevi, Jan Christian Blaise Cruz, Kareem Elzeky, Fermin Cristobal, Alham Fikri Aji, Skyler Wang, Rada Mihalcea, Thamar Solorio
| Challenge: | MoMentS is a benchmark designed to assess the ToM capabilities of multimodal large language models (LLMs) in short films. |
| Approach: | They introduce a benchmark to assess the ToM capabilities of multimodal large language models (LLMs) through realistic, narrative-rich scenarios presented in short films. |
| Outcome: | The proposed benchmark features long video context windows and realistic social interactions that provide deeper insight into characters’ mental states. |
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks of social language are lacking for large language models. |
| Approach: | They propose a new benchmark that measures how well large language models understand social language by grouping 58 tasks into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
| Outcome: | The proposed model performs well at 58 tasks that are divided into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
LongMP-Bench: A Benchmark for Multimodal Persona Understanding in Long-Term Dialogues (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets suffer from limited persona diversity and static, overly simplified settings, making them insufficient for capturing the complexity of real-world interactions. |
| Approach: | They propose a benchmark to evaluate models' ability to understand evolving user personas within long-term multimodal dialogues by using a dataset that contains long conversations from 150 users. |
| Outcome: | The proposed benchmark aims to assess models' ability to track persona evolution, integrate visual and textual inputs, and apply persona understanding in realistic dialogue scenarios. |
Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding (2026.acl-long)
Copied to clipboard
Jaehyun Jeon, Min Soo Kim, Janghan Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu
| Challenge: | Recent studies focus on surface-level features, overlooking how design choices influence user behavior at scale. |
| Approach: | They propose a benchmark for multimodal understanding of how UI/UX design affects user behavior built on 300 real-world UI image pairs from industry A/B tests. |
| Outcome: | The proposed benchmarks show that models exhibit limited understanding of the behavioral impact of UI/UX design. |
Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on single image settings, but some focus on multi-image settings. |
| Approach: | They introduce the TempVS benchmark which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models in image sequences. |
| Outcome: | The proposed model performs poorly compared to human models in vision and language tasks. |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension (2026.acl-long)
Copied to clipboard
| Challenge: | a gap exists between human embodied logic and machine statistical learning . authors: models internalize statistical patterns or mimic static recognition . |
| Approach: | They propose a cognitive linguistic benchmark to test whether large language models internalize statistical logic or not . they find that models function as "Super-Associators" expert at static recognition yet fail at causal reasoning . |
| Outcome: | The proposed model fails at causal reasoning and has a high-fidelity concept representation but lacks transformational operators essential for true relational understanding. |