AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models (MLLMs) have garnered significant attention, offering a promising pathway toward artificial general intelligence (AGI). |
| Approach: | They propose a benchmark to evaluate associative ability while circumventing the inherent ambiguity in association tasks by decomposing ambiguities into two types and propose 'assoCiAm' they conduct extensive experiments on MLLMs, revealing a strong positive correlation between cognition and association. |
| Outcome: | The proposed method shows that ambiguity in association evaluations makes MLLMs more random-like and the model's behavior more random. |
Similar Papers
Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies evaluate the creative capabilities of large language models (LLMs) through diverse tasks, aiming to understand their strengths and limitations. |
| Approach: | They propose to ask LLMs to generate Parallel Chains of Associations to Evaluate their creativity. |
| Outcome: | The proposed framework minimizes the risk of data contamination and offers a highly efficient evaluation. |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
Probing the “Creativity” of Large Language Models: Can models produce divergent semantic association? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models possess remarkable capacity for processing language, but it remains unclear whether they can further generate creative content. |
| Approach: | They utilize the divergent association task (DAT) to examine the creative thinking of large language models through a cognitive perspective. |
| Outcome: | The proposed model outperforms the greedy search strategy while outperforming the average human level. |
AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Evaluating multimodal large language models (MLLMs) is becoming increasingly expensive as benchmarks grow in scale and cross-modality complexity. |
| Approach: | They propose an adaptive evaluation framework for efficient benchmarking that treats evaluation as an interview-like process by keeping a hypothesized ability structure of the evaluated model and actively selecting the informative questions. |
| Outcome: | Experiments on four representative multimodal benchmarks show that **A2-Judger significantly improves sample efficiency while maintaining reliable evaluation results. |
UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work lacks direct and fair evaluation of Large Language Models’ ability to express uncertainty effectively in long-form generation. |
| Approach: | They propose a benchmark to evaluate uncertainty expression in both long- and short-form question answering (QA) they propose prompt-based and training-based methods to improve models’ performance. |
| Outcome: | The proposed method mitigates this issue but a misalignment persists in uncertainty expression between long- and short-form generation. |
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain (2024.findings-acl)
Copied to clipboard
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
| Challenge: | a new multimodal decision-making benchmark evaluates the integrated capabilities of multimodal large language models. |
| Approach: | They propose a multimodal decision-making benchmark for evaluating MLLMs . they propose an automatic evaluation protocol to assess 10 prevalent ML models . |
| Outcome: | The proposed benchmark improves performance of multimodal large language models in three scenarios . the model is required to integrate multiple capabilities to make accurate decisions . |
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are increasingly used as automatic judges . however, their reliability and vulnerabilities to biases remain underexplored . |
| Approach: | They propose a benchmark to evaluate MLLMs that fail to integrate visual cues . they also introduce a test to evaluate the reliability of MLMLs based on a set of asymmetric evaluation tendencies. |
| Outcome: | Experiments on 26 state-of-the-art MLLMs reveal modality neglect and asymmetric evaluation tendencies . a standardized model with a benchmark enables a fine-grained diagnosis of nine bias types . |
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks (2026.acl-long)
Copied to clipboard
Tan Min Sen, Zachary Choy Kit Chun, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Banerjee Mohor, Swaagat Bikash Saikia, Alvin Chan
| Challenge: | Existing methods for evaluating creativity are tightly coupled to specific tasks and limiting scalability and generality. |
| Approach: | They propose a domain-agnostic framework for quantifying LLM creativity across open-ended tasks. |
| Outcome: | The proposed framework captures key facets of creativity including novelty, diversity, and task fulfilment with over 60% improved efficiency. |
CONSTRUCTURE: Benchmarking CONcept STRUCTUre REasoning for Multimodal Large Language Models (2024.findings-emnlp)
Copied to clipboard
Zhiwei Zha, Xiangru Zhu, Yuanyi Xu, Chenghua Huang, Jingping Liu, Zhixu Li, Xuwu Wang, Yanghua Xiao, Bei Yang, Xiaoxiao Xu
| Challenge: | Existing multimodal large language models lack the ability to perceive the visual world with a deep concept structure cognition. |
| Approach: | They propose a concept-level benchmark to assess MLLMs’ hierarchical concept understanding and reasoning abilities. |
| Outcome: | The proposed model outperforms state-of-the-art models in concept structure reasoning evaluation. |
AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models are increasingly used in Personalized Image Aesthetic Assessment (PIAA) however, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. |
| Approach: | They propose to evaluate MLLMs along two complementary dimensions: (1) stereotype bias and (2) alignment between model outputs and genuine human aesthetic preferences. |
| Outcome: | The proposed benchmark covers three subtasks: aesthetic perception, assessment, empathy and alignment between outputs and genuine human aesthetic preferences. |