WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging (2025.acl-short)
Copied to clipboard
| Challenge: | Multiple choice question (MCQ) benchmarks are widely used to evaluate Large Language Models (LLMs). |
| Approach: | They propose a method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with “None of the above”. |
| Outcome: | The proposed method can be applied to 6 popular benchmarks and evaluate 18 open-weight LLMs. |
Similar Papers
None of the Above, Less of the Right Parallel Patterns in Human and LLM Performance on Multi-Choice Questions Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice exam questions with “None of the above” (NA) options have been extensively studied in educational testing . however, their impact on Large Language Models (LLMs) evaluation remains underexplored . |
| Approach: | They conduct systematic experiments with 28 LLMs on the MMLU benchmark to examine how NA options affect model performance and confidence calibration. |
| Outcome: | The results highlight important implications for benchmark design and raise questions about LLMs’ ability to handle uncertainty in real-world applications. |
Morables: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables (2025.emnlp-main)
Copied to clipboard
Matteo Marcuzzo, Alessandro Zangari, Andrea Albarelli, Jose Camacho-Collados, Mohammad Taher Pilehvar
| Challenge: | Literature-based benchmarks provide a compelling framework for evaluating LLMs' capacity for complex abstract reasoning and inference. |
| Approach: | They propose a novel moral reasoning benchmark built from fables and short stories that uses adversarial variants to stress-test model robustness. |
| Outcome: | The proposed model outperforms models on fables and short stories, but is susceptible to adversarial manipulation and rely on superficial patterns rather than true moral reasoning. |
ARC ‘Challenge’ Is Not That Challenging (2025.findings-acl)
Copied to clipboard
| Challenge: | ARC Challenge appears to be more difficult than ARC Easy for modern LLMs due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. |
| Approach: | They propose a setup where multiple choice problems are evaluated and the one with the highest likelihood is compared against the gold standard to determine accuracy. |
| Outcome: | The proposed evaluation setup is more difficult than ARC Easy for modern LLMs because it prevents direct comparison of answer choices rather than inherent complexity. |
Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities? (2025.findings-acl)
Copied to clipboard
| Challenge: | Current 3D LLMs are evaluated on Q&A or captioning tasks rather than specific downstream tasks like object detection. |
| Approach: | They propose principles for better assessing genuine 3D understanding by explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs. |
| Outcome: | The proposed methods are based on the “2D-Cheating” problem in 3D LLM evaluation, suggesting that they are ineffective . |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Many commonsense reasoning questions require a hard selection of a single correct answer . ambiguity and semantic mismatches are common in many MCQs . |
| Approach: | They collect plausibility judgments on 5 000 commonsense reasoning questions . they find that the answer rated most plausible does not match the benchmark gold answers . |
| Outcome: | Experiments with LLMS reveal low accuracy and high variation in performance on the subset . high plausibility rating for the most plausible answer is highlighted in bold . |
Curse of Knowledge: Your Guidance and Provided Knowledge are biasing LLM Judges in Complex Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has focused on simple settings, but their reliability in complex tasks remains understudied. |
| Approach: | They propose to use large language models as judges to evaluate reliability in complex tasks . they use a challenge benchmark to expose and quantify Auxiliary Information Induced Biases . |
| Outcome: | The proposed benchmark exposes and quantifies Auxiliary Information Induced Biases across 12 basic and 3 advanced scenarios. |
In Benchmarks We Trust ... Or Not? (2025.emnlp-main)
Copied to clipboard
Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens, Andriy Kosar, Ehsan Lotfi, Nikolay Banar, Pieter Fivez, Luna De Bruyne, Walter Daelemans
| Challenge: | Existing benchmarks for Large Language Models (LLMs) are inadequate and lack a clear solution. |
| Approach: | They propose checklists to cover all aspects of benchmarking issues, both for benchmark creation and usage. |
| Outcome: | The proposed checklists cover all aspects of benchmarking issues, both for benchmark creation and usage. |
MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation (2025.naacl-long)
Copied to clipboard
Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, Jingyang Yuan, Wei Ju, Luchen Liu, Tianyu Liu, Baobao Chang, Ming Zhang
| Challenge: | Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, but many benchmarks suffer from systematic biases. |
| Approach: | They propose a benchmark to avoid Type-I errors by creating one perception question and one knowledge anchor question through a meticulous annotation process. |
| Outcome: | The proposed benchmark avoids Type-I errors while maintaining reliability of MCQ evaluations. |
Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It (2026.findings-eacl)
Copied to clipboard
| Challenge: | Using social reasoning benchmarks, we uncover pervasive flaws in both benchmark items and evaluation methodology. |
| Approach: | They audit three widely used social reasoning benchmarks and identify flaws in their design and evaluation methodology. |
| Outcome: | The results challenge the validity of current benchmark-based claims about social reasoning in large language models. |