Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts . |
| Approach: | They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone . |
| Outcome: | The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain. |
Similar Papers
LLMs May Perform MCQA by Selecting the Least Incorrect Option (2025.coling-main)
Copied to clipboard
| Challenge: | Multiple Choice Question Answering (MCQA) is a fundamental format for various tasks in NLP, such as commonsense reasoning. |
| Approach: | They propose a method to increase the number of correct options in a dataset. |
| Outcome: | The proposed method improves the performance of multiple choice question answering (MCQA) and improves its accuracy. |
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering tasks are one of the most commonly used tasks for evaluating Large Language Models (LLMs). |
| Approach: | They analyze whether existing answer extraction methods are aligned with human judgment and how they are influenced by answer constraints in the prompt across different domains. |
| Outcome: | The proposed evaluation strategies can be inconsistent with human judgment, and can lead to inaccurate and misleading comparisons. |
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above (2025.acl-long)
Copied to clipboard
| Challenge: | Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing. |
| Approach: | They argue for a reform of multiple choice question answering (MCQA) they argue for more generative formats based on human testing . |
| Outcome: | The proposed reforms improve the quality of MCQA, the authors argue . they show that even when MCQ is a useful format, its datasets suffer from leakage, unanswerability, shortcuts and saturation. |
Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs? (2024.lrec-main)
Copied to clipboard
| Challenge: | Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) however, there are concerns about whether MCQ can truly measure LLM’s capabilities. |
| Approach: | They propose to use multiple choice questions to evaluate large language models (LLMs) to assess their capabilities. |
| Outcome: | The proposed methods show that MCQs are less reliable than LFGQs in terms of expected calibration error. |
Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted the significant performance variation that can arise from minor changes in prompt design. |
| Approach: | They propose to tokenize the space following the colon to facilitate automated answer extraction via next-token probabilities. |
| Outcome: | The proposed tokenization improves model calibration and improves confidence estimates. |
When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) is easy to evaluate but adds a meta-task . prior work has shown that language models exhibit selection biases for particular option identifiers such as the label "A" |
| Approach: | They find that option-boundary residual states contain strong linearly decodable signals . winning content position becomes decoded after final option is processed . |
| Outcome: | The proposed model solves the problem and outputs the symbol that represents the answer. |
Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers (2026.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) give reasoning before answering, excelling in multiple-choice question answering (MCQA) . but, some studies find that LLMs sans reasoning fail in MCQA without using the question, i.e., choices-only. |
| Approach: | They propose to use reasoning LLMs to separate problematic data from less problematic strategies by examining reasoning traces. |
| Outcome: | The proposed models perform well in multiple-choice question answering without the question, but they fail to use the question. |
Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on Large Language Models (LLMs) have not investigated the impact of question types on LLM performance. |
| Approach: | They evaluate the performance of five Large Language Models on reasoning tasks . they use quantitative reasoning tasks and deductive reasoning tasks to evaluate the models . |
| Outcome: | The results show that Reasoning accuracy does not correlate with final selection accuracy. |
Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questions (2026.eacl-long)
Copied to clipboard
| Challenge: | Multiple-choice questions (MCQs) are used to assess knowledge, reasoning abilities, and even values encoded in large language models. |
| Approach: | They propose to test whether multilingual LLMs are consistent in their responses across languages . they also use human-translated questions aligned in 8 European languages to test their robustness . |
| Outcome: | The proposed corpus of questions is aligned in 8 European languages and compared with previous studies. |
Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Multiple-Choice Question Answering (MCQA) is a widely used task in the evaluation of large language models (LLMs). |
| Approach: | They propose a tuning-free, causal effect driven debiasing method which intervenes the activations of identified components according to their causal effects. |
| Outcome: | The proposed method alleviates the aforementioned bias and improves the performance of LLMs. |