Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated impressive capabilities in various NLP tasks, but previous studies have shown they are sensitive to prompt wording and few-shot demonstrations and their order. |
| Approach: | They focus on LLMs robustness on multiple-choice questions . they find a performance gap of 13% to 85% when options are reordered . |
| Outcome: | The proposed model outperforms supervised models on multiple choice questions even outperforming humans. |
Similar Papers
Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Using zero-shot or few-shot prompting, Large Language Models have been widely adopted in downstream applications. |
| Approach: | They propose to quantify the impact of option order and token usage on LLMs and propose mitigation strategies to enhance model performance. |
| Outcome: | The proposed mitigation strategies improve model performance and reduce the impact of token and order sensitivity on LLMs. |
Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models can be influenced by various forms of biases, says a new study . positional bias affects how LLMs interpret and weigh information, the authors say . |
| Approach: | a new study examines the impact of positional bias on large language models . positional biased models prioritize items based on their position rather than content or quality . |
| Outcome: | a new study shows that LLMs prioritize items based on their position rather than content or quality . the positional bias affects how LLM interpret and weigh information, the authors say . |
Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs? (2024.lrec-main)
Copied to clipboard
| Challenge: | Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) however, there are concerns about whether MCQ can truly measure LLM’s capabilities. |
| Approach: | They propose to use multiple choice questions to evaluate large language models (LLMs) to assess their capabilities. |
| Outcome: | The proposed methods show that MCQs are less reliable than LFGQs in terms of expected calibration error. |
LLMs May Perform MCQA by Selecting the Least Incorrect Option (2025.coling-main)
Copied to clipboard
| Challenge: | Multiple Choice Question Answering (MCQA) is a fundamental format for various tasks in NLP, such as commonsense reasoning. |
| Approach: | They propose a method to increase the number of correct options in a dataset. |
| Outcome: | The proposed method improves the performance of multiple choice question answering (MCQA) and improves its accuracy. |
Wait, that’s not an option: LLMs Robustness with Incorrect Multiple-Choice Options (2025.acl-long)
Copied to clipboard
| Challenge: | Using a framework that combines instruction-following with critical reasoning, we show that the ability of LLMs to override defaults when faced with invalid options is impaired by alignment techniques. |
| Approach: | They propose a framework for evaluating LLMs’ capacity to balance instruction-following with critical reasoning when presented with multiple-choice questions containing no valid answers. |
| Outcome: | The proposed framework improves models' ability to override defaults when faced with invalid options while minimizing the impact of model size and training techniques on the model. |
Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to follow multi-constraint instructions in real-world applications. |
| Approach: | They propose to quantify the difficulty distribution of constraints by a novel Difficulty Distribution Index (CDDI) they find that LLMs are more performant when presented with constraints in a “hard-to-easy” order. |
| Outcome: | The proposed model is more performant when presented with constraints in a “hard-to-easy” order, compared with existing models with different architectures and sizes of parameters. |
Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on Large Language Models (LLMs) have not investigated the impact of question types on LLM performance. |
| Approach: | They evaluate the performance of five Large Language Models on reasoning tasks . they use quantitative reasoning tasks and deductive reasoning tasks to evaluate the models . |
| Outcome: | The results show that Reasoning accuracy does not correlate with final selection accuracy. |
7 Points to Tsinghua but 10 Points to ? Assessing Large Language Models in Agentic Multilingual National Bias (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, but studies on risks associated with cross biases are limited to immediate context preferences. |
| Approach: | They investigate multilingual bias in state-of-the-art Large Language Models by analyzing their responses to decision-making tasks across multiple languages. |
| Outcome: | The proposed model can provide personalized advice across university applications, travel, and relocation scenarios. |
Exploring the Sensitivity of LLMs’ Decision-Making Capabilities: Insights from Prompt Variations and Hyperparameters (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior studies have compared the decision-making abilities of large language models with those of humans from a psychological perspective. |
| Approach: | They examine LLMs' performance on the Horizon decision-making task studied by Binz and Schulz (2023) they observe that the decision- making abilities fluctuate based on input prompts and temperature settings. |
| Outcome: | The results show that LLMs display a human-like exploration–exploitation tradeoff after simple adjustments to the prompt. |
Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Multiple-Choice Question Answering (MCQA) is a widely used task in the evaluation of large language models (LLMs). |
| Approach: | They propose a tuning-free, causal effect driven debiasing method which intervenes the activations of identified components according to their causal effects. |
| Outcome: | The proposed method alleviates the aforementioned bias and improves the performance of LLMs. |