| Challenge: | Current language models perform well on multiple choice reasoning tasks, but the options are not treated equally. |
| Approach: | They propose a two-step scoring method that scores options and masks them to make the final prediction from the remaining options. |
| Outcome: | The proposed method is especially performant on logical reasoning tasks. |
Similar Papers
It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. |
| Approach: | They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting. |
| Outcome: | The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets. |
Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing task decomposition methods focus on memory, tool usage, and feedback mechanisms, but they often overlook the trade-off between performance and cost. |
| Approach: | They propose a strategy that selects the most suitable decomposition approach based on task characteristics and enhances the reliability of the results through a verification module. |
| Outcome: | The proposed strategy is based on categories of approaches, characteristics of tasks, and configuration of decomposition and execution models. |
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering tasks are one of the most commonly used tasks for evaluating Large Language Models (LLMs). |
| Approach: | They analyze whether existing answer extraction methods are aligned with human judgment and how they are influenced by answer constraints in the prompt across different domains. |
| Outcome: | The proposed evaluation strategies can be inconsistent with human judgment, and can lead to inaccurate and misleading comparisons. |
LM2: A Simple Society of Language Models Solves Complex Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that providing guidance via decomposing the original question into multiple subproblems elicits more robustness in LLM reasoning. |
| Approach: | They propose a language-based decomposition, solution and verification framework that modularizes the decomposer, solution, and verification into three different language models. |
| Outcome: | The proposed model outperforms existing methods on in- and out-domain reasoning problems, outperforming the best baselines by 8.1% on MATH, 7.71% on JEEBench, and 9.7% on MedQA problems. |
LLMs May Perform MCQA by Selecting the Least Incorrect Option (2025.coling-main)
Copied to clipboard
| Challenge: | Multiple Choice Question Answering (MCQA) is a fundamental format for various tasks in NLP, such as commonsense reasoning. |
| Approach: | They propose a method to increase the number of correct options in a dataset. |
| Outcome: | The proposed method improves the performance of multiple choice question answering (MCQA) and improves its accuracy. |
Efficient LLM Comparative Assessment: A Product of Experts Framework for Pairwise Comparisons (2024.emnlp-main)
Copied to clipboard
| Challenge: | LLM-as-a-judge approaches are effective but cost scales quadratically with number of candidates, which has practical limitations. |
| Approach: | They propose a Product of Expert (PoE) framework for efficient LLM Comparative Assessment where individual comparisons are considered experts that provide information on a pair’s score difference. |
| Outcome: | The proposed framework can generate score predictions that correlate well with human judgements on multiple NLG tasks with as few as 2% of comparisons. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Reasons to Reject? Aligning Language Models with Judgments (2024.findings-acl)
Copied to clipboard
| Challenge: | a new framework for aligning large language models with judgments is proposed to help with alignment . a framework that allows for fine-grained inappropriate content detection and correction based on judgments . large language model alignment is critical for making artificial intelligence a reliable ally for humanity . |
| Approach: | They propose a framework that allows for fine-grained inappropriate content detection and correction based on judgments. |
| Outcome: | The proposed framework beats the 175B DaVinci003 and improves on AlpacaEval using judgments. |
Current Advances in LLM Reasoning (2026.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial examines comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) advanced inference time methods and post-training methods that aim to make LLMs think more like humans are discussed in this tutorial. |
| Approach: | This tutorial explores comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) and discusses two types of methods to improve models’ reasoning: advanced inference time methods, structured and self-improvement inference methods, and post-training methods, such as RLHF, DPO, and GRPO. |
| Outcome: | This tutorial examines evaluation strategies to assess the reasoning abilities of large language models and discusses two types of methods to improve models’ reasoning. |
How do LLMs’ Preferences Affect Event Argument Extraction? CAT: Addressing Preference Traps in Unsupervised EAE (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to supervised EAE suffer from preference traps due to misalignments between prior knowledge, instructions, or output constraints and LLMs’ preferences. |
| Approach: | They propose an unsupervised EAE framework that handles LLMs' preference traps by targeting their prior knowledge and instructions. |
| Outcome: | The proposed framework matches the best DeepSeek-R1 API model with a significantly lower time cost. |