It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. |
| Approach: | They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting. |
| Outcome: | The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets. |
Similar Papers
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023.acl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). |
| Approach: | They propose to use Chain-of-Thought (CoT) prompting to encourage the LLM to generate intermediate rationales for solving a problem by providing a series of reasoning steps in the demonstrations. |
| Outcome: | The proposed model can generate coherent lines of reasoning even with invalid demonstrations while still generating coherent lines during inference. |
Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open-domain question-answering methods lack quality assurance . existing methods lack scalability and poor diversity, hindering LLMs' capabilities . |
| Approach: | They propose an open-domain multi-hop reasoning framework to answer multi-choice questions . they propose an adaptive sampler for in-context selection and self-prompted inference . |
| Outcome: | The proposed framework surpasses the existing SOTA methods on large-scale datasets and doubles the zero-shot performance of small-scale LLMs. |
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference. |
| Approach: | They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task. |
| Outcome: | The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets. |
Large Language Models are Better Reasoners with Self-Verification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve complex natural language processing tasks require multiple steps to verify the answers. |
| Approach: | They propose to use chain of thought prompting to solve reasoning tasks with large language models. |
| Outcome: | The proposed method can improve reasoning performance on arithmetic, commonsense, and logical reasoning datasets. |
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) is a key technique for enhancing the performance of Large Language Models. |
| Approach: | They propose a framework that optimizes outputs by utilizing wrong information and multi-perspective verification. |
| Outcome: | The proposed framework surpasses all baselines on 8 datasets and 5 LLMs. |
ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing CoT synthesis approaches focus on simpler reasoning tasks and result in inconsistent CoT prompts. |
| Approach: | They propose a framework for automatic generation of superior CoT prompts based on three major evolution strategies . they propose 'step-level debating' method where multiple debaters discuss each reasoning step to arrive at the correct answer. |
| Outcome: | The proposed framework can generate superior CoT prompts from a CoT dataset. |
Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a number of shortcomings, including lack of factual correctness. |
| Approach: | They propose a framework to increase prediction factuality by post-editing reasoning chains . they propose to use large language models to generate interpretable reasoning chains. |
| Outcome: | The proposed framework leads to accuracy improvements in open-domain question-answering tasks. |
CAC-CoT: Connector-Aware Compact Chain-of-Thought for Efficient Reasoning Data Synthesis Across Dual-System Cognitive Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Long chain-of-thought (CoT) prompting often slows or even degrades performance on fast, intuitive "System-1" tasks. |
| Approach: | They introduce a method that deliberately restricts reasoning to a small, fixed set of connector phrases, steering the model toward concise and well-structured explanations. |
| Outcome: | The method achieves 85% on GSM8K and 40% on GPQA while also surpassing the baseline by over 20%. |
Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Step (2025.findings-naacl)
Copied to clipboard
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Yufei Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Current research found the issue of Early Answering in large language models where the models already have an answer before generating the Chain-of-Thought (CoT). |
| Approach: | They propose a method to probe changes in confidence during the model’s reasoning and prioritize answers with correct reasoning among multiple candidates. |
| Outcome: | The proposed method reveals that in a significant number of question-answer cases, CoT appears to be unnecessary and this necessity correlates with the simplicity of the task, defined by the reasoning steps required. |
CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought prompting is popular in reasoning tasks, but its application to Large Language Models (LLMs) in Natural Language Understanding (NLU) is under-explored. |
| Approach: | They propose a Coarse-to-Fine Chain-of-Thought approach that breaks down NLU tasks into multiple reasoning steps where LLMs can learn to acquire essential concepts. |
| Outcome: | The proposed approach is effective in assisting the LLMs adapt to multi-grained NLU tasks under zero-shot and few-shot multi-domain settings. |