Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Step (2025.findings-naacl)
Copied to clipboard
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Yufei Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Current research found the issue of Early Answering in large language models where the models already have an answer before generating the Chain-of-Thought (CoT). |
| Approach: | They propose a method to probe changes in confidence during the model’s reasoning and prioritize answers with correct reasoning among multiple candidates. |
| Outcome: | The proposed method reveals that in a significant number of question-answer cases, CoT appears to be unnecessary and this necessity correlates with the simplicity of the task, defined by the reasoning steps required. |
Similar Papers
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023.acl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). |
| Approach: | They propose to use Chain-of-Thought (CoT) prompting to encourage the LLM to generate intermediate rationales for solving a problem by providing a series of reasoning steps in the demonstrations. |
| Outcome: | The proposed model can generate coherent lines of reasoning even with invalid demonstrations while still generating coherent lines during inference. |
Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a number of shortcomings, including lack of factual correctness. |
| Approach: | They propose a framework to increase prediction factuality by post-editing reasoning chains . they propose to use large language models to generate interpretable reasoning chains. |
| Outcome: | The proposed framework leads to accuracy improvements in open-domain question-answering tasks. |
It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. |
| Approach: | They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting. |
| Outcome: | The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets. |
Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens (2026.findings-acl)
Copied to clipboard
Chengshuai Zhao, Zhen Tan, Pingchuan Ma, Dawei Li, Bohan Jiang, Yancheng Wang, Yingzhen Yang, Huan Liu
| Challenge: | Chain-of-Thought (CoT) prompting has been shown to be effective in eliciting structured reasoning from large language models (LLMs). |
| Approach: | They propose a data distribution lens to understand when and why CoT reasoning fails . they propose 'data-based' training that trains LLMs from scratch . |
| Outcome: | The proposed model enables models to generate reasoning trajectories that approximate those observed during training. |
Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive reasoning abilities when prompted with Chain-of-Thought (CoT). |
| Approach: | They propose to categorize Chain-of-X methods by taxonomies of nodes, i.e., the X in CoX, and application tasks, and then categorise them by taxanomies and discuss potential future directions. |
| Outcome: | The proposed methods are categorised by taxonomies of nodes, i.e., the X in CoX, and application tasks. |
Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness (2025.findings-acl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting demonstrates varying performance under different reasoning tasks. |
| Approach: | They propose to recall extra information from the question to enhance CoT generation and evaluate CoTs based on their information gain. |
| Outcome: | The proposed method improves both the faithfulness and effectiveness of CoT and evaluates it based on their information gain. |
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference. |
| Approach: | They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task. |
| Outcome: | The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets. |
Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting has been shown to enhance the multi-step reasoning capabilities of Large Language Models (LLMs). |
| Approach: | They propose to use CoT prompting to analyze a symbolic reasoning task where letters are shifted forward some number of steps in the alphabet. |
| Outcome: | The proposed model performs well on a symbolic reasoning task, with three LLMs performing the task using CoT prompts. |
Probe Then Retrieve and Reason: Distilling Probing and Reasoning Capabilities into Smaller Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research efforts have focused on distilling Large Language Models into Small Language Model (SLMs) however, the results of CoT distillation are inadequate for knowledge-intensive reasoning tasks. |
| Approach: | They propose a retrieval-based framework which distills question probing and reasoning capabilities from Large Language Models into SLMs. |
| Outcome: | The proposed framework improves probing and reasoning capabilities of large language models in knowledge-intensive reasoning tasks. |
Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit high-level commonsense reasoning abilities, especially with enhancement methods like Chain-of-Thought (CoT). |
| Approach: | They propose a chain-of-thought-like method to elicit models' potential abilities to generate rationales and answers that are based on attribution tracing and causal tracers to probe the internal working mechanism of the LLM. |
| Outcome: | The proposed method eliminates Toxic CoT problems and improves the model’s overall commonsense reasoning performance by 5.5%. |