No Need for Explanations: LLMs can implicitly learn from mistakes in-context (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature assumes that correct answers to large language models must be accompanied by comprehensive rationales to be helpful. |
| Approach: | They propose to show incorrect answers to Large Language Models (LLMs) as a popular strategy to improve their performance in reasoning-intensive tasks. |
| Outcome: | The proposed approach outperforms chain-of-thought prompting in math reasoning tasks. |
Similar Papers
Can LLMs Learn from Previous Mistakes? Investigating LLMs’ Errors to Boost for Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown the benefits to LLMs from fine-tuning golden-standard Chain-of-Thought rationales or using them as correct examples in few-shot prompting. |
| Approach: | They propose a new benchmark to test the effectiveness of large language models by leveraging errors to enhance reasoning capabilities. |
| Outcome: | The proposed methods can be used to fine-tune models in correct and incorrect domains, rather than tuning models to learn ground truth in traditional methods. |
Unraveling Misinformation Propagation in LLM Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning, but how they propagate within their reasoning process remains underexplored. |
| Approach: | They propose a practical approach to mitigating misinformation propagation in LLMs by applying factual corrections early in the reasoning process and fine-tuning on synthesized data with early-stage corrections significantly improves reasoning factuality. |
| Outcome: | The proposed model can correct misinformation when explicitly instructed, but fails to correct misinformation less than half the time even with explicit instructions. |
Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work has shown that simple learning can enhance the chain-of-thought (CoT) reasoning of large language models. |
| Approach: | They construct mistake-correction datasets to identify and correct mistakes in CoTs . they conclude that LLMs can learn from mistakes to enhance their CoT reasoning . |
| Outcome: | The proposed datasets show that LLMs can learn from mistakes to enhance their CoT reasoning performance. |
LLMs cannot find reasoning errors, but can correct them given the error location (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in poor performance overall. |
| Approach: | They propose to use a backtracking setup to test the correction abilities of LLMs on their mistake-finding ability to find logical mistakes. |
| Outcome: | The proposed model improves on 5 reasoning tasks, showing that it can correct logical mistakes without ground truth labels or training data. |
Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown great potential for in-context learning, but their robustness and performance on downstream tasks remains limited. |
| Approach: | They propose to examine the reliance of LLMs on shortcuts or spurious correlations within prompts for downstream tasks and find larger models are more likely to utilize shortcuts in prompts during inference. |
| Outcome: | The proposed model is “lazy learner” and more likely to use shortcuts in prompts during inference. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus on final accuracy, neglecting the critical aspect of reasoning capabilities. |
| Approach: | They propose to evaluate LLMs’ abilities to detect and correct reasoning mistakes by using rule-based methods and smaller language models. |
| Outcome: | The proposed model outperforms existing models such as GPT-4o and GPT4 in both accuracy and accuracy, but lacks data contamination and memorization concerns. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024.emnlp-main)
Copied to clipboard
| Challenge: | syllogistic reasoning is a deductive reasoning skill that is crucial in everyday problem-solving and decision-making experiences. |
| Approach: | They propose to study the reasoning abilities of Large Language Models (LLMs) they propose to use supervised fine-tuning and chain-of-thought reasoning to investigate their results. |
| Outcome: | The proposed models exhibit reasoning biases, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning. |
Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly required to selectively unlearn specific information. |
| Approach: | They propose a method which fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context while preserving unrelated information. |
| Outcome: | The proposed method achieves up to 95% forget accuracy while retaining 80% of unrelated knowledge, significantly outperforming baselines in both in-domain and out-of-domain scenarios. |