Boosting Logical Fallacy Reasoning in LLMs via Logical Structure Tree (2024.emnlp-main)
Copied to clipboard
| Challenge: | Logical fallacy is the use of invalid or flawed reasoning in the construction of a statement. |
| Approach: | They propose to build a logical structure tree to represent hierarchical logic flow among relation connectives and their arguments in a statement. |
| Outcome: | The proposed model significantly improves accuracy and recall for fallacy detection and fallacy classification. |
Similar Papers
Logical Fallacy Detection (2022.findings-emnlp)
Copied to clipboard
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, Bernhard Schoelkopf
| Challenge: | Existing language models perform poorly on logical fallacy detection . fallacious arguments can lead to disagreements, conflicts, endless debates, and a lack of consensus . |
| Approach: | They propose a task of logical fallacy detection and propose LogicClimate to detect fallacies in text. |
| Outcome: | The proposed task outperforms the best language model on Logic and LogicClimate . human reasoning is marred by logical fallacies, and some exacerbate misinformation . |
Reason from Fallacy: Enhancing Large Language Models’ Logical Reasoning through Logical Fallacy Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but struggle with some more complex reasoning tasks including logical reasoning. |
| Approach: | They propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW to evaluate LLMs’ capability of logical fallacy understanding. |
| Outcome: | The proposed dataset can be used to evaluate LLMs’ LFU capability and to fine-tune LLM models to obtain significantly enhanced performance on logical reasoning. |
Large Language Models Are Better Logical Fallacy Reasoners with Counterargument, Explanation, and Goal-Aware Prompt Formulation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent large language models have demonstrated improved processing of complex language, but detecting logical fallacies remains a challenge. |
| Approach: | They propose a prompt formulation approach for logical fallacy detection that integrates contextual information into input text and queries for validity within the argument’s context. |
| Outcome: | The proposed approach improves over state-of-the-art models by 0.57 in F1-scores and 0.45 in fine-tuned models. |
Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has focused on the ability of Large Language Models to **identify** or **classify** fallacies, but their robustness against these fallacias in persuasive contexts remains largely unexplored. |
| Approach: | They propose a new metric to assess LLM robustness against fallacies by pairing factual questions with fallacious arguments and developing a multi-round debate framework to assess model resilience. |
| Outcome: | The proposed metric disentangles robustness from a model’s knowledge limitations and demonstrates unique vulnerability profiles across models. |
CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowds (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing algorithms for detecting logical fallacies in texts are expensive and require large-scale labeled datasets. |
| Approach: | They introduce CoCoLoFa, the largest known logical fallacy dataset, with 7,706 comments for 648 news articles labeled for fallacy presence and type. |
| Outcome: | The proposed dataset outperforms state-of-the-art LLMs in fallacy detection and classification. |
Flee the Flaw: Annotating the Underlying Logic of Fallacious Arguments Through Templates and Slot-filling (2024.emnlp-main)
Copied to clipboard
Irfan Robbani, Paul Reisert, Surawat Pothong, Naoya Inoue, Camélia Guerraoui, Wenzhi Wang, Shoichi Naito, Jungmin Choi, Kentaro Inui
| Challenge: | Prior work on quality assessment has focused on numerical scoring and fallacy type-labeling tasks, without aiming to analyze fallacy logic structures. |
| Approach: | They propose four sets of explainable templates for common informal logical fallacies designed to explicate a fallacy’s implicit logic. |
| Outcome: | The proposed models achieve a high agreement score and reasonable coverage 83% on 400 fallacious arguments and state-of-the-art language models struggle with detecting fallacy templates (0.47 accuracy). |
A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing models of large language models struggle with complex logical reasoning problems. |
| Approach: | They propose to use large language models to identify their own errors to improve their models' performance. |
| Outcome: | The proposed models can identify logical fallacies accurately and improve by themselves. |
A Logical Fallacy-Informed Framework for Argument Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Argument generation is crucial in daily life and has numerous online and offline applications. |
| Approach: | They propose a fallacy-informed preference optimization that includes a classification loss to capture the fine-grained information on fallacy types to help LLMs generate logically sound arguments. |
| Outcome: | The proposed method reduces fallacy errors by 17.5% on argument generation tasks and outperforms fine-tuned baselines and other preference optimization methods, such as DPO. |
Where Reasoning Breaks: Logic-Aware Path Selection by Controlling Logical Connectives in LLMs Reasoning Chains (2026.findings-acl)
Copied to clipboard
| Challenge: | a single transition error can propagate through the entire reasoning chain, leading to unstable performance. |
| Approach: | They propose a framework that intervenes at logical connective junctions to improve LLMs' reasoning. |
| Outcome: | The proposed framework achieves favorable accuracy–efficiency trade-off compared to global inference time scaling methods like beam search and self-consistency. |
How Susceptible Are LLMs to Logical Fallacies? (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies have focused on LLMs' reasoning abilities, but their rational thinking capacity is not as robust as that of other NLP downstream tasks. |
| Approach: | They propose a diagnostic benchmark to assess the robustness of Large Language Models against logical fallacies by comparing their performance against a scenario where the persuader employs logical fallsacie. |
| Outcome: | The proposed benchmark compares the performance of LLMs in debates on controversial topics against logical fallacies. |