CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowds (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing algorithms for detecting logical fallacies in texts are expensive and require large-scale labeled datasets. |
| Approach: | They introduce CoCoLoFa, the largest known logical fallacy dataset, with 7,706 comments for 648 news articles labeled for fallacy presence and type. |
| Outcome: | The proposed dataset outperforms state-of-the-art LLMs in fallacy detection and classification. |
Similar Papers
Debiasing Logical Fallacy Detection for Real-World Robustness via Counterfactually Augmented Data (2026.acl-srw)
Copied to clipboard
| Challenge: | Logical fallacy detection models often over-flag valid reasoning due to spurious correlations. |
| Approach: | They propose to augment CAD models with counterfactually-augmented data to debias them . they found that the models often over-flag valid reasoning due to spurious correlations . |
| Outcome: | The proposed approach reduces false positive rate by 58% on a 300-sample set. |
Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has focused on the ability of Large Language Models to **identify** or **classify** fallacies, but their robustness against these fallacias in persuasive contexts remains largely unexplored. |
| Approach: | They propose a new metric to assess LLM robustness against fallacies by pairing factual questions with fallacious arguments and developing a multi-round debate framework to assess model resilience. |
| Outcome: | The proposed metric disentangles robustness from a model’s knowledge limitations and demonstrates unique vulnerability profiles across models. |
Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing systems that detect logical fallacies in public discourse do not help people recognize them independently. |
| Approach: | They propose an intelligent tutoring system which uses large language models to help humans learn about logical fallacies. |
| Outcome: | The proposed system outperforms baseline LLMs lacking such pedagogical strategies. |
Reason from Fallacy: Enhancing Large Language Models’ Logical Reasoning through Logical Fallacy Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but struggle with some more complex reasoning tasks including logical reasoning. |
| Approach: | They propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW to evaluate LLMs’ capability of logical fallacy understanding. |
| Outcome: | The proposed dataset can be used to evaluate LLMs’ LFU capability and to fine-tune LLM models to obtain significantly enhanced performance on logical reasoning. |
Missci: Reconstructing Fallacies in Misrepresented Science (2024.acl-long)
Copied to clipboard
| Challenge: | False or misleading narratives spread rapidly on social networks, posing challenges for non-experts in discerning credible information. |
| Approach: | They propose a model for fallacious reasoning that focuses on implicit fallacies between relevant content and the inaccurate claim and requires models to verbalize the fallacious thinking in addition to classifying it. |
| Outcome: | The proposed model focuses on implicit fallacies between relevant content and the inaccurate claim and requires models to verbalize the fallacious reasoning in addition to classifying it. |
How Susceptible Are LLMs to Logical Fallacies? (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies have focused on LLMs' reasoning abilities, but their rational thinking capacity is not as robust as that of other NLP downstream tasks. |
| Approach: | They propose a diagnostic benchmark to assess the robustness of Large Language Models against logical fallacies by comparing their performance against a scenario where the persuader employs logical fallsacie. |
| Outcome: | The proposed benchmark compares the performance of LLMs in debates on controversial topics against logical fallacies. |
Fine-grained Fallacy Detection with Human Label Variation (2025.naacl-long)
Copied to clipboard
| Challenge: | Fallacy detection is an open challenge in NLP and has shown to be intrinsically difficult for both humans and machines. |
| Approach: | They propose a framework that minimizes annotation errors whilst keeping signals of human label variation. |
| Outcome: | The proposed framework minimizes annotation errors while keeping signals of human label variation. |
Logical Fallacy Detection (2022.findings-emnlp)
Copied to clipboard
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, Bernhard Schoelkopf
| Challenge: | Existing language models perform poorly on logical fallacy detection . fallacious arguments can lead to disagreements, conflicts, endless debates, and a lack of consensus . |
| Approach: | They propose a task of logical fallacy detection and propose LogicClimate to detect fallacies in text. |
| Outcome: | The proposed task outperforms the best language model on Logic and LogicClimate . human reasoning is marred by logical fallacies, and some exacerbate misinformation . |
Boosting Logical Fallacy Reasoning in LLMs via Logical Structure Tree (2024.emnlp-main)
Copied to clipboard
| Challenge: | Logical fallacy is the use of invalid or flawed reasoning in the construction of a statement. |
| Approach: | They propose to build a logical structure tree to represent hierarchical logic flow among relation connectives and their arguments in a statement. |
| Outcome: | The proposed model significantly improves accuracy and recall for fallacy detection and fallacy classification. |
The Search for Agreement on Logical Fallacy Annotation of an Infodemic (2022.lrec-1)
Copied to clipboard
Claire Bonial, Austin Blodgett, Taylor Hudson, Stephanie M. Lukin, Jeffrey Micher, Douglas Summers-Stay, Peter Sutor, Clare Voss
| Challenge: | a parallel "infodemic" has emerged with the COVID-19 pandemic . logical fallacies can be subtly encoded in the structure of a document across multiple sentences . |
| Approach: | They evaluate an annotation schema for labeling logical fallacy types using linguist annotations . they propose to use a machine learning algorithm to train annotators for fallacy detection . |
| Outcome: | The proposed annotation schema is clear and non-overlapping for manual and system assignment. |