Can LLMs Judge Debates? Evaluating Non-Linear Reasoning via Argumentation Theory Semantics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models excel at linear reasoning tasks but are underexplored on non-linear structures such as natural debates. |
| Approach: | They evaluate whether Large Language Models can approximate structured reasoning from Computational Argumentation Theory. |
| Outcome: | The proposed model performs well on dialogue-formatted debates without access to the underlying graph. |
Similar Papers
Natural Language Reasoning in Large Language Models: Analysis and Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Argumentative reasoning presents unique challenges due to its reliance on context, implicit assumptions, and value judgments. |
| Approach: | They propose a large-scale evaluation of LLMs' unconstrained natural language reasoning capabilities . they formalise a new strategy designed to evaluate argumentative reasoning in LLM . |
| Outcome: | The proposed model performs better on a range of reasoning tasks than other models. |
Can LLMs Really Judge? A Progressive Argumentation-Mining Framework for Distinguishing Understanding from Aggregation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of large language models rely on dataset-based generation accuracy . however, generative correctness does not guarantee discriminative capability to verify solutions . |
| Approach: | They propose a diagnostic framework that explicitly controls context and isolates discriminative behaviors. |
| Outcome: | The proposed framework explicitly controls context and isolates discriminative behaviors. |
Exploring the Potential of Large Language Models in Computational Argumentation (2024.acl-long)
Copied to clipboard
| Challenge: | Argumentation is an essential tool in various domains, including law, public policy, and artificial intelligence. |
| Approach: | They propose to evaluate LLMs on various computational argumentation tasks . they organize existing tasks into six main categories and standardize the format of 14 datasets . |
| Outcome: | The proposed model performs well on argument mining and argument generation tasks. |
ArgBench: Benchmarking LLMs on Computational Argumentation Tasks (2026.findings-acl)
Copied to clipboard
| Challenge: | Argumentation skills are an essential toolkit for large language models (LLMs). |
| Approach: | They propose a benchmark to evaluate the generalizability of five LLM families across 46 computational argumentation tasks. |
| Outcome: | The proposed benchmark evaluates the generalizability of five LLM families across 46 computational argumentation tasks covering mining arguments, assessing perspectives, evaluating argument quality, reasoning about arguments, and generating arguments. |
Evaluating the Performance of Large Language Models via Debates (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are evolving and impacting various fields . current methods for evaluation are based on fixed, domain-specific questions or rely on human input, making them unscalable. |
| Approach: | They propose a benchmarking framework based on debates between LLMs, judged by another LLM. |
| Outcome: | The proposed framework achieves rankings that align closely with popular rankings based on human input eliminating the need for costly crowdsourcing. |
Can LLMs Extract Frame-Semantic Arguments? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Frame-semantic parsing is a critical task in natural language understanding . however, the ability of large language models to extract frame-sensical arguments remains unexplored . |
| Approach: | They propose a framework to extract frame-semantic arguments from large language models . they use JSON representations to enhance performance, but smaller models can achieve competitive results . |
| Outcome: | The proposed model achieves state-of-the-art on ambiguous targets while limiting generalization to out-of domain data. |
Can Language Models Recognize Convincing Arguments? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have found that large language models can generate persuasive content without engaging in human experimentation. |
| Approach: | They extend a dataset with debates, votes, and user traits to measure LLMs' ability to distinguish between strong and weak arguments, predict stances based on beliefs and demographic characteristics, and determine appeal of argument to individual based upon their traits. |
| Outcome: | The proposed tasks outperform human predictions in detecting convincing arguments in debates, votes, and user traits. |
Argument Quality Assessment in the Age of Instruction-Following Large Language Models (2024.lrec-main)
Copied to clipboard
Henning Wachsmuth, Gabriella Lapesa, Elena Cabrio, Anne Lauscher, Joonsuk Park, Eva Maria Vecchi, Serena Villata, Timon Ziegenbein
| Challenge: | Argument quality assessment is critical for opinion formation, decision making, writing education, and the like. |
| Approach: | They propose to use large language models to leverage knowledge across contexts to enable a much more reliable assessment. |
| Outcome: | The proposed approach improves the quality of argumentation and the ability to leverage knowledge across contexts. |
Can Large Language Models Understand Argument Schemes? (2025.findings-acl)
Copied to clipboard
| Challenge: | Argument schemes are stereotypical forms of reasoning that occur in everyday arguments. |
| Approach: | They propose to use large language models (LLMs) to classify argument schemes based on Walton’s taxonomy to employ formal definitions and LLM-generated descriptions to enhance task instructions. |
| Outcome: | The proposed models perform well on annotated and automatically generated arguments, and provide insights for advancing reasoning capabilities in computational argumentation. |
LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in mathematical reasoning, but their effectiveness is limited to specific mathematical topics. |
| Approach: | They propose to use the MaTT benchmark to assess large language models' accuracy in multiple-choice scenarios. |
| Outcome: | The proposed model achieved 54% accuracy in a multiple-choice scenario, while the Chain-of-Thought prompting did not improve. |