LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning (2026.acl-short)
Copied to clipboard
| Challenge: | LOGICAL-COMMONSENSEQA benchmarks evaluate commonsense reasoning as logical composition over pairs of atomic statements . commonsensible reasoning is central to human cognition and a long-standing challenge in artificial intelligence and natural language understanding. |
| Approach: | They propose a benchmark that reframes commonsense reasoning as logical composition over pairs of atomic statements using plausibility-level operators. |
| Outcome: | LOGICAL-COMMONSENSEQA exposes fundamental reasoning limitations and provides a framework for advancing compositional commonsense reasoning. |
Similar Papers
Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Many commonsense reasoning questions require a hard selection of a single correct answer . ambiguity and semantic mismatches are common in many MCQs . |
| Approach: | They collect plausibility judgments on 5 000 commonsense reasoning questions . they find that the answer rated most plausible does not match the benchmark gold answers . |
| Outcome: | Experiments with LLMS reveal low accuracy and high variation in performance on the subset . high plausibility rating for the most plausible answer is highlighted in bold . |
CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing commonsense reasoning datasets target different knowledge types, modalities, and formats, but how to help machines acquire and infer over commonsensical knowledge is still unclear. |
| Approach: | They propose a commonsense reasoning benchmark to motivate commonsensing progress from two perspectives: (1) Evaluating whether models can distinguish knowledge quality by predicting if the knowledge is enough to answer the question or not. |
| Outcome: | The proposed model outperforms existing models in evaluating their generalization capabilities across tasks while demonstrating that distinguishing knowledge quality remains challenging for current models. |
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing (D19-60)
Copied to clipboard
| Challenge: | Workshop on Commonsense Inference in Natural Language Processing focuses on commonsense knowledge representation and application in NLP tasks. |
| Approach: | COIN is a workshop on commonsense inference in natural language processing . workshop included two shared tasks on reading comprehension using commonsensense knowledge . |
| Outcome: | the workshop focused on modeling commonsense knowledge and commonsensing in natural language processing tasks. |
The Odyssey of Commonsense Causality: From Foundational Benchmarks to Cutting-Edge Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite its significance, a systematic exploration of commonsense causality is lacking. |
| Approach: | They focus on taxonomies, benchmarks, acquisition methods, qualitative reasoning, and quantitative measurements in commonsense causality. |
| Outcome: | The proposed method synthesizes insights from over 200 representative articles and provides a practical guide for beginners. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pretrained language models have shown promising results on commonsense reasoning benchmark datasets. |
| Approach: | They propose a commonsense reasoning benchmark dataset with 4k sentence pairs . they propose 'gamified' model-in-the-loop setup to incentivize challenging samples . |
| Outcome: | The proposed benchmarks show that the proposed model achieves 71% standard accuracy and 51% pairwise accuracy, well below human performance. |
Every Answer Matters: Evaluating Commonsense with Probabilistic Measures (2024.acl-long)
Copied to clipboard
Qi Cheng, Michael Boratko, Pranay Kumar Yelugam, Tim O’Gorman, Nalini Singh, Andrew McCallum, Xiang Li
| Challenge: | Existing commonsense evaluations are often posed as multiple-choice questions, allowing models to exploit systematic biases. |
| Approach: | They propose a generative task that evaluates common sense via multiple open-ended generations and a method that strongly correlates with human judgments. |
| Outcome: | The proposed method outperforms strong language model baselines on a dataset of human and machine common sense. |
HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning (2025.findings-acl)
Copied to clipboard
Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin
| Challenge: | Existing studies show that large language models are robust in commonsense reasoning . however, some variations in questions can lead to incorrect responses . |
| Approach: | They propose a large-scale bilingual benchmark consisting of 11,200 cases . they conduct extensive experiments on 41 representative LLMs . |
| Outcome: | The proposed benchmark systematically evaluates the robustness of large language models in commonsense reasoning. |
Beyond Surface Simplicity: Revealing Hidden Reasoning Attributes for Precise Commonsense Diagnosis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing commonsense question answering benchmarks often treat these aspects in isolation, resulting in evaluation accuracy differences of up to 24.8% across different difficulty levels. |
| Approach: | They propose a framework that reveals hidden reasoning attributes behind commonsense questions by leveraging the knowledge generated during the reasoning process. |
| Outcome: | The proposed framework reveals hidden reasoning attributes behind commonsense questions by leveraging the knowledge generated during the reasoning process. |
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (N19-1)
Copied to clipboard
| Challenge: | Recent work on question answering relies on factoid questions with little general knowledge. |
| Approach: | They propose a dataset to capture commonsense question answering with prior knowledge . they extract multiple-choice questions that discriminate between the source and target concepts . |
| Outcome: | The proposed dataset captures commonsense reasoning beyond associations . it obtains 56% accuracy, well below human performance, which is 89% . |