ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning (2023.acl-short)
Copied to clipboard
| Challenge: | Negation is a ubiquitous but complex linguistic phenomenon that poses a significant challenge for NLP systems. |
| Approach: | They propose a benchmark that measures how well models handle natural language negation . they extend ScoNe-NLI to embed negation reasoning in short narratives . |
| Outcome: | The proposed model can reason about negation, but struggles to do so on NLI examples outside of its core pretraining regime. |
Similar Papers
This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have grammatical knowledge but fail to interpret negation . a recent study shows that LLMs struggle with negative sentences . |
| Approach: | They propose to use a dataset to grasp LLMs' generalization and inference capability . they also fine-tuned models to assess whether the understanding of negation can be trained . |
| Outcome: | The proposed model is able to generalize and infer negation in 400,000 sentences . but it is suboptimal when it comes to negation, a key step in natural language processing . |
Understanding by Understanding Not: Modeling Negation in Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Negation is a core construction in natural language, but state-of-the-art pre-trained language models often handle it incorrectly. |
| Approach: | They propose to augment language modeling objective with unlikelihood objective based on negated generic sentences from a raw text corpus. |
| Outcome: | The proposed approach reduces the top 1 error rate to 4% on negated LAMA dataset and improves on negating NLI benchmarks. |
Towards the Roots of the Negation Problem: A Multilingual NLI Dataset and Model Scaling Analysis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Negations are key to determining sentence meaning, making them essential for logical reasoning. |
| Approach: | They construct and publish two new textual entailment datasets in four languages with paired examples differing in negation. |
| Outcome: | The results show that increasing the model size may improve the models’ ability to handle negations. |
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation (2022.aacl-main)
Copied to clipboard
| Challenge: | Negation is an important linguistic phenomenon which denotes non-existence, denial, or contradiction. |
| Approach: | They propose a natural language inference test suite to test models for negation . they use a linguistic framework to analyze negation types and constructions . |
| Outcome: | The proposed test suite is more challenging than existing benchmarks on negation . it includes annotation of negation types and constructions grounded in linguistic theory . |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
Negation, Coordination, and Quantifiers in Contextualized Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work has focused on specific tasks and on the learning outcome. |
| Approach: | They propose to decouple the weaknesses from specific tasks and focus on the embeddings per se and their mode of learning. |
| Outcome: | The proposed model can learn semantic constraints and how the context impacts their embeddings. |
Thunder-NUBench: A Benchmark for LLMs’ Sentence-Level Negation Understanding (2026.findings-eacl)
Copied to clipboard
| Challenge: | Negation is a fundamental linguistic phenomenon that poses ongoing challenges for Large Language Models (LLMs) Current benchmarks treat negation as a minor detail within broader tasks, such as natural language inference. |
| Approach: | They propose a novel benchmark specifically created to assess sentence-level understanding of negation in Large Language Models (LLMs). |
| Outcome: | The proposed benchmark compares standard negation with structurally diverse alternatives, such as local negation, contradiction, and paraphrase. |
Developmental Negation Processing in Transformer Language Models (2022.acl-short)
Copied to clipboard
| Challenge: | Negation is an important construct in language for reasoning over the truth of propositions, garnering interest from philosophy (Horn, 1989) and psycholinguistics (Zwaan, 2012). |
| Approach: | They propose to frame a natural language inference task as a problem and examine how well transformers can process negation categories. |
| Outcome: | The proposed models perform better on certain categories, suggesting clear differences in how they are processed. |
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
An Analysis of Natural Language Inference Benchmarks through the Lens of Negation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language inference ignore negations and can make inferences that are difficult to make. |
| Approach: | They propose a new benchmark for natural language inference in which negation plays a critical role. |
| Outcome: | The proposed benchmarks show that negation plays a critical role in inference judgments. |