Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been studied for their ability to store and utilize positive knowledge. |
| Approach: | They propose to use a constrained keywords-to-sentence generation task and a Boolean question answering task to probe large language models on negative commonsense knowledge. |
| Outcome: | The proposed tasks show that LLMs fail to generate valid sentences grounded in negative commonsense knowledge, yet they can correctly answer yes-or-no questions. |
Similar Papers
This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have grammatical knowledge but fail to interpret negation . a recent study shows that LLMs struggle with negative sentences . |
| Approach: | They propose to use a dataset to grasp LLMs' generalization and inference capability . they also fine-tuned models to assess whether the understanding of negation can be trained . |
| Outcome: | The proposed model is able to generalize and infer negation in 400,000 sentences . but it is suboptimal when it comes to negation, a key step in natural language processing . |
Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Negation is a common and important semantic feature in natural language, yet Large Language Models struggle when negation is involved in natural learning tasks. |
| Approach: | They propose to augment existing corpora with negation by automatically augmenting existing ones with negations by combining multiple triples with if-then relations. |
| Outcome: | The proposed approach yields two new corpora containing over 2M triples with if-then relations. |
Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. |
| Approach: | They conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. |
| Outcome: | The language used to query the LLMs can impact their performance on cultural-related tasks. |
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used for accessing information on the web. |
| Approach: | They conduct experiments with 80 crowdworkers to compare LLMs with search engines . they ask LLM to provide contrastive information to reduce over-reliance on LLM . |
| Outcome: | The results show that LLMs can outperform search engines but not LLM explanations . the study shows that LMS explanations are not reliable replacements for reading retrieved passages compared to search engines alone. |
NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in knowledge base construction techniques focus on the acquisition of positive (true) KB statements, but negative (false) statements are important for discriminative reasoning. |
| Approach: | They propose a framework that ranks potential negatives in commonsense KBs using a contextual language model. |
| Outcome: | The proposed framework ranks negatives in commonsense KBs using a language model . it yields positives that are more grammatical, coherent, and informative . |
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method (2024.naacl-long)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Recent literature reveals that Large Language Models (LLMs) hallucinate intermittently, which impedes their reliability for further utilization. |
| Approach: | They propose a self-detection method to detect which questions an LLM does not know by combining the two components to identify whether the model generates a non-factual response to the question. |
| Outcome: | The proposed method can detect which questions an LLM does not know across factoid question-answering, arithmetic reasoning, and commonsense reasoning tasks. |
Conservative Bias in Large Language Models: Measuring Relation Predictions (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit pronounced conservative bias in relation extraction tasks, often defaulting to no_relation label when an appropriate option is unavailable. |
| Approach: | They systematically evaluate the trade-off between conservative bias and hallucination in relation extraction tasks by using SBERT and LLM prompts to quantify this effect. |
| Outcome: | The proposed model defaults to no_relation label twice as often as hallucination, resulting in significant information loss when reasoning is not explicitly included in the output. |
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024.emnlp-main)
Copied to clipboard
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, Dan Roth
| Challenge: | a new hypothesis-testing framework is developed to assess whether large language models possess genuine reasoning abilities or primarily depend on token bias. |
| Approach: | They propose a framework to assess whether large language models have genuine reasoning abilities or primarily depend on token bias. |
| Outcome: | The proposed framework outlines a list of hypotheses where token biases are readily identifiable . the results suggest that most LLMs still struggle with logical reasoning . |
Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. |
| Approach: | They argue that LLMs only parrot statistical patterns in training data and that language learning in LLM cannot inform human language learning. |
| Outcome: | The proposed model can generate grammatically correct, fluent text without requiring human intervention. |
What Evidence Do Language Models Find Convincing? (2024.acl-long)
Copied to clipboard
| Challenge: | Current retrieval-augmented language models are tasked with subjective, contentious, and conflicting queries. |
| Approach: | They construct a dataset that pairs controversial queries with real-world evidence documents . they find current models rely heavily on relevance of a website to the query . |
| Outcome: | The proposed dataset pairs controversial queries with real-world evidence documents that contain different facts, arguments, and answers. |