Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)
Copied to clipboard
| Challenge: | Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively. |
| Approach: | They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models. |
| Outcome: | The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation. |
Similar Papers
Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study examined how large language models handle interactions in meaning across words and larger syntactic forms. |
| Approach: | They propose to use a dataset to examine the linguistic properties of optionally transitive English verbs to examine their agentivity. |
| Outcome: | The proposed model outperforms all other models in the evaluation dataset . the results are better correlated with human judgements than syntactic and semantic corpus statistics . |
Learning Negation Scope from Syntactic Structure (2020.starsem-1)
Copied to clipboard
| Challenge: | a semi-supervised model learns the semantics of negation purely through syntactic analysis . Negation is a semantic phenomenon in natural language which varies significantly . |
| Approach: | They propose a semi-supervised model which learns negation semantics purely through syntactic analysis. |
| Outcome: | The proposed model achieves state-of-the-art on a Negation Scope Detection task without identifying individual words or extracting features beyond syntax. |
Limits for learning with language models (2023.starsem-1)
Copied to clipboard
| Challenge: | Recent studies show that large language models fail to capture important aspects of linguistic meaning . authors argue that LLMs cannot learn fundamental semantic properties defined in formal semantics . |
| Approach: | They propose a theoretical explanation for some of the observed failings of large language models . they show that LLMs cannot learn certain fundamental semantic properties . |
| Outcome: | The proposed model fails to learn semantic entailment and consistency as defined in formal semantics, the authors argue . their model fails on tasks that require engorgements and deep linguistic understanding, they argue - but not on universal quantification. |
Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning. |
| Approach: | They used a script-based story to manipulate event B in a story which causally depends on a previous event A. |
| Outcome: | The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition. |
LEXPLAIN: Improving Model Explanations via Lexicon Supervision (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing methods that extract features from input text to explain a classifier's prediction are limiting to models that are faithful to their predictions. |
| Approach: | They propose a framework for guiding model explanations by supervising them explicitly using task-related lexicons to direct supervise model explanation. |
| Outcome: | The proposed method improves model explanations without sacrificing performance on sentiment analysis and toxicity detection tasks while demoting spurious correlations with African American English dialects. |
Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021.starsem-1)
Copied to clipboard
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache
| Challenge: | Prior work has explored the ability of computational models to predict word semantic fit with a given predicate. |
| Approach: | They compare Transformers Language Models to SDM to assess their performance . they found that TLMs do not capture important aspects of event knowledge . people can discriminate between typical and atypical events, they say . |
| Outcome: | The proposed models can achieve comparable performance to SDM, but they lack important aspects of event knowledge. |
Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent attempt at language modeling with predicted semantic structure failed to establish empirical lower bounds on what could have made the attempt successful. |
| Approach: | They propose a concise binary vector representation of semantic structure at the lexical level and evaluate how good an incremental tagger needs to be to achieve better-than-baseline performance. |
| Outcome: | The proposed model can achieve better-than-baseline performance without losing its main advantages and lower bounds on prediction quality can't be established via a single score alone. |
MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection (2024.starsem-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic formalism that captures the core meaning of an utterance. |
| Approach: | They propose to use AMR to map meanings of 1,685 utterances to 50+ languages to build a dataset 20 times larger than existing resources. |
| Outcome: | The proposed dataset covers more languages, has more utterances, and has localized or translated entities for each language. |
Overcoming Poor Word Embeddings with Word Definitions (2021.starsem-1)
Copied to clipboard
| Challenge: | Modern natural language understanding models depend on pretrained word embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. |
| Approach: | They propose a method to improve a model's ability to learn to use definitions in natural text to overcome this handicap. |
| Outcome: | The proposed model learns to use definitions in natural text to overcome this handicap. |
Lexical Substitution as Causal Language Modeling (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for lexical substitution task lacks autoregressive decoding capabilities. |
| Approach: | They propose a framework that uses causal language modeling (CLM) for lexical substitution task. |
| Outcome: | The proposed system outperforms GeneSis, the best previously published supervised LST method. |