Are Language Models Sensitive to Semantic Attraction? A Study on Surprisal (2023.starsem-1)
Copied to clipboard
| Challenge: | In psycholinguistics, semantic attraction is a sentence processing phenomenon in which a given argument violates the selectional requirements of a verb but is not perceived by comprehenders due to its attraction to another noun in the same sentence. |
| Approach: | They used autoregressive language models to compute the sentence-level and target phrase-level Surprisal scores of a psycholinguistic dataset on semantic attraction. |
| Outcome: | The proposed models are sensitive to semantic attraction, leading to reduced Surprisal scores, although none perfectly matches the human behavioral pattern. |
Similar Papers
Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study examined how large language models handle interactions in meaning across words and larger syntactic forms. |
| Approach: | They propose to use a dataset to examine the linguistic properties of optionally transitive English verbs to examine their agentivity. |
| Outcome: | The proposed model outperforms all other models in the evaluation dataset . the results are better correlated with human judgements than syntactic and semantic corpus statistics . |
Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models (2024.starsem-1)
Copied to clipboard
| Challenge: | Polysemes are words that can have different senses depending on context . traditionally, NLP models assume that each sense should be given a separate representation in a lexicon, thus limiting the amount of evidence that can be gained from their use. |
| Approach: | They propose a framework to model polysemes as a continuous variation in psycholinguistic properties of a word in context without postulating jumps between senses. |
| Outcome: | The proposed framework accommodates different sense interpretations, without postulating clear-cut jumps between senses. |
Automatic Learning of Modality Exclusivity Norms with Crosslingual Word Embeddings (2020.starsem-1)
Copied to clipboard
| Challenge: | Normative studies on modality for English words are relatively common . however, they are limited to a relatively small number of languages and require costly ratings. |
| Approach: | They aim to learn a mapping between word embeddings and modality norms by training on a high-resource language and testing on . monolingual and crosslingual word embeds are used to predict modality association scores . |
| Outcome: | The proposed model predicts modality associations even when trained on an English resource and tested on a completely unseen language. |
Disambiguating Emotional Connotations of Words Using Contextualized Word Representations (2024.starsem-1)
Copied to clipboard
| Challenge: | BERT, RoBERTa, XLNet, and GPT-2 models effectively discern emotional connotations of words, demonstrating superior performance and greater resilience against biases. |
| Approach: | They propose to use contextualized word representations to examine how words can be used to distinguish emotional connotations across contexts. |
| Outcome: | The proposed models show that they can distinguish emotional connotations of words in different contexts. |
Assessing Polyseme Sense Similarity through Co-predication Acceptability and Contextualised Embedding Distance (2020.starsem-1)
Copied to clipboard
| Challenge: | Co-predication is a commonly used linguistic test to tell apart shifts in polysemic sense from changes in homonymic meaning. |
| Approach: | They examine how co-predication acceptability relates to explicit ratings of polyseme word sense similarity and how well they can be predicted through the distance between target words’ contextualised word embeddings. |
| Outcome: | The proposed measures can be predicted through the distance between target words’ contextualised word embeddings. |
Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021.starsem-1)
Copied to clipboard
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache
| Challenge: | Prior work has explored the ability of computational models to predict word semantic fit with a given predicate. |
| Approach: | They compare Transformers Language Models to SDM to assess their performance . they found that TLMs do not capture important aspects of event knowledge . people can discriminate between typical and atypical events, they say . |
| Outcome: | The proposed models can achieve comparable performance to SDM, but they lack important aspects of event knowledge. |
When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent work using word embeddings to model semantic categorization has shown that static models outperform contextual models. |
| Approach: | They consider polysemy as a possible confounding factor in categorization decisions . they compare sense-level embeddings with previously studied static embedds . |
| Outcome: | The proposed model outperforms static models on coarse- and fine-grained categorization tasks. |
What Drives the Use of Metaphorical Language? Negative Insights from Abstractness, Affect, Discourse Coherence and Contextualized Word Representations (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing models of discourse coherence, abstractness and affect are not sufficient to explain metaphorical vs. literal language choices. |
| Approach: | They build five models that predict the use of a metaphorical vs. literal expression in context . they use established cognitive and linguistic properties to define discourse coherence, abstractness and affect . |
| Outcome: | The proposed models show that they do not explain metaphorical vs. literal language choices . metaphors are "not just nice", but represent a "necessary" element of everyday thought and communication . |
Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)
Copied to clipboard
| Challenge: | Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively. |
| Approach: | They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models. |
| Outcome: | The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation. |
Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning. |
| Approach: | They used a script-based story to manipulate event B in a story which causally depends on a previous event A. |
| Outcome: | The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition. |