On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020.starsem-1)
Copied to clipboard
| Challenge: | Existing studies have found that BERT can correctly retrieve noun hypernyms in cloze tasks, but this does not correspond to systematic knowledge in BERT. |
| Approach: | They propose to use BERT to probe for hypernymy knowledge encoded in representations for cloze tasks to find out whether it is systematic or not . |
| Outcome: | The proposed model can retrieve hypernyms in cloze tasks, but not systematic knowledge in BERT. |
Similar Papers
Seeking Clozure: Robust Hypernym extraction from BERT with Anchored Prompts (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for extracting hypernym knowledge from large language models are unclear whether they fail due to a lack of knowledge or shortcomings. |
| Approach: | They propose to use pattern-based hypernym extraction as a diagnostic tool to examine hypernomy knowledge encoded in BERT. |
| Outcome: | The proposed method compares the results of two different methods on six English data sets and on challenge sets of rare and abstract concepts. |
Assessing Polyseme Sense Similarity through Co-predication Acceptability and Contextualised Embedding Distance (2020.starsem-1)
Copied to clipboard
| Challenge: | Co-predication is a commonly used linguistic test to tell apart shifts in polysemic sense from changes in homonymic meaning. |
| Approach: | They examine how co-predication acceptability relates to explicit ratings of polyseme word sense similarity and how well they can be predicted through the distance between target words’ contextualised word embeddings. |
| Outcome: | The proposed measures can be predicted through the distance between target words’ contextualised word embeddings. |
Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models (2024.starsem-1)
Copied to clipboard
| Challenge: | Polysemes are words that can have different senses depending on context . traditionally, NLP models assume that each sense should be given a separate representation in a lexicon, thus limiting the amount of evidence that can be gained from their use. |
| Approach: | They propose a framework to model polysemes as a continuous variation in psycholinguistic properties of a word in context without postulating jumps between senses. |
| Outcome: | The proposed framework accommodates different sense interpretations, without postulating clear-cut jumps between senses. |
Investigating Aspect Features in Contextualized Embeddings with Semantic Scales and Distributional Similarity (2024.starsem-1)
Copied to clipboard
| Challenge: | Aspect is a linguistic category describing how actions and events unfold over time. |
| Approach: | They propose to use semantic projections to examine whether the vector dimensions of annotated verbs reflect human linguistic distinctions. |
| Outcome: | The proposed models encode the aspects of stativity, durativity and telicity in most of their layers, while durativité is the most challenging feature. |
Fine-tuning BERT with Focus Words for Explanation Regeneration (2020.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to explain the correct answer in multiple-choice QA are low in F-scores and lack of performance. |
| Approach: | They introduce a lightweight focus feature in a transformer-based NLP model and examine performance improvements. |
| Outcome: | The proposed model achieves the highest scores, second only to a computationally intensive system. |
How Are Idioms Processed Inside Transformer Language Models? (2023.starsem-1)
Copied to clipboard
| Challenge: | idioms are prevalent in natural language, but how do they be processed? |
| Approach: | They analyze the embeddings of idiomatic and literal expressions across all layers of the networks at both the sentence and word levels. |
| Outcome: | The proposed models represent idioms distinctively compared to literal language, the study finds . |
Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021.starsem-1)
Copied to clipboard
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache
| Challenge: | Prior work has explored the ability of computational models to predict word semantic fit with a given predicate. |
| Approach: | They compare Transformers Language Models to SDM to assess their performance . they found that TLMs do not capture important aspects of event knowledge . people can discriminate between typical and atypical events, they say . |
| Outcome: | The proposed models can achieve comparable performance to SDM, but they lack important aspects of event knowledge. |
Disambiguating Emotional Connotations of Words Using Contextualized Word Representations (2024.starsem-1)
Copied to clipboard
| Challenge: | BERT, RoBERTa, XLNet, and GPT-2 models effectively discern emotional connotations of words, demonstrating superior performance and greater resilience against biases. |
| Approach: | They propose to use contextualized word representations to examine how words can be used to distinguish emotional connotations across contexts. |
| Outcome: | The proposed models show that they can distinguish emotional connotations of words in different contexts. |
Representation of Lexical Stylistic Features in Language Models’ Embedding Space (2023.starsem-1)
Copied to clipboard
| Challenge: | lexical stylistic notions such as complexity, formality, and figurativeness can be identified in pretrained Language Models . static embeddings encode these features more accurately at the level of words and phrases whereas contextualized LMs perform better on sentences. |
| Approach: | They propose to derive a vector representation for stylistic notions from seed pairs . they find that static embeddings encode stylistic features more accurately . |
| Outcome: | The proposed representations can be used to characterize new texts in terms of these dimensions using a small number of seed pairs. |
When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent work using word embeddings to model semantic categorization has shown that static models outperform contextual models. |
| Approach: | They consider polysemy as a possible confounding factor in categorization decisions . they compare sense-level embeddings with previously studied static embedds . |
| Outcome: | The proposed model outperforms static models on coarse- and fine-grained categorization tasks. |