Does BERT Recognize an Agent? Modeling Dowty’s Proto-Roles with Contextual Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | Contextual embeddings build multidimensional representations of word tokens based on their context of occurrence. |
| Approach: | They propose to map the verb embeddings to an interpretable space of semantic properties built from a linguistic dataset and test their ability to model the semantic properties of the agent of the verbs participating in the alternation. |
| Outcome: | The proposed models can model the semantic properties of the verbs participating in the so-called causative alternation. |
Similar Papers
Using Paraphrases to Study Properties of Contextual Embeddings (2022.naacl-main)
Copied to clipboard
| Challenge: | Previously, paraphrases have been used to probe whether compositionality is accurately captured by BERT, but we believe they can be used to explore many other questions. |
| Approach: | They propose to use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT. |
| Outcome: | The proposed analysis of paraphrases and paraphrase representations using the Paraphrase Database shows that BERT handles polysemous words, but different representations in many cases. |
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself. |
| Approach: | They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces. |
| Outcome: | The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks. |
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Contextualized word embeddings can incorporate contextual information, whereas other embeddables cannot. |
| Approach: | They propose an approach to address this question using Representational Similarity Analysis (RSA) they investigate whether verb embeddings encode verb’s subject, pronoun embedds antecedent and full-sentence representations encode sentence’s head word . |
| Outcome: | The proposed approach can adjudicate between hypotheses about which aspects of context are encoded in representations of language. |
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation. |
| Approach: | They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. |
| Outcome: | The results show that the representations of all words are not isotropic in any layer of the contextualizing model. |
Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020.acl-main)
Copied to clipboard
| Challenge: | a suite of probing tasks test contextual embeddings for encoding of information about surrounding words . authors: little is known about what information embeddables encode about the context words encode . a recent study shows that contextual embeds can be powerful for many tasks . |
| Approach: | They propose probing tasks that enable fine-grained testing of contextual embeddings . they examine popular contextual encoders and find that each encodes contextual information across tokens a little different . |
| Outcome: | The proposed probing tasks show that word embeddings encode information about words . the tests show that the encoded information is encoded across tokens with near-perfect recoverability . |
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task (2022.findings-acl)
Copied to clipboard
| Challenge: | lexically-independent subject-verb number agreement (NA) is performed by transformer-based neural language models (NLMs) . but when as little as one attractor is present, the model fails to perform lexical generalization . |
| Approach: | They propose to disrupt lexical patterns found in naturally occurring stimuli for each targeted structure in a novel fine-grained analysis of BERT's behavior. |
| Outcome: | The proposed model generalizes well for simple templates, but fails to perform lexically-independent generalization when as little as one attractor is present. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations (2020.coling-main)
Copied to clipboard
| Challenge: | a recent study has shown that metonymy is a productive and systematic process . linguistic and psycholinguistic studies support the idea that metnomic interpretations are based on lexical ambiguity . |
| Approach: | They compare BERT to a generalized event knowledge model to capture the meaning shift associated with metonymy. |
| Outcome: | The proposed model is good at predicting the meaning of metonymic expressions, the authors say . they show that the model can capture the meaning shift associated with metonymy . |
BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? (2021.acl-long)
Copied to clipboard
| Challenge: | Analogies play a central role in human commonsense reasoning. |
| Approach: | They analyze the capabilities of transformer-based language models on an unsupervised task . they find off-the-shelf language models can identify analogies to a certain extent . |
| Outcome: | The proposed language models outperform word embedding models on an unsupervised task . the best results were obtained with GPT-2 and RoBERTa . |
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (D19-60)
Copied to clipboard
| Challenge: | Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of their capabilities. |
| Approach: | They investigate BERT's ability to encode various commonsense features in its embedding space, but are still deficient in many areas. |
| Outcome: | The proposed model improves performance on a downstream commonsense reasoning task while using minimal data. |