A Psycholinguistic Analysis of BERT’s Representations of Compounds (2023.eacl-main)
Copied to clipboard
| Challenge: | 'compound' semantic representations are based on the semantics of constituent words, and are lexical items like any other word. |
| Approach: | They leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD) and semantic transparency (ST). |
| Outcome: | The proposed representations are based on a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD) and semantic transparency (ST). |
Similar Papers
A Systematic Search for Compound Semantics in Pretrained BERT Architectures (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing models for noun compounds have been less successful in predicting compositionality than transformers . authors: suboptimal use of encoded information may be a contributing factor . performance of transformer-based models is poor, authors say . |
| Approach: | They propose to use semantic knowledge derived from pretrained BERT to predict compositionality . they find distinct linguistic roles of heads and modifiers are reflected by differences in BERT representations . |
| Outcome: | The proposed model improves on unsupervised implementations of pretrained BERT . empirical properties such as frequency, productivity, and ambiguity affect performance . |
Modeling the Evolution of English Noun Compounds with Feature-Rich Diachronic Compositionality Prediction (2025.acl-long)
Copied to clipboard
| Challenge: | Empirical research directly addressing these issues is limited to a small number of studies suggesting that compounding is a highly productive process. |
| Approach: | They represent English noun compounds as vectors of time-specific values and implement a set of features to classify them for present-day compositionality and assess the informativeness of the corresponding linguistic patterns. |
| Outcome: | The proposed method captures relevant and complementary information across approaches and shows that low-compositional meanings are reflected by a parallel drop in compositionality and sustained semantic change. |
What Does BERT Learn about the Structure of Language? (P19-1)
Copied to clipboard
| Challenge: | BERT is a language representation model that has performed well in diverse language understanding benchmarks. |
| Approach: | They perform experiments to unpack the elements of English language structure learned by BERT. |
| Outcome: | The proposed model outperforms state-of-the-art models in the GLUE benchmark by a significant margin. |
Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds (2024.acl-long)
Copied to clipboard
| Challenge: | Noun-noun compounds represent an important challenge for Natural Language Understanding . correct interpretation of noun-nomin compounds is essential for many applications . |
| Approach: | They test whether Large Language Models can interpret the semantic relation between nouns . they also test whether they can abstract from such knowledge to predict the relation . |
| Outcome: | The proposed models can interpret the semantic relation between nouns and compounds using analogical comparisons. |
Using Collostructional Analysis to evaluate BERT’s representation of linguistic constructions (2023.findings-acl)
Copied to clipboard
| Challenge: | Collostructional analysis is a technique devised to find correlations between words and linguistic constructions in order to analyse meaning associations of these constructions. |
| Approach: | They propose to compare English BERT’s meaning representations to known constructions from the linguistics literature by predicting words that can be used in open slots of constructions and finding similar sequences using BERT's output embeddings. |
| Outcome: | The proposed methods show that English BERT represents constructional meaning to a certain extent, but does not separate instances of a construction from a near-synonymous construction that has a different form. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction (2022.naacl-srw)
Copied to clipboard
| Challenge: | Among studies on localization of linguistic knowledge, it is unclear what information is encoded in each layer. |
| Approach: | They analyze BERT’s layer-wise masked word prediction on an English corpus and find syntactic and semantic information is encoded at different layers for words of different syntaktic categories. |
| Outcome: | The proposed model outperforms state-of-the-art models in many downstream tasks. |
From BERT‘s Point of View: Revealing the Prevailing Contextual Differences (2022.findings-acl)
Copied to clipboard
| Challenge: | BERTology is a new approach to understanding the inner workings of large pretraining language models. |
| Approach: | They propose to invert the probing design to analyze the prevailing differences and clusters in BERT’s high dimensional space by extracting coarse features from masked token representations and predicting them by probing models with access to only partial information. |
| Outcome: | The proposed method extracts coarse features from masked token representations and predicts them by probing models with access to only partial information. |
Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations (2020.coling-main)
Copied to clipboard
| Challenge: | a recent study has shown that metonymy is a productive and systematic process . linguistic and psycholinguistic studies support the idea that metnomic interpretations are based on lexical ambiguity . |
| Approach: | They compare BERT to a generalized event knowledge model to capture the meaning shift associated with metonymy. |
| Outcome: | The proposed model is good at predicting the meaning of metonymic expressions, the authors say . they show that the model can capture the meaning shift associated with metonymy . |
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task (2022.findings-acl)
Copied to clipboard
| Challenge: | lexically-independent subject-verb number agreement (NA) is performed by transformer-based neural language models (NLMs) . but when as little as one attractor is present, the model fails to perform lexical generalization . |
| Approach: | They propose to disrupt lexical patterns found in naturally occurring stimuli for each targeted structure in a novel fine-grained analysis of BERT's behavior. |
| Outcome: | The proposed model generalizes well for simple templates, but fails to perform lexically-independent generalization when as little as one attractor is present. |