Explaining Contextualization in Language Models using Visual Analytics (2021.acl-long)
Copied to clipboard
| Challenge: | Contextualized language models (LMs) have learned highly transferable and task-agnostic properties of language, even to a degree of imitating the classical NLP pipeline. |
| Approach: | They propose to use an existing similarity-based score to measure contextualization and integrate it into a visual analytics technique that combines the model's layers simultaneously and highlighting intra-layer properties and inter-layer differences. |
| Outcome: | The proposed approach combines linguistically-informed insights with scoring and visual analytics to show that contextualization is neither driven by polysemy nor by pure context variation. |
Similar Papers
Quantifying the Contextualization of Word Representations with Semantic Class Probing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models are effective in solving NLP tasks, but there are still questions about how and why they work so well. |
| Approach: | They use BERT to quantify contextualization by studying the extent of inference . they show that top layer representations support highly accurate inference of semantic classes . |
| Outcome: | The proposed model is highly accurate, but weak in the lower layers . it is more task-specific after finetuning while lower layers are more transferable . |
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation. |
| Approach: | They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. |
| Outcome: | The results show that the representations of all words are not isotropic in any layer of the contextualizing model. |
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Contextualized word embeddings can incorporate contextual information, whereas other embeddables cannot. |
| Approach: | They propose an approach to address this question using Representational Similarity Analysis (RSA) they investigate whether verb embeddings encode verb’s subject, pronoun embedds antecedent and full-sentence representations encode sentence’s head word . |
| Outcome: | The proposed approach can adjudicate between hypotheses about which aspects of context are encoded in representations of language. |
Language Models and Semantic Relations: A Dual Relationship (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on language models for the extraction of semantic relations have focused on injecting semantic knowledge into these models to enhance them. |
| Approach: | They propose to extract lexical semantic relations from a BERT model and inject them into it using unsupervised methods based on semantic similarity at word and sentence levels. |
| Outcome: | The proposed method allows to enrich a BERT model without using any external semantic resource. |
Negation, Coordination, and Quantifiers in Contextualized Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work has focused on specific tasks and on the learning outcome. |
| Approach: | They propose to decouple the weaknesses from specific tasks and focus on the embeddings per se and their mode of learning. |
| Outcome: | The proposed model can learn semantic constraints and how the context impacts their embeddings. |
Analysing Lexical Semantic Change with Contextualised Word Representations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies on lexical semantic change have focused on detecting and characterising word meaning shifts using distributional semantic models. |
| Approach: | They propose a method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. |
| Outcome: | The proposed method captures a variety of synchronic and diachronic linguistic phenomena and is highly reproducible and reproducible. |
Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts (2022.coling-1)
Copied to clipboard
Maria Lymperaiou, George Manoliadis, Orfeas Menis Mastromichalakis, Edmund G. Dervakos, Giorgos Stamou
| Challenge: | Recent advances in NLP research have focused on robustness and explainability issues of their evaluation strategies. |
| Approach: | They propose to use pre-trained transformers to evaluate semantic similarity for visual vocabularies . they propose to provide explainable metrics for understanding the quality of retrieved instances . |
| Outcome: | The proposed metrics highlight inabilities of widely used evaluation methods and highlight weaknesses in learned linguistic representations. |
Measuring Context-Word Biases in Lexical Semantic Datasets (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pretrained contextualized models have been used to evaluate word-in-context representations in many lexical semantic tasks. |
| Approach: | They propose to quantify the degree of context or word biases in existing datasets by probing masked input. |
| Outcome: | The proposed model performs better when both word and context are available than with masked input. |
From BERT‘s Point of View: Revealing the Prevailing Contextual Differences (2022.findings-acl)
Copied to clipboard
| Challenge: | BERTology is a new approach to understanding the inner workings of large pretraining language models. |
| Approach: | They propose to invert the probing design to analyze the prevailing differences and clusters in BERT’s high dimensional space by extracting coarse features from masked token representations and predicting them by probing models with access to only partial information. |
| Outcome: | The proposed method extracts coarse features from masked token representations and predicts them by probing models with access to only partial information. |
Which *BERT? A Survey Organizing Contextualized Encoders (2020.emnlp-main)
Copied to clipboard
| Challenge: | a survey on language representation learning aims to highlight common themes . we focus on the areas of progress, compared to other fields, and discuss how each area is evaluated. |
| Approach: | They present a survey on language representation learning to highlight common themes . they compare contributions in contextualized text encoders to ideas from other fields . |
| Outcome: | The proposed survey aims to highlight common themes in the field of language representation learning. |