| Challenge: | Static word embeddings that represent words by a single vector cannot capture word meaning in different linguistic and extralinguistic contexts. |
| Approach: | They propose dynamic contextualized word embeddings that represent words as a function of linguistic and extralinguistic contexts. |
| Outcome: | The proposed model models time and social space jointly, making them attractive for NLP tasks involving semantic variability. |
Similar Papers
Analyzing the Surprising Variability in Word Embedding Stability Across Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are powerful representations that form the foundation of many natural language processing architectures. |
| Approach: | They explore word embedding stability in a wide range of languages to gain insight into their stability. |
| Outcome: | The proposed results provide insights into word embedding stability in English and other languages. |
What Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition (D19-1)
Copied to clipboard
| Challenge: | Contextualized word embeddings have boosted many NLP tasks compared with static word embeds. |
| Approach: | They propose a framework that can explain word meanings given contextualized word embeddings for better interpretation. |
| Outcome: | The proposed framework can explain word meanings given contextualized word embeddings for better interpretation. |
Contextual String Embeddings for Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | Recent advances in language modeling have made it viable to model language as distributions over characters. |
| Approach: | They propose to leverage internal states of a trained character language model to produce a new type of word embeddings. |
| Outcome: | The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks. |
Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that contextualized models generate dynamic embeddings for words in context, but static embedds are often overlooked in this trend towards contextualized modeling. |
| Approach: | They propose a method that learns a transformation through static anchors and requires only another pre-trained model. |
| Outcome: | The proposed method improves a range of benchmark tasks that test contextual variations of meaning across different usages of a word and across different words as they are used in context. |
KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus (L18-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings are representations of words across languages in a shared continuous vector space. |
| Approach: | They propose a multilingual word embedding corpus which is acquired by neural machine translation and is based on monolingual data. |
| Outcome: | The proposed method is competitive with existing methods but on the cross-lingual document classification task, it obtains the best figures. |
Exploring the Value of Personalized Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | a subset of words belonging to specific psycholinguistic categories vary more in their representations across users . combining generic and personalized word embeddings yields the best performance . |
| Approach: | They propose personalized word embeddings and compare their performance to generic ones . they show that personalized word representations can be leveraged for improved performance . |
| Outcome: | The proposed model can be used for authorship attribution. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
Combining Static and Contextualised Multilingual Embeddings (2022.findings-acl)
Copied to clipboard
| Challenge: | Static embeddings are less expressive than contextual language models, but can be more straightforwardly aligned across multiple languages. |
| Approach: | They extract static embeddings for 40 languages from XLM-R and validate them with cross-lingual word retrieval and then align them using VecMap. |
| Outcome: | The proposed approach improves multilingual representations by leveraging static embeddings and a pre-training code. |