Self-supervised Post-processing Method to Enrich Pretrained Word Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods that use external resources to make word vectors specialize depend on the lexicon. |
| Approach: | They propose a self-supervised extension of extrofitting by its own word vector distribution. |
| Outcome: | The proposed method improves word similarity embeddings on similarity tasks without external resources. |
Similar Papers
Enhancing Word Embeddings with Knowledge Extracted from Lexical Resources (2020.acl-srw)
Copied to clipboard
| Challenge: | In this paper, we present an effective method for semantic specialization of word vector representations. |
| Approach: | They propose a method for semantic specialization of word vector representations using BabelNet. |
| Outcome: | The proposed method improves on word similarity and dialog state tracking tasks. |
Explicit Retrofitting of Distributional Word Vectors (P18-1)
Copied to clipboard
| Challenge: | Existing models for word vector specialization focus on word co-occurrences from large text corpora, resulting in a tendency to fuse semantic similarity with other types of semantic relatedness. |
| Approach: | They propose to transform external lexico-semantic relations into training examples and learn an explicit retrofitting model. |
| Outcome: | The proposed model can specialize vector spaces of new languages and translate them to other languages. |
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Contextualized embeddings are expensive and resource-demanding, hence environmentally unfriendly. |
| Approach: | They propose a method to convert contextualized embeddings from pre-trained models into static embeddables using synonym knowledge and weighted vector distribution. |
| Outcome: | The proposed method outperforms baseline embeddings by a large margin through extrinsic and intrinsic tasks. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Post-Specialisation: Retrofitting Vectors of Words Unseen in Lexical Resources (N18-1)
Copied to clipboard
| Challenge: | Word vector specialisation is a portable, light-weight approach to fine-tuning distributional word vector spaces by injecting external knowledge from rich lexical resources such as WordNet. |
| Approach: | They propose a constraint-driven vector space specialisation method that embeds external knowledge into lexical resources into a deep neural network to specialise unseen words. |
| Outcome: | The proposed method preserves useful linguistic knowledge for seen words while propagating external signal to unseen words to improve their vector representations. |
Semantic Specialization of Distributional Word Vectors (D19-2)
Copied to clipboard
| Challenge: | Distributional word vectors conflate various paradigmatic and syntagmatic lexico-semantic relations. |
| Approach: | This tutorial provides an overview of specialization methods for distributional word vectors . a common solution is to include external lexico-semantic knowledge in a reshaped vector space . |
| Outcome: | This paper provides an overview of specialization methods for distributional word vectors . the most recent developments include a new method for asymmetric relations in Euclidean . |
A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Existing word embedding models mix semantic similarity with other types of relatedness. |
| Approach: | They propose a model that leverages relational knowledge available in a knowledge resource to improve word embeddings. |
| Outcome: | The proposed model improves word embeddings on synonymy, antonymy and hypernymy relations in WordNet and significantly improves lexical entailment detection task. |
Adversarial Propagation and Zero-Shot Cross-Lingual Transfer of Word Vector Specialization (D18-1)
Copied to clipboard
| Challenge: | Semantic specialization is a process of fine-tuning pre-trained distributional word vectors using external lexical knowledge to accentuate a particular semantic relation in the specialized vector space. |
| Approach: | They propose a method for specializing distributional word vectors using external lexical knowledge. |
| Outcome: | The proposed method improves on word similarity, dialog state tracking, and lexical simplification across three languages and on three tasks. |
Self-supervised Rewiring of Pre-trained Speech Encoders:
Towards Faster Fine-tuning with Less Labels in Speech Processing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech encoders have facilitated great success across various speech processing tasks, but fine-tuning them for downstream tasks requires large training data to converge or to achieve state-of-the-art. |
| Approach: | They propose to rewire pre-trained speech encoders to improve their representation space without task-specific labels by neutrally synthesising audio inputs and frame masking. |
| Outcome: | The proposed model shows consistent improvement in isotropy in the representation space on 6 speech processing tasks. |
What just happened? Evaluating retrofitted distributional word vectors (N19-1)
Copied to clipboard
| Challenge: | Recent work has attempted to enhance vector space representations using information from structured semantic resources. |
| Approach: | They propose a root-mean-square error evaluation metric to evaluate the utility of different lexical resources for retrofitting. |
| Outcome: | The proposed method improves word similarity performance by using root-mean-square error (RMSE) and root-macro-error (RMME) metric. |