| Challenge: | Existing approaches to embedding imputation use vector space properties or subword information to learn representations for rare or unseen words. |
| Approach: | They propose an online method to construct a knowledge graph from grounded information and an algorithm to map from the resulting graph to the space of the pre-trained embeddings. |
| Outcome: | The proposed method improves on a card-660 task by 11% and 17.8% respectively using GloVe embeddings. |
Similar Papers
Knowledge Graph Embedding with Atrous Convolution and Residual Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding methods are complex and require time for training and inference. |
| Approach: | They propose an atrous convolution based knowledge graph embedding method that increases feature interactions by using atrous . they evaluate method on six benchmark datasets with different evaluation metrics . |
| Outcome: | The proposed method achieves better results on six benchmark datasets than state-of-the-art methods on most evaluation metrics. |
Learning Word Embeddings for Low-Resource Languages by PU Learning (N18-1)
Copied to clipboard
| Challenge: | Existing approaches to learn word embedding on a corpus with only a few million tokens are limited to low-resource languages. |
| Approach: | They propose to use a sparse co-occurrence matrix to factorize the co-existence matrix and validate the proposed approaches in four different languages. |
| Outcome: | The proposed model is validated in four different languages. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Subword Attention and Post-Processing for Rare and Unknown Contextualized Embeddings (2024.findings-naacl)
Copied to clipboard
| Challenge: | Word embeddings are useful, but struggle on rare and unknown words. |
| Approach: | They propose a rare/unknown embedding architecture that focuses on contextualized representations. |
| Outcome: | The proposed architecture improves performance in most intrinsic and downstream tasks. |
An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have been shown to improve the performance of neural networks across a wide variety of tasks. |
| Approach: | They propose two new metrics to understand the downstream reliability of word embeddings. |
| Outcome: | The proposed model can improve performance with slight changes to the training data, but it can also fail with multiple neural network architectures. |
Embedding Words in Non-Vector Space with Unsupervised Graph Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | GraphGlove is an unsupervised graph word representations that are learned end-to-end. |
| Approach: | They propose a method to learn weighted graph word representations end-to-end using a weighteable weighte . they adopt a hierarchical graph representation method and modify the GloVe training algorithm to learn graph representations. |
| Outcome: | The proposed method outperforms vector-based methods on word similarity and analogy tasks. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
Distilling Relation Embeddings from Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture a surprisingly rich amount of lexical knowledge, but it is unclear to what extent relation embeddings can be used to encode relational knowledge. |
| Approach: | They found that word vector differences capture lexical relations . relationship embeddings can be used to encode relational knowledge . |
| Outcome: | The results are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. |
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Contextualized embeddings are expensive and resource-demanding, hence environmentally unfriendly. |
| Approach: | They propose a method to convert contextualized embeddings from pre-trained models into static embeddables using synonym knowledge and weighted vector distribution. |
| Outcome: | The proposed method outperforms baseline embeddings by a large margin through extrinsic and intrinsic tasks. |