| Challenge: | Unsupervised word embeddings have limitations to the semantics of words and inadequate fine-tuning of embedded word can lead to suboptimal performance. |
| Approach: | They propose a method that optimizes word embeddings by regularizing them incrementally to ensure they are tuned in an incremental way. |
| Outcome: | The proposed method improves performance on various NLP tasks and shows that it absorbs semantic information without "forging" |
Similar Papers
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
Delta-training: Simple Semi-Supervised Text Classification using Pretrained Word Embeddings (D19-1)
Copied to clipboard
| Challenge: | Pretrained word embeddings outperforms classifiers with randomly initialized word embeds, a new method is proposed for semi-supervised text classification. |
| Approach: | They propose a method that uses pretrained word embeddings to predict text classification . they use unlabeled data to build a classifier, and use early-stopping to improve performance . |
| Outcome: | The proposed method outperforms self-training and co-training frameworks on unlabeled data. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Improving Word Embeddings through Iterative Refinement of Word- and Character-level Models (2020.coling-main)
Copied to clipboard
| Challenge: | Embedding of rare and out-of-vocabulary words is an important open NLP problem . standard embedding models are not useful for recommending jobs to users with rare or unseen words . |
| Approach: | They propose to train a character-level neural network to reproduce word embeddings . they then use the model to assign vectors to any input string, including rare words . |
| Outcome: | The proposed method outperforms existing methods on word similarity data sets and can be applied to job title normalization in the e-recruitment domain. |
Word Embedding Binarization with Semantic Information Preservation (2020.coling-main)
Copied to clipboard
| Challenge: | Word and Sentence embeddings are one of the most common starting points of any NLP task. |
| Approach: | They propose a way to convert word embedding to binary representation to reduce overall size . they propose different approaches suitable for different downstream tasks based on contextual and semantic information. |
| Outcome: | The proposed method reduces the size of the embedding while keeping the semantic and relational knowledge intact. |
Self-Discriminative Learning for Unsupervised Document Embedding (N19-1)
Copied to clipboard
| Challenge: | Existing methods for document embedding learning do not consider inter-document relationships. |
| Approach: | They propose to exploit the inter-document information and directly model the relations of documents in embedding space with a discriminative network and a novel objective. |
| Outcome: | The proposed method has errors that are 5 to 13% lower than state-of-the-art models and is even more pronounced in scarce label setting. |
WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models perform well on learning sentence semantics when fine-tuned with supervised data. |
| Approach: | They conduct a thorough examination of pretrained model based unsupervised sentence embeddings. |
| Outcome: | The proposed approach improves on whitening-based vector normalization with less than 10 lines of code. |
Composition-contrastive Learning for Sentence Embeddings (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows potential to learn vector representations from unlabelled data without task-specific fine-tuning. |
| Approach: | They propose to maximize alignment between textual embeddings and a composition of their phrasal constituents. |
| Outcome: | The proposed approach improves on similarity tasks comparable to state-of-the-art approaches. |