Towards Incremental Learning of Word Embeddings Using Context Informativeness (P19-2)
Copied to clipboard
| Challenge: | In this paper, we investigate the task of learning word embeddings from very sparse data in an incremental, cognitively-plausible way. |
| Approach: | They propose a model that incorporates informativeness into a proposed model of nonce learning, using it for context selection and learning rate modulation. |
| Outcome: | The proposed model is based on a proposed model of nonce learning, and it performs well on the task of learning new words from definitions and potentially uninformative contexts. |
Similar Papers
Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts (N19-1)
Copied to clipboard
| Challenge: | Mimicking has been proposed as a solution to learning high-quality embeddings for rare words because of sparse context information. |
| Approach: | They propose a method to reproduce embeddings of frequent words from their surface form and then use it to compute embedds for rare words. |
| Outcome: | The proposed model outperforms previous work on rare and medium-frequency words. |
Interpretable Word Embeddings via Informative Priors (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings lack interpretability and are unsupervised . this limitation limits their use within computational social science and digital humanities. |
| Approach: | They propose to use informative priors to create interpretable dimensions for probabilistic word embeddings using a priori model. |
| Outcome: | The proposed models capture latent semantic concepts better than or on-par with the current state of the art while maintaining the simplicity and generalizability of priors. |
Learned Incremental Representations for Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | a new syntactic representation that commits to syntakic choices is proposed for humans . we use a system that uses only incremental processing of a prefix to predict the word in a sentence . |
| Approach: | They propose a syntactic representation that commits to syntakic choices incrementally . they say the system can achieve 93.72 F1 on the Penn Treebank with as few as 5 bits per word . |
| Outcome: | The proposed representation achieves 93.72 F1 on the Penn Treebank with as few as 5 bits per word . the analysis of the representations shows they have interpretable features and deferred resolution of syntactic ambiguities. |
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |
Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing word embedding methods for natural language processing are limited in their ability to produce dense word embeds. |
| Approach: | They propose a word embedding SentiVec which is infused with sentiment information from a lexical resource and outperforms baselines on subjectivity-sensitive tasks. |
| Outcome: | The proposed word embedding SentiVec outperforms baselines on subjectivity-sensitive tasks. |
Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that contextualized models generate dynamic embeddings for words in context, but static embedds are often overlooked in this trend towards contextualized modeling. |
| Approach: | They propose a method that learns a transformation through static anchors and requires only another pre-trained model. |
| Outcome: | The proposed method improves a range of benchmark tasks that test contextual variations of meaning across different usages of a word and across different words as they are used in context. |
Word2Sense: Sparse Interpretable Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Word2Sense embeddings are interpretable, but they are sparse and fast to compute . a unitary rotation can be applied to many of these embeddables retaining their utility for computational tasks while changing the values of individual coordinates. |
| Approach: | They propose an unsupervised method to generate Word2Sense word embeddings that are interpretable. |
| Outcome: | The proposed method compares well with other unsupervised word embeddings on NLP tasks. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
Contextual String Embeddings for Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | Recent advances in language modeling have made it viable to model language as distributions over characters. |
| Approach: | They propose to leverage internal states of a trained character language model to produce a new type of word embeddings. |
| Outcome: | The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks. |