| Challenge: | Recent work shows that better word representations can be obtained by concatenating different types of embeddings. |
| Approach: | They propose to automate the process of finding better concatenated embeddings for structured prediction tasks by concatending different types of embeddables. |
| Outcome: | The proposed approach outperforms baselines and achieves state-of-the-art with fine-tuned embeddings on 6 tasks and 21 datasets. |
Similar Papers
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |
Leveraging the Structure of Pre-trained Embeddings to Minimize Annotation Effort (2024.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for text classification are based on fine-tuning the representations computed by large language models. |
| Approach: | They propose to exploit structural properties of pre-trained embeddings to spread information . they use a semisupervised strategy to train models with minimal annotation effort . |
| Outcome: | The proposed method outperforms self-training and random walk labels on different datasets. |
Contextual String Embeddings for Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | Recent advances in language modeling have made it viable to model language as distributions over characters. |
| Approach: | They propose to leverage internal states of a trained character language model to produce a new type of word embeddings. |
| Outcome: | The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks. |
Autoencoding Improves Pre-trained Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work has shown that word embeddings are distributed in a narrow cone and that centering and projection can improve the accuracy of pre-trained word embeds without requiring additional training data. |
| Approach: | They propose to remove the top principal components from pre-trained word embeddings and center and project them onto principal component vectors to reinstate isotropy in the embeddable space. |
| Outcome: | The proposed method is equivalent to applying a linear autoencoder to minimize the squared L2 reconstruction error. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation (2024.naacl-short)
Copied to clipboard
| Challenge: | Continuous-output neural machine translation models are trained to predict the continuous representation based on distances between vectors. |
| Approach: | They propose a continuous-output neural machine translation (CoNMT) approach that uses random output embeddings to outperform laboriously pre-trained models. |
| Outcome: | The proposed strategy outperforms pre-trained embeddings on large datasets and is strongest for rare words due to the geometry of their embedders. |
Domain-Specific Word Embeddings with Structure Prediction (2023.tacl-1)
Copied to clipboard
| Challenge: | Current word embedding methods do not provide a way to use or predict information on structure between sub-corpora, time or domain. |
| Approach: | They propose a word embedding method that provides general word representations for the whole corpus, domain-specific representations and embeddable alignment simultaneously. |
| Outcome: | The proposed method provides better performance than baselines on a dataset of science and philosophy articles. |