| Challenge: | Word embeddings dominate overall model sizes in neural methods for natural language processing, especially when large vocabularies and high dimensions are used. |
| Approach: | They propose a Gumbel-Softmax distribution to maximize over the latent clustering while minimizing the task loss. |
| Outcome: | The proposed method minimizes the task loss while maximizing over the latent clustering while remaining parameter-efficient. |
Similar Papers
Tiny Word Embeddings Using Globally Informed Reconstruction (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for word embedding reconstruction use only local information of subwords and pre-trained word embeds. |
| Approach: | They propose a global loss function that uses words other than the target word to improve word embedding reconstruction by a factor of 200. |
| Outcome: | The proposed method reduces the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. |
Tensorized Embedding Layers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using the Tensor Train decomposition, embeddings layers occupy large portion of model weights, preventing their deployment in limited resource settings. |
| Approach: | They propose a method for parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. |
| Outcome: | The proposed method can be plugged into any model and trained end-to-end. |
Adaptive Compression of Word Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Distributed representations of words have been an indispensable component for natural language processing (NLP) tasks. |
| Approach: | They propose a method that uses a code-book approach to represent words as discrete codes such as (8, 5, 2, 4). |
| Outcome: | The proposed method makes the highly compressed word embeddings without hurting the task accuracy. |
Improving Text Embeddings with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages . |
| Approach: | They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. |
| Outcome: | The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data. |
Leveraging the Structure of Pre-trained Embeddings to Minimize Annotation Effort (2024.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for text classification are based on fine-tuning the representations computed by large language models. |
| Approach: | They propose to exploit structural properties of pre-trained embeddings to spread information . they use a semisupervised strategy to train models with minimal annotation effort . |
| Outcome: | The proposed method outperforms self-training and random walk labels on different datasets. |
Block-wise Word Embedding Compression Revisited: Better Weighting and Structuring (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for word embedding compression are limited . word embeds have a considerable size and need to be compressed to deploy on edge devices . |
| Approach: | They propose a block-wise low-rank approximation method for word embedding called GroupReduce . they propose 'frequency-inverse document frequency method' and a differentiable method for weighting . |
| Outcome: | The proposed algorithm more effectively finds word weights than competitors in most cases. |
Latent-Variable Generative Models for Data-Efficient Text Classification (D19-1)
Copied to clipboard
| Challenge: | Generative classifiers offer potential advantages over discriminative classifications, including data efficiency and zero-shot learning. |
| Approach: | They introduce discrete latent variables into generative story to improve classifiers' performance . they empirically characterize performance of their models on six text classification datasets . |
| Outcome: | The proposed model outperforms discriminative and generative classifiers on six text classification datasets. |
Text Classification with Few Examples using Controlled Generalization (N19-1)
Copied to clipboard
| Challenge: | Current training data for text classification is limited, resulting in limited generalization capacity. |
| Approach: | They propose a feed-forward network that can generalize from unlabeled parsed corpora to produce task-specific semantic vectors. |
| Outcome: | The proposed approach is especially effective in low-data scenarios compared to state-of-the-art methods. |
Reusing Weights in Subword-Aware Neural Language Models (N18-1)
Copied to clipboard
| Challenge: | a statistical language model assigns a probability to a sequence of words . data sparsity is a major problem in building traditional n-gram language models . |
| Approach: | They propose several ways to reuse subword embeddings and other weights in subword-aware neural language models. |
| Outcome: | The proposed techniques do not benefit a competitive character-aware model . but they show significant reductions in model sizes and performance. |
A Simple and Effective Usage of Word Clusters for CBOW Model (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing word clustering algorithms can be used to obtain word embeddings without additional language resources. |
| Approach: | They propose to replace infrequent input and output words with clusters to produce word embeddings. |
| Outcome: | The proposed method produces embeddings of frequent words and small amount of cluster embeddables, which can be fine-tuned on downstream tasks. |