| Challenge: | Using the Tensor Train decomposition, embeddings layers occupy large portion of model weights, preventing their deployment in limited resource settings. |
| Approach: | They propose a method for parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. |
| Outcome: | The proposed method can be plugged into any model and trained end-to-end. |
Similar Papers
Smaller Text Classifiers with Discriminative Cluster Embeddings (N18-2)
Copied to clipboard
| Challenge: | Word embeddings dominate overall model sizes in neural methods for natural language processing, especially when large vocabularies and high dimensions are used. |
| Approach: | They propose a Gumbel-Softmax distribution to maximize over the latent clustering while minimizing the task loss. |
| Outcome: | The proposed method minimizes the task loss while maximizing over the latent clustering while remaining parameter-efficient. |
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word-embeddings are vital components of natural language processing (NLP) but they consume a lot of memory which poses a challenge for edge deployment. |
| Approach: | They propose an embedding compression method based on matrix decomposition and knowledge distillation that initializes weights of pre-trained word-embeddings and fine-tunes end-to-end. |
| Outcome: | The proposed method has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune methods. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Block-wise Word Embedding Compression Revisited: Better Weighting and Structuring (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for word embedding compression are limited . word embeds have a considerable size and need to be compressed to deploy on edge devices . |
| Approach: | They propose a block-wise low-rank approximation method for word embedding called GroupReduce . they propose 'frequency-inverse document frequency method' and a differentiable method for weighting . |
| Outcome: | The proposed algorithm more effectively finds word weights than competitors in most cases. |
Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | Using a new architecture, alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning. |
| Approach: | They propose a new architecture where alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning. |
| Outcome: | The proposed architecture achieves competitive performance on three popular benchmarks, SNLI, MultiNLI and SciTail, while maintaining lightweight parameter size. |
Do We Really Need All Those Dimensions? An Intrinsic Evaluation Framework for Compressed Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for compressed text embeddings are either expensive or too simplistic. |
| Approach: | They propose a task-agnostic intrinsic evaluation framework that provides a reliable proxy for downstream performance. |
| Outcome: | The proposed framework provides a reliable proxy for downstream performance. |
EmByte: Decomposition and Compression Learning for Small yet Private NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EMBYTE is a byte-level tokenization model that reduces embedding parameters by up to 94% . it is also resilient to privacy threats such as gradient inversion attacks . |
| Approach: | EMBYTE is a byte-level tokenization model that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. |
| Outcome: | EMBYTE achieves substantial embedding compression while preserving accuracy and enhancing privacy. |
EmbedTextNet: Dimension Reduction with Weighted Reconstruction and Correlation Losses for Efficient Text Embedding (2023.findings-acl)
Copied to clipboard
| Challenge: | EmbedTextNet is a light add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. |
| Approach: | They propose an add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. |
| Outcome: | The proposed network can be appended to an arbitrary language model to generate a compact embedding without any changes in its architecture or training procedure. |
Refining Pretrained Word Embeddings Using Layer-wise Relevance Propagation (D18-1)
Copied to clipboard
| Challenge: | Recent research trend is to refine or fine-tune pretrained word embeddings. |
| Approach: | They propose a method for refining pretrained word embeddings using layer-wise relevance propagation using a neural network. |
| Outcome: | The proposed method achieves higher performance than the original vectors. |
Efficient Vocabulary Reduction for Small Language Models (2025.coling-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have high computational costs and energy consumption, making their deployment in industrial settings difficult. |
| Approach: | They propose a small language model that compresses the embedding layer and reduces model size without significant loss of performance. |
| Outcome: | The proposed model reduces the embedding layer while maintaining performance while improving accuracy and performance. |