Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word-embeddings are vital components of natural language processing (NLP) but they consume a lot of memory which poses a challenge for edge deployment. |
| Approach: | They propose an embedding compression method based on matrix decomposition and knowledge distillation that initializes weights of pre-trained word-embeddings and fine-tunes end-to-end. |
| Outcome: | The proposed method has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune methods. |
Similar Papers
Block-wise Word Embedding Compression Revisited: Better Weighting and Structuring (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for word embedding compression are limited . word embeds have a considerable size and need to be compressed to deploy on edge devices . |
| Approach: | They propose a block-wise low-rank approximation method for word embedding called GroupReduce . they propose 'frequency-inverse document frequency method' and a differentiable method for weighting . |
| Outcome: | The proposed algorithm more effectively finds word weights than competitors in most cases. |
Tensorized Embedding Layers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using the Tensor Train decomposition, embeddings layers occupy large portion of model weights, preventing their deployment in limited resource settings. |
| Approach: | They propose a method for parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. |
| Outcome: | The proposed method can be plugged into any model and trained end-to-end. |
EmByte: Decomposition and Compression Learning for Small yet Private NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EMBYTE is a byte-level tokenization model that reduces embedding parameters by up to 94% . it is also resilient to privacy threats such as gradient inversion attacks . |
| Approach: | EMBYTE is a byte-level tokenization model that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. |
| Outcome: | EMBYTE achieves substantial embedding compression while preserving accuracy and enhancing privacy. |
Distilling Relation Embeddings from Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture a surprisingly rich amount of lexical knowledge, but it is unclear to what extent relation embeddings can be used to encode relational knowledge. |
| Approach: | They found that word vector differences capture lexical relations . relationship embeddings can be used to encode relational knowledge . |
| Outcome: | The results are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models produce large sentence embeddings, resulting in performance gap between large and small models. |
| Approach: | They propose a method that augments a small Transformer encoder model with learnable projection layers to produce compact sentences while mimicking a large pre-trained language model to retain the sentence representation quality. |
| Outcome: | The proposed method achieves 2.7-4.5 points performance gain on STS and SR tasks while maintaining the quality of the pre-trained language models. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Textual Dataset Distillation via Language Model Embedding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | prevailing methods for dataset distillation generate distilled data as embedding vectors, which are not human-readable. |
| Approach: | They propose a model-agnostic, data-efficient method that leverages Language Model embeddings . their method offers enhanced flexibility and improved transferability . |
| Outcome: | The proposed method achieves comparable performance with faster processing times compared to other methods . it offers enhanced flexibility and improved transferability, expanding the range of potential applications . |
Multi-Sense Embeddings for Language Models and Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Transformer-based large language models generate different representations for the same token depending on context . however, words and tokens typically have only a limited number of senses . a knowledge distillation method can be used to learn a smaller student model . |
| Approach: | They propose a multi-sense embedding method that uses a clustering algorithm to generate a sense embeddable dictionary. |
| Outcome: | The proposed method offers significant space and inference time savings while maintaining competitive performance. |