Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models produce large sentence embeddings, resulting in performance gap between large and small models. |
| Approach: | They propose a method that augments a small Transformer encoder model with learnable projection layers to produce compact sentences while mimicking a large pre-trained language model to retain the sentence representation quality. |
| Outcome: | The proposed method achieves 2.7-4.5 points performance gain on STS and SR tasks while maintaining the quality of the pre-trained language models. |
Similar Papers
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases. |
| Approach: | They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time. |
| Outcome: | The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings. |
Learning Compressed Sentence Representations for On-Device Text Processing (P19-1)
Copied to clipboard
Dinghan Shen, Pengyu Cheng, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Meng Tang, Asli Celikyilmaz, Lawrence Carin
| Challenge: | Existing methods for learning sentence embeddings assume they are continuous and real-valued. |
| Approach: | They propose four different strategies to transform continuous and generic sentence embeddings into a binarized form while preserving their rich semantic information. |
| Outcome: | The proposed methods reduce storage requirements by over 98% and improve performance on downstream tasks. |
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word-embeddings are vital components of natural language processing (NLP) but they consume a lot of memory which poses a challenge for edge deployment. |
| Approach: | They propose an embedding compression method based on matrix decomposition and knowledge distillation that initializes weights of pre-trained word-embeddings and fine-tunes end-to-end. |
| Outcome: | The proposed method has higher BLEU score on translation and lower perplexity on language modeling compared to complex, difficult to tune methods. |
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |
Pretraining Context Compressor for Large Language Models with Embedding-Based Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Efficient processing of long contexts in large language models is essential for real-world applications such as retrieval-augmented generation and in-context learning. |
| Approach: | They propose a decoupled compressor-LLM framework that preserves contextual information within condensed embedding representations. |
| Outcome: | The proposed framework outperforms baseline models in three domains and across eight datasets while adapting to different downstream LLMs. |
Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings (2020.findings-emnlp)
Copied to clipboard
Prafull Prakash, Saurabh Kumar Shashidhar, Wenlong Zhao, Subendhu Rongali, Haidar Khan, Michael Kayser
| Challenge: | Existing task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders. |
| Approach: | They propose to learn compositional code embeddings to greatly reduce the sizes of BERT and RoBERTa encoders. |
| Outcome: | The proposed model reduces the size of BERT and RoBERTa encoders while maintaining performance. |
Towards Lossless Encoding of Sentences (P19-1)
Copied to clipboard
| Challenge: | Existing methods for encoding text into lossless representations focus on performing well on downstream tasks and are unable to reconstruct original sequence from learned embedding. |
| Approach: | They propose a lossless method for encoding long sequences of texts into feature rich representations by recursive autoencoding. |
| Outcome: | The proposed method performs well on sentiment analysis and sentiment classification tasks. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
ConGen: Unsupervised Control and Generalization Distillation For Sentence Representation (2022.findings-emnlp)
Copied to clipboard
Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Lalita Lowphansirikul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Sentence representations are essential in many NLP tasks operating at the sentence level. |
| Approach: | They propose an unsupervised sentence representation method to reduce the supervised-unsupervised performance gap for smaller models. |
| Outcome: | The proposed method outperforms supervised training on STS, text classification, and natural language inference tasks on smaller models. |
Condenser: a Pre-training Architecture for Dense Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work fine-tunes deep LMs to encode text sequences into single dense vector representations, but dense encoders require a lot of data and sophisticated techniques to train and suffer in low data situations. |
| Approach: | They propose to pre-train Transformer language models (LMs) with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. |
| Outcome: | The proposed model improves on various text retrieval and similarity tasks by large margins over standard LMs. |