LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation (2023.eacl-main)
Copied to clipboard
| Challenge: | Large-scale language-agnostic sentence embedding models suffer from inference speed and computation overhead. |
| Approach: | They propose to train a lightweight sentence embedding model to achieve this by incorporating knowledge from a teacher model. |
| Outcome: | The proposed model can build low-dimensional sentences for 109 languages with a thin-deep encoder. |
Similar Papers
Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in text-to-speech systems allow for speech synthesis with unprecedented quality and controllability. |
| Approach: | They use embeddings derived from articulatory vectors rather than phoneme identities to learn phoneme representations that hold across languages. |
| Outcome: | The proposed models fine-tuned on 30 minutes of data in a previously unseen language with language agnostic meta learning. |
Textual Dataset Distillation via Language Model Embedding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | prevailing methods for dataset distillation generate distilled data as embedding vectors, which are not human-readable. |
| Approach: | They propose a model-agnostic, data-efficient method that leverages Language Model embeddings . their method offers enhanced flexibility and improved transferability . |
| Outcome: | The proposed method achieves comparable performance with faster processing times compared to other methods . it offers enhanced flexibility and improved transferability, expanding the range of potential applications . |
Disentangling Meaning and Language Components in Diverse Multilingual Sentence Embeddings (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing studies have reported language specificity in multilingual sentence embeddings, resulting in language-specific subspaces. |
| Approach: | They propose to disentangle multilingual sentence embeddings into language-dependent and language-agnostic components to improve cross-lingual similarity estimation. |
| Outcome: | The proposed methods improve cross-lingual similarity estimation across multiple embeddings. |
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing sentence embeddings models are monolingual, and only for English . a new method allows to create multilingual versions from monolingual models . |
| Approach: | They propose a method to extend existing sentence embedding models to new languages . they use a translated sentence to generate sentence embeds for the source language . |
| Outcome: | The proposed method improves accuracy for multilingual setups and languages. |
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases. |
| Approach: | They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time. |
| Outcome: | The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings. |
InferLite: Simple Universal Sentence Representations from Natural Language Inference Data (D18-1)
Copied to clipboard
| Challenge: | InferLite is a lightweight version of InferSent that does not use recurrent layers and can generalize to multiple pre-trained word embeddings. |
| Approach: | They propose a lightweight version of InferSent that does not use recurrent layers and operates on a collection of pre-trained word embeddings. |
| Outcome: | The proposed model outperforms existing models that learn generic embeddings in an unsupervised setting, often requiring several days or weeks to train. |
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |
Language-agnostic BERT Sentence Embedding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning bilingual sentence embeddings are not well explored. |
| Approach: | They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba. |
| Outcome: | The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER. |
WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models perform well on learning sentence semantics when fine-tuned with supervised data. |
| Approach: | They conduct a thorough examination of pretrained model based unsupervised sentence embeddings. |
| Outcome: | The proposed approach improves on whitening-based vector normalization with less than 10 lines of code. |
Language-agnostic Representation from Multilingual Sentence Encoders for Cross-lingual Similarity Estimation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract language-specific information from multilingual sentence embeddings are remarkably successful in cross-lingual and multilingual NLU tasks. |
| Approach: | They propose to extract language-specific information from the original embedding and use it to retrieve an embeddable that fully represents the sentence’s meaning. |
| Outcome: | The proposed method outperforms baselines on cross-lingual sentences even in low-resource language pairs where only tens of thousands of parallel sentence pairs are available. |