Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior work on multilingual sentence embedding has demonstrated that the efficient use of natural language inference data to build high-performance models can outperform conventional methods. |
| Approach: | They propose a multilingual sentence embedding model by extending an existing monolingual model by using the low-rank adaptation technique. |
| Outcome: | The proposed model outperforms the previous approach and shows that languages with fewer resources or those with less linguistic similarity to English benefit more from the parameter increase. |
Similar Papers
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
Multilingual Universal Sentence Encoder for Semantic Retrieval (2020.acl-demos)
Copied to clipboard
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, Ray Kurzweil
| Challenge: | Using a multi-task trained dual-encoder, our models embed text from 16 languages into a shared semantic space. |
| Approach: | They propose retrieval focused multilingual sentence embedding models on TensorFlow Hub. |
| Outcome: | The models achieve state-of-the-art on monolingual and cross-lingual retrieval (SR) and retrieval question answering (ReQA) competitive performance is obtained on related tasks of translation pair bitext retrieval and retrieving question answering. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Multilingual sentence encoders are often trained to map sentences from different languages into a shared semantic vector space . cross-lingual alignment training distorts optimal monolingual structure of semantic spaces of individual languages . a modular solution can be used for cross-linguistic tasks such as cross-language semantic similarity and zero-shot transfer . |
| Approach: | They propose a modular training system that embeds sentences from different languages into a shared semantic vector space. |
| Outcome: | The proposed solution achieves better performance across all tasks compared to monolithic models. |
Language-agnostic BERT Sentence Embedding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning bilingual sentence embeddings are not well explored. |
| Approach: | They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba. |
| Outcome: | The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER. |
Multi-Source Text Classification for Multilingual Sentence Encoder with Machine Translation (2024.naacl-srw)
Copied to clipboard
| Challenge: | Pre-trained multilingual sentence encoders suffer from performance degradation for non-English languages. |
| Approach: | They propose a method of machine translating a source sentence into English and then inputting it together with the source sentence in a multi-source manner. |
| Outcome: | The proposed method improves the performance of pre-trained multilingual sentence encoders in Japanese on sentiment analysis and topic classification tasks. |
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing sentence embeddings models are monolingual, and only for English . a new method allows to create multilingual versions from monolingual models . |
| Approach: | They propose a method to extend existing sentence embedding models to new languages . they use a translated sentence to generate sentence embeds for the source language . |
| Outcome: | The proposed method improves accuracy for multilingual setups and languages. |
Cross-lingual Sentence Embedding using Multi-Task Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual sentence embedding models require large parallel corpora to learn efficiently, limiting their scope. |
| Approach: | They propose a sentence embedding framework based on an unsupervised loss function . they capture semantic similarity and relatedness between sentences using a multi-task loss function. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on STS, BUCC and Tatoeba benchmarks and on a monolingual benchmark. |
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages. |
| Approach: | They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data . |
| Outcome: | The proposed model performs well on multilingual summarization and question-answering tasks. |