MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in text-video retrieval neglect the crucial user perspective, leading to discrepancies between user queries and content retrieved. |
| Approach: | They propose a novel, training-free pipeline that leverages Large Language Models for iterative feedback learning. |
| Outcome: | Experimental results show that MERLIN significantly outperforms existing systems in video retrieval. |
Similar Papers
MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning (2026.eacl-long)
Copied to clipboard
Kosei Uemura, David Guzmán, Quang Phuoc Nguyen, Jesujoba Oluwadara Alabi, En-Shiun Annie Lee, David Ifeoluwa Adelani
| Challenge: | Existing methods to align large language models with multilingual encoders raise accuracy for low-resource languages (LRLs) but performance of LLMs in low- and high-resourced languages remains a problem. |
| Approach: | They propose a model-stacking framework that iteratively refines in 2-stages based on a curriculum strategy and adapts only a small set of DoRA weights. |
| Outcome: | The proposed framework improves exact-match accuracy by +12.9 pp over MindMerger and outperforms GPT-4o-mini by 15.2 pp on the AfriMGSM benchmark. |
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Dense retrieval requires discriminative embeddings to represent the semantic relationship between query and document. |
| Approach: | They propose an unsupervised approach that performs unsupervised adaptation of large language models for dense retrieval. |
| Outcome: | The proposed model improves on a variety of dense retrieval benchmarks and is available on github. |
ViLL-E: Video LLM Embeddings for Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Video Large Language Models excel at video understanding tasks where outputs are textual . however, they underperform specialized embedding-based models in Retrieval tasks . |
| Approach: | They propose a video-LLM-based model with an embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. |
| Outcome: | The proposed model outperforms specialized embedding-based models in video understanding tasks while remaining competitive on VideoQA tasks. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
ITERATE: Image-Text Enhancement, Retrieval, and Alignment for Transmodal Evolution with LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | a new framework for visual annotation of text-based questions is needed to improve performance . obtaining corresponding images through manual annotation often entails high costs . |
| Approach: | They propose a framework that uses visual modality to enhance the performance of text-based questions. |
| Outcome: | The proposed framework improves the alignment between text and images by using search engines or web scraping techniques. |
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LMM-based embedding models exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. |
| Approach: | They propose a framework that improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. |
| Outcome: | The proposed framework improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. |
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks. |
| Approach: | They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training. |
| Outcome: | The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks. |
Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Experiments with AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-text retrieval performance to state-of-the-art M2D-CLAP. |
| Approach: | They propose a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding that allows users to express their queries in five different ways. |
| Outcome: | Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP while demonstrating clear advantages in two critical areas. |
A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens (2025.acl-long)
Copied to clipboard
| Challenge: | et al., 2023) show that text embeddings from large language models can be aligned with key tokens in input text. |
| Approach: | They propose a sparse retrieval method based on aligned tokens for large language models . they show that this phenomenon is universal and is not affected by model architecture . |
| Outcome: | The proposed method can achieve 80% of the dense retrieval effect of the same model while reducing the computation significantly. |
Unified Embeddings for Multimodal Retrieval via Frozen LLMs (2024.findings-eacl)
Copied to clipboard
| Challenge: | Unified Embeddings for Multimodal Retrieval (UniMuR) is a simple but effective approach that embeds multimodal inputs and retrieves visual and textual outputs via frozen Large Language Models (LLMs). |
| Approach: | They propose a method that embeds multimodal inputs and retrieves visual and textual outputs via frozen Large Language Models (LLMs). |
| Outcome: | The proposed method significantly reduces LLM’s modality bias towards generating text-only outputs and achieves strong image/text retrieval ability. |