MATE: Meet At The Embedding - Connecting Images with Long Texts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Vision Language Models (VLMs) focus on aligning images with short descriptive captions. |
| Approach: | They propose a method that combines VLMs with Large Language Models to efficiently align images with long texts without additional text pairs. |
| Outcome: | The proposed method bridges the gap between VLM and LLM without additional image-long text pairs. |
Similar Papers
Enhancing Lexicon-Based Text Embeddings with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. |
| Approach: | They introduce the first lexicon-based embeddings that consolidates the vocabulary space through token embeddation clustering to handle the issue of token redundancy in LLM vocabularies. |
| Outcome: | The proposed model outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) it also supports efficient dimension pruning without any specialized objectives like Matryoshka Representation Learning. |
ViLL-E: Video LLM Embeddings for Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Video Large Language Models excel at video understanding tasks where outputs are textual . however, they underperform specialized embedding-based models in Retrieval tasks . |
| Approach: | They propose a video-LLM-based model with an embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. |
| Outcome: | The proposed model outperforms specialized embedding-based models in video understanding tasks while remaining competitive on VideoQA tasks. |
A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens (2025.acl-long)
Copied to clipboard
| Challenge: | et al., 2023) show that text embeddings from large language models can be aligned with key tokens in input text. |
| Approach: | They propose a sparse retrieval method based on aligned tokens for large language models . they show that this phenomenon is universal and is not affected by model architecture . |
| Outcome: | The proposed method can achieve 80% of the dense retrieval effect of the same model while reducing the computation significantly. |
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs (2025.emnlp-main)
Copied to clipboard
Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao
| Challenge: | Multimodal Large Language Models (MLLMs) integrate visual and textual inputs, yet modality alignment remains one of the most challenging aspects. |
| Approach: | They propose a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. |
| Outcome: | The proposed method improves performance across various model sizes, with smaller models benefiting the most. |
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Enabling LLMs to handle lengthy context is currently a research hotspot . a notable challenge limiting further customization is the inability of LLM to utilize context beyond pretrained length due to the inherent flaw of rotary position embedding (RoPE). |
| Approach: | They propose to extend the RoPE from an attention perspective and on two benchmarking tasks. |
| Outcome: | The proposed extension of the RoPE improves extrapolation and retrieval errors. |
Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method (2025.acl-long)
Copied to clipboard
| Challenge: | Existing long-context training data is scarce and requires substantial GPU resources for training. |
| Approach: | They propose a training-free plug-and-play method to enhance long-context understanding in existing large language models. |
| Outcome: | The proposed method outperforms existing LLMs on various tasks and surpasses baseline methods. |
Training LLMs to be Better Text Embedders through Bidirectional Reconstruction (2025.emnlp-main)
Copied to clipboard
Chang Su, Dengliang Shi, Siyuan Huang, Jintao Du, Changhua Meng, Yu Cheng, Weiqiang Wang, Zhouhan Lin
| Challenge: | Existing text embedding approaches often leverage the embeddment of the final token, typically a reserved special token such as ‘[EOS]‘. |
| Approach: | They propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. |
| Outcome: | The proposed training stage improves performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales. |
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks. |
| Approach: | They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training. |
| Outcome: | The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks. |
Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability. |
| Approach: | They propose a novel multi-modal in-context learning methodology to enhance LLMs’ contextual understanding and reasoning. |
| Outcome: | The proposed model achieves SOTA performance among all visual reasoning tasks and achieves a 'higher level of accuracy' than previous models. |
Improving Text Embeddings with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages . |
| Approach: | They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. |
| Outcome: | The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data. |