DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong effectiveness and robustness when fine-tuned as dense retrievers. |
| Approach: | They propose a training framework that leverages pruned LLMs to train smaller generalizable dense retrievers. |
| Outcome: | The proposed training framework offers better multilingual and long-context capabilities than traditional encoder-based retrievers and achieves strong performance across multiple tasks and languages. |
Similar Papers
Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for training contrastive learning based sentence embedding models are largely influenced by the quality of sentence pairs. |
| Approach: | They propose a framework that decomposes LLMs into three stages for training . they propose to refine the generated content at these stages to ensure only high-quality sentence pairs are utilized to train a base contrastive learning model. |
| Outcome: | The proposed framework surpasses ChatGPT and ChatGPP in terms of performance. |
Making Large Language Models Efficient Dense Retrievers (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that fine-tuning large language models for dense retrieval yields strong performance, but their substantial parameter counts make them computationally inefficient. |
| Approach: | They propose a framework for developing efficient retrievers that performs coarse-to-fine compression through a coarse-grained coarse-tuning strategy. |
| Outcome: | The proposed framework reduces model size and inference cost while preserving performance of full-size models. |
Contrastive Learning for Task-Independent SpeechLLM-Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in speech processing tasks but their reliance on written text limits their application in real-world scenarios. |
| Approach: | They propose a task-independent speech pretraining stage and task-specific fine-tuning stage to adapt LLMs to speech processing tasks. |
| Outcome: | The proposed model outperforms models specialized on speech translation and question answering while being trained on 10% of the task-specific data. |
ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing dense retrieval with query augmentation ignore the alignment between generation and ranking objectives. |
| Approach: | They propose a unified LLM-augmented dense retrieval framework that jointly optimizes both the LLM and the retriever. |
| Outcome: | Experimental results show that ExpandR outperforms strong baselines, achieving more than 5% improvement in retrieval performance. |
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have limited generalization capabilities and performance challenges. |
| Approach: | They evaluate 15 different backbone LLMs and non-LLMs to evaluate their performance . larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency . |
| Outcome: | The results show that larger models and extensive pre-training enhance in-domain accuracy and data efficiency. |
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized natural language processing with impressive capabilities, but they lack domain specificity, real-time information and face challenges in solving specialized problems. |
| Approach: | They propose a multi-LLM approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer. |
| Outcome: | The proposed model outperforms existing models by demonstrating its effectiveness and advantages in tool learning. |
Diversity-oriented Data Augmentation with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing data augmentation methods focus on increasing sample numbers while neglecting sample distribution diversity, which can lead to model overfitting. |
| Approach: | They propose a data augmentation framework that focuses on sample distribution diversity and trains a large language model as a diverse paraphraser. |
| Outcome: | The proposed framework achieves an average performance gain of 10.52% surpassing the runner-up baseline with more than three percentage points. |
ARL2: Aligning Retrievers with Black-box Large Language Models via Self-guided Adaptive Relevance Labeling (2024.acl-long)
Copied to clipboard
| Challenge: | Existing retrievers are misaligned with large language models due to separate training processes and inherent black-box nature of LLMs. |
| Approach: | They propose a retriever learning technique that harnesses LLMs as labelers to annotate and score adaptive relevance evidence. |
| Outcome: | Extensive experiments show that ARL2 improves accuracy and reduces the cost of API calls. |
Data and Model Centric Approaches for Expansion of Large Language Models to New languages (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | Existing LLMs mainly support English alongside a handful of high resource languages . this leaves a major gap for most low-resource languages despite increasing pace of research . |
| Approach: | This tutorial examines approaches to expand the language coverage of LLMs . they look at tokenizer training, pre-training, instruction tuning, alignment, evaluation, etc. |
| Outcome: | This tutorial examines approaches to expand the language coverage of LLMs . it provides an efficient and viable path to bring LLM technologies to low-resource languages . |
Retrieval-Augmented Retrieval: Large Language Models are Strong Zero-Shot Retriever (2024.findings-acl)
Copied to clipboard
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, Daxin Jiang
| Challenge: | Large-scale retrieval is indispensable in information-seeking tasks such as open-domain question answering and knowledgegrounded dialogue. |
| Approach: | They propose to use a large language model (LLM) to augment a query with its potential answers by prompting LLMs with a composition of the query and the query’s in-domain candidates. |
| Outcome: | The proposed method breaks brute-force combinations of retrievers with LLMs and lifts the performance of zero-shot retrieval to be very competitive on benchmark datasets. |