Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, Daniel Fried
| Challenge: | Language models excel at generating code, but many programs are difficult to generate using only parametric knowledge. |
| Approach: | They propose a retrieval-augmented code generation benchmark that provides reproducible evaluations on retrieval and end-to-end code generation performance. |
| Outcome: | The proposed benchmark covers programming, open-domain, and repository-level tasks and provides reproducible evaluations on retrieval and end-to-end code generation performance. |
Similar Papers
FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation (2025.acl-demo)
Copied to clipboard
| Challenge: | Existing frameworks for retrieval-augmented generation (RAG) lack new techniques, difficulties in algorithm reproduction and sharing, and high system overhead. |
| Approach: | They propose a retrieval-augmented generation framework specifically designed for research and prototyping that supports text-based, multimodal, and network-based RAG. |
| Outcome: | The proposed framework supports text-based, multimodal, and network-based RAG, providing comprehensive lifecycle support alongside efficient asynchronous processing and persistent caching capabilities. |
CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in code large language models have produced repository-level code completion methods that automatically predict the unfinished code based on the broader information from the repository. |
| Approach: | They propose a framework to identify relevant knowledge for retrieval-augmented repository-level code completion. |
| Outcome: | The proposed framework significantly outperforms state-of-the-art methods on ReccEval and CCEval. |
XRAG: Cross-lingual Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | XRAG evaluates the generation abilities of LLMs in cross-lingual RAG settings where the user language does not match retrieval results. |
| Approach: | They propose a benchmark to evaluate the generation abilities of LLMs in cross-lingual RAG settings where the user language does not match retrieval results. |
| Outcome: | XRAG is a benchmark designed to evaluate the generation abilities of LLMs in cross-lingual RAG settings where the user language does not match retrieval results. |
BERGEN: A Benchmarking Library for Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Stéphane Clinchant, Vassilina Nikoulina
| Challenge: | Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. |
| Approach: | They propose a library that allows to benchmark and standardize RAG experiments. |
| Outcome: | The proposed library is an end-to-end library for reproducible research standardizing RAG experiments. |
LightRAG: Simple and Fast Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing RAG systems rely on flat data representations and inadequate contextual awareness . lightRAG framework incorporates graph structures into text indexing and retrieval processes . |
| Approach: | LightRAG is a framework that integrates graph structures into text indexing and retrieval processes. |
| Outcome: | The proposed framework incorporates graph structures into text indexing and retrieval processes. |
RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies focus on question scenarios with clear user intents and concise answers, but it is prevalent that users issue broad, open-ended queries with diverse sub-intents. |
| Approach: | They propose a framework that includes a sub-aspect explorer and a multi-faceted retriever to build a candidate pool of diverse external documents related to these sub-intents. |
| Outcome: | The proposed framework provides comprehensive and satisfying responses to users on two publicly available datasets. |
Enhancing Retrieval-Augmented Generation: A Study of Best Practices (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented generation systems have shown remarkable advancements by integrating retrieval mechanisms into language models, enhancing their ability to produce more accurate and contextually relevant responses. |
| Approach: | They propose to integrate query expansion, various novel retrieval strategies, and a Contrastive In-Context Learning RAG to improve response quality. |
| Outcome: | The proposed RAGs incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG. |
Retrieval Augmented Code Generation and Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Software developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them. |
| Approach: | They propose a retrieval augmented framework that retrieves relevant code or summaries from a database and provides them as a supplement to code generation or summarization models. |
| Outcome: | The proposed framework can search for relevant code or summaries from retrieval databases and can work with unimodal (only code or natural language description) or bimodal instances (code-description pairs). |
VideoRAG: Retrieval-Augmented Generation over Video Corpus (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to generating models rely on text and images, but video content is a rich source of multimodal knowledge. |
| Approach: | They propose a framework that dynamically retrieves videos based on their relevance with queries . they use large video language models to represent video content for retrieval . |
| Outcome: | The proposed framework retrieves videos based on relevance with queries and integrates both visual and textual information. |
FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAG (2025.findings-naacl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) is widely adopted in Large Language Models, but is flat and has limitations such as a significant burden on one retriever and constant granularity limits the ceiling of retrieval performance. |
| Approach: | They propose a progressive retrieval paradigm with coarse-to-fine granularity for RAG, termed FunnelRAG, so as to balance effectiveness and efficiency. |
| Outcome: | The proposed paradigm achieves comparable retrieval performance while the time overhead is reduced by nearly 40%. |