SWE-QA: Can Language Models Answer Repository-level Code Questions? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for understanding and reasoning about entire soft-ware repositories focus on small, self-contained code snippets. |
| Approach: | They propose a repository-level code question answering benchmark to facilitate research on automated QA systems in real-world repositories. |
| Outcome: | The proposed benchmarks are designed to facilitate research on automated QA systems in real-world repositories. |
Similar Papers
SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding (2026.findings-acl)
Copied to clipboard
Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, Wenhu Chen
| Challenge: | Existing benchmarks for agentic repository-level code understanding overlook long tail topics and rely on memorized knowledge. |
| Approach: | They propose a repository-level agentic code understanding benchmark that uses long-tail repositories with executable environments to enforce topical balance. |
| Outcome: | Empirically, a Qwen3-8B model trained with the proposed benchmark outperforms GPT-4o by 2.3 points. |
On Improving Repository-Level Code QA for Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Commercial AI-assisted programming Chatbots may generate incorrect information when requests go beyond the model training data or require additional knowledge. |
| Approach: | They propose to implement different self-alignment processes and retrieval-augmented generation pipelines to improve the copilot performance. |
| Outcome: | The proposed model improves the copilot performance on repository-level semantics, dependency between files, and meta-information about the repository. |
Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Code large language models (LLMs) enhance programming by understanding and generating code across languages. |
| Approach: | a new benchmark evaluates code understanding and generation in repositories using code large language models. |
| Outcome: | The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages. |
DependEval: Benchmarking LLMs for Repository Dependency Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | a benchmark is designed to evaluate the repository-level dependency understanding of large language models (LLMs) based on 2683 repositories from real-world websites. |
| Approach: | They propose a benchmark to evaluate repository dependency understanding for large language models . DEPENDEVAL evaluates models on three core tasks across 8 programming languages . |
| Outcome: | The benchmark evaluates models on three core tasks across 8 programming languages from real-world repositories. |
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation (2025.acl-long)
Copied to clipboard
Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, Scarlett Li
| Challenge: | Existing benchmarks focus on standalone programming problems, such as HumanEval, MBPP, and LiveCodeBench. |
| Approach: | They propose to use large language models to evaluate their ability to perform incremental development within code repositories by collecting pull requests from 83 GitHub repositorias and using rule-based and intent-based filtering to construct task instances focused on new feature development. |
| Outcome: | The proposed benchmarks show that large language models perform significantly worse in the FEA-Bench, highlighting considerable challenges in repository-level incremental code development. |
Repo4QA: Answering Coding Questions via Dense Retrieval on GitHub Repositories (2022.coling-1)
Copied to clipboard
| Challenge: | Stack Overflow and GitHub are open source communities that are gaining popularity . developers need to raise programming questions in coding forums and navigate to GitHub repositories . |
| Approach: | They propose a questionrepository matching task that bridges the gap between repositories and real-world coding questions. |
| Outcome: | The proposed model outperforms state-of-the-art methods on coding questions and repositories . it can find suitable coding repositoriels and bridge the gap between them . |
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced rapidly from conversational problem solving to addressing real-world tasks involving tool use, such as software engineering (SWE). |
| Approach: | They propose to build an LLM-based software engineering agent that synthesizes test cases and scales up agent trajectories to build training data. |
| Outcome: | The proposed model outperforms state-of-the-art models on the SWE-bench-Verified benchmark. |
SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in code generation benchmarks, but these benchmarks focus on single-file scenarios with constrained context scope. |
| Approach: | They propose an open-source framework to effectively resolve GitHub issues using a code file retrieval module and a model-based code editing module. |
| Outcome: | The proposed approach achieves state-of-the-art performance on two GitHub benchmarks. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |