ProConSuL: Project Context for Code Summarization with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Experimental results show that ProConSuL significantly improves code summaries and reduces the number of hallucinations. |
| Approach: | They propose a framework to provide a large language model with precise information about the code structure from program analysis methods. |
| Outcome: | The proposed framework significantly improves code summaries and reduces hallucinations compared to the base model. |
Similar Papers
DependEval: Benchmarking LLMs for Repository Dependency Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | a benchmark is designed to evaluate the repository-level dependency understanding of large language models (LLMs) based on 2683 repositories from real-world websites. |
| Approach: | They propose a benchmark to evaluate repository dependency understanding for large language models . DEPENDEVAL evaluates models on three core tasks across 8 programming languages . |
| Outcome: | The benchmark evaluates models on three core tasks across 8 programming languages from real-world repositories. |
Analyzing the Performance of Large Language Models on Code Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models perform very well on tasks that involve both natural language and source code. |
| Approach: | They show that large language models perform very well on tasks that involve both natural language and source code. |
| Outcome: | The proposed models perform very well on tasks that involve both natural language and source code. |
Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Code Large Language Models have limited ability to reason about runtime behavior and understand functionality . authors present a generic framework to support integrating semantic information to code task-relevant prompts . |
| Approach: | a study examines the role of trace-based semantic information in boosting supervised fine-tuning and post-phase inference of Code LLMs. |
| Outcome: | a new framework integrates semantic information to code task-relevant prompts . the proposed framework shows that trace-based semantic information boosts reasoning ability . |
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (2024.eacl-demo)
Copied to clipboard
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali
| Challenge: | Recent development and success of Large Language Models necessitate evaluation of their performance across diverse NLP tasks in different languages. |
| Approach: | They propose a framework that can be customized to evaluate LLMs for any NLP task, regardless of language. |
| Outcome: | The LLMeBench framework can be customized to evaluate LLMs for any NLP task, regardless of language. |
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks such as revising source code to address code reviews. |
| Approach: | They propose a benchmark to evaluate large language models' ability to bridge both technical and conversational contexts by decomposing the generation task of code refinement into three essential reasoning steps. |
| Outcome: | The proposed benchmark exposes specific model weaknesses in code review comprehension disentangled from their generative automated code refinement results. |
On Context Utilization in Summarization with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models excel in abstractive summarization tasks, delivering fluent and pertinent summaries. |
| Approach: | They conduct the first comprehensive study on context utilization and position bias in summarization. |
| Outcome: | The proposed benchmark compares two methods to alleviate position bias in summarization tasks. |
What can Large Language Models Capture about Code Functional Equivalence? (2025.findings-naacl)
Copied to clipboard
| Challenge: | SeqCoBench is a benchmark to assess how Code-LLMs can capture code semantics. |
| Approach: | They propose a benchmark to assess how Code-LLMs capture code semantics . they use seqCoBench to evaluate whether they can discern semantically equivalent or different pairs of programs . |
| Outcome: | The proposed benchmarks show that they can capture code semantics better than classical match-based retrieval scores. |
Re-FRAME the Meeting Summarization SCOPE: Fact-Based Summarization and Personalization via Questions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | FRAME reframes summarization as a semantic enrichment task . SCOPE is a reason-out-loud protocol that has the model build a reasoning trace . |
| Approach: | They propose a modular pipeline that reframes summarization as a semantic enrichment task. |
| Outcome: | The proposed pipeline reduces hallucinations and omissions by 2 out of 5 points . SCOPE improves knowledge fit and goal alignment over prompt-only baselines . |
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2024.acl-long)
Copied to clipboard
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng
| Challenge: | Existing benchmarks for evaluating the code understanding and generation capacities of Large Language Models are insufficient . existing benchmarks focus on a narrow range of popular programming languages and specific tasks . |
| Approach: | They propose an execution-based, multilingual, multitask evaluation benchmark for LLMs . they evaluate coding performance from three dimensions: length, difficulty, efficiency . |
| Outcome: | The proposed benchmark covers 43 programming languages and eight coding tasks. |
Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective (2023.emnlp-industry)
Copied to clipboard
| Challenge: | a study examines how to build meeting summarization systems using large language models . closed-source models are generally better in terms of performance, but open-source ones are more advantageous for industrial use . |
| Approach: | They compare closed-source and open-source meeting summarization models for real-world use . they find that closed-sourced models are generally better in terms of performance . however, smaller open-sourced LLMs could still achieve comparable performance if they are open . |
| Outcome: | The proposed model is more efficient for industrial use than closed-source models due to privacy concerns and high cost. |