| Challenge: | Existing evaluation methodologies for code summarization tasks do not consider timestamps of code and comments. |
| Approach: | They propose a time-segmented evaluation methodology for code summarization that considers timestamps of code and comments during evaluation. |
| Outcome: | The proposed evaluation methodology compares with other evaluation methodologies that have been widely used. |
Similar Papers
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
A Critical Look at Meta-evaluating Summarisation Evaluation Metrics (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Effective summarisation evaluation metrics enable researchers and practitioners to compare different summarization systems efficiently. |
| Approach: | They argue that evaluation metrics are primarily meta-evaluated on news summarisation datasets and that there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. |
| Outcome: | The evaluation metrics are primarily meta-evaluated on news summarisation datasets and there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. |
Recommendations for Datasets for Source Code Summarization (N19-1)
Copied to clipboard
| Challenge: | Code summarization is the task of writing short, natural language descriptions of source code. |
| Approach: | They propose to use a dataset based on 2.1m pairs of Java methods and one sentence method descriptions from over 28k Java projects to write short, natural language code summarizations. |
| Outcome: | The proposed dataset shows that the proposed standards are more effective than previous versions. |
ReflectSumm: A Benchmark for Course Reflection Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research has focused on standard summarization benchmarks within domains like news, scientific articles, and opinions. |
| Approach: | They propose a summarization dataset specifically designed for summarizing students’ reflective writing. |
| Outcome: | The proposed summarization dataset can be used in opinion summarizing scenarios and in educational domains. |
Facet-Aware Evaluation for Extractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | lexical overlap is a common evaluation metric for extractive summarization, but recent studies reveal its limitations. |
| Approach: | They propose a facet-aware evaluation setup for better assessment of information coverage in extractive summaries. |
| Outcome: | The proposed evaluation setup improves human correlation with extractive summarization datasets and improves comparative analysis. |
Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages (2025.acl-long)
Copied to clipboard
Hyangsuk Min, Yuho Lee, Minjeong Ban, Jiaqi Deng, Nicole Hee-Yeon Kim, Taewon Yun, Hang Su, Jason Cai, Hwanjun Song
| Challenge: | Existing evaluation frameworks for text summarization lack domain-specific assessment criteria and are predominantly English-centric. |
| Approach: | They propose a multi-dimensional, multi-domain evaluation of summarization in English and Chinese that incorporates specialized assessment criteria for each domain and leverages a debate system to enhance annotation quality. |
| Outcome: | The proposed evaluation framework provides a multi-dimensional, multi-domain evaluation of summarization in English and Chinese. |
On Leakage of Code Generation Evaluation Datasets (2024.findings-emnlp)
Copied to clipboard
Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, Matthias Gallé
| Challenge: | In this paper, we discuss contamination by code generation test sets in large language models. |
| Approach: | They propose to use Python to test code generation test sets for contamination . they find that code generation is an important skill for large language models to master . |
| Outcome: | The proposed benchmarks are uncontaminated and provide a new insight into code generation. |
CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for text summarization systems are limited to in-domain setting, where supervised pre-trained models are evaluated on the same dataset. |
| Approach: | They propose to use a cross-dataset evaluation approach to evaluate different summarization systems in a multi-domain setting. |
| Outcome: | The proposed model can be used to evaluate text summarization systems on different datasets. |
In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | citation counts are a shallow view that fails to capture how a paper has influenced subsequent work. |
| Approach: | They propose a task to generate nuanced, expressive, and time-aware impact summaries . they analyze fine-grained confirmatory and correction citation intents to generate summary . |
| Outcome: | The proposed task shows moderate to strong human correlation on subjective metrics such as insightfulness. |
SummerTime: Text Summarization Toolkit for Non-experts (2021.emnlp-demo)
Copied to clipboard
Ansong Ni, Zhangir Azerbayev, Mutethia Mutuma, Troy Feng, Yusen Zhang, Tao Yu, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Recent advances in summarization provide models that can generate high quality summaries . a new toolkit for summarizing text is being developed to make it easier for non-experts to keep track of them. |
| Approach: | They develop a toolkit for text summarization that integrates with libraries designed for NLP researchers. |
| Outcome: | SummerTime is a toolkit for text summarization, including models, datasets, and evaluation metrics. |