Papers with ROUGE
Abstractive Timeline Summarization (D19-54)
Copied to clipboard
| Challenge: | Prior approaches to TLS focus on extractive methods, which generate extractive timelines . a study with human judges shows that our abstractive system also produces output that is easy to read and understand. |
| Approach: | They propose an abstractive timeline summarization system that is unsupervised . their system outperforms extractive systems in terms of ROUGE scores . |
| Outcome: | The proposed system outperforms extractive systems in terms of ROUGE scores . it produces output that is easy to read and understand, the authors say . |
Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate concise summarizations require extensive modifications to the architecture. |
| Approach: | They propose a lightweight model that can be finetuned to extract salient keyphrases from the source document to enhance ROUGE F1 and recall. |
| Outcome: | The proposed model can be finetuned to extract salient keyphrases without any LLM customization. |
Generating Text through Adversarial Training Using Skip-Thought Vectors (N19-3)
Copied to clipboard
| Challenge: | Existing approaches to use word embeddings for text generation have been limited. |
| Approach: | They propose to use GANs with word embeddings to reproduce writing style in text . they use a sentence embeddable vector to model people's way of expression . |
| Outcome: | The proposed model outperforms baseline text generation networks across several metrics including BLEU-n, METEOR and ROUGE. |
Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing lexical or semantic metrics cannot accurately capture the discourse coherence of long-form text generation. |
| Approach: | They propose to use automatic metric to quantify the discourse divergence between two long-form articles . they use a theory of functional discourse structure to model the coherence of long-formed texts . |
| Outcome: | The proposed metric outperforms existing evaluation methods on three datasets from representative domains. |
Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation (2022.emnlp-industry)
Copied to clipboard
Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Anya Belz, Ehud Reiter
| Challenge: | Existing studies have shown that note generation is difficult due to subjective nature of many aspects of output quality. |
| Approach: | They propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists, which are created in a preliminary step and then used as a common point of reference during quality assessment. |
| Outcome: | The proposed protocol shows that the evaluations produced in the study are more objective than the original human note. |
Multi-Document Summarization with Centroid-Based Pretraining (2023.acl-short)
Copied to clipboard
| Challenge: | In Multi-Document Summarization, the input is a set of documents, and the output is its summary. |
| Approach: | They propose a novel pretraining objective that uses the ROUGE-based centroid of each document cluster as a proxy for its summary. |
| Outcome: | The proposed model is better or comparable to state-of-the-art models. |
MULSUM: A Multimodal Summarization System with Vis-Aligner and Diversity-Aware Image Selection (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing systems that condense text and images into concise, faithful digests are inefficient and require large fusion transformers. |
| Approach: | They propose a framework that uses image embeddings to generate a visually informed text summary and a Diversity-Aware Image Selector to maximize images-relevance to the summary. |
| Outcome: | The proposed framework outperforms baselines on automatic metrics such as ROUGE and human evaluation shows that selected images act as explanatory evidence rather than ornamental add-ons. |
Evaluating Question Answering Evaluation (D19-58)
Copied to clipboard
| Challenge: | Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks. |
| Approach: | They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA . |
| Outcome: | The proposed BERTScore metric fails to provide stronger correlation with human judgements . |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |
GREEN: Generative Radiology Report Evaluation and Error Notation (2024.findings-emnlp)
Copied to clipboard
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Md, Michael Moseley, Curtis Langlotz, Akshay Chaudhari, Jean-Benoit Delbrouck
| Challenge: | Existing automated evaluation metrics fail to consider factual correctness or are limited in their interpretability. |
| Approach: | They propose a radiology report evaluation metric that leverages natural language understanding of language models to identify and explain clinically significant errors. |
| Outcome: | The proposed method demonstrates higher correlation with expert error counts and higher alignment with expert preferences when compared to previous methods. |
string2string: A Modern Python Library for String-to-String Algorithms (2024.acl-demos)
Copied to clipboard
| Challenge: | Notable algorithms include the Smith-Waterman algorithm for pairwise local alignment, the Hirschberg algorithm for global alignment, and the Wagner-Fischer algorithm for edit distance. |
| Approach: | **string2string** is an open-source library that offers efficient algorithms for string-to-string problems. |
| Outcome: | **string2string** is an open-source library that offers efficient algorithms for string-to-string problems. |
Evaluating the Effects of Embedding with Speaker Identity Information in Dialogue Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for automatic dialogue summarization do not take into account speaker identity information, but instead use sinusoidal functions to embed speaker information at the less informative part of the position embedding. |
| Approach: | They propose to embed speaker identity information into a dialogue transcript encoder to address this issue and reduce the "who said what"-related errors. |
| Outcome: | The proposed method improves the convergence of the model in training and increases the average ROUGE scores of the generated summaries in comparison to existing methods. |
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
RadEval: A framework for radiology text evaluation (2025.emnlp-demos)
Copied to clipboard
Justin Xu, Xi Zhang, Javid Abderezaei, Julie Bauml, Roger Boodoo, Fatemeh Haghighi, Ali Ganjizadeh, Eric Brattain, Dave Van Veen, Zaiqiao Meng, David W Eyre, Jean-Benoit Delbrouck
| Challenge: | Evaluating automated radiology report generation systems remains a fundamental challenge in the development of safe, accurate, and clinically useful medical AI. |
| Approach: | They propose a unified, open-source framework for evaluating radiology texts that consolidates a diverse range of metrics from classic ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (GREEN). |
| Outcome: | The framework consolidates a diverse range of metrics from ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1) and advanced LLMbased evaluators (GREEN). |
VizSeq: a visual analysis toolkit for text generation tasks (D19-3)
Copied to clipboard
| Challenge: | Several softwares for text evaluation are available that do not provide detailed examples. |
| Approach: | They propose a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks. |
| Outcome: | The proposed toolkit covers most common n-gram metrics and latest embedding-based metrics such as BERTScore. |
Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues (2024.findings-eacl)
Copied to clipboard
| Challenge: | blending multiple languages within a single conversation presents a formidable challenge, given the wide-ranging variations influenced by individual speaking styles and cultural backgrounds. |
| Approach: | They propose a novel approach to harness the Big Five personality traits acquired in an unsupervised manner from code-mixed conversations to bolster the performance of response generation. |
| Outcome: | The proposed approach enhances contextual relevance and performance of the proposed model by combining personality traits with dialogue context. |
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing text overlap based evaluation metrics are limited to matching tokens, either lexically or via embeddings. |
| Approach: | They propose a metric to evaluate the content quality of a summary using question-answering (QA) QA-based methods directly measure a summary’s information overlap with a reference, making them fundamentally different from text overlap metrics. |
| Outcome: | The proposed metric outperforms current state-of-the-art metrics on most evaluations using benchmark datasets while being competitive on others due to limitations of state- of-the art models. |
Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations (2022.aacl-main)
Copied to clipboard
Francesco Dalla Serra, William Clackett, Hamish MacKinnon, Chaoyang Wang, Fani Deligianni, Jeff Dalton, Alison Q. O’Neil
| Challenge: | Existing approaches to generate text radiology reports are prone to errors and poor clinical accuracy. |
| Approach: | They propose a two-step pipeline that subdivides the problem into factual triple extraction followed by free-text report generation. |
| Outcome: | The proposed pipeline shows that the generated reports exhibit realistic style but lack clinical accuracy. |
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator (2025.coling-industry)
Copied to clipboard
| Challenge: | Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. |
| Approach: | They propose a framework that uses a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. |
| Outcome: | The proposed framework achieves high correlation with human judgment and a consistent rating and adaptability to custom error guidelines. |
SEM-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. |
| Approach: | They propose to use a sentence-level precision-recall style automated evaluation metric to evaluate a new NLP task called Semantic Overlap Summarization (SOS) they propose to employ the popular ROUGE metric and use it to compare the two tasks. |
| Outcome: | The proposed metric yields higher correlation with human judgment and higher inter-rater agreement compared to the existing metric. |
FineSurE: Fine-grained Summarization Evaluation using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text summarization evaluation do not correlate well with human judgments . evaluators that use Likert scale scores are limited in their ability to perform deeper analysis. |
| Approach: | They propose a fine-grained evaluator specifically tailored for the summarization task using large language models. |
| Outcome: | The proposed method improves on open-source and proprietary LLMs and shows better completeness and conciseness than existing methods. |
Liputan6: A Large-scale Indonesian Dataset for Text Summarization (2020.aacl-main)
Copied to clipboard
| Challenge: | Despite having the fourth largest speaker population in the world, 1 Indonesian is under-represented in NLP. |
| Approach: | They propose to use a large-scale Indonesian summarization dataset to test extractive and abstractive summarizing methods. |
| Outcome: | The proposed methods are compared with multilingual and monolingual BERT-based models. |
A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for summarization evaluations that approximate human judgments are lacking for accuracy and reliability. |
| Approach: | They propose methods for calculating confidence intervals and running hypothesis tests for correlations using bootstrapping and permutation. |
| Outcome: | The proposed methods show that the confidence intervals are wide, demonstrating high uncertainty in the reliability of automatic metrics. |
A Graph-theoretic Summary Evaluation for ROUGE (D18-1)
Copied to clipboard
| Challenge: | ROUGE is one of the most widely used evaluation metrics for text summarization. |
| Approach: | They propose to use ROUGE to evaluate summaries based on lexical and semantic similarities. |
| Outcome: | The proposed method improves ROUGE's correlation with human judgments by exploiting lexical and semantic similarities. |
Global Optimization under Length Constraint for Neural Text Summarization (P19-1)
Copied to clipboard
| Challenge: | GOLC increases the probabilities of generating summaries that have high evaluation scores within a desired length. |
| Approach: | They propose a global optimization method under length constraint for neural text summarization models. |
| Outcome: | The proposed method generates fewer overlength summaries while maintaining the fastest processing speed. |
White-Box Multi-Objective Adversarial Attack on Dialogue Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained transformers are popular in state-of-the-art dialogue generation systems . however, they are vulnerable to adversarial samples crafted by small and imperceptible perturbations. |
| Approach: | They propose a multi-objective attack method that balances two objectives: generation accuracy and length. |
| Outcome: | The proposed method significantly degrades state-of-the-art DG models with a higher success rate than traditional accuracy-based methods. |
Multi-Reward Reinforced Summarization with Saliency and Entailment (N18-2)
Copied to clipboard
| Challenge: | Abstractive text summarization is the task of compressing and rewriting a long document into a short summary while maintaining saliency, directed logical entailment, and non-redundancy. |
| Approach: | They propose a novel reward function for ROUGESal and Entail to improve abstractive summarization . they use a coverage-based reward function to combine ROUGE and En Tail . |
| Outcome: | The proposed method achieves state-of-the-art results on CNN/Daily Mail dataset and strong improvements in a test-only transfer setup on DUC-2002. |
Objective Function Learning to Match Human Judgements for Optimization-Based Summarization (N18-2)
Copied to clipboard
| Challenge: | In previous work on summarization, the objective function is based on ad-hoc assumptions about which quality aspects of a summary are relevant. |
| Approach: | They learn a summary-level scoring function including human judgments as supervision and automatically generated data as regularization. |
| Outcome: | The proposed method performs well across automatic and manual evaluations. |
Pruning Basic Elements for Better Automatic Evaluation of Summaries (N18-2)
Copied to clipboard
| Challenge: | Summarization studies work on increasing the scores that are given by automatic evaluation measures. |
| Approach: | They propose a simple but highly effective automatic evaluation measure of summarization, pruned Basic Elements. |
| Outcome: | The proposed measure outperforms ROUGE and BE in most cases and achieves highest correlation coefficient in TAC 2011 AESOP task. |
Rogue Scores (2023.acl-long)
Copied to clipboard
| Challenge: | Critical evaluation decisions and parameters are routinely omitted, making most reports irreproducible . Thousands of papers use nonstandard evaluation packages with software defects that produce incorrect scores. |
| Approach: | a systematic review of over two thousand papers using a popular metric called ROUGE finds errors . critical evaluation decisions and parameters are routinely omitted, making most reported scores irreproducible . a large number of ROUGEE model evaluation scores have been incorrectly computed . |
| Outcome: | a systematic review of over two thousand papers finds that ROUGE scores are incorrect . the metric is widely used in machine learning and is inconsistent with human evaluations . |
SKGSum: Structured Knowledge-Guided Document Summarization (2024.findings-acl)
Copied to clipboard
Qiqi Wang, Ruofan Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, Jiamou Liu, Xianda Zheng, Zijian Huang
| Challenge: | Existing summarization methods ignore the importance of summary structure, resulting in summaries that emphasize the most prominent information while omitting essential details from other sections. |
| Approach: | They propose a method that uses automatically extracted summary points to generate summaries. |
| Outcome: | The proposed methods improve quality and BERTScore of summaries and broaden the types of documents that can be effectively summarized. |
On the Evaluation of Vision-and-Language Navigation Instructions (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing instruction generators have not been evaluated using human wayfinders . BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions. |
| Approach: | They propose an instruction-trajectory compatibility model that operates without reference instructions to improve wayfinding performance. |
| Outcome: | The proposed model shows the highest correlation with human wayfinding outcomes when scoring individual instructions. |
The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for summarization systems measure semantic overlap between a system summary and a human reference on word-string level. |
| Approach: | They propose to use distributed representations to evaluate system summary and human reference on word-string level. |
| Outcome: | The proposed representations outperform ROUGE on recent corpora but are less good on test data used in previous studies. |
A Finer-grain Universal Dialogue Semantic Structures based Model For Abstractive Dialogue Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive summarization models have achieved impressive results on document summarizing tasks, but their performance on dialogue modeling is poor due to the crude and straight methods for dialogue encoding. |
| Approach: | They propose a model that leverages Finer-grain universal Dialogue semantic Structures to model dialogue and generate better summaries. |
| Outcome: | The proposed model outperforms various dialogue summarization approaches and achieves state-of-the-art (SOTA) ROUGE results on a SAMsum dataset. |
DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization (2022.acl-long)
Copied to clipboard
Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, Dragomir Radev
| Challenge: | Existing models struggle with summarizing long text due to high memory complexity of the full self-attention. |
| Approach: | They propose a dynamic latent extraction approach for abstractive long-input summarization that treats extracted text snippets as latent variables and allows dynamic attention weights during decoding. |
| Outcome: | The proposed method outperforms existing methods on GovReport, QMSum, and arXiv while yielding strong results on arX. |
A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding (2021.acl-long)
Copied to clipboard
Khalil Mrini, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter Chang, Emilia Farcas, Ndapa Nakashole
| Challenge: | Existing methods for medical question understanding often fail to provide high recall in answer retrieval. |
| Approach: | They propose a multi-task learning method with data augmentation for medical question understanding that uses just one dataset to optimize for both tasks. |
| Outcome: | The proposed method outperforms existing MTL methods across 4 datasets of medical question pairs in ROUGE scores, RQE accuracy and human evaluation. |
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate text summarization tasks using ROUGE have been criticized for lack of semantic understanding. |
| Approach: | They propose a semantic-aware metric for extractive summarization task that is semantic-based . they use CNN/DailyMail dataset to study the new metric . |
| Outcome: | The proposed metric is semantic-aware and shows higher correlation with human judgement and yields a large number of disagreements with the original ROUGE metric. |
From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for large language models yield numerical scores that ignore user experience. |
| Approach: | They propose a metric that suggests revision edits that mimic the human writing process . their results show that the metric offers more insightful feedback and distinguishes between texts . |
| Outcome: | The proposed metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. |
Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses (2025.emnlp-industry)
Copied to clipboard
| Challenge: | DeCE is model-agnostic and domain-general, requiring no predefined taxonomies or handcrafted rubrics. |
| Approach: | They propose a decomposed LLM evaluation framework that separates accuracy and recall from accuracy and relevance. |
| Outcome: | The proposed framework achieves stronger correlation with expert judgments than traditional metrics and pointwise LLM scoring. |
CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs (2023.acl-long)
Copied to clipboard
| Challenge: | a large-scale cross-lingual summarization dataset is available for free . a cross-linguistic summarizing model can be trained in any target language . |
| Approach: | They propose a multistage data sampling algorithm to train a cross-lingual summarization model capable of summarizing an article in any target language. |
| Outcome: | The proposed model outperforms baseline models on ROUGE and LaSE. |
A Timestep aware Sentence Embedding and Acme Coverage for Brief but Informative Title Generation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for title generation are based on timestep aware sentence embeddings, but they are not effective for generating a title with appropriate information in the content. |
| Approach: | They propose a Timestep aware Sentence Embedding mechanism which refreshes the sentences’ embeddings with corresponding key words in different decoding timesteps. |
| Outcome: | The proposed framework outperforms existing methods on various title generation tasks and the evaluation scores are significantly higher than previous approaches. |
Generating EDU Extracts for Plan-Guided Summary Re-Ranking (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to generate summary candidates for re-ranking produce redundant, and often low quality, content. |
| Approach: | They propose a method to generate candidates for re-ranking that addresses these issues by grounding each abstract on its own unique content plan and creating distinct plan-guided abstracts using a model's top beam. |
| Outcome: | The proposed method outperforms baseline decoding methods on CNN, NYT, and Xsum and shows that prompting GPT-3 to follow EDU plans outperformed sampling-based methods by 1.05 points. |
Modeling Exemplification in Long-form Question Answering via Retrieval (2022.naacl-main)
Copied to clipboard
| Challenge: | Exemplification is a process by which writers explain or clarify a concept by providing an example. |
| Approach: | They propose to use a partially-written answer to query a large set of human-written examples extracted from a corpus to determine exemplification quality. |
| Outcome: | The proposed model is able to retrieve human-written examples from a corpus and show that it is more relevant than state-of-the-art models. |
Estimating Summary Quality with Pairwise Preferences (N18-1)
Copied to clipboard
| Challenge: | Existing evaluation systems rely on gold standard summaries but they are expensive and require the availability of experts to achieve high quality. |
| Approach: | They propose an alternative evaluation approach based on pairwise preferences of sentences to provide useful feedback in the form of pairwise preference. |
| Outcome: | The proposed evaluation framework performs better than the three most popular versions of ROUGE with less expensive human input. |
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks. |
| Approach: | They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs. |
| Outcome: | The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments. |
Which Scores to Predict in Sentence Regression for Text Summarization? (N18-1)
Copied to clipboard
| Challenge: | Sentence regression is an emerging branch in automatic text summarizations. |
| Approach: | They propose to estimate the importance of information via learned utility scores for individual sentences. |
| Outcome: | The proposed models learn to predict ROUGE recall scores of individual sentences . the models show that following intuition leads to suboptimal results . |
Accuracy is not enough: Evaluating Personalization in Summarizers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing accuracy measures cannot evaluate the degree of personalization of summarization models. |
| Approach: | They propose to use a PENS dataset to analyze the degree of personalization of ten different summarization models. |
| Outcome: | The proposed measure can evaluate the degree of personalization of summarization models using the PENS dataset. |
How Far are We from Robust Long Abstractive Summarization? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization has made tremendous progress in recent years . however, even under a short document setting, abstractive models often generate summaries that are repetitive, ungrammatical, and factually inconsistent with the source. |
| Approach: | They perform fine-grained human annotations to evaluate long document abstractive summarization systems and develop factual consistency metrics. |
| Outcome: | The proposed model can generate more relevant summaries but not factual ones. |
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on automatic summary evaluation metrics focus on lexical similarity and require a reference summary which is expensive to obtain. |
| Approach: | They propose to use a weakly supervised summary evaluation approach without the presence of reference summaries to transform existing summarization datasets into corrupted reference summarizers. |
| Outcome: | The proposed method outperforms baselines and shows that it improves linguistic quality over all metrics. |
Leveraging Language Models for Summarizing Mental State Examinations: A Comprehensive Evaluation and Dataset Release (2025.coling-main)
Copied to clipboard
| Challenge: | Mental health disorders affect a significant portion of the global population . access to mental health support is limited in developing countries . |
| Approach: | They evaluated a 12-item descriptive MSE questionnaire and five well-known summarization models . they found that language models can generate coherent MSE summaries for doctors . |
| Outcome: | The proposed model can generate coherent summaries from MSEs in a conversational format. |
GEAR: A Scalable and Interpretable Evaluation Framework for RAG-Based Car Assistant Systems (2025.emnlp-industry)
Copied to clipboard
Niloufar Beyranvand, Hamidreza Dastmalchi, Aijun An, Heidar Davoudi, Winston Chan, Ron DiCarlantonio
| Challenge: | Large language models (LLMs) increasingly power car assistants, but evaluating response quality remains a challenge. |
| Approach: | They propose a framework that uses large language models as evaluators to compare assistant responses against ground-truth counterparts. |
| Outcome: | The proposed framework compares assistant responses against ground-truth counterparts, assessing coverage, correctness, and other dimensions of answer quality. |
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021.eacl-main)
Copied to clipboard
| Challenge: | a lack of transparency is a key issue for robotics and AI. |
| Approach: | They propose to map existing automatic evaluation methods for natural language generation onto explanations. |
| Outcome: | The proposed model shows that embedding-based evaluation methods have higher correlations with human ratings than word-overlap metrics. |
Reducing Quantity Hallucinations in Abstractive Summarization (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive summaries are subject to hallucination, but they are not very informative. |
| Approach: | They propose to use a beam-worth of abstractive summaries to up-rank summary that is not supported by the original text. |
| Outcome: | The proposed system up-ranks summaries whose quantity terms are supported by the original text without losing Recall, and shows higher Precision. |
Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing models for extractive summarization of meetings are unfocused and lack content coverage. |
| Approach: | They propose a multi-modal hierarchical attention model that prioritizes segmentation and summarization . they propose to use multi-level hierarchies to narrow down the focus into topically-relevant segments . |
| Outcome: | The proposed model outperforms the state-of-the-art with BLEU and ROUGE measures. |
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent generative language models have shown promise in abstractive summarization tasks. |
| Approach: | They propose to use Fr echet embedding distance and angular embeddable similarity to evaluate the performance of generative language models in abstractive summarization tasks. |
| Outcome: | The proposed metric shows close relation with human judgments and has overall better correlations with them. |
Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to exploit PrLMs for NLG tasks do not get as much performance gain as in the NLU task. |
| Approach: | They propose a method to integrate public checkpoints of PrLMs for the most convenience. |
| Outcome: | The proposed method significantly improves the quality of the language generation tasks on 6 different kinds of PrLMs. |
Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision (2022.coling-1)
Copied to clipboard
Khalil Mrini, Harpreet Singh, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter W. Chang, Emilia Farcas, Ndapa Nakashole
| Challenge: | Current medical question answering systems have difficulty processing long, detailed and informally worded questions . a growing number of approaches attempt to enhance the processing of consumer health questions - or medical question understanding . |
| Approach: | They propose a medical question understanding and answering system with knowledge grounding and semantic self-supervision that matches a user question with a trusted medical knowledge base and retrieves a fixed number of relevant sentences from the corresponding answer document. |
| Outcome: | The proposed system retrieves more relevant answers while achieving 20 times faster. |
Reinforced Multi-task Approach for Multi-hop Question Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Empirical evaluation shows our model to outperform the single-hop question generation models on both automatic evaluation metrics such as BLEU, METEOR, and ROUGE and human evaluation metrics for quality and coverage of the generated questions. |
| Approach: | They propose a question-aware reward function to maximize the utilization of supporting facts in the context. |
| Outcome: | The proposed model outperforms single-hop neural question generation models on automatic evaluation metrics and human evaluation metrics for quality and coverage of the generated questions. |
Argument Generation with Retrieval, Planning, and Realization (P19-1)
Copied to clipboard
| Challenge: | a novel argument generation framework is used to generate counter-arguments . CANDELA uses a text planning decoder to retrieve arguments of different perspectives . |
| Approach: | They propose a powerful retrieval system and a novel two-step argument generation framework . they use a retrieval-based retrieval platform indexed with 12 million articles from Wikipedia . |
| Outcome: | The proposed framework yields higher BLEU, ROUGE, and METEOR scores than state-of-the-art models. |
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, Noah A. Smith
| Challenge: | Recent advances on models and metrics should benefit and inform each other, authors argue . bidimensional leaderboards allow for fast, accurate evaluation of language generation models . |
| Approach: | They propose a bidimensional leaderboard that tracks progress in language generation models and metrics for their evaluation. |
| Outcome: | The proposed leaderboards track progress in language generation models and metrics for their evaluation. |
Narrate Dialogues for Better Summarization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on dialogue summarization models focuses on generating concise summaries for multi-party dialogues. |
| Approach: | They propose several ways to convert dialogue into a third-person narrative style . they propose to use narration as a valuable annotation for LLMs . |
| Outcome: | Empirical results show that the proposed approach achieves higher scores on ROUGE and a factual correctness metric. |
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing text decoding methods struggle to produce high-quality text . Greedy and beam search suffer from text degeneration and linguistic diversity issues . |
| Approach: | They propose a family of decoding methods based on minimum bayes risk minimization to address diversity-quality trade-offs in open-ended natural-language generation. |
| Outcome: | The proposed methods improve diversity-quality trade-offs on open-ended natural-language generation tasks. |
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts (P19-1)
Copied to clipboard
| Challenge: | Existing automatic metrics for evaluating text are expensive and time-consuming. |
| Approach: | They propose automatic metrics that evaluate text in a continuous space using word and sentence embeddings. |
| Outcome: | The proposed method outperforms ROUGE on machine-generated summaries and human-authored essays on human-generated texts. |
Summarizing Patients’ Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models (2022.coling-1)
Copied to clipboard
| Challenge: | Problem list summarization requires a model to understand, abstract, and generate clinical documentation. |
| Approach: | They propose a task that summarises patients' main problems from daily progress notes using input from the provider's progress notes during hospitalization. |
| Outcome: | The proposed model outperforms two state-of-the-art seq2seq transformer architectures in summarizing patients' main problems from daily progress notes in the medical information mart for Intensive Care (MIMIC)-III. |
ACE-M3: Automatic Capability Evaluator for Multimodal Medical Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing metrics for multimodal large language models only focus on token overlap and may not align with human judgment. |
| Approach: | They propose an open-source model that assesses the question answering abilities of multimodal large language models. |
| Outcome: | Experiments show that the ACE-M3 model performs better than existing models and is more reliable than existing metrics. |
CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluation methods do not capture complex attributes of counterspeech quality, such as contextual relevance, aggressiveness, or argumentative coherence. |
| Approach: | They propose to use a dataset and framework to evaluate counterspeech quality across four dimensions: contextual relevance, aggressiveness, argument-coherence, and suitability. |
| Outcome: | The proposed method outperforms ROUGE, METEOR, and BertScore in correlating with human judgement, indicating a significant improvement in automated counterspeech evaluation. |
Geo-Aware Image Caption Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Standard image caption generation systems do not take contextual information or world knowledge into account. |
| Approach: | They propose to build an image-specific representation of the geographic context and adapt the caption generation network to produce appropriate geographic names in the image descriptions. |
| Outcome: | The proposed system achieves significant improvements on a dataset that contains contextualized captions and geographic metadata and improves BLEU, ROUGE, METEOR and CIDEr scores. |
Generative Pretraining for Paraphrase Evaluation (2022.acl-long)
Copied to clipboard
| Challenge: | ParaBLEU is a paraphrase representation learning model and evaluation metric for text generation. |
| Approach: | They propose a paraphrase representation learning model and evaluation metric for text generation that uses generative conditioning as a pretraining objective. |
| Outcome: | The proposed model outperforms existing models on the 2017 WMT Metrics Shared Task using only 50% of the available training data and surpasses BLEU, ROUGE and METEOR with only 40 examples. |
Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document summarization consider the informativeness of the assessed summary and require human-generated references for each test summary. |
| Approach: | They propose to evaluate summary qualities without reference summaries by unsupervised contrastive learning. |
| Outcome: | The proposed method outperforms other evaluation metrics even without reference summaries. |
To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text (2021.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models have seen great improvements in recent years, but there is limited understanding of the strategies different models employ and how they relate their understanding of language. |
| Approach: | They characterize how one popular abstractive model uses an explicit copy/generation switch to control its level of abstraction vs extraction . they find that abstractive summarization models lack the semantic understanding necessary to generate paraphrases that are both abstractive and faithful to the source document. |
| Outcome: | The proposed model uses syntactic boundaries to truncate sentences that are often copied verbatim. |
How well do you know your summarization datasets? (2021.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art summarization systems are trained on massive datasets scraped from the web. |
| Approach: | They manually analyse 600 samples from three popular summarization datasets . they use a six-class typology which captures different noise types and degrees of summarizing difficulty. |
| Outcome: | The proposed model performs better on large datasets than on the current models. |
Better Rewards Yield Better Summaries: Learning to Summarise Without References (D19-1)
Copied to clipboard
| Challenge: | Reinforcement Learning (RL)-based document summarisation systems produce state-of-the-art performance in terms of ROUGE scores, but high summaries receive low human judgement. |
| Approach: | They propose to learn a reward function from human ratings on 2,500 summaries to generate human-appealing summary. |
| Outcome: | The proposed reward function can generate human-appealing summaries without reference summary input. |
Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations (2021.findings-emnlp)
Copied to clipboard
Longxiang Zhang, Renato Negrinho, Arindam Ghosh, Vasudevan Jagannathan, Hamid Reza Hassanzadeh, Thomas Schaaf, Matthew R. Gormley
| Challenge: | Using pretrained transformer models for automatically summarizing doctor-patient conversations presents challenges . limited training data, domain shift, long and noisy transcripts, and high target summary variability are challenges compared to human annotators. |
| Approach: | They propose a method for fine-tuning pretrained transformer models for automatically summarizing doctor-patient conversations directly from transcripts. |
| Outcome: | The proposed method surpasses the performance of an average human annotator and the quality of previous published work for the task. |
AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation (2025.acl-long)
Copied to clipboard
Xiechi Zhang, Zetian Ouyang, Linlin Wang, Gerard De Melo, Zhu Cao, Xiaoling Wang, Ya Zhang, Yanfeng Wang, Liang He
| Challenge: | Existing evaluation methods based on large language models (LLMs) are expensive and lack expertise due to limitations in human expertise. |
| Approach: | They propose an open-source automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs. |
| Outcome: | The proposed model surpasses baselines in terms of correlation with human judgments. |
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (D19-1)
Copied to clipboard
| Challenge: | Abstractive summarization approaches based on Reinforcement Learning (RL) have been proposed to overcome classical likelihood maximization. |
| Approach: | They propose to use Reinforcement Learning to learn the model parameters through RL techniques to overcome classical likelihood maximization. |
| Outcome: | The proposed measures favor ROUGE with the additional property of not requiring reference summaries. |
An Entity-Driven Framework for Abstractive Summarization (D19-1)
Copied to clipboard
| Challenge: | Popular neural summarization models produce incoherent and unfaithful summaries . however, their outputs are often incohérent and incoerent . |
| Approach: | They propose a system for ENtity-drivEn Coherent Abstractive summarization framework that leverages entity information to generate informative and coherent abstracts. |
| Outcome: | The proposed framework outperforms existing state-of-the-art models on New York Times and CNN/Daily Mail datasets. |
Neural Extractive Text Summarization with Syntactic Compression (D19-1)
Copied to clipboard
| Challenge: | Recent approaches to summarization are either selection-based extraction or generation-based abstraction. |
| Approach: | They propose a neural model for single-document summarization based on joint extraction and syntactic compression. |
| Outcome: | The proposed model outperforms an off-the-shelf compression module and its output generally remains grammatical. |
Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sentence extractive summarization shortens a document by selecting sentences for a summary while preserving its important contents. |
| Approach: | They propose a nested tree-based extractive summarization model on RoBERTa that uses syntactic and discourse trees to represent sentences in a given document. |
| Outcome: | The proposed model outperforms baseline models on the CNN/DailyMail dataset and achieves significantly better scores than the baseline models in terms of coherence and comparable scores to the state-of-the-art models. |
Identifying Reliable Evaluation Metrics for Scientific Text Revision (2025.acl-long)
Copied to clipboard
| Challenge: | Effective revision is a critical step in scientific writing, ensuring clarity, coherence, and adherence to academic standards. |
| Approach: | They propose to use ROUGE and BERTScore to assess revision quality . they also examine LLM-as-a-judge approaches to assess instruction-following revisions . |
| Outcome: | The proposed method improves the accuracy of revision tasks with and without a gold reference. |
Transformer-based Lexically Constrained Headline Generation (2021.emnlp-main)
Copied to clipboard
Kosuke Yamada, Yuta Hitomi, Hideaki Tamori, Ryohei Sasano, Naoaki Okazaki, Kentaro Inui, Koichi Takeda
| Challenge: | Existing automatic headline generation methods cannot include a given phrase in the generated headline. |
| Approach: | They propose a Transformer-based method that guarantees to include a given phrase in a generated headline. |
| Outcome: | The proposed method achieves ROUGE scores comparable to previous methods with Japanese news corpus. |
A Multi-Document Coverage Reward for RELAXed Multi-Document Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Multi-document summarization models are limited by limited references and with maximum-likelihood objectives. |
| Approach: | They propose to fine-tune an MDS baseline with a reward that balances a reference-based metric such as ROUGE with coverage of the input documents. |
| Outcome: | The proposed model improves on the Multi-News and WCEP datasets with a low-variance estimator . the proposed model also improves the coverage of the input documents . |
Diverse dialogue generation with context dependent dynamic loss function (2020.coling-main)
Copied to clipboard
| Challenge: | Dialogue systems using deep learning have achieved generation of fluent response sentences to user utterances, but they tend to produce responses that are not diverse and less context-dependent. |
| Approach: | They propose an Inverse N-gram loss function which incorporates contextual fluency and diversity at the same time by a simple formula. |
| Outcome: | The proposed loss function outperforms baseline models in automatic evaluations such as DIST-N and ROUGE and achieves higher scores on human evaluations of coherence and richness. |
Active Learning for Abstractive Text Summarization (2022.findings-emnlp)
Copied to clipboard
Akim Tsvigun, Ivan Lysenko, Danila Sedashov, Ivan Lazichny, Eldar Damirov, Vladimir Karlov, Artemy Belousov, Leonid Sanochkin, Maxim Panov, Alexander Panchenko, Mikhail Burtsev, Artem Shelmanov
| Challenge: | Abstractive text summarization (ATS) requires a long document and short summaries. |
| Approach: | They propose a query strategy for AL in abstractive text summarization that uses uncertainty estimation to reduce model performance. |
| Outcome: | The proposed query strategy improves ROUGE and consistency scores for annotated datasets . it also increases the performance of the model, compared to passive annotation. |
GSum: A General Framework for Guided Neural Abstractive Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Abstractive summarization models are flexible, but they can be difficult to control. |
| Approach: | They propose a general and extensible guided summarization framework that takes different kinds of guidance as input and perform experiments across different varieties. |
| Outcome: | The proposed framework can generate more faithful summaries and different types of guidance generate qualitatively different summary. |
Logic-Consistency Text Generation from Semantic Parses (2021.findings-acl)
Copied to clipboard
| Challenge: | Text generation from semantic parses is challenging due to the complexity of the inner logic and the lack of automatic evaluation metrics for logic consistency. |
| Approach: | They propose a framework for logic consistent text generation from semantic parses that employs iterative training procedures and quality control. |
| Outcome: | The proposed framework enhances logic consistency and human evaluation on two benchmark datasets. |
Question Answering as an Automatic Evaluation Metric for News Article Summarization (N19-1)
Copied to clipboard
| Challenge: | Recent work on summarization and headline generation focuses on maximizing ROUGE scores. |
| Approach: | They propose an extrinsic evaluation metric that maximizes ROUGE scores for automatic summarization and headline generation. |
| Outcome: | The proposed model maximizes ROUGE scores while increasing competitive results. |
CaLcs: Continuously Approximating Longest Common Subsequence for Sequence Level Optimization (D18-1)
Copied to clipboard
| Challenge: | Maximum-likelihood estimation (MLE) is widely used for text-generation based natural language processing applications. |
| Approach: | They propose a method to train models with maximum-likelihood estimation using a differentiable surrogate of longest common subsequence measure that captures sequence-level structure similarity. |
| Outcome: | Experimental results show that the proposed approach improves on the current MLE approach for downstream tasks like text summarization and machine translation. |
LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Non-factoid (NF) question answering is challenging to evaluate due to diverse potential answers and no objective criterion. |
| Approach: | They propose a listwise NFQA evaluation approach that uses Large Language Models to rank candidate answers in a descending list of reference answers sorted by descending quality. |
| Outcome: | The proposed method has higher correlations with human annotations than standard methods. |
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)
Copied to clipboard
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev
| Challenge: | Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. |
| Approach: | They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries. |
| Outcome: | The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets. |
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022.naacl-main)
Copied to clipboard
| Challenge: | Current models for dialogue summarization have flaws that may not be well exposed by frequently used metrics such as ROUGE. |
| Approach: | They propose to re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models for the first time. |
| Outcome: | The proposed dataset will be used to evaluate 18 categories of metrics in terms of coherence, consistency, fluency and relevance, and a unified human evaluation of various models for the first time. |
Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to detect false claims ignore the characteristics of FC-articles . claims are often quoted to describe checked events, providing lexical information . sentence templates to introduce or debunk claims are common across articles, providing pattern information. |
| Approach: | They propose a model to rerank FC-articles using key sentences and pattern information. |
| Outcome: | The proposed model outperforms existing methods on two real-world datasets showing that key sentences can be used to predict if an article fact-checks the given claim. |
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (2023.findings-acl)
Copied to clipboard
Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, Marzieh Saeidi
| Challenge: | Existing metrics for evaluating the quality of automatically generated questions are expensive and penalise valid questions that may not have high lexical or semantic similarity to the reference questions. |
| Approach: | They propose a question-answering and span scorer metric based on the answerability of the candidate question given the context. |
| Outcome: | The proposed metric has higher correlation with human judgment without relying on the reference question. |
Multimodal Differential Network for Visual Question Generation (D18-1)
Copied to clipboard
| Challenge: | Current dialog systems show improvement in visual question answering but this does not translate to improved human-AI dialog. |
| Approach: | They propose to use a Multimodal Differential Network to generate natural questions from images using a multimodal differential network. |
| Outcome: | The proposed approach significantly improves over state-of-the-art benchmarks on the quantitative metrics. |
Inverse Reinforcement Learning for Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that inverse reinforcement learning (RL) training has certain disadvantages such as object mismatch and exposure bias. |
| Approach: | They propose inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models. |
| Outcome: | The proposed model outperforms MLE and RL baselines on ROUGE, coverage, novelty, compression ratio, factuality, and human evaluations. |
Closed-Book Training to Improve Summarization Encoder Memory (D18-1)
Copied to clipboard
| Challenge: | Extractive summarization models pick words, phrases, and sentences from the source text to form a summary, while abstractive models generate (samples) words from a fixed-size vocabulary instead of copying from text directly. |
| Approach: | They propose to add an additional ‘closed-book’ decoder without attention and pointer mechanisms to the model to improve its memorization capabilities. |
| Outcome: | The proposed model outperforms the baseline model on the CNN/Daily Mail dataset in terms of ROUGE and METEOR metrics, and achieves higher scores in a test-only setup. |
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing definitions of system-level correlations are inconsistent with how they are used to evaluate systems. |
| Approach: | They propose to calculate correlations only on pairs of systems separated by small differences in automatic scores . they propose to use the full test set instead of the subset of summaries judged by humans . |
| Outcome: | The proposed changes improve the accuracy of the estimated correlations on pairs of systems separated by small differences in automatic scores. |
Bottom-Up Abstractive Summarization (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to summarize text using end-to-end content selectors have had mixed success in content selection, for example copying full sentences from the source document. |
| Approach: | They propose to use content selectors to over-determine phrases in a source document that should be part of the summary. |
| Outcome: | The proposed model over-determines phrases in a source document that should be part of the summary while generating fluent summaries. |
Facet-Aware Evaluation for Extractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | lexical overlap is a common evaluation metric for extractive summarization, but recent studies reveal its limitations. |
| Approach: | They propose a facet-aware evaluation setup for better assessment of information coverage in extractive summaries. |
| Outcome: | The proposed evaluation setup improves human correlation with extractive summarization datasets and improves comparative analysis. |
A Modular Approach for Multimodal Summarization of TV Shows (2024.acl-long)
Copied to clipboard
| Challenge: | In this paper, we address the task of summarizing television shows, which touches key areas in AI research. |
| Approach: | They propose a modular approach where separate components perform specialized sub-tasks . they propose atomic facts to measure precision and recall of generated summaries . |
| Outcome: | The proposed method produces higher quality summaries than comparison models on a recently released dataset. |
MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes (2022.acl-long)
Copied to clipboard
| Challenge: | MemSum is a reinforcement-learning-based extractive summarizer that considers the text content of the sentence, the global context of the rest of the document, and the extraction history of the sentences that have already been extracted. |
| Approach: | They propose a reinforcement-learning-based extractive summarizer that iteratively selects sentences from a broad set of information that would intuitively be used by humans. |
| Outcome: | The proposed extractive summarizer is enriched with information on the extraction history and local, global, and historical information. |
Enhancing Argument Summarization: Prioritizing Exhaustiveness in Key Point Generation and Introducing an Automatic Coverage Evaluation Metric (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for summarizing arguments are incapable of distinguishing between generated key points of different qualities. |
| Approach: | They propose an extractive approach that generates concise, high quality key points . they propose to use a clustering approach to generate key points from raw arguments . |
| Outcome: | The proposed method outperforms state-of-the-art methods for key point generation . it offers concise, high quality generated key points with higher coverage of reference summaries . |
Improving Accessibility of SCOTUS Opinions: A Benchmark Study and a New Dataset for Generic Heading Prediction and Specific Heading Generation (2025.coling-main)
Copied to clipboard
| Challenge: | SCOTUS opinions are notoriously long and use specialised language, making them laborious to read and understand. |
| Approach: | They propose generic and specific headings for each section to be trained automatically . they compare the performance of different systems trained for each subtask . |
| Outcome: | The proposed system can achieve a score of 0.90% in predicting generic headings . the proposed system also achieves similar scores in generating specific headings. |
Joint Enhancement of Relational Reasoning for Long-Context LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | JERR is a graph-based reasoning framework for large language models . it enables LLMs to handle extended contexts with improved reliability and transparency . |
| Approach: | They propose a graph-based reasoning framework that integrates synopsis extraction, graph construction, and relational reasoning. |
| Outcome: | The proposed framework outperforms baselines on ROUGE and F1 metrics and achieves the highest scores on the LLM-Rater evaluation. |
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are slow and abstractive, but we propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries. |
| Approach: | They propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries by exposing diverse pseudo summary with two supervision to the student model. |
| Outcome: | The proposed method outperforms previous methods in abstractive summarization distillation, producing highly abstractive and informative summaries. |
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, document summarization is challenging even for humans. |
| Approach: | They propose a focus attention mechanism which encourages decoders to generate tokens that are topically similar to the input document. |
| Outcome: | The proposed method outperforms top-k and nucleus sampling methods on the BBC extreme summarization task and is more accurate than focus attention-based models. |
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (P19-1)
Copied to clipboard
| Challenge: | Existing text generation metrics rely on reference texts, such as BLEU and ROUGE, but they are too expensive to apply repeatedly. |
| Approach: | They propose a metric which aligns n-grams from the generated texts to the semi-structured data before computing their precision and recall. |
| Outcome: | The proposed metric correlates with human judgments better than existing text generation metrics while being easier to use. |
HARE: an entity and relation centric evaluation framework for histopathology reports (2025.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the clinical quality of medical domain automated text generation remains a challenge. |
| Approach: | They propose a framework for histopathology automated report evaluation that prioritizes clinically relevant content by aligning critical histo pathology entities and relations between reference and generated reports. |
| Outcome: | The proposed framework outperforms existing metrics in histopathology report evaluations. |
Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization (2020.coling-main)
Copied to clipboard
Dongyub Lee, Myeong Cheol Shin, Taesun Whang, Seungwoo Cho, Byeongil Ko, Daniel Lee, EungGyun Kim, Jaechoon Jo
| Challenge: | Existing methods for text summarization are based on recall-oriented understudy for gisting evaluation (ROUGE) scores do not reflect semantic meaning correspondences between generated and reference summaries. |
| Approach: | They propose to use Korean as a summarization language to generate a shorter form of text from the source document preserving salient information. |
| Outcome: | The proposed evaluation metrics improve the correlation between the metrics and human judgment. |
Autoregressive Knowledge Distillation through Imitation Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive models are ubiquitous in natural language processing due to the sequential nature of text generation. |
| Approach: | They propose a compression technique for autoregressive models driven by an imitation learning perspective on knowledge distillation. |
| Outcome: | The proposed method outperforms other distillation algorithms on translation and summarization tasks while increasing inference speed 14 times. |
Controllable Abstractive Sentence Summarization with Guiding Entities (2020.coling-main)
Copied to clipboard
| Challenge: | Existing text summarization models lack guiding entities to ensure that entities are present in summaries. |
| Approach: | They propose a controllable abstractive sentence summarization model which generates summaries with guiding entities. |
| Outcome: | The proposed model outperforms the state-of-the-art models in evaluation scores and informativeness metrics. |
HOLMS: Alternative Summary Evaluation with Large Language Models (2020.coling-main)
Copied to clipboard
| Challenge: | Efficient document summarization requires evaluation measures that can rank a set of systems based on an average score and highlight which individual summary is better than another. |
| Approach: | They propose a hybrid evaluation measure for document summarization called HOLMS that combines both language models pre-trained on large corpora and lexical similarity measures. |
| Outcome: | The proposed measure outperforms ROUGE and BLEU on several extractive summarization datasets for both linguistic quality and pyramid scores. |
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (P19-1)
Copied to clipboard
| Challenge: | Neural extractive summarization models employ hierarchical encoders with inaccurate sentence-level labels. |
| Approach: | They propose a method to pre-train a hierarchical encoder with unlabeled data. |
| Outcome: | The proposed model outperforms its initialized counterpart by 1.25 ROUGE on CNN and 2.0 ROUGEE on a version of New York Times dataset. |
On the Importance of Diversity in Question Generation for QA (2020.acl-main)
Copied to clipboard
| Challenge: | In this paper, we show that textual diversity in automatic question generation is beneficial for downstream QA. |
| Approach: | They propose to use textual diversity to promote automatic question generation as a quality measure for QA. |
| Outcome: | The proposed measure of QG quality correlates well with evaluation on QA. |
An Anchor-Based Automatic Evaluation Metric for Document Summarization (2020.coling-main)
Copied to clipboard
| Challenge: | Existing reference-based evaluation metrics such as ROUGE have their own drawbacks. |
| Approach: | They propose a protocol for a reference-based automatic evaluation metric that requires the endorsement of source document. |
| Outcome: | The proposed metric is anchored on source document and has higher correlation with human judgments. |
A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications (2022.lrec-1)
Copied to clipboard
| Challenge: | Automated text generation has achieved performance levels that make the generated text almost indistinguishable from those written by humans. |
| Approach: | They propose to use a completely synthetic dataset and a partial text substitution dataset to evaluate the quality of the generated research content. |
| Outcome: | The proposed datasets compare the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE. |
Compressive Summarization with Plausibility and Salience Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new method to learn which compressions to apply is based on syntactic rules for deleting spans . plausibility and salience are the two main criteria for determining which compression to apply . a recent study shows that the plausability model generally selects for grammatical and factual deletions compared to extractive methods . |
| Approach: | They propose to leave the decision about what to delete to two data-driven criteria . they show that plausibility and salience are the most important criteria if a span is deleted . |
| Outcome: | The proposed method achieves strong in-domain results on benchmark datasets and human evaluation shows that plausibility model generally selects for grammatical and factual deletions. |
Language Model Augmented Relevance Score (2021.acl-long)
Copied to clipboard
| Challenge: | Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements. |
| Approach: | They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements. |
| Outcome: | The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree. |
Boosting Code Summarization by Embedding Code Structures (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work on code summarization relies on structural information from the abstract syntax tree (AST) of source codes. |
| Approach: | They propose a program dependency graph (PDG) that represents the structure of a code more effectively. |
| Outcome: | The proposed model improves the performance of an out-of-domain benchmark dataset and the measure SBERT score. |
QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)
Copied to clipboard
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari
| Challenge: | Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments. |
| Approach: | They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document. |
| Outcome: | The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions. |
EmailSum: Abstractive Email Thread Summarization (2021.acl-long)
Copied to clipboard
| Challenge: | Recent years have brought about interest in the task of summarizing conversation threads. |
| Approach: | They develop an email thread summarization dataset that contains human-annotated short and long email threads over a wide variety of topics. |
| Outcome: | The proposed dataset contains human-annotated short (30 words) and long (100 words) summaries of 2,549 email threads over a wide variety of topics. |
Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study (2022.coling-1)
Copied to clipboard
| Challenge: | Existing tasks for summarizing multiple alternate narratives with different perspectives are under-explored. |
| Approach: | They propose a task which entails generating a single summary from multiple alternative narratives . they use a web-based dataset and human annotations to evaluate the task . |
| Outcome: | The proposed task is based on a novel dataset and human annotations. |
MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods overlook the distinction between factoid and non-factoidic questions. |
| Approach: | They propose a method that distinguishes open-ended questions and ranks candidate answers . they propose QA requires longer answer statements and nuanced reasoning processes . |
| Outcome: | The proposed method better aligns with human annotations and offers more interpretable results. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |
SumeCzech: Large Czech News-Based Summarization Dataset (L18-1)
Copied to clipboard
| Challenge: | Summarization of documents is a well-studied NLP task, but only a few datasets are available for Czech. |
| Approach: | They propose to use a Czech news-based summarization dataset to evaluate document summarizing . they propose a language-agnostic variant of the ROUGE metric to enable automatic evaluation . |
| Outcome: | The proposed dataset contains more than a million Czech news articles . the proposed approach is strong abstractive and language-agnostic . |
Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability. |
| Approach: | They propose a framework that combines ensemble methods with code refinement methodology to address these challenges. |
| Outcome: | The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus. |
One Document, Many Revisions: A Dataset for Classification and Description of Edit Intents (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to understand revisions have failed to provide a deeper understanding of the nature of these edits. |
| Approach: | They propose to use a Wikipedia revision history dataset to train a classifier that achieves a 90% accuracy in identifying edit intent and a distantly-supervised model that generates . |
| Outcome: | The proposed model achieves 90% accuracy in identifying edit intent and a best score of 28 ROUGE. |
AmbigNLG: Addressing Task Ambiguity in Instruction for NLG (2024.emnlp-main)
Copied to clipboard
| Challenge: | AmbigNLG is a novel task designed to tackle task ambiguity in instructions for NLG . ambiguous instructions often impede the performance of Large Language Models (LLMs) . |
| Approach: | They propose an ambiguity taxonomy that categorizes different types of instruction ambiguities and refines initial instructions with clearer specifications. |
| Outcome: | The proposed task improves alignment of generated text with user expectations, achieving 15.02-point increase in ROUGE scores. |
Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
MATCH: Task-Driven Code Evaluation through Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GitHub Copilot generates 46% of the code on GitHub. |
| Approach: | They propose a reference-free metric that uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions. |
| Outcome: | This paper compares the performance of a new similarity score with existing metrics. |
Abstract Text Summarization: A Low Resource Challenge (D19-1)
Copied to clipboard
| Challenge: | Existing datasets for multilingual text summarization are difficult to construct and lack of human knowledge and language processing abilities in computers makes text summaries a challenging task. |
| Approach: | They propose an iterative data augmentation approach which uses synthetic data along with the real summarization data for the German language. |
| Outcome: | The proposed system improves on the development and test sets on the German language text using the state-of-the-art “Transformer” model. |
Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses (D19-1)
Copied to clipboard
| Challenge: | Sentence position is a strong feature for news summarization, since the lead often summarizes the key points of the article. |
| Approach: | They propose two techniques to make neural systems sensitive to the importance of content in different parts of the article by using random shuffled sentences to pretrain the model. |
| Outcome: | The proposed techniques improve the performance of a competitive reinforcement learning based extractive system, with the auxiliary loss being more powerful than pretraining. |
Improving Factual Consistency in Summarization with Compression-Based Post-Editing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve factual consistency of summarization models fail to remove entity errors if a suitable input entity replacement is not available or insert erroneous content. |
| Approach: | They propose to remove extrinsic entity errors, or entities not in the source, to improve consistency while retaining the summary’s essential information and form. |
| Outcome: | The proposed model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and can be applied on top of another post-editor, improving accuracy by 38%. |
Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics, such as ROUGE and BLEU, rely on exact word matching and fail to capture semantic similarity. |
| Approach: | They propose to use contextualized word or sentence embeddings to capture semantic similarity between sentences to evaluate text summarization methods. |
| Outcome: | The proposed evaluation metric shows that it performs faster than the current state-of-the-art on the SummEval dataset. |
Multimodal Abstractive Summarization for How2 Videos (P19-1)
Copied to clipboard
| Challenge: | Abstractive summarization is a task of producing a shorter version of the content in the document while preserving its information. |
| Approach: | They propose a new evaluation metric that measures semantic adequacy rather than fluency of abstractive summarization tasks. |
| Outcome: | The proposed model integrates information from different sources into a coherent output. |
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work suggests that the degree of hallucination depends on factual errors in training data. |
| Approach: | They propose a method to use training data to reduce hallucination by ensembling parameter variations in training data. |
| Outcome: | The proposed method improves on XSUM and CNN/DM datasets on human evaluations and factual metrics. |
COSMIC: Mutual Information for Task-Agnostic Summarization Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for summarizing text are not well aligned with human judgments. |
| Approach: | They propose a task-oriented evaluation approach that assesses the quality of summarizers based on their capacity to produce summaries while preserving task outcomes. |
| Outcome: | The proposed method is able to predict task performance in a variety of contexts and tasks. |
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored . |
| Approach: | They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE . |
| Outcome: | The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show . |
Don’t Take This Out of Context!: On the Need for Contextual Models and Evaluations for Stylistic Rewriting (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing stylistic text rewriting methods ignore the context of the text, causing generic, incoherent, and generic outputs. |
| Approach: | They propose a contextual evaluation metric that integrates preceding context into stylistic text rewriting. |
| Outcome: | The proposed metric integrates the preceding textual context into rewriting and evaluation stages . human preferences are better reflected by the proposed criterio and other metrics . |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
BLEURT: Learning Robust Metrics for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Text generation has made significant advances, but evaluation metrics have lagged behind. |
| Approach: | They propose a learning evaluation metric for English based on BERT . BLEURT can model human judgment with a few thousand possibly biased training examples . |
| Outcome: | The proposed model can model human judgment with a few thousand potentially biased training examples. |
Evaluating the Knowledge Dependency of Questions (2022.emnlp-main)
Copied to clipboard
Hyeongdon Moon, Yoonseok Yang, Hangyeol Yu, Seunghyun Lee, Myeongho Jeong, Juneyoung Park, Jamin Shin, Minsam Kim, Seungtaek Choi
| Challenge: | Existing evaluation metrics for MCQ generation focus on the n-gram based similarity of the generated MCq to the gold sample and disregard their educational value. |
| Approach: | They propose to use a human survey to measure the MCQ’s answerability given knowledge of the target fact. |
| Outcome: | The proposed methods measure the MCQ’s answerability given knowledge of the target fact. |
ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization is one of the areas influenced by pre-trained language models. |
| Approach: | They propose a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue. |
| Outcome: | The proposed model outperforms previous models on six Persian summarization tasks . it also outperformed previous models in textual entailment, question paraphrasing, and question answering . |
Multi-Fact Correction in Abstractive Text Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing abstractive summarization systems generate incorrect facts with respect to the source text. |
| Approach: | They propose a suite of two factual correction models that leverages question-answering knowledge to make corrections in system-generated summaries via span selection. |
| Outcome: | The proposed model improves factuality of news summarization without sacrificing summary quality. |
Re-evaluating Evaluation in Text Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | Automated evaluation metrics are an essential part of the development of text-generation tasks such as summarization. |
| Approach: | They propose to use top-scoring system outputs to assess the reliability of automatic evaluation metrics for text summarization. |
| Outcome: | The proposed evaluation method is based on human judgments from 25 top-scoring neural summarization systems. |
Lost in Translation: Chemical Language Models and the Misunderstanding of Molecule Structures (2024.findings-emnlp)
Copied to clipboard
Veronika Ganeeva, Andrey Sakhovskiy, Kuzma Khrabrov, Andrey Savchenko, Artur Kadurin, Elena Tutubalina
| Challenge: | chemistry and natural language processing (NLP) have advanced drug discovery. |
| Approach: | They propose a framework for assessment of Chemistry LMs of different natures that relies on augmentations that preserve an underlying chemical. |
| Outcome: | The proposed framework relies on augmentations that preserve an underlying chemical, such as kekulization and cycle replacements. |
Improving Copy-oriented Text Generation via EDU Copy Mechanism (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing extractive models generate texts through word-by-word decoding, causing factual inconsistencies and slow inference. |
| Approach: | They propose a framework that integrates the behavior of copying EDUs into generative models. |
| Outcome: | The proposed framework reduces the number of generated tokens significantly. |
Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation (2023.acl-long)
Copied to clipboard
| Challenge: | Argument summarisation is a promising but currently under-explored field. |
| Approach: | They propose a framework to generate key points from short texts in a task known as Key Point Analysis. |
| Outcome: | The proposed framework improves state-of-the-art in argument summarisation with performance improvement of 14 percentage points compared to ROUGE and human evaluation scores. |
ECIS-VQG: Generation of Entity-centric Information-seeking Questions from Videos (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on question generation from videos are mostly focused on generating questions about common objects and attributes. |
| Approach: | They propose a model architecture combining Transformers, rich context signals and a combination of cross-entropy and contrastive loss function to encourage entity-centric question generation. |
| Outcome: | The proposed system yields BLEU, ROUGE, CIDEr, and METEOR scores of 71.3, 78.6, 7.31, and 81.9. |
Language Agnostic Automatic Summarization Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for summarization of documents have been primarily focused on the English language. |
| Approach: | They propose to use ROUGE and PYRAMID to evaluate non-English data using English and non- English data sets. |
| Outcome: | The proposed evaluation methods can be adapted to non-English data, and the results show that they can perform well on non- English data. |
A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for summarization evaluation focus on a metric to represent the quality of the text, but they focus on only a few quality dimensions. |
| Approach: | They analyze the depen-dencies between various quality dimensions on automatically created multi-document summaries and which are best suited for summarization. |
| Outcome: | The proposed method achieves higher quality summaries than other methods on a large-scale heterogeneous data set. |
AXCEL: Automated eXplainable Consistency Evaluation using LLMs (2024.findings-emnlp)
Copied to clipboard
P Aditya Sreekar, Sahil Verma, Suransh Chopra, Abhishek Persad, Sarik Ghazarian, Narayanan Sadagopan
| Challenge: | Large Language Models (LLMs) are widely used for various tasks but evaluating the consistency of generated text remains a challenge. |
| Approach: | They propose a prompt-based consistency metric which provides explanations for consistency scores by providing detailed reasoning and pinpointing inconsistent text spans. |
| Outcome: | The proposed metric outperforms state-of-the-art metrics in summarization, free text generation and data-to-text conversion tasks by 8.7% and 6.2%. |
Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization (2025.acl-long)
Copied to clipboard
| Challenge: | n-gram-based metrics are considered indicative (even if imperfect) of human evaluation for English, but their suitability for other languages remains unclear. |
| Approach: | They systematically assess evaluation metrics for generation for languages and tasks using n-gram-based and neural-based metrics. |
| Outcome: | The proposed evaluation suite is based on eight languages from four typological families and shows that it is sensitivity to the language type at hand. |
LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for text summarisation have restrictive token limits, limiting their effectiveness. |
| Approach: | They propose a human-annotated data set for evaluating automatic factuality metrics . they propose 'longDocFACTScore' framework which can be extended to any length document . |
| Outcome: | The proposed framework outperforms state-of-the-art metrics in evaluating long document summarisation data sets. |
Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Query-focused Summarization (QfS) is a system that generates summaries from document(s) based on a query. |
| Approach: | They propose a Query-focused Summarization approach that uses a generalization of Reinforcement Learning (RL) for Natural Language Generation and a better semantic similarity reward. |
| Outcome: | The proposed approach improves on the ROUGE-L metric and in a benchmark dataset. |
Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks, but their adaptation to domain-specific intricacies remains challenging. |
| Approach: | They propose to use a planning engine to orchestrate structuring knowledge alignment to achieve high-order planning by encapsulating domain knowledge and leveraging sheaf convolution learning to enhance its understanding of the dialogue’s structural nuances. |
| Outcome: | The proposed framework improves on existing LLMs and shows that it can generate better summaries with better quality and better execution. |
MATCHA: Matching Text via Contrastive Semantic Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | MATCHA is an automatic metric that rewards semantic agreement with a reference and penalizes contradictions. |
| Approach: | They introduce a metric that jointly rewards semantic agreement with a reference and penalizes contradictions. |
| Outcome: | The proposed metric outperforms popular metrics on eight public benchmarks compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks. |
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multi-role dialogue summarization favor surface-level imitation of references rather than genuine gains in faithfulness or alignment with human preferences. |
| Approach: | They propose a framework that couples explicit cognitive-style reasoning with reward-based optimization for multi-role dialogue summarization. |
| Outcome: | The proposed framework matches strong baselines on ROUGE and BERTScore, while in-depth analysis on SAMSum shows clear gains in factual faithfulness and model-based preference alignment. |
Agent-as-Judge for Factual Summarization of Long Narratives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore . however, these metrics do not adequately capture critical aspects of summarizing quality, such as factual accuracy, especially for long narratives. |
| Approach: | They propose a framework that evaluates and refines factuality in narrative summarization by leveraging a Character Knowledge Graph extracted from input narrative. |
| Outcome: | The proposed framework evaluates factuality and provides actionable guidance for refinement. |
Replace, Paraphrase or Fine-tune? Evaluating Automatic Simplification for Medical Texts in Spanish (2024.lrec-main)
Copied to clipboard
Leonardo Campillos-Llanos, Ana Rosa Terroba, Rocío Bartolomé, Ana Valverde-Mateos, Cristina González, Adrián Capllonch-Carrión, Jonathan Heras
| Challenge: | lexicon-based simplification methods can help patients understand medical documents . but they must ensure that the content is transmitted rigorously and not creating wrong information. |
| Approach: | They tested automatic simplification techniques using a Spanish lexicon of technical and laymen terms. |
| Outcome: | The proposed methods improve the quantitative results and the human evaluation of medical documents. |
CourtEval: A Courtroom-Based Multi-Agent Evaluation Framework (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing automated evaluation metrics like ROUGE and BLEU show low correlation with human judgments. |
| Approach: | They propose a multi-agent evaluation framework that integrates multiple agents . they use ROUGE and BLEU to evaluate natural language models . |
| Outcome: | The proposed evaluation framework outperforms the current state-of-the-art methods in two meta-evaluation benchmarks. |
QUIDS: Query Intent Description for Exploratory Search via Dual Space Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using QUIDS, we generate user-facing query intent descriptions that surface what the search engine likely inferred the query to mean based on post-retrieval evidence. |
| Approach: | They propose a method that leverages dual-space contrastive learning to isolate intent-relevant information while suppressing irrelevant content. |
| Outcome: | The proposed method outperforms state-of-the-art methods across ROUGE, BERTScore, and human/LLM evaluations. |
The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs (2025.emnlp-main)
Copied to clipboard
Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, Tomasz Jan Kajdanowicz
| Challenge: | Large language models (LLMs) have revolutionized natural language processing, but their tendency to hallucinate poses serious challenges for reliable deployment. |
| Approach: | They propose to use ROUGE to assess lexical overlap to determine accuracy of hallucination detection methods. |
| Outcome: | The proposed evaluation frameworks can rival complex methods, exposing a fundamental flaw in current evaluation practices. |
Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews (2025.emnlp-main)
Copied to clipboard
Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, Juho Kim
| Challenge: | Large Language Models (LLMs) can automatically draft reviews, but determining whether they are trustworthy requires systematic evaluation. |
| Approach: | They propose an automatic focus-level evaluation pipeline based on two sets of facets . authors evaluated LLM reviews at surface-level or content-level . |
| Outcome: | The proposed framework enables automatic evaluation of paper reviews based on two sets of facets . the framework compared open review paper reviews with human experts on validity, clarity, novelty . |
GROKE: Vision-Free Navigation Instruction Evaluation via Graph Reasoning on OpenStreetMap (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for navigation instructions are limited by licensing constraints and computational costs. |
| Approach: | They propose a vision-free training-free hierarchical LLM-based framework for evaluating navigation instructions using OpenStreetMap data. |
| Outcome: | The proposed framework reduces navigation error by 68.5% compared to baselines on the Map2Seq dataset. |
SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing ranking strategies for large language models suffer from instability and lack of information content. |
| Approach: | They propose a framework that enhances summarization by leveraging Summary Content Units (SCUs) they investigate the effectiveness of SCURank in distilling summaries from multiple LLMs . |
| Outcome: | The proposed framework outperforms traditional metrics and LLM-based ranking methods in summarization tasks. |
Budget-Aware Routing for Long Clinical Text (2026.findings-acl)
Copied to clipboard
| Challenge: | Long-context capability is now a headline feature of large language models . clinical inputs are long because they are templated, redundant, and stitched from multiple sources. |
| Approach: | They propose a token-constrained subset selection problem with two design choices . they propose heuristics that balance relevance, coverage, diversity and a monotone submodular objective . |
| Outcome: | The proposed model is based on a subset selection problem with two design choices . positional heuristics perform best at low budgets in extractive tasks, while diversity-aware methods improve LLM generation. |