Papers by Alexander Fabbri
DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often don’t perform as expected under Domain Shift or after Instruct-tuning. |
| Approach: | They propose a method that uses the known performance in high-resource domains and fine-tuning settings to predict performance in low-resourced domains or base models. |
| Outcome: | The proposed method can help researchers decide if resources should be allocated for data labeling and LLM Instruct-tuning. |
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on text summarization factual consistency are divided into two categories . entailment-based and question answering-based metrics are the most efficient . |
| Approach: | They propose an optimized QA-based metric that improves factual consistency by 14% . they compare entailment-based and QA metrics to find the best fit . |
| Outcome: | The proposed metric outperforms the best performing entailment-based metric on the SummaC factual consistency benchmark. |
FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)
Copied to clipboard
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
| Challenge: | Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity. |
| Approach: | They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models. |
| Outcome: | The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models. |
On Learning to Summarize with Large Language Models as References (2024.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan
| Challenge: | Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets. |
| Approach: | They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved. |
| Outcome: | The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance. |
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in efficient attention mechanisms have led to the expansion of the context length of large language models. |
| Approach: | They propose a procedure to synthesize Haystacks of documents and generate a summary that identifies relevant insights and precisely cites the source documents. |
| Outcome: | The proposed evaluation can score summaries on Coverage and Citation . the proposed evaluation lags human performance estimates by 10+ points on SummHay . |
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, Noah A. Smith
| Challenge: | Recent advances on models and metrics should benefit and inform each other, authors argue . bidimensional leaderboards allow for fast, accurate evaluation of language generation models . |
| Approach: | They propose a bidimensional leaderboard that tracks progress in language generation models and metrics for their evaluation. |
| Outcome: | The proposed leaderboards track progress in language generation models and metrics for their evaluation. |
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2021.naacl-main)
Copied to clipboard
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, Yashar Mehdad
| Challenge: | Recent advances in summarization are driven by the availability of large datasets such as the CNN-DailyMail corpus and the New York Times corpus. |
| Approach: | They propose a method for fine-tuning pretrained models for summarization in unsupervised manner . they use Wikipedia data to produce pseudo-summaries which contain characteristics of target dataset . |
| Outcome: | The proposed method achieves state-of-the-art, zero-shot abstractive summarization performance on CNN-DailyMail dataset and compares with other methods on other datasets. |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
Evaluating Cultural and Social Awareness of LLM Web Agents (2025.findings-naacl)
Copied to clipboard
Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu
| Challenge: | Existing benchmarks often overlook cultural and social awareness . current evaluations focus on task completion, often ignoring the diverse cultural and socio-cultural backgrounds. |
| Approach: | They propose a benchmark to assess LLM agents’ sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums. |
| Outcome: | The proposed framework evaluates LLM agents’ ability to detect and appropriately respond to norm-violating user queries and observations across two web-based tasks. |
Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Question Answering (QA) is a field of increasing demand due to the availability of information online. |
| Approach: | They propose an unsupervised approach to training QA models with generated pseudo-training data by applying a simple template on a related sentence rather than the original context sentence. |
| Outcome: | The proposed approach improves the performance of a QA model on generated pseudo-training data. |
ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
| Challenge: | Existing evaluations of large language models (LLMs) for instruction following are incomplete. |
| Approach: | They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets. |
| Outcome: | The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. |
R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Concept prerequisite chain learning is an unsupervised task with no access to labeled concept pairs during training. |
| Approach: | They propose a model that uses deep learning representations to predict concept relations . they frame concept prerequisite chain learning as an unsupervised task with no labeled concept pairs . |
| Outcome: | The proposed model outperforms semi-supervised methods in terms of accuracy and F1 score. |
Fair Abstractive Summarization of Diverse Perspectives (2024.naacl-long)
Copied to clipboard
Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang
| Challenge: | Existing work on summarization metrics and large language models has not explored fair abstractive summarizing. |
| Approach: | They propose four reference-free automatic metrics to measure the differences between target and source perspectives. |
| Outcome: | The proposed methods alleviate fair abstractive summarization on user-generated data. |
ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining (2021.acl-long)
Copied to clipboard
| Challenge: | Abstractive text summarization has primarily focused on modeling news articles . lack of standardized datasets for summarizing online conversations is a major problem . |
| Approach: | They propose to crowdsource four new datasets for summarizing online conversations . they incorporate argument mining through graph construction to directly model issues, viewpoints, and assertions present in a conversation. |
| Outcome: | The proposed models are compared against widely-used conversation summarization datasets and show comparable or improved results. |
TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendation (P18-1)
Copied to clipboard
Alexander Fabbri, Irene Li, Prawat Trairatvorakul, Yijiao He, Weitai Ting, Robert Tung, Caitlin Westerfield, Dragomir Radev
| Challenge: | TutorialBank is a publicly available dataset that aims to facilitate NLP education and research . a google search of "Natural Language Processing" returns over 100 million hits with papers, tutorials, 1 http://aan.how blog posts, codebases and other related online resources. |
| Approach: | They have manually collected and categorized over 5,600 resources on NLP . they have created a search engine and command-line tool to search the corpus . |
| Outcome: | The tutorial bank dataset is the largest manually-picked corpus of resources intended for NLP education . it includes lists of research topics, relevant resources for each topic, prerequisite relations among topics . |
Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (P19-1)
Copied to clipboard
| Challenge: | Multi-document summarization (MDS) of news articles has been limited to datasets of a couple of hundred examples. |
| Approach: | They propose a model which integrates a traditional extractive summarization model with a standard SDS model and achieves competitive results on MDS datasets. |
| Outcome: | The proposed model achieves competitive results on large-scale datasets. |
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)
Copied to clipboard
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu
| Challenge: | Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs. |
| Approach: | They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. |
| Outcome: | The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9. |
Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt leakage poses a compelling security and privacy threat in LLM applications. |
| Approach: | They propose a model which leverages the LLM sycophancy effect and a threat model which fine tunes an open-source model to defend against prompt leakage attempts. |
| Outcome: | The proposed model elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. |
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2022.naacl-main)
Copied to clipboard
Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Adams, Borui Wang, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev
| Challenge: | Existing pre-trained summarization models produce text that is factually inconsistent with the input. |
| Approach: | They present a scale-based scale for Likert rating and a scoring algorithm for Best-Worst Scaling to improve crowdsourcing reliability. |
| Outcome: | The proposed model is more reliable than existing models on two news summarization datasets. |
Exploring Neural Models for Query-Focused Summarization (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work in Query-focused summarization lacks a comprehensive study of the broad space of applicable modeling methods. |
| Approach: | They propose to explore two general classes of methods for Query-focused summarization: extractive-abstractive solutions and end-to-end models. |
| Outcome: | The proposed models achieve state-of-the-art on the QMSum dataset, with a margin of 3.38 ROUGE-1, 3.72 ROUGe2 and 3.28 ROUGEL-L. |
Improving Low-Resource Cross-lingual Document Retrieval by Reranking with Deep Bilingual Representations (P19-1)
Copied to clipboard
Rui Zhang, Caitlin Westerfield, Sungrok Shim, Garrett Bingham, Alexander Fabbri, William Hu, Neha Verma, Dragomir Radev
| Challenge: | Experimental results show that our model outperforms competitive translation-based baselines on cross-lingual relevance ranking tasks. |
| Approach: | They propose to match queries and documents in both source and target languages with deep bilingual query-document representations. |
| Outcome: | The proposed model outperforms translation-based baselines on English-Swahili, English-Tagalog, and English-Somali cross-lingual retrieval tasks. |
Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles (2024.naacl-long)
Copied to clipboard
Kung-Hsiang Huang, Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu
| Challenge: | Existing studies on multi-document summarization focus on collating information that all sources agree upon, but the task of summarizing diverse information remains underexplored. |
| Approach: | They propose a task of summarizing diverse information encountered in multiple news articles encompassing the same event using a dataset curated by a large language model. |
| Outcome: | The proposed task aims to summarize diverse information in multiple news articles encompassing the same event . the proposed task is difficult due to its limited coverage and verbosity biases . |
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
| Challenge: | Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied. |
| Approach: | They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance . |
| Outcome: | The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs. |
AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization (2022.naacl-main)
Copied to clipboard
| Challenge: | Community Question Answering (CQA) fora lack a dataset to produce answer summarizations . a novel dataset of 4,631 CQA threads is used to generate answer summaries . |
| Approach: | They propose a dataset of 4,631 CQA threads for answer summarization curated by professional linguists. |
| Outcome: | The proposed approach boosts summarization performance according to automatic evaluation. |
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases (D19-1)
Copied to clipboard
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, Dragomir Radev
| Challenge: | CoSQL is a corpus for building cross-domain, general-purpose database querying dialogue systems. |
| Approach: | They present a corpus for building cross-domain, general-purpose database querying dialogue systems . they use a Wizard-of-Oz collection of 3k turns plus 10k+ annotated SQL queries . |
| Outcome: | The proposed corpus is based on a Wizard-of-Oz dataset of 3k dialogues querying 200 complex DBs spanning 138 domains. |
Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. |
| Approach: | They propose to partition test set based on lexical similarity of reference test summaries with training summary to determine model competencies. |
| Outcome: | The proposed evaluation protocol improves generalization and generalization on novel test cases while maintaining average performance. |