Papers by Dragomir Radev

FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)

Challenge:	Empirical results show that a modified beam decoding implementation improves decoding performance of strong, neural language generation models.
Approach:	They propose a modification to a beam decoding implementation that generalizes the stopping criterion and provides flexibility to the depth of search.
Outcome:	The proposed method improves decoding performance of strong models on news text summarization and machine translation over diverse language pairs with negligible inference slowdown.

Copied to clipboard

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev

Reasoning

Challenge:	Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity.
Approach:	They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models.
Outcome:	The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models.

Twist Decoding: Diverse Generators Guide Each Other (2022.emnlp-main)

Copied to clipboard

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, Noah A. Smith

On Learning to Summarize with Large Language Models as References (2024.naacl-long)

Challenge:	Using a variety of language generation models, ensembling models is challenging during inference.
Approach:	They propose a method that decodes text models that do not assume a shared vocabulary, tokenization or generation order.
Outcome:	The proposed method outperforms models decoded in isolation over various scenarios.

Copied to clipboard

Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan

PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022.acl-demo)

Challenge:	Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets.
Approach:	They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved.
Outcome:	The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance.

Copied to clipboard

Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, Alexander Rush

DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)

Challenge:	PromptSource is a system for creating, sharing, and using natural language prompts . prompts are used to train and query language models in zero-shot learning settings .
Approach:	PromptSource is a system for creating, sharing, and using natural language prompts . et al.: using prompts to train and query language models is emerging area in NLP . they propose a templating language for defining data-linked prompts, a user interface that iterates on prompt development .
Outcome:	PromptSource is a system for creating, sharing, and using natural language prompts . it has a templating language for defining data-linked prompts and a community-driven set of guidelines .

Copied to clipboard

Wenpeng Yin, Dragomir Radev, Caiming Xiong

Improving Text-to-SQL Evaluation Methodology (P18-1)

Challenge:	Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems.
Approach:	They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems.
Outcome:	The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity.

Copied to clipboard

Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev

OpenRT: An Open-source Framework for Reasoning Over Tabular Data (2023.acl-demo)

Challenge:	Current evaluations of text-to-SQL systems are limited by the way they divide data into training and test sets.
Approach:	They propose to standardize and improve existing and new text-to-SQL datasets . they propose a template-based slot-filling baseline that cannot generalize to new queries .
Outcome:	The proposed system is competitive with prior work on multiple datasets and can be used on training and test sets.

Copied to clipboard

Yilun Zhao, Boyu Mi, Zhenting Qi, Linyong Nan, Minghao Guo, Arman Cohan, Dragomir Radev

Unsupervised Cross-Domain Prerequisite Chain Learning using Variational Graph Autoencoders (2021.acl-short)

Challenge:	Existing table pre-training methods are benchmarked on a limited number of datasets with varying configurations, resulting in a lack of unified, standardized, fair, and comprehensive comparison between methods.
Approach:	They propose to use OpenRT to reproduce existing table pre-training models and develop new models quickly.
Outcome:	The proposed framework reproduces existing table pre-training models and compares them against four question answering, one fact checking, and one faithful text generation datasets.

Copied to clipboard

Irene Li, Vanessa Yan, Tianxiao Li, Rihao Qu, Dragomir Radev

P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)

Challenge:	Existing methods to learn prerequisite relations between concepts require annotated concept pairs during training.
Approach:	They propose to use an optimized variational graph autoencoder to learn prerequisite chains in unsupervised manner using an information-rich domain and an information poor domain.
Outcome:	The proposed model learns to transfer concept prerequisite relations from an information-rich domain (source domain) to an information poor domain (target domain) the annotated data and resources as well as the code will be made publicly available.

Copied to clipboard

Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan

An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (2021.findings-emnlp)

Challenge:	Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales.
Approach:	They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans.
Outcome:	The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more.

Copied to clipboard

Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, Dragomir Radev

Information Retrieval

Challenge:	Existing models for dialogue summarization focus on extracting the main events of short conversations, but real-world dialogues are difficult to train.
Approach:	They propose three strategies to deal with the lengthy input problem and locate relevant information using long dialogue datasets.
Outcome:	The retrieve-then-summarize pipeline models yield the best performance on three long dialogue datasets.

DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization (2022.acl-long)

Copied to clipboard

Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, Dragomir Radev

SParC: Cross-Domain Semantic Parsing in Context (P19-1)

Challenge:	Existing models struggle with summarizing long text due to high memory complexity of the full self-attention.
Approach:	They propose a dynamic latent extraction approach for abstractive long-input summarization that treats extracted text snippets as latent variables and allows dynamic attention weights during decoding.
Outcome:	The proposed method outperforms existing methods on GovReport, QMSum, and arXiv while yielding strong results on arX.

Copied to clipboard

Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, Dragomir Radev

DART: Open-Domain Structured Data Record to Text Generation (2021.naacl-main)

Challenge:	Xu et al., 2017): a dataset for cross-domain semantic parsing in context with 4,298 question sequences.
Approach:	They present a dataset for cross-domainSemanticParsing inContext that consists of 4,298 coherent question sequences.
Outcome:	The proposed dataset demonstrates that it has greater semantic diversity and can be generalized to unseen domains due to its cross-domain nature and the unseened databases at test time.

Copied to clipboard

Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, Nazneen Fatema Rajani

Structured Data in NLP

Challenge:	Data-to-text annotations can be costly when dealing with tables with nontrivial structures.
Approach:	They propose a procedure for extracting semantic triples from tables that encodes their structures by exploiting table headers and table title.
Outcome:	The proposed method exploits the semantic dependencies between table headers and title to extract semantic triples from tables.

ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples (2022.emnlp-main)

Copied to clipboard

Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, Dragomir Radev

Leveraging Locality in Abstractive Text Summarization (2022.emnlp-main)

Challenge:	Existing models with table-specific architectures and pre-training methods perform well on understanding table structures but lack table reasoning skills.
Approach:	They propose to pre-train tables with table reasoning skills without complex architectures . they define 7 table reasoning skill, and then pre-teach them to generate tables .
Outcome:	The proposed model improves on four tasks and is available on github.

Copied to clipboard

Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah, Dragomir Radev

Summarization

Challenge:	Neural attention models have improved on many natural language processing tasks, but their quadratic memory complexity hinders their applications in long text summarization.
Approach:	They propose to use a restricted context to study locality in text summarization . they propose to employ a quadratic memory growth with respect to the input length .
Outcome:	The proposed model has better performance than baseline models with efficient attention modules.

SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task (D18-1)

Copied to clipboard

Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, Dragomir Radev

bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark (2023.acl-long)

Challenge:	Existing studies in text-to-SQL do not require generating complex SQL queries with multiple clauses or sub-queries.
Approach:	They propose a syntax tree network to address the complex text-to-SQL generation task.
Outcome:	The proposed model outperforms the current state-of-the-art model by 9.5% on a large text-to-SQL corpus.

Copied to clipboard

Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev

Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2021.naacl-main)

Challenge:	bgGLUE is a benchmark for evaluating language models on natural language understanding (NLU) tasks in Bulgarian.
Approach:	They propose to use a benchmark to evaluate language models on NLU tasks in Bulgarian.
Outcome:	The proposed model performs well on sequence labeling tasks, but there is room for improvement for tasks that require more complex reasoning.

Copied to clipboard

Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, Yashar Mehdad

Responsible & Trustworthy NLP

Challenge:	Recent advances in summarization are driven by the availability of large datasets such as the CNN-DailyMail corpus and the New York Times corpus.
Approach:	They propose a method for fine-tuning pretrained models for summarization in unsupervised manner . they use Wikipedia data to produce pseudo-summaries which contain characteristics of target dataset .
Outcome:	The proposed method achieves state-of-the-art, zero-shot abstractive summarization performance on CNN-DailyMail dataset and compares with other methods on other datasets.

R2D2: Robust Data-to-Text with Replacement Detection (2022.emnlp-main)

Copied to clipboard

Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou, Dragomir Radev

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)

Challenge:	Existing methods to mitigate unfaithful text generation are inadequate . data-to-text generation requires a structured input format .
Approach:	They propose a training framework that addresses unfaithful Data-to-Text generation by training a system as a generator and faithfulness discriminator with additional replacement detection and unlikelihood learning tasks.
Outcome:	The proposed training framework improves FeTaQA, LogicNLG, and ToTTo fidelity on D2T systems.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Responsible & Trustworthy NLP

Challenge:	Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality.
Approach:	They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence.
Outcome:	The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability.

Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions (D19-1)

Copied to clipboard

Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, Dragomir Radev

R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning (2020.coling-main)

Challenge:	Generating SQL queries from user utterances is an important task to help end users acquire information from databases.
Approach:	They propose a context-dependent text-to-SQL generation task that edits previous queries . they use an utterance-table encoder and a table-aware decoder to incorporate context .
Outcome:	The proposed model is flexible to change individual tokens and robust to error propagation.

Copied to clipboard

Irene Li, Alexander Fabbri, Swapnil Hingmire, Dragomir Radev

Low-Resource NLP

Challenge:	Concept prerequisite chain learning is an unsupervised task with no access to labeled concept pairs during training.
Approach:	They propose a model that uses deep learning representations to predict concept relations . they frame concept prerequisite chain learning as an unsupervised task with no labeled concept pairs .
Outcome:	The proposed model outperforms semi-supervised methods in terms of accuracy and F1 score.

FeTaQA: Free-form Table Question Answering (2022.tacl-1)

Copied to clipboard

Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, Dragomir Radev

Fair Abstractive Summarization of Diverse Perspectives (2024.naacl-long)

Challenge:	Existing table-based question answering datasets lack advanced information-based questions that require reasoning and integration of information pieces retrieved from structured knowledge sources.
Approach:	They propose a dataset with 10K Wikipedia-based table, question, free-form answer, supporting table cells pairs that can be used to generate an answer.
Outcome:	The proposed dataset has 10K Wikipedia-based table, question, free-form answer, supporting table cells pairs.

Copied to clipboard

Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang

BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization (2022.findings-emnlp)

Challenge:	Existing work on summarization metrics and large language models has not explored fair abstractive summarizing.
Approach:	They propose four reference-free automatic metrics to measure the differences between target and source perspectives.
Outcome:	The proposed methods alleviate fair abstractive summarization on user-generated data.

Copied to clipboard

Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, Dragomir Radev

Summarization

Challenge:	Existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies and contain strong layout and stylistic biases.
Approach:	They propose a dataset for long-form narrative summarization that uses human written summaries on three levels of difficulty.
Outcome:	The proposed dataset covers documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of difficulty.

ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020.acl-main)

Copied to clipboard

Nazneen Fatema Rajani, Rui Zhang, Yi Chern Tan, Stephan Zheng, Jeremy Weiss, Aadit Vyas, Abhijit Gupta, Caiming Xiong, Richard Socher, Dragomir Radev

Explainability & Interpretability in NLP

Challenge:	Neural networks lack the ability to reason about qualitative physics and cannot generalize to scenarios and tasks unseen during training.
Approach:	They propose a framework for reasoning about qualitative physics in natural language that generates interpretable descriptions of physical events.
Outcome:	The proposed framework generates explanations of how the physical simulation will causally evolve so that an agent or a human can reason about a solution using interpretable descriptions.

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (D18-1)

Copied to clipboard

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, Dragomir Radev

MACSum: Controllable Summarization with Mixed Attributes (2023.tacl-1)

Challenge:	Existing datasets for semantic parsing are too small in terms of number of programs for training modern data-intensive models.
Approach:	They propose a large-scale complex and cross-domain semantic parsing task for a database . they use a dataset with 10,181 questions and 5,693 unique complex SQL queries .
Outcome:	The proposed task is different from previous tasks because it uses the same database and program . the best model achieves only 9.7% exact matching accuracy on a database split setting.

Copied to clipboard

Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, Rui Zhang

ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining (2021.acl-long)

Challenge:	Existing work on controllable summarization with mixed attributes lacks designated annotations.
Approach:	They propose a human-annotated summarization benchmark for controllable summarizing with mixed attributes based on news and dialogue sources .
Outcome:	The proposed dataset contains human-annotated summarization datasets with mixed attributes . hard prompt models yield the best performance on most metrics and human evaluations . mixed-attribute control is still challenging for summarizing tasks .

Copied to clipboard

Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, Dragomir Radev

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models (2023.findings-acl)

Challenge:	Abstractive text summarization has primarily focused on modeling news articles . lack of standardized datasets for summarizing online conversations is a major problem .
Approach:	They propose to crowdsource four new datasets for summarizing online conversations . they incorporate argument mining through graph construction to directly model issues, viewpoints, and assertions present in a conversation.
Outcome:	The proposed models are compared against widely-used conversation summarization datasets and show comparable or improved results.

Copied to clipboard

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron Halfaker, Liqun Shao, Daniel McDuff, Ahmed Hassan Awadallah, Dragomir Radev, Jianfeng Gao

TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendation (P18-1)

Challenge:	Existing pre-trained language models that ignore the logical structures underlying natural language text often lack the ability to capture and encode key logical information in the input sequences.
Approach:	They propose to construct logic-aware input embeddings for transformer language models through logic detection, logic mapping and hierarchical logical projections and then develop a new modeling paradigm that can upgrade existing transformer language model into logical transformers to boost their performance.
Outcome:	The proposed model can achieve superior performance on four important and challenging tasks.

Copied to clipboard

Alexander Fabbri, Irene Li, Prawat Trairatvorakul, Yijiao He, Weitai Ting, Robert Tung, Caitlin Westerfield, Dragomir Radev

On Improving Summarization Factual Consistency from Natural Language Feedback (2023.acl-long)

Challenge:	TutorialBank is a publicly available dataset that aims to facilitate NLP education and research . a google search of "Natural Language Processing" returns over 100 million hits with papers, tutorials, 1 http://aan.how blog posts, codebases and other related online resources.
Approach:	They have manually collected and categorized over 5,600 resources on NLP . they have created a search engine and command-line tool to search the corpus .
Outcome:	The tutorial bank dataset is the largest manually-picked corpus of resources intended for NLP education . it includes lists of research topics, relevant resources for each topic, prerequisite relations among topics .

Copied to clipboard

Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, Ahmed Hassan Awadallah

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)

Challenge:	Recent work shows that language generation models can make errors on fine-grained qualities such as factual consistency.
Approach:	They propose to use natural language feedback to improve generation quality and user preference alignment.
Outcome:	The proposed model can provide factual consistency in human-edited summaries and further insights into summarization factual consistentness.

Copied to clipboard

Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou

RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations (2023.acl-long)

Challenge:	Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work.
Approach:	They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations.
Outcome:	The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work.

Copied to clipboard

Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, Dragomir Radev

Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (P19-1)

Challenge:	Existing Table QA models are vulnerable to task-specific perturbations, such as replacing key question entities or shuffling table columns.
Approach:	They propose to use large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models.
Outcome:	The proposed model significantly improves on existing Table QA models against human-annotated adversarial perturbations.

Copied to clipboard

Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, Dragomir Radev

Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020.emnlp-main)

Challenge:	Multi-document summarization (MDS) of news articles has been limited to datasets of a couple of hundred examples.
Approach:	They propose a model which integrates a traditional extractive summarization model with a standard SDS model and achieves competitive results on MDS datasets.
Outcome:	The proposed model achieves competitive results on large-scale datasets.

Copied to clipboard

Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, Caiming Xiong

Crosslingual Generalization through Multitask Finetuning (2023.acl-long)

Challenge:	a current approach to solving NLP problems is to build a problem-specific dataset . current approaches do not allow for transforming tasks into textual entailment .
Approach:	They propose a pretrained textual entailment system that can generalize across domains . they argue that when is it worth transforming an NLP task into textual detailment?
Outcome:	The proposed model can generalize across domains with few examples, the authors argue . they show that it can be used for several downstream NLP tasks with limited annotations .

Copied to clipboard

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel

Multilinguality

Challenge:	Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models.
Approach:	They apply multitask prompted finetuning to pretrained multilingual models and generate variants called BLOOMZ and mT0.
Outcome:	The proposed models can generalize to non-English languages that have never been seen before.

Uni-Parser: Unified Semantic Parser for Question Answering on Knowledge Base and Database (2022.emnlp-main)

Copied to clipboard

Ye Liu, Semih Yavuz, Rui Meng, Dragomir Radev, Caiming Xiong, Yingbo Zhou

Question Answering

Challenge:	Existing approaches on semantic parsing suffer from exponential growth of logical form candidates and can hardly generalize to unseen data.
Approach:	They propose a unified semantic parser for question answering on KB and DB . they define the primitive as the essential element in their framework .
Outcome:	The proposed framework can predict logical forms by altering and composing top-ranked primitives with different operations.

Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2022.naacl-main)

Copied to clipboard

Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Adams, Borui Wang, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

SummerTime: Text Summarization Toolkit for Non-experts (2021.emnlp-demo)

Challenge:	Existing pre-trained summarization models produce text that is factually inconsistent with the input.
Approach:	They present a scale-based scale for Likert rating and a scoring algorithm for Best-Worst Scaling to improve crowdsourcing reliability.
Outcome:	The proposed model is more reliable than existing models on two news summarization datasets.

Copied to clipboard

Ansong Ni, Zhangir Azerbayev, Mutethia Mutuma, Troy Feng, Yusen Zhang, Tao Yu, Ahmed Hassan Awadallah, Dragomir Radev

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)

Challenge:	Recent advances in summarization provide models that can generate high quality summaries . a new toolkit for summarizing text is being developed to make it easier for non-experts to keep track of them.
Approach:	They develop a toolkit for text summarization that integrates with libraries designed for NLP researchers.
Outcome:	SummerTime is a toolkit for text summarization, including models, datasets, and evaluation metrics.

Copied to clipboard

Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

Summarization

Challenge:	Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization.
Approach:	They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries.
Outcome:	The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets.

QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021.naacl-main)

Copied to clipboard

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, Dragomir Radev

LoFT: Enhancing Faithfulness and Diversity for Table-to-Text Generation via Logic Form Control (2023.eacl-main)

Challenge:	Existing work on meeting summarization tasks is limited to short summaries that cover all the content of a meeting.
Approach:	They propose a query-based multi-domain meeting summarization task that generates a single short summary of meetings based on a transcript.
Outcome:	The proposed task is based on 1,808 query-summary pairs over 232 meetings in multiple domains.

Copied to clipboard

Yilun Zhao, Zhenting Qi, Linyong Nan, Lorenzo Jaime Flores, Dragomir Radev

Improving Low-Resource Cross-lingual Document Retrieval by Reranking with Deep Bilingual Representations (P19-1)

Challenge:	Existing models for LT2T generation focus on surface-level realizations without much logical inference.
Approach:	They propose a model that uses logic forms as fact verifiers and content planners to control LT2T generation.
Outcome:	Experimental results show that the proposed model addresses unfaithfulness and diversity issues simultaneously.

Copied to clipboard

Rui Zhang, Caitlin Westerfield, Sungrok Shim, Garrett Bingham, Alexander Fabbri, William Hu, Neha Verma, Dragomir Radev

Information Retrieval

Challenge:	Experimental results show that our model outperforms competitive translation-based baselines on cross-lingual relevance ranking tasks.
Approach:	They propose to match queries and documents in both source and target languages with deep bilingual query-document representations.
Outcome:	The proposed model outperforms translation-based baselines on English-Swahili, English-Tagalog, and English-Somali cross-lingual retrieval tasks.

BRIO: Bringing Order to Abstractive Summarization (2022.acl-long)

Copied to clipboard

Yixin Liu, Pengfei Liu, Dragomir Radev, Graham Neubig

Syntax-aware Neural Semantic Role Labeling with Supertags (N19-1)

Challenge:	Abstractive summarization models are often trained with maximum likelihood estimation (MLE) . mLE assumes a deterministic (one-point) target distribution, but can cause performance degradation .
Approach:	They propose a new training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality.
Outcome:	The proposed model can estimate probabilities of candidate summaries that are more correlated with their level of quality.

Copied to clipboard

Jungo Kasai, Dan Friedman, Robert Frank, Dragomir Radev, Owen Rambow

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)

Challenge:	a new syntax-aware model for dependency-based semantic role labeling outperforms syntax-based models for English and Spanish.
Approach:	They propose a syntax-aware model for dependency-based semantic role labeling that outperforms syntax-based models for English and Spanish.
Outcome:	The proposed model outperforms syntax-agnostic models for English and Spanish.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022.emnlp-main)

Challenge:	Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied.
Approach:	They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance .
Outcome:	The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs.

Copied to clipboard

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu

SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents (2022.acl-long)

Challenge:	Structured knowledge grounding (SKG) uses structured knowledge to complete user requests . since inputs and outputs of SKG tasks are heterogeneous, they have been studied separately .
Approach:	They propose a framework that unifies 21 SKG tasks into a text-to-text format . they use unifiedSKG to benchmark T5 with different sizes .
Outcome:	The proposed framework unifies 21 SKG tasks into a text-to-text format . it achieves state-of-the-art performance on almost all of the 21 tasks, the authors show .

Copied to clipboard

Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, Rui Zhang

HiPool: Modeling Long Documents Using Graph Neural Networks (2023.acl-short)

Challenge:	Existing methods to handle long text are limited due to time and memory complexity and limited input lengths.
Approach:	They propose a multi-stage split-then-summarize framework for long input summarization . their framework can process input text of arbitrary length by adjusting the number of stages .
Outcome:	The proposed framework outperforms existing methods on three long meeting summarization datasets and on a long document summarizing dataset.

Copied to clipboard

Irene Li, Aosong Feng, Dragomir Radev, Rex Ying

Structured Data in NLP

Challenge:	Recent work on pretraining languages have achieved satisfying results in many NLP tasks, but they are still restricted by a pre-defined maximum length.
Approach:	They propose a graph-based method to model sentence-level information using a fixed length and graphs to model intra- and cross-sentence correlations.
Outcome:	The proposed model outperforms baseline models by 2.6% in F1 score, and 4.8% on the longest sequence dataset.

Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)

Copied to clipboard

Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev

Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering (P18-2)

Challenge:	In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory.
Approach:	They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs.
Outcome:	The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset.

Copied to clipboard

Rui Zhang, Cícero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, Dragomir Radev

HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution (2023.findings-emnlp)

Challenge:	Existing methods for identifying and clustering mentions in text are complex and require heuristics to solve.
Approach:	They propose to use a biaffine attention model to get antecedent scores for each possible mention and optimize mention detection and mention clustering accuracy given the mention cluster labels.
Outcome:	The proposed model achieves the state-of-the-art performance on the CoNLL-2012 shared task English test set.

Copied to clipboard

Ye Liu, Semih Yavuz, Rui Meng, Dragomir Radev, Caiming Xiong, Shafiq Joty, Yingbo Zhou

Question Answering

Challenge:	End-to-end neural networks excel at answering natural language questions but fail on complex ones . a proposed framework for question parsing and execution on textual QA is designed to combine the strengths of neural and symbolic methods.
Approach:	They propose a framework for question parsing and execution on textual QA . they parse questions into an intermediate representation and use deterministic rules to translate them .
Outcome:	The proposed framework outperforms existing methods in supervised, few-shot, and zero-shot settings while preserving its underlying reasoning process.

CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases (D19-1)

Copied to clipboard

Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, Dragomir Radev

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)

Challenge:	CoSQL is a corpus for building cross-domain, general-purpose database querying dialogue systems.
Approach:	They present a corpus for building cross-domain, general-purpose database querying dialogue systems . they use a Wizard-of-Oz collection of 3k turns plus 10k+ annotated SQL queries .
Outcome:	The proposed corpus is based on a Wizard-of-Oz dataset of 3k dialogues querying 200 complex DBs spanning 138 domains.

Copied to clipboard

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Robustness in NLP

Challenge:	Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale.
Approach:	They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark.
Outcome:	The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement.

QTSumm: Query-Focused Summarization over Tabular Data (2023.emnlp-main)

Copied to clipboard

Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, Dragomir Radev, Arman Cohan

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023.acl-long)

Challenge:	Existing text generation systems that can provide accurate table summaries can facilitate more efficient access to relevant data insights.
Approach:	They propose a query-focused task where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored table summary.
Outcome:	The proposed method improves existing baselines on table-to-text generation and large language models by concatenating generated facts to the model input.

Copied to clipboard

Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina

TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Generation (N18-2)

Challenge:	Existing language adaptation strategies for multilingual models are limited to 46 languages . a new language is added to the model to improve zero-shot prompting performance .
Approach:	They apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting.
Outcome:	The proposed model can be extended to other languages without incurring prohibitively large costs.

Copied to clipboard

Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, Dragomir Radev

STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension (2022.emnlp-main)

Challenge:	Existing systems that can understand natural language questions and generate corresponding SQL queries are not able to do this.
Approach:	They propose a novel approach which formats the problem as a slot filling task in a more reasonable way and utilizes type information to better understand rare entities and numbers in the questions.
Outcome:	The proposed approach outperforms the prior art on the WikiSQL dataset and can reach 82.6% accuracy, a 17.5% improvement compared to the previous content-sensitive model.

Copied to clipboard

Borui Wang, Chengcheng Feng, Arjun Nair, Madelyn Mao, Jai Desai, Asli Celikyilmaz, Haoran Li, Yashar Mehdad, Dragomir Radev

Natural Language Interfaces

Challenge:	Abstractive dialogue summarization is an important standalone task in natural language processing, but no previous work has explored whether it can be used to boost an NLP system's performance on other important dialogue comprehension tasks.
Approach:	They propose a novel type of dialogue summarization task that decomposes and imitates the hierarchical, systematic and structured mental process that human beings usually go through when understanding and analyzing dialogues.
Outcome:	The proposed model improves the performance of transformer encoder language models on two important dialogue comprehension tasks.

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models (2025.naacl-long)

Copied to clipboard

Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, Nikita Nangia, Anaelia Ovalle, Giada Pistilli, Dragomir Radev, Beatrice Savoldi, Vipul Raheja, Jeremy Qin, Esther Ploeger, Arjun Subramonian, Kaustubh Dhole, Kaiser Sun, Amirbek Djanibekov, Jonibek Mansurov, Kayo Yin, Emilio Villa Cueva, Sagnik Mukherjee, Jerry Huang, Xudong Shen, Jay Gala, Hamdan Al-Ali, null Tair Djanibekov, Nurdaulet Mukhituly, Shangrui Nie, Shanya Sharma, Karolina Stanczak, Eliza Szczechla, Tiago Timponi Torrent, Deepak Tunuguntla, Marcelo Viridiano, Oskar Van Der Wal, Adina Yakefu, Aurélie Névéol, Mike Zhang, Sydney Zink, Zeerak Talat

SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)

Challenge:	Large Language Models reproduce and exacerbate social biases present in training data, and resources to quantify this issue are limited.
Approach:	They propose a multilingual parallel dataset to examine culturally-specific stereotypes that may be learned by LLMs.
Outcome:	The proposed dataset includes stereotypes from 20 regions around the world and 16 languages, spanning multiple identity categories subject to discrimination worldwide.

Copied to clipboard

Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev