Papers by Chien-Sheng Wu

MixQG: Neural Question Generation with Mixed Answer Types (2022.findings-naacl)

Challenge:	Using a model to generate summary sketches, we improve abstractive dialogue summarization quality and enable granularity control.
Approach:	They propose a model that generates a preliminary summary sketch and a strategy to control granularity.
Outcome:	The proposed model achieves state-of-the-art on the largest dialogue summarization corpus with as high as 50.79 in ROUGE-L score.

Copied to clipboard

Lidiya Murakhovs’ka, Chien-Sheng Wu, Philippe Laban, Tong Niu, Wenhao Liu, Caiming Xiong

Question Generation

Challenge:	Existing neural question generation approaches focus on short factoid type of answers.
Approach:	They propose a neural question generator that trains a single generative model by combining multiple question types with different answer types.
Outcome:	The proposed model outperforms existing models in both seen and unseen domains and can generate questions with different cognitive levels when conditioned on different answer types.

Personalizing Dialogue Agents via Meta-Learning (P19-1)

Copied to clipboard

Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, Pascale Fung

Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding (2025.findings-acl)

Challenge:	Existing personalized dialogue models use human designed persona descriptions to improve dialogue consistency.
Approach:	They propose to extend Model-Agnostic Meta-Learning (MAML) to personalized dialogue learning without using persona descriptions.
Outcome:	The proposed model outperforms baseline models in terms of human-evaluated fluency and consistency on a persona-chat dataset.

Copied to clipboard

Kung-Hsiang Huang, Can Qin, Haoyi Qiu, Philippe Laban, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu

Multimodality

Challenge:	Vision Language Models struggle with visual arithmetic, seemingly simple tasks like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning.
Approach:	They propose a novel post-training strategy inspired by Piaget’s theory of cognitive development that trains VLMs to recognize invariant properties under visual transformations.
Outcome:	The proposed approach outperforms supervised fine-tuning methods while requiring 60% less training data.

SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages (2023.acl-long)

Copied to clipboard

Philippe Laban, Jesse Vig, Wojciech Kryscinski, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu

DialFact: A Benchmark for Fact-Checking in Dialogue (2022.acl-long)

Challenge:	Prior work on document-level simplification has focused on sentence-level edits, while many desirable edits require document- level context.
Approach:	They propose a dataset that reconstructs the document-level editing process from English Wikipedia to paired Simple Wikipedia articles.
Outcome:	The proposed dataset reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) pages.

Copied to clipboard

Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong

From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models (2026.findings-acl)

Challenge:	Existing fact-checking models trained on non-dialogue data fail to perform well on this task.
Approach:	They propose a task of fact-checking in dialogue to improve fact- checking performance . they propose to use an annotated conversational claim and Wikipedia snippets as evidence .
Outcome:	The proposed task improves fact-checking performance in dialogue.

Copied to clipboard

Jiaxin Zhang, Wendi Cui, Zhuohang Li, Lifu Huang, Bradley A. Malin, Caiming Xiong, Chien-Sheng Wu

Language Models

Challenge:	Large Language Models (LLMs) have remarkable capabilities, but unreliability remains a barrier to deployment in high-stakes domains.
Approach:	They propose to transform uncertainty from a passive diagnostic metric to an active control signal guiding real-time model behavior.
Outcome:	The proposed model evolution from passive diagnostic metric to active control signal is critical for high-stakes applications.

Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems (2024.emnlp-main)

Copied to clipboard

Philippe Laban, Alexander Fabbri, Caiming Xiong, Chien-Sheng Wu

Summarization

Challenge:	Recent advances in efficient attention mechanisms have led to the expansion of the context length of large language models.
Approach:	They propose a procedure to synthesize Haystacks of documents and generate a summary that identifies relevant insights and precisely cites the source documents.
Outcome:	The proposed evaluation can score summaries on Coverage and Citation . the proposed evaluation lags human performance estimates by 10+ points on SummHay .

Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems (P18-1)

Copied to clipboard

Andrea Madotto, Chien-Sheng Wu, Pascale Fung

Don’t Stop Early: Scalable Enterprise Deep Research with Controlled Information Flow and Evidence-Aware Termination (2026.acl-industry)

Challenge:	End-to-end task-oriented dialog systems often suffer from the challenge of incorporating knowledge bases.
Approach:	They propose a novel yet simple end-to-end differentiable model called memory-tosequence to address this issue.
Outcome:	The proposed model can be trained faster and achieve state-of-the-art performance on three different task-oriented dialog datasets.

Copied to clipboard

Prafulla Kumar Choubey, Kung-Hsiang Huang, Pranav Narayanan Venkit, Jiaxin Zhang, Vaibhav Vats, Yu Li, Xiangyu Peng, Chien-Sheng Wu

Challenge:	Enterprise deep research systems fail to produce decision-ready reports due to uneven information coverage, context explosion, and premature stopping.
Approach:	They propose a scalable Enterprise Deep Research (EDR) architecture that decomposes requests into coverage-driven objectives via outline generation with reflection and localizes context with dependency-guided execution and explicit information sharing.
Outcome:	The proposed system achieves the strongest overall performance compared with competitive deep-research baselines on internal sales enablement tasks and the public DeepResearch Bench benchmark.

Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023.acl-long)

Copied to clipboard

Artidoro Pagnoni, Alex Fabbri, Wojciech Kryscinski, Chien-Sheng Wu

Language Models

Challenge:	Existing methods to control document controllable summarization lack abundant labeled data.
Approach:	They propose a question-driven, unsupervised pretraining objective to improve controllability in document controllable summarization tasks.
Outcome:	The proposed method outperforms pre-finetuning approaches on QMSum and SQuALITY.

Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation (2022.findings-naacl)

Copied to clipboard

Philippe Laban, Chien-Sheng Wu, Lidiya Murakhovs’ka, Wenhao Liu, Caiming Xiong

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)

Challenge:	Question generation models are often evaluated with standardized NLG metrics that are based on n-gram overlap.
Approach:	They propose to use QGen to help teachers automate the generation of reading comprehension quizzes by comparing n-gram overlap with BLEU to compare system-generated questions with heldout human-written references.
Outcome:	The best model had only 68.4% of its questions accepted by the ten teachers who participated in the study.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Evaluating Cultural and Social Awareness of LLM Web Agents (2025.findings-naacl)

Challenge:	Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality.
Approach:	They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence.
Outcome:	The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability.

Copied to clipboard

Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu

Language Models

Challenge:	Existing benchmarks often overlook cultural and social awareness . current evaluations focus on task completion, often ignoring the diverse cultural and socio-cultural backgrounds.
Approach:	They propose a benchmark to assess LLM agents’ sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums.
Outcome:	The proposed framework evaluates LLM agents’ ability to detect and appropriately respond to norm-violating user queries and observations across two web-based tasks.

Getting To Know You: User Attribute Extraction from Dialogues (2020.lrec-1)

Copied to clipboard

Chien-Sheng Wu, Andrea Madotto, Zhaojiang Lin, Peng Xu, Pascale Fung

Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning (D19-1)

Challenge:	a new method to extract user attributes from dialogues is needed to improve user understanding.
Approach:	They propose to leverage dialogues with conversational agents to automatically extract user attributes from dialogues.
Outcome:	The proposed model surpasses retrieval and generation baselines on human evaluation.

Copied to clipboard

Peng Xu, Chien-Sheng Wu, Andrea Madotto, Pascale Fung

Explicit Memory Tracker with Coarse-to-Fine Reasoning for Conversational Machine Reading (2020.acl-main)

Challenge:	Conventional abstractive headline generation methods do not optimize for maximum reader attention.
Approach:	They propose a model that generates sensational headlines without labeled data by classifying online headlines with many comments against a summarization model.
Outcome:	The proposed model generates sensational headlines without labeled data.

Copied to clipboard

Yifan Gao, Chien-Sheng Wu, Shafiq Joty, Caiming Xiong, Richard Socher, Irwin King, Michael Lyu, Steven C.H. Hoi

ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)

Challenge:	Existing approaches to answer user questions are limited in their decision making due to struggles in extracting question-related rules and reasoning about them.
Approach:	They propose a conversational machine reading framework that uses a Explicit Memory Tracker to track whether conditions in the rule text have already been satisfied to make a decision.
Outcome:	The proposed framework achieves state-of-the-art on the ShARC benchmark and is more interpretable by visualizing the entailment-oriented reasoning process as the conversation flows.

Copied to clipboard

Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan

Robustness in NLP

Challenge:	Existing evaluations of large language models (LLMs) for instruction following are incomplete.
Approach:	They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets.
Outcome:	The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness.

Improving Large-Scale Fact-Checking using Decomposable Attention Models and Lexical Tagging (D18-1)

Copied to clipboard

Nayeon Lee, Chien-Sheng Wu, Pascale Fung

Tagging

Challenge:	Existing pipelines for fact-checking of textual sources are limited . fact- checking of text sources requires a large knowledge base to extract relevant information .
Approach:	They propose a neural ranker that dynamically selects sentences to improve evidence retrieval . they incorporate lexical tagging methods into the pipeline framework to simplify the tasks .
Outcome:	The proposed model outperforms the existing TF-IDF method on a large-scale fact extraction and verification dataset with speedup.

Improving Limited Labeled Dialogue State Tracking with Self-Supervision (2020.findings-emnlp)

Copied to clipboard

Chien-Sheng Wu, Steven C.H. Hoi, Caiming Xiong

Discern: Discourse-Aware Entailment Reasoning Network for Conversational Machine Reading (2020.emnlp-main)

Challenge:	Existing dialogue state tracking models require plenty of labeled data, but collecting labels is expensive.
Approach:	They propose to use only 1% labeled data to train dialogue state tracking models . they encourage a model to have consistent latent distributions given a perturbed input .
Outcome:	The proposed self-supervised signals improve goal accuracy by 8.95% when only 1% labeled data is used on the MultiWOZ dataset.

Copied to clipboard

Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty, Steven C.H. Hoi, Caiming Xiong, Irwin King, Michael Lyu

Conformal Predictor for Improving Zero-Shot Text Classification Efficiency (2022.emnlp-main)

Challenge:	Document interpretation and dialog understanding are the two major challenges for conversational machine reading.
Approach:	They propose a discourse-aware entailment reasoning network to strengthen the connection and enhance the understanding of document and dialog.
Outcome:	The proposed model improves document interpretation and dialog understanding on the ShARC benchmark.

Copied to clipboard

Prafulla Kumar Choubey, Yu Bai, Chien-Sheng Wu, Wenhao Liu, Nazneen Rajani

GTA: Generating Long-horizon Tasks for Web Agents at Scale (2026.acl-long)

Challenge:	Pre-trained language models (PLMs) have been shown effective for zero-shot (0shot) text classification.
Approach:	They propose to limit the number of likely labels using a fast base classifier-based conformal predictor calibrated on samples labeled by the 0shot model.
Outcome:	The proposed models reduce the average inference time for NLI- and NSP-based models by 25.6% and 22.2% without dropping performance below the predefined error rate of 1%.

Copied to clipboard

Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu

Green & Sustainable NLP

Challenge:	Existing benchmarks provide only coarse start–goal annotations without intermediate trajectories . Existing frameworks provide no supervision over the agent's latent decision process .
Approach:	They propose a framework that integrates crawling, retrieval-based seeding, in-context generation and automated quality control to produce realistic tasks paired with executable trajectories.
Outcome:	The proposed framework decouples crawling from generation for greater efficiency and ensures dense supervision through deterministic replays and systematic validation.

QAConv: Question Answering on Informative Conversations (2022.acl-long)

Copied to clipboard

Chien-Sheng Wu, Andrea Madotto, Wenhao Liu, Pascale Fung, Caiming Xiong

Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking (2020.starsem-1)

Challenge:	Experimental results show that state-of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable.
Approach:	They propose a question-answering dataset that uses conversations as a knowledge source.
Outcome:	The proposed dataset provides a training and evaluation testbed to facilitate QA on conversations research.

Copied to clipboard

Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip Yu, Richard Socher, Caiming Xiong

LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback (2025.findings-acl)

Challenge:	Existing methods for dialog state tracking are ontology-based and ontologie-free . however, it is not clear enough which slots are better handled by either of the two methods .
Approach:	They propose a dual-strategy model that integrates both ontology-based and ontological-free methods.
Outcome:	The proposed model outperforms the existing model on noisy and cleaner datasets.

Copied to clipboard

Thai Quoc Hoang, Kung-Hsiang Huang, Shirley Kokane, Jianguo Zhang, Zuxin Liu, Ming Zhu, Jake Grigsby, Tian Lan, Michael S Ryoo, Chien-Sheng Wu, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong, Juan Carlos Niebles

Challenge:	Large Action Models (LAMs) face challenges due to the need for high-quality training data, especially for multi-steps tasks that involve planning, executing tool calls, and responding to feedback.
Approach:	They propose a framework for online exploration of agentic tasks with high-quality feedback . they use a dynamic task query generator and an extensive collection of tools to create a high-level feedback environment for LLM Agents.
Outcome:	The proposed framework achieves 49.3% performance improvement over baselines on toolbench and CRMArena.

Benchmarking Deep Search over Heterogeneous Enterprise Data (2025.emnlp-industry)

Copied to clipboard

Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Kung-Hsiang Huang, Caiming Xiong, Chien-Sheng Wu

Information Retrieval

Challenge:	Existing methods struggle to conduct deep searches and retrieve all necessary evidence.
Approach:	They propose a benchmark for evaluating deep search, a retrieval-augmented generation that requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.
Outcome:	The proposed benchmarks show that even the best-performing agentic RAG methods achieve an average performance score of 32.96 on the benchmark.

SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)

Copied to clipboard

Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020.emnlp-main)

Challenge:	Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs.
Approach:	They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits.
Outcome:	The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9.

Copied to clipboard

Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, Caiming Xiong

Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications (2024.emnlp-industry)

Challenge:	Existing pre-trained language models with self-attention encoder architectures are less useful in practice.
Approach:	They propose to use user and system tokens to model dialogue behavior during pre-training . they propose a contrastive objective function to simulate the response selection task .
Outcome:	The proposed model outperforms baseline models on four downstream tasks . it also has a few-shot ability that can mitigate the data scarcity problem .

Copied to clipboard

Divyansh Agarwal, Alexander Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, Chien-Sheng Wu

INTELMO: Enhancing Models’ Adoption of Interactive Interfaces (2023.emnlp-demo)

Challenge:	Prompt leakage poses a compelling security and privacy threat in LLM applications.
Approach:	They propose a model which leverages the LLM sycophancy effect and a threat model which fine tunes an open-source model to defend against prompt leakage attempts.
Outcome:	The proposed model elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting.

Copied to clipboard

Chunxu Yang, Chien-Sheng Wu, Lidiya Murakhovs’ka, Philippe Laban, Xiang Chen

Challenge:	INTELMO is an easy-to-use library to help model developers adopt user-faced interactive interfaces for their language models.
Approach:	They propose a library to help model developers adopt user-faced interactive interfaces and articles from real-time RSS sources for their language models.
Outcome:	The proposed library categorizes common NLP tasks and provides default style patterns . it provides developers with fine-grained and flexible control over user interfaces .

Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents (2025.findings-acl)

Copied to clipboard

Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Caiming Xiong, Shiva Kumar Pentyala, Chien-Sheng Wu

Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning (2023.acl-long)

Challenge:	Existing workflow extraction methods for service agents are time-consuming and outdated, causing inconsistent and inconsistent results.
Approach:	They propose a framework for extracting and evaluating dialog workflows from historical interactions.
Outcome:	The proposed framework improves workflow extraction by 12.16% over baseline.

Copied to clipboard

Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu

Exploring Neural Models for Query-Focused Summarization (2022.findings-naacl)

Challenge:	Large language models have shown impressive performance in following natural language instructions to solve unseen tasks.
Approach:	They propose two strategies to help large language models better leverage task instructions . they propose to remove 60% of tokens from the task definitions while maintaining model performance .
Outcome:	The proposed approach achieves 4.2 Rouge-L improvement over 119 unseen test tasks.

Copied to clipboard

Jesse Vig, Alexander Fabbri, Wojciech Kryscinski, Chien-Sheng Wu, Wenhao Liu

Information Extraction & Text Mining

Challenge:	Recent work in Query-focused summarization lacks a comprehensive study of the broad space of applicable modeling methods.
Approach:	They propose to explore two general classes of methods for Query-focused summarization: extractive-abstractive solutions and end-to-end models.
Outcome:	The proposed models achieve state-of-the-art on the QMSum dataset, with a margin of 3.38 ROUGE-1, 3.72 ROUGe2 and 3.28 ROUGEL-L.

Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles (2024.naacl-long)

Copied to clipboard

Kung-Hsiang Huang, Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu

Summarization

Challenge:	Existing studies on multi-document summarization focus on collating information that all sources agree upon, but the task of summarizing diverse information remains underexplored.
Approach:	They propose a task of summarizing diverse information encountered in multiple news articles encompassing the same event using a dataset curated by a large language model.
Outcome:	The proposed task aims to summarize diverse information in multiple news articles encompassing the same event . the proposed task is difficult due to its limited coverage and verbosity biases .

Unanswerability Evaluation for Retrieval Augmented Generation (2025.acl-long)

Copied to clipboard

Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu

Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020.emnlp-main)

Challenge:	Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but ignore the importance of appropriately rejecting unanswerable requests.
Approach:	They propose a framework to evaluate whether retrieval-augmented generation systems handle unanswerable queries specific to a given knowledge base.
Outcome:	The proposed framework synthesizes diverse and challenging queries for any given knowledge base and evaluates them with unanswered ratio and acceptable ratio metrics.

Copied to clipboard

Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, Caiming Xiong

Probing Task-Oriented Dialogue Representation from Language Models (2020.emnlp-main)

Challenge:	Existing work on few-shot intent classification without OOS has focused on the few-shot intent classification with out-of-scope intents.
Approach:	They propose to use BERT-style pairwise encoding to train a binary classifier that estimates the best matched training example for a user input.
Outcome:	The proposed approach achieves more stable and accurate in-domain and OOS detection accuracy than RoBERTa-based classifiers and embedding-based nearest neighbor approaches.

Copied to clipboard

Chien-Sheng Wu, Caiming Xiong

Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems (P19-1)

Challenge:	Using pre-trained language models, we find out which model has the most informative representation for task-oriented dialogue tasks.
Approach:	They propose a supervised classifier probe and unsupervised mutual information probe to investigate the mutual dependence between a real clustering and a representation clustering.
Outcome:	The proposed model is a supervised classifier probe and unsupervised mutual information probe.

Copied to clipboard

Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, Pascale Fung

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)

Challenge:	Existing approaches to dialogue state tracking are dependent on domain ontology and lack of sharing knowledge across domains.
Approach:	They propose a transferable dialogue state generator that generates dialogue states from utterances using copy mechanism.
Outcome:	Empirical results show that TRADE achieves state-of-the-art 48.62% joint goal accuracy for the five domains of MultiWOZ.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan

Information Extraction & Text Mining

Challenge:	Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied.
Approach:	They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance .
Outcome:	The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs.

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022.emnlp-main)

Copied to clipboard

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu

Improving Factual Consistency in Summarization with Compression-Based Post-Editing (2022.emnlp-main)

Challenge:	Structured knowledge grounding (SKG) uses structured knowledge to complete user requests . since inputs and outputs of SKG tasks are heterogeneous, they have been studied separately .
Approach:	They propose a framework that unifies 21 SKG tasks into a text-to-text format . they use unifiedSKG to benchmark T5 with different sizes .
Outcome:	The proposed framework unifies 21 SKG tasks into a text-to-text format . it achieves state-of-the-art performance on almost all of the 21 tasks, the authors show .

Copied to clipboard

Alex Fabbri, Prafulla Kumar Choubey, Jesse Vig, Chien-Sheng Wu, Caiming Xiong

Discord Questions: A Computational Approach To Diversity Analysis in News Coverage (2022.findings-emnlp)

Challenge:	Existing methods to improve factual consistency of summarization models fail to remove entity errors if a suitable input entity replacement is not available or insert erroneous content.
Approach:	They propose to remove extrinsic entity errors, or entities not in the source, to improve consistency while retaining the summary’s essential information and form.
Outcome:	The proposed model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and can be applied on top of another post-editor, improving accuracy by 38%.

Copied to clipboard

Philippe Laban, Chien-Sheng Wu, Lidiya Murakhovs’ka, Xiang Chen, Caiming Xiong

CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments (2025.naacl-long)

Challenge:	Modern news aggregators do the hard work of organizing the news, but choosing which source to read remains challenging.
Approach:	They propose a framework to help readers identify source differences and gain an understanding of news coverage diversity by generating questions with a diverse answer pool and reusing existing methods.
Outcome:	The proposed framework improves performance from current question generation methods by 5% and achieves 81% balanced accuracy on a realistic test set.

Copied to clipboard

Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, Chien-Sheng Wu

Challenge:	Existing benchmarks for evaluating CRM agents on work-related tasks are limited due to data privacy concerns.
Approach:	They propose a benchmark to evaluate AI agents on real-world CRM tasks . they use 16 commonly used industrial objects with high interconnectivity to simulate real data distributions.
Outcome:	The new benchmark evaluates AI agents on real-world customer service tasks . it includes 16 commonly used industrial objects with high interconnectivity . the results highlight the need for enhanced agent capabilities in function-calling and rule-following .

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)

Copied to clipboard

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Robustness in NLP

Challenge:	Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale.
Approach:	They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark.
Outcome:	The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement.

CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization (2023.findings-acl)

Copied to clipboard

Prafulla Kumar Choubey, Alex Fabbri, Jesse Vig, Chien-Sheng Wu, Wenhao Liu, Nazneen Rajani

Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage (2025.naacl-long)

Challenge:	Existing work suggests that the degree of hallucination depends on factual errors in training data.
Approach:	They propose a method to use training data to reduce hallucination by ensembling parameter variations in training data.
Outcome:	The proposed method improves on XSUM and CNN/DM datasets on human evaluations and factual metrics.

Copied to clipboard

Kaige Xie, Philippe Laban, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu

Information Retrieval

Challenge:	Existing evaluations of retrieval-augmented generation systems are limited . sub-question coverage measures how well a RAG system addresses different facets of a question.
Approach:	They propose a framework for evaluation based on sub-question coverage . they propose to decompose questions into sub-questions and classify them into three types .
Outcome:	The proposed evaluation framework measures how well a RAG system addresses different facets of a question.

Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems (2023.findings-emnlp)

Copied to clipboard

Lidiya Murakhovs’ka, Philippe Laban, Tian Xie, Caiming Xiong, Chien-Sheng Wu

Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets (2022.emnlp-main)

Challenge:	Existing conversational recommender systems focus on a single-shot approach to understand user preferences and provide recommendations.
Approach:	They propose a problem space for conversational agents that aim to provide both product recommendations and educational value through mixed-type mixed-initiative dialog.
Outcome:	The proposed framework can simulate salesbot and shopperbot agents and provide both product recommendations and educational value through mixed-type mixed-initiative dialog.

Copied to clipboard

Philippe Laban, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong