Papers by Chenguang Zhu

PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents (2024.eacl-long)

Challenge:	Existing datasets only annotate a binary label for each sentence pair. Existing models only annnotate binary labels for each phrase pair.
Approach:	They propose a novel binary paraphrase classification task that annotates the degree of paraphrase between sentences and a new annotation schema that labels the minimum spans of tokens in a sentence that don't have the corresponding paraphrases in the other sentence.
Outcome:	The proposed dataset can be used to train an automatic scorer for language generation evaluation.

Copied to clipboard

Simeng Sun, Yang Liu, Shuohang Wang, Dan Iter, Chenguang Zhu, Mohit Iyyer

Retrieval Augmentation for Commonsense Reasoning: A Unified Approach (2022.emnlp-main)

Challenge:	Using chain-of-thought prompting, large language models perform better on complex reasoning tasks.
Approach:	They propose a prompting framework that decomposes a question into a sequence of actions and executes them over the document to obtain the answer.
Outcome:	The proposed framework outperforms zero-shot and chain-of-thought prompting on a QuALITY dataset . it proposes a plan based on actions mined from a training set and executes it step by step .

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Zhihan Zhang, Shuohang Wang, Zhuosheng Zhang, Yuwei Fang, Meng Jiang

Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022.emnlp-main)

Challenge:	Existing methods for retrieving encyclopedic knowledge lack a large corpus and effective commonsense retriever.
Approach:	They propose a framework for retrieval-augmented commonsense reasoning with a large commonsensense corpus and a commonseense retriever.
Outcome:	The proposed framework outperforms existing methods on commonsense reasoning tasks.

Copied to clipboard

Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han

End-to-End Segmentation-based News Summarization (2022.findings-acl)

Challenge:	Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics.
Approach:	They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer.
Outcome:	The proposed evaluator improves on three typical NLG tasks and improves with external knowledge.

Copied to clipboard

Yang Liu, Chenguang Zhu, Michael Zeng

Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)

Challenge:	Existing summarization systems only provide one genetic summary of the whole article, making it difficult for users to navigate the reading.
Approach:	They propose a task of segmenting a news article into multiple sections and generating the corresponding summary to each section.
Outcome:	The proposed model outperforms state-of-the-art models on a 27k news article dataset . it can jointly segment a document and produce the summary for each section .

Copied to clipboard

Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang

Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022.findings-acl)

Challenge:	Z-Code++ is a pre-trained language model optimized for abstractive text summarization.
Approach:	They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance.
Outcome:	The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings.

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, Meng Jiang

i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2024.findings-naacl)

Challenge:	Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks.
Approach:	They propose a method that diversifies the generative reasoning by a mixture of expert strategy on commonsense knowledge graphs to encourage various generation outputs.
Outcome:	The proposed method improves diversity while achieving on par performance on two GCR benchmarks, based on both automatic and human evaluations.

Copied to clipboard

Ziyi Yang, Mahmoud Khademi, Yichong Xu, Reid Pryzant, Yuwei Fang, Chenguang Zhu, Dongdong Chen, Yao Qian, Xuemei Gao, Yi-Ling Chen, Robert Gmyr, Naoyuki Kanda, Noel Codella, Bin Xiao, Yu Shi, Lu Yuan, Takuya Yoshioka, Michael Zeng, Xuedong Huang

Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering (2022.emnlp-main)

Challenge:	i-Code V2 is one of the first models capable of generating natural language from any combination of Vision, Language, and Speech data.
Approach:	They propose to create a model that can generate natural language from any combination of Vision, Language, and Speech data.
Outcome:	i-Code V2 matches or outperforms state-of-the-art single- and dual-modality baselines on 7 multimodal tasks.

Copied to clipboard

Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, Yizhou Sun

An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (2021.findings-emnlp)

Challenge:	Existing Language Models lack the power to store all required knowledge, resulting in a lack of ability to infer out-of-context knowledge.
Approach:	They propose a Knowledge Interaction Layer that can be flexibly plugged into existing Transformer-based LMs to interact with a differentiable Knowledge Graph Reasoning module collaboratively.
Outcome:	The proposed model can be plugged into existing Transformer-based LMs to interact with a differentiable Knowledge Graph Reasoning module collaboratively.

Copied to clipboard

Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, Dragomir Radev

Information Retrieval

Challenge:	Existing models for dialogue summarization focus on extracting the main events of short conversations, but real-world dialogues are difficult to train.
Approach:	They propose three strategies to deal with the lengthy input problem and locate relevant information using long dialogue datasets.
Outcome:	The retrieve-then-summarize pipeline models yield the best performance on three long dialogue datasets.

DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization (2022.acl-long)

Copied to clipboard

Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, Dragomir Radev

Summarization of Dialogues and Conversations At Scale (2023.eacl-tutorials)

Challenge:	Existing models struggle with summarizing long text due to high memory complexity of the full self-attention.
Approach:	They propose a dynamic latent extraction approach for abstractive long-input summarization that treats extracted text snippets as latent variables and allows dynamic attention weights during decoding.
Outcome:	The proposed method outperforms existing methods on GovReport, QMSum, and arXiv while yielding strong results on arX.

Copied to clipboard

Diyi Yang, Chenguang Zhu

Natural Language Interfaces

Challenge:	Conversations are the natural communication format for people.
Approach:	This tutorial will survey the cutting-edge methods for summarizing written and spoken conversation.
Outcome:	This tutorial will examine the cutting-edge methods for summarizing written and spoken conversations, covering key sub-areas whose combination is needed for a successful solution.

The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions (2023.emnlp-main)

Copied to clipboard

Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, Jiawei Han

RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems (2021.acl-long)

Challenge:	Recent advances in large language models (LLMs) have produced models that exhibit remarkable performance across a variety of NLP tasks.
Approach:	They analyze a large-scale collection of user-GPT conversations to identify a significant gap between academic research in NLP and the needs of real-world NLP applications.
Outcome:	The proposed model outperforms existing models in a large-scale collection of user-GPT conversations and identifies a significant gap between the tasks that users frequently request from LLMs and the tasks commonly studied in academic research.

Copied to clipboard

Baolin Peng, Chunyuan Li, Zhu Zhang, Chenguang Zhu, Jinchao Li, Jianfeng Gao

Robustness in NLP

Challenge:	Existing task-oriented dialog systems are less than satisfactory in robustness evaluation . existing systems are weak in robustity evaluation based on pre-training and fine-tuning .
Approach:	They propose to use a set of training examples to evaluate model generalization ability . they propose to include tasks with limited training data to favor models with strong generalization abilities .
Outcome:	The proposed model generalizes well with limited training data and is robust to user input across domains.

Fusing Context Into Knowledge Graph for Commonsense Question Answering (2021.findings-acl)

Copied to clipboard

Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, Xuedong Huang

In-Context Demonstration Selection with Cross Entropy Difference (2023.findings-emnlp)

Challenge:	Existing methods to combine language modeling and knowledge graphs (KG) lack the context to provide a more precise understanding of the concepts.
Approach:	They propose to use external entity descriptions to provide contextual information for commonsense question answering models.
Outcome:	The proposed model achieves state-of-the-art among non-generative models in OpenBookQA and is the first of its kind in the field.

Copied to clipboard

Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu

Parameter-free Sentence Embedding via Orthogonal Basis (D19-1)

Challenge:	Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks.
Approach:	They propose a cross-entropy difference method for selecting in-context demonstrations that uses parameter efficient finetuning to train small models on training data.
Outcome:	The proposed method outperforms baseline selection methods on a mix-domain dataset and shows that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example.

Copied to clipboard

Ziyi Yang, Chenguang Zhu, Weizhu Chen

Representation Learning

Challenge:	Existing methods to build sentence embeddings are parameterized and require training to optimize their parameters.
Approach:	They propose a non-parameterized method to combine pre-trained word embeddings into sentence representations using an orthogonal basis of the word vector subspace and its surrounding context.
Outcome:	The proposed method shows superior performance on 11 downstream NLP tasks and is competitive to other methods relying on large amounts of labelled data or prolonged training time.

Leveraging Locality in Abstractive Text Summarization (2022.emnlp-main)

Copied to clipboard

Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah, Dragomir Radev

Summarization

Challenge:	Neural attention models have improved on many natural language processing tasks, but their quadratic memory complexity hinders their applications in long text summarization.
Approach:	They propose to use a restricted context to study locality in text summarization . they propose to employ a quadratic memory growth with respect to the input length .
Outcome:	The proposed model has better performance than baseline models with efficient attention modules.

Retrieval Enhanced Model for Commonsense Generation (2021.findings-acl)

Copied to clipboard

Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, Michael Zeng

Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022.acl-long)

Challenge:	Existing frameworks for commonsense generation are lacking for pre-trained models.
Approach:	They propose a framework that uses concept matching to retrieve prototype sentences and trainable sentence retriever to enhance pre-training and fine-tuning.
Outcome:	The proposed framework achieves state-of-the-art on the large-scale Common-Gen benchmark.

Copied to clipboard

Woojeong Jin, Dong-Ho Lee, Chenguang Zhu, Jay Pujara, Xiang Ren

MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization (2021.naacl-main)

Challenge:	Pre-trained language models lack visual knowledge of common objects due to reporting bias.
Approach:	They investigate whether integrating visual knowledge into a language model can fill the gap . they use captions and images to transfer visual knowledge to 5 downstream tasks .
Outcome:	The proposed model can improve performance on 5 tasks that may need visual knowledge to solve the problem.

Copied to clipboard

Chenguang Zhu, Yang Liu, Jie Mei, Michael Zeng

Dialogue Systems & Conversational Agents

Challenge:	Existing datasets for dialogue summarization are limited to their small sizes and are built from a narrow domain.
Approach:	They propose a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries.
Outcome:	The proposed dataset is larger and contains multi-party conversations from multiple domains.

APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning (2023.acl-long)

Copied to clipboard

Soumya Sanyal, Yichong Xu, Shuohang Wang, Ziyi Yang, Reid Pryzant, Wenhao Yu, Chenguang Zhu, Xiang Ren

Automatic Rule Induction for Efficient Semi-Supervised Learning (2022.findings-emnlp)

Challenge:	Existing methods to improve logical reasoning skills require complex data processing.
Approach:	They propose an adaptive pretraining approach to improve logical reasoning over text . they use a subset of Wikipedia sentences for pretraining and a sentence-level classification loss .
Outcome:	The proposed model outperforms baselines on LogiQA and ReClor.

Copied to clipboard

Reid Pryzant, Ziyi Yang, Yichong Xu, Chenguang Zhu, Michael Zeng

Responsible & Trustworthy NLP

Challenge:	Existing approaches to generalize from labeled and unlabeled data are difficult to explain and behave unreliably.
Approach:	They propose a framework for automatic discovery and integration of symbolic rules into pretrained transformer models by using an attention mechanism.
Outcome:	The proposed framework can improve state-of-the-art methods with no manual effort and minimal computational overhead.

Few-shot Natural Language Generation for Task-Oriented Dialog (2020.findings-emnlp)

Copied to clipboard

Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, Jianfeng Gao

GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)

Challenge:	Existing methods for NLG depend on heavily annotated data, which is infeasible for new domains.
Approach:	They propose a system that converts a dialog act into a response in natural language . they propose 'nuclear language generation' to simulate a few-shot learning setting .
Outcome:	The proposed model outperforms existing methods on a large set of annotated datasets.

Copied to clipboard

Shen Zheng, Yuyu Zhang, Yijie Zhu, Chenguang Xi, Pengyang Gao, Zhou Xun, Kevin Chang

UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization (2023.acl-long)

Challenge:	Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results.
Approach:	They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories.
Outcome:	The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories.

Copied to clipboard

Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang, Chenguang Zhu, Michael Zeng, Yue Zhang

Responsible & Trustworthy NLP

Challenge:	a new benchmark summarization model is being developed to train few-shot summarizers . a large number of summarizing tasks are required to perform well in heterogeneous datasets.
Approach:	They propose a few-shot summarization model pre-trained with multiple summarizing tasks . they propose 'uniSumm' to be prefix-tuned to excel at any few-shot summarisation task .
Outcome:	The proposed model outperforms baseline models under automatic and human evaluations and achieves comparable results in human evaluation.

MACSum: Controllable Summarization with Mixed Attributes (2023.tacl-1)

Copied to clipboard

Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, Rui Zhang

A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining (2020.findings-emnlp)

Challenge:	Existing work on controllable summarization with mixed attributes lacks designated annotations.
Approach:	They propose a human-annotated summarization benchmark for controllable summarizing with mixed attributes based on news and dialogue sources .
Outcome:	The proposed dataset contains human-annotated summarization datasets with mixed attributes . hard prompt models yield the best performance on most metrics and human evaluations . mixed-attribute control is still challenging for summarizing tasks .

Copied to clipboard

Chenguang Zhu, Ruochen Xu, Michael Zeng, Xuedong Huang

Self-Generated Critiques Boost Reward Modeling for Language Models (2025.naacl-long)

Challenge:	Existing methods of summarizing meetings require complex multi-step pipelines that are intractable.
Approach:	They propose an abstractive summary network that adapts to meeting transcripts by hierarchical structure and role vectors.
Outcome:	The proposed model outperforms existing methods in both metrics and human evaluation.

Copied to clipboard

Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou

Knowledge-Augmented Methods for Natural Language Processing (2022.acl-tutorials)

Challenge:	Existing reward models produce scalar scores and struggle to incorporate critiques in a natural language format.
Approach:	They propose a framework that predicts critiques and rewards using self-generated critiques without extra supervision.
Outcome:	The proposed framework improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges.

Copied to clipboard

Chenguang Zhu, Yichong Xu, Xiang Ren, Bill Yuchen Lin, Meng Jiang, Wenhao Yu

Mixed-Lingual Pre-training for Cross-lingual Summarization (2020.aacl-main)

Challenge:	Knowledge in natural language processing (NLP) is a rising trend especially after the advent of large scale pre-trained models.
Approach:	This tutorial introduces the key steps in integrating knowledge into natural language processing (NLP) it introduces knowledge grounding from text, knowledge representation and fusing.
Outcome:	This tutorial introduces the key steps in integrating knowledge into natural language processing including knowledge grounding from text, knowledge representation and fusing.

Copied to clipboard

Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng, Xuedong Huang

Leveraging Knowledge in Multilingual Commonsense Reasoning (2022.findings-acl)

Challenge:	Cross-lingual summarization (CLS) aims at producing a summary in the target language for an article in the source language.
Approach:	They propose a mixed-lingual pre-training scheme that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models.
Outcome:	The proposed model improves on the translation and masked language models with no task-specific components and saves memory.

Copied to clipboard

Yuwei Fang, Shuohang Wang, Yichong Xu, Ruochen Xu, Siqi Sun, Chenguang Zhu, Michael Zeng

Multilinguality

Challenge:	Commonsense reasoning is a language-agnostic process, but most comprehensive knowledge sources are limited to a small number of languages, especially English.
Approach:	They propose to use English as a pivot language to integrate commonsense reasoning into models using a translate-retrieve-translate strategy.
Outcome:	The proposed model outperforms the state-of-the-art on the XCSR benchmarks.

WPO: Enhancing RLHF with Weighted Preference Optimization (2024.emnlp-main)

Copied to clipboard

Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu

Sentence-Permuted Paragraph Generation (2021.emnlp-main)

Challenge:	Off-policy preference optimization suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization.
Approach:	They propose a method to simulate on-policy learning with off-police preference data.
Outcome:	The proposed method outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 and MT-bench.

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Tong Zhao, Zhichun Guo, Meng Jiang

Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models (2023.findings-emnlp)

Challenge:	Existing models produce similar contents from homogenized contexts due to the fixed left-to-right sentence order.
Approach:	They propose a framework permuting sentence orders to improve content diversity of multi-sentence paragraphs by permutating the sentence orders.
Outcome:	The proposed framework produces more diverse outputs with higher quality than existing models.

Copied to clipboard

Zhihan Zhang, Shuohang Wang, Wenhao Yu, Yichong Xu, Dan Iter, Qingkai Zeng, Yang Liu, Chenguang Zhu, Meng Jiang

Automatic Prompt Optimization with “Gradient Descent” and Beam Search (2023.emnlp-main)

Challenge:	Large language models can perform a wide range of tasks by following natural language instructions without task-specific fine-tuning.
Approach:	They propose a method to automatically improve the quality of LLM instructions . they leverage the generative ability of LMS to generate diverse candidate instructions based on a scoring model trained on 575 existing NLP tasks.
Outcome:	The proposed method surpasses human-written and LLM-generated instructions on 118 out-of-domain tasks.

Copied to clipboard

Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, Michael Zeng

Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)

Challenge:	Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort.
Approach:	They propose an algorithm that uses numerical gradient descent to automatically improve prompts by rewriting vague task descriptions into more precise annotation instructions.
Outcome:	The proposed algorithm outperforms previous methods and improves performance on three benchmark NLP tasks and the novel problem of LLM jailbreak detection.

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang

MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering (2023.findings-emnlp)

Challenge:	Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized.
Approach:	They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions.
Outcome:	The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks.

Copied to clipboard

Mahmoud Khademi, Ziyi Yang, Felipe Frujeri, Chenguang Zhu

SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents (2022.acl-long)

Challenge:	Recent knowledge-based visual question answering approaches miss visual information captured by captions and cannot fully utilize the visual information required to answer the question.
Approach:	They propose a framework that extracts visual information from an image and prompts an LLM to extract query-specific knowledge from the extracted textual information.
Outcome:	Empirical results show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.

Copied to clipboard

Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, Rui Zhang

AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022.findings-emnlp)

Challenge:	Existing methods to handle long text are limited due to time and memory complexity and limited input lengths.
Approach:	They propose a multi-stage split-then-summarize framework for long input summarization . their framework can process input text of arbitrary length by adjusting the number of stages .
Outcome:	The proposed framework outperforms existing methods on three long meeting summarization datasets and on a long document summarizing dataset.

Copied to clipboard

Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang

Small Models are Valuable Plug-ins for Large Language Models (2024.findings-acl)

Challenge:	Prompt-based learning can tackle zero-shot and few-shot NLP tasks . authors propose a method that makes use of pre-trained language models .
Approach:	They propose to map NLP tasks into natural language prompts, which are then filled by pre-trained language models.
Outcome:	The proposed method outperforms standard prompt-based methods in few-shot settings.

Copied to clipboard

Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, Julian McAuley

Narrate Dialogues for Better Summarization (2022.findings-emnlp)

Challenge:	Large-scale pre-trained language models are difficult to fine-tune due to their huge weights and limited context length.
Approach:	They propose an approach which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks.
Outcome:	The proposed approach overcomes the challenges of poor performance and instability of In-Context Learning (ICL) while reducing the complexity of in-context learning.

Copied to clipboard

Ruochen Xu, Chenguang Zhu, Michael Zeng

Dialogue Systems & Conversational Agents

Challenge:	Recent work on dialogue summarization models focuses on generating concise summaries for multi-party dialogues.
Approach:	They propose several ways to convert dialogue into a third-person narrative style . they propose to use narration as a valuable annotation for LLMs .
Outcome:	Empirical results show that the proposed approach achieves higher scores on ROUGE and a factual correctness metric.

Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering (N19-1)

Copied to clipboard

Jianmo Ni, Chenguang Zhu, Weizhu Chen, Julian McAuley

Question Answering

Challenge:	Existing approaches to open-domain question answering struggle to retrieve indirectly related evidence when no direct evidence is provided.
Approach:	They propose a retriever-reader model that learns to attend on essential terms during the question answering process.
Outcome:	The proposed model achieves the state-of-the-art on multiple open-domain QA datasets and achieves a 'reader-reader' level.

Multi-task Learning for Natural Language Generation in Task-Oriented Dialogue (D19-1)

Copied to clipboard

Chenguang Zhu, Michael Zeng, Xuedong Huang

Enhancing Factual Consistency of Abstractive Summarization (2021.naacl-main)

Challenge:	Existing methods to generate natural language for task-oriented dialogues lack naturalness and variation in language.
Approach:	They propose a multi-task learning framework for natural language generation that explicitly targets for naturalness in generated responses via an unconditioned language model.
Outcome:	The proposed framework outperforms existing models across multiple datasets in the study of natural language generation.

Copied to clipboard

Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, Meng Jiang

Embedding Imputation with Grounded Language Information (P19-1)

Challenge:	Abstractive summarization models often distort or fabricate facts in articles . factual inconsistency is a common problem with abstractive summaries .
Approach:	They propose a fact-aware summarization model FASum to extract factual relations into the summary generation process via graph attention.
Outcome:	The proposed model can produce abstractive summaries with higher factual consistency compared with existing systems and corrects factual errors via modifying only a few keywords.

Copied to clipboard

Ziyi Yang, Chenguang Zhu, Vin Sachidananda, Eric Darve

Representation Learning

Challenge:	Existing approaches to embedding imputation use vector space properties or subword information to learn representations for rare or unseen words.
Approach:	They propose an online method to construct a knowledge graph from grounded information and an algorithm to map from the resulting graph to the space of the pre-trained embeddings.
Outcome:	The proposed method improves on a card-660 task by 11% and 17.8% respectively using GloVe embeddings.

InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT (2023.findings-emnlp)

Copied to clipboard

Yichong Xu, Ruochen Xu, Dan Iter, Yang Liu, Shuohang Wang, Chenguang Zhu, Michael Zeng

A Unified Encoder-Decoder Framework with Entity Memory (2022.emnlp-main)

Challenge:	Recent advances in large language models have revolutionized the way summarization is generated.
Approach:	They propose a summarization model derived from GPT-3.5 through distillation that is compact and has comparable summarizing capabilities to GPT-3.
Outcome:	The proposed model outperforms the established best small models in prefix-tuning and full-data fine-tuned scenarios.

Copied to clipboard

Zhihan Zhang, Wenhao Yu, Chenguang Zhu, Meng Jiang

Injecting Entity Types into Entity-Guided Text Generation (2021.emnlp-main)

Challenge:	Existing approaches to index, retrieve, and read documents as evidence suffer from large computational overheads.
Approach:	They propose an encoder-decoder framework with an entity memory that stores entity knowledge as latent representations and pre-trained on Wikipedia along with encoder parameters.
Outcome:	The proposed framework outperforms memory-based and non-memory encoder-decoder models on various entity-intensive question answering and generation tasks.

Copied to clipboard

Xiangyu Dong, Wenhao Yu, Chenguang Zhu, Meng Jiang

LMGQS: A Large-scale Dataset for Query-focused Summarization (2023.findings-emnlp)

Challenge:	Recent advances in deep generative modeling have led to significant advances in natural language generation (NLG).
Approach:	They propose to model the entity type carefully in the decoding phase to generate contextual words accurately.
Outcome:	The proposed model produces a target sequence based on a given list of entities.

Copied to clipboard

Ruochen Xu, Song Wang, Yang Liu, Shuohang Wang, Yichong Xu, Dan Iter, Pengcheng He, Chenguang Zhu, Michael Zeng

Summarization

Challenge:	Lack of large-scale datasets for query-focused summarization hinders model development . lack of data limits the ability of QFS models to train robust neural models .
Approach:	They propose to generate a query for each summary sentence in a generic summarization annotation using a pretrained language model.
Outcome:	The proposed model achieves state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks.

Unsupervised Multi-Granularity Summarization (2022.findings-emnlp)

Copied to clipboard

Ming Zhong, Yang Liu, Suyu Ge, Yuning Mao, Yizhu Jiao, Xingxing Zhang, Yichong Xu, Chenguang Zhu, Michael Zeng, Jiawei Han

TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising (2020.findings-emnlp)

Challenge:	Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines.
Approach:	They propose to rank events by their salience and annotate a benchmark for GranuSum that contains multiple summaries at different granularities for each document cluster.
Outcome:	The proposed framework is capable of producing multi-granular summaries in unsupervised manner over strong baselines.

Copied to clipboard

Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, Eric Darve

Responsible & Trustworthy NLP

Challenge:	Existing abstractive summarization models ignore abundant unlabeled corpora resources . TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets .
Approach:	They propose a transformer-based unsupervised text summarization system with pretraining on large-scale data.
Outcome:	The proposed system outperforms baseline models on NYT, CNN/DM and English Gigaword datasets with various document styles.

SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021.naacl-main)

Copied to clipboard

Yu-An Chung, Chenguang Zhu, Michael Zeng

Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets (2024.findings-emnlp)

Challenge:	Experimental results show that SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%.
Approach:	They propose a semi-supervised learning framework to jointly pre-train the speech and language modules using unpaired speech and text.
Outcome:	The proposed framework improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%.

Copied to clipboard

Sathish Reddy Indurthi, Wenxuan Zhou, Shamil Chollampatt, Ravi Agrawal, Kaiqiang Song, Lingxiao Zhao, Chenguang Zhu

Multilinguality

Challenge:	Advancements in Large Language Models (LLMs) have significantly enhanced instruction-following capabilities, but most IFT datasets are predominantly in English, limiting model performance in other languages.
Approach:	They propose a method for collecting multilingual IFT datasets that preserves linguistic naturalness and ensures prompt diversity.
Outcome:	Experiments show that LLMs fine-tuned using this method show significant improvements in generative and discriminative tasks.

i-Code Studio: A Configurable and Composable Framework for Integrative AI (2024.emnlp-demo)

Copied to clipboard

Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang

KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering (2022.acl-long)

Challenge:	Existing frameworks for Integrative AI lack flexibility and composability to handle multimodal tasks.
Approach:	They propose a configurable framework for Integrative AI that orchestrates multiple pre-trained models to conduct complex multimodal tasks.
Outcome:	The proposed framework achieves impressive results on zero-shot multimodal tasks . it can communicate and personalize for users, and it can be used in a multimodal agent .

Copied to clipboard

Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng

Task Compass: Scaling Multi-task Pre-training with Task Prefix (2022.findings-emnlp)

Challenge:	Open-Domain Question Answering (ODQA) models typically include a retrieving module and a reading module.
Approach:	They propose a new open-domain question-answering framework that uses a knowledge-enhanced version of FiD to improve the approach.
Outcome:	The proposed model improves on ODQA benchmark datasets with less than 40% computation cost.

Copied to clipboard

Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng

G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)

Challenge:	Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks.
Approach:	They propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks.
Outcome:	The proposed model can be used as a foundation backbone for a wide range of tasks and as augmentation tool for data augmentation with complementary tasks.

Copied to clipboard

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu

How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)

Challenge:	Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks.
Approach:	They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs.
Outcome:	The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments.

Copied to clipboard

Simeng Sun, Yang Liu, Dan Iter, Chenguang Zhu, Mohit Iyyer