Papers by Michael Zeng

End-to-End Segmentation-based News Summarization (2022.findings-acl)

Challenge:	Existing datasets only annotate a binary label for each sentence pair. Existing models only annnotate binary labels for each phrase pair.
Approach:	They propose a novel binary paraphrase classification task that annotates the degree of paraphrase between sentences and a new annotation schema that labels the minimum spans of tokens in a sentence that don't have the corresponding paraphrases in the other sentence.
Outcome:	The proposed dataset can be used to train an automatic scorer for language generation evaluation.

Copied to clipboard

Yang Liu, Chenguang Zhu, Michael Zeng

Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)

Challenge:	Existing summarization systems only provide one genetic summary of the whole article, making it difficult for users to navigate the reading.
Approach:	They propose a task of segmenting a news article into multiple sections and generating the corresponding summary to each section.
Outcome:	The proposed model outperforms state-of-the-art models on a 27k news article dataset . it can jointly segment a document and produce the summary for each section .

Copied to clipboard

Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang

i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2024.findings-naacl)

Challenge:	Z-Code++ is a pre-trained language model optimized for abstractive text summarization.
Approach:	They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance.
Outcome:	The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings.

Copied to clipboard

Ziyi Yang, Mahmoud Khademi, Yichong Xu, Reid Pryzant, Yuwei Fang, Chenguang Zhu, Dongdong Chen, Yao Qian, Xuemei Gao, Yi-Ling Chen, Robert Gmyr, Naoyuki Kanda, Noel Codella, Bin Xiao, Yu Shi, Lu Yuan, Takuya Yoshioka, Michael Zeng, Xuedong Huang

Multimodality

Challenge:	i-Code V2 is one of the first models capable of generating natural language from any combination of Vision, Language, and Speech data.
Approach:	They propose to create a model that can generate natural language from any combination of Vision, Language, and Speech data.
Outcome:	i-Code V2 matches or outperforms state-of-the-art single- and dual-modality baselines on 7 multimodal tasks.

Fusing Context Into Knowledge Graph for Commonsense Question Answering (2021.findings-acl)

Copied to clipboard

Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, Xuedong Huang

Retrieval Enhanced Model for Commonsense Generation (2021.findings-acl)

Challenge:	Existing methods to combine language modeling and knowledge graphs (KG) lack the context to provide a more precise understanding of the concepts.
Approach:	They propose to use external entity descriptions to provide contextual information for commonsense question answering models.
Outcome:	The proposed model achieves state-of-the-art among non-generative models in OpenBookQA and is the first of its kind in the field.

Copied to clipboard

Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, Michael Zeng

Reasoning

Challenge:	Existing frameworks for commonsense generation are lacking for pre-trained models.
Approach:	They propose a framework that uses concept matching to retrieve prototype sentences and trainable sentence retriever to enhance pre-training and fine-tuning.
Outcome:	The proposed framework achieves state-of-the-art on the large-scale Common-Gen benchmark.

MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization (2021.naacl-main)

Copied to clipboard

Chenguang Zhu, Yang Liu, Jie Mei, Michael Zeng

Dialogue Systems & Conversational Agents

Challenge:	Existing datasets for dialogue summarization are limited to their small sizes and are built from a narrow domain.
Approach:	They propose a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries.
Outcome:	The proposed dataset is larger and contains multi-party conversations from multiple domains.

RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)

Copied to clipboard

Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji

Automatic Rule Induction for Efficient Semi-Supervised Learning (2022.findings-emnlp)

Challenge:	Existing methods for event prediction are incomplete and noisy.
Approach:	They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework .
Outcome:	The proposed framework can be applied to a wide variety of newsworthy scenarios.

Copied to clipboard

Reid Pryzant, Ziyi Yang, Yichong Xu, Chenguang Zhu, Michael Zeng

Responsible & Trustworthy NLP

Challenge:	Existing approaches to generalize from labeled and unlabeled data are difficult to explain and behave unreliably.
Approach:	They propose a framework for automatic discovery and integration of symbolic rules into pretrained transformer models by using an attention mechanism.
Outcome:	The proposed framework can improve state-of-the-art methods with no manual effort and minimal computational overhead.

Few-shot Natural Language Generation for Task-Oriented Dialog (2020.findings-emnlp)

Copied to clipboard

Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, Jianfeng Gao

UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization (2023.acl-long)

Challenge:	Existing methods for NLG depend on heavily annotated data, which is infeasible for new domains.
Approach:	They propose a system that converts a dialog act into a response in natural language . they propose 'nuclear language generation' to simulate a few-shot learning setting .
Outcome:	The proposed model outperforms existing methods on a large set of annotated datasets.

Copied to clipboard

Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang, Chenguang Zhu, Michael Zeng, Yue Zhang

Responsible & Trustworthy NLP

Challenge:	a new benchmark summarization model is being developed to train few-shot summarizers . a large number of summarizing tasks are required to perform well in heterogeneous datasets.
Approach:	They propose a few-shot summarization model pre-trained with multiple summarizing tasks . they propose 'uniSumm' to be prefix-tuned to excel at any few-shot summarisation task .
Outcome:	The proposed model outperforms baseline models under automatic and human evaluations and achieves comparable results in human evaluation.

MACSum: Controllable Summarization with Mixed Attributes (2023.tacl-1)

Copied to clipboard

Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, Rui Zhang

A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining (2020.findings-emnlp)

Challenge:	Existing work on controllable summarization with mixed attributes lacks designated annotations.
Approach:	They propose a human-annotated summarization benchmark for controllable summarizing with mixed attributes based on news and dialogue sources .
Outcome:	The proposed dataset contains human-annotated summarization datasets with mixed attributes . hard prompt models yield the best performance on most metrics and human evaluations . mixed-attribute control is still challenging for summarizing tasks .

Copied to clipboard

Chenguang Zhu, Ruochen Xu, Michael Zeng, Xuedong Huang

Mixed-Lingual Pre-training for Cross-lingual Summarization (2020.aacl-main)

Challenge:	Existing methods of summarizing meetings require complex multi-step pipelines that are intractable.
Approach:	They propose an abstractive summary network that adapts to meeting transcripts by hierarchical structure and role vectors.
Outcome:	The proposed model outperforms existing methods in both metrics and human evaluation.

Copied to clipboard

Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng, Xuedong Huang

Leveraging Knowledge in Multilingual Commonsense Reasoning (2022.findings-acl)

Challenge:	Cross-lingual summarization (CLS) aims at producing a summary in the target language for an article in the source language.
Approach:	They propose a mixed-lingual pre-training scheme that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models.
Outcome:	The proposed model improves on the translation and masked language models with no task-specific components and saves memory.

Copied to clipboard

Yuwei Fang, Shuohang Wang, Yichong Xu, Ruochen Xu, Siqi Sun, Chenguang Zhu, Michael Zeng

Multilinguality

Challenge:	Commonsense reasoning is a language-agnostic process, but most comprehensive knowledge sources are limited to a small number of languages, especially English.
Approach:	They propose to use English as a pivot language to integrate commonsense reasoning into models using a translate-retrieve-translate strategy.
Outcome:	The proposed model outperforms the state-of-the-art on the XCSR benchmarks.

Automatic Prompt Optimization with “Gradient Descent” and Beam Search (2023.emnlp-main)

Copied to clipboard

Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, Michael Zeng

Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)

Challenge:	Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort.
Approach:	They propose an algorithm that uses numerical gradient descent to automatically improve prompts by rewriting vague task descriptions into more precise annotation instructions.
Outcome:	The proposed algorithm outperforms previous methods and improves performance on three benchmark NLP tasks and the novel problem of LLM jailbreak detection.

Copied to clipboard

Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang

AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022.findings-emnlp)

Challenge:	Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized.
Approach:	They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions.
Outcome:	The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks.

Copied to clipboard

Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang

Narrate Dialogues for Better Summarization (2022.findings-emnlp)

Challenge:	Prompt-based learning can tackle zero-shot and few-shot NLP tasks . authors propose a method that makes use of pre-trained language models .
Approach:	They propose to map NLP tasks into natural language prompts, which are then filled by pre-trained language models.
Outcome:	The proposed method outperforms standard prompt-based methods in few-shot settings.

Copied to clipboard

Ruochen Xu, Chenguang Zhu, Michael Zeng

Dialogue Systems & Conversational Agents

Challenge:	Recent work on dialogue summarization models focuses on generating concise summaries for multi-party dialogues.
Approach:	They propose several ways to convert dialogue into a third-person narrative style . they propose to use narration as a valuable annotation for LLMs .
Outcome:	Empirical results show that the proposed approach achieves higher scores on ROUGE and a factual correctness metric.

Multi-task Learning for Natural Language Generation in Task-Oriented Dialogue (D19-1)

Copied to clipboard

Chenguang Zhu, Michael Zeng, Xuedong Huang

Enhancing Factual Consistency of Abstractive Summarization (2021.naacl-main)

Challenge:	Existing methods to generate natural language for task-oriented dialogues lack naturalness and variation in language.
Approach:	They propose a multi-task learning framework for natural language generation that explicitly targets for naturalness in generated responses via an unconditioned language model.
Outcome:	The proposed framework outperforms existing models across multiple datasets in the study of natural language generation.

Copied to clipboard

Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, Meng Jiang

InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT (2023.findings-emnlp)

Challenge:	Abstractive summarization models often distort or fabricate facts in articles . factual inconsistency is a common problem with abstractive summaries .
Approach:	They propose a fact-aware summarization model FASum to extract factual relations into the summary generation process via graph attention.
Outcome:	The proposed model can produce abstractive summaries with higher factual consistency compared with existing systems and corrects factual errors via modifying only a few keywords.

Copied to clipboard

Yichong Xu, Ruochen Xu, Dan Iter, Yang Liu, Shuohang Wang, Chenguang Zhu, Michael Zeng

Photon: A Robust Cross-Domain Text-to-SQL System (2020.acl-demos)

Challenge:	Recent advances in large language models have revolutionized the way summarization is generated.
Approach:	They propose a summarization model derived from GPT-3.5 through distillation that is compact and has comparable summarizing capabilities to GPT-3.
Outcome:	The proposed model outperforms the established best small models in prefix-tuning and full-data fine-tuned scenarios.

Copied to clipboard

Jichuan Zeng, Xi Victoria Lin, Steven C.H. Hoi, Richard Socher, Caiming Xiong, Michael Lyu, Irwin King

Programming Languages in NLP

Challenge:	Existing natural language interfaces to databases are ambiguous or untranslatable . we present a robust, modular cross-domain text-to-SQL system .
Approach:	They propose a system that flags natural language input to which a SQL mapping cannot be immediately determined.
Outcome:	The proposed system can flag natural language input to which a SQL mapping cannot be determined.

LMGQS: A Large-scale Dataset for Query-focused Summarization (2023.findings-emnlp)

Copied to clipboard

Ruochen Xu, Song Wang, Yang Liu, Shuohang Wang, Yichong Xu, Dan Iter, Pengcheng He, Chenguang Zhu, Michael Zeng

Summarization

Challenge:	Lack of large-scale datasets for query-focused summarization hinders model development . lack of data limits the ability of QFS models to train robust neural models .
Approach:	They propose to generate a query for each summary sentence in a generic summarization annotation using a pretrained language model.
Outcome:	The proposed model achieves state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks.

Unsupervised Multi-Granularity Summarization (2022.findings-emnlp)

Copied to clipboard

Ming Zhong, Yang Liu, Suyu Ge, Yuning Mao, Yizhu Jiao, Xingxing Zhang, Yichong Xu, Chenguang Zhu, Michael Zeng, Jiawei Han

TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising (2020.findings-emnlp)

Challenge:	Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines.
Approach:	They propose to rank events by their salience and annotate a benchmark for GranuSum that contains multiple summaries at different granularities for each document cluster.
Outcome:	The proposed framework is capable of producing multi-granular summaries in unsupervised manner over strong baselines.

Copied to clipboard

Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, Eric Darve

Responsible & Trustworthy NLP

Challenge:	Existing abstractive summarization models ignore abundant unlabeled corpora resources . TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets .
Approach:	They propose a transformer-based unsupervised text summarization system with pretraining on large-scale data.
Outcome:	The proposed system outperforms baseline models on NYT, CNN/DM and English Gigaword datasets with various document styles.

SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021.naacl-main)

Copied to clipboard

Yu-An Chung, Chenguang Zhu, Michael Zeng

Multimodality

Challenge:	Experimental results show that SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%.
Approach:	They propose a semi-supervised learning framework to jointly pre-train the speech and language modules using unpaired speech and text.
Outcome:	The proposed framework improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%.

i-Code Studio: A Configurable and Composable Framework for Integrative AI (2024.emnlp-demo)

Copied to clipboard

Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang

Multimodality

Challenge:	Existing frameworks for Integrative AI lack flexibility and composability to handle multimodal tasks.
Approach:	They propose a configurable framework for Integrative AI that orchestrates multiple pre-trained models to conduct complex multimodal tasks.
Outcome:	The proposed framework achieves impressive results on zero-shot multimodal tasks . it can communicate and personalize for users, and it can be used in a multimodal agent .

KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering (2022.acl-long)

Copied to clipboard

Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng

Task Compass: Scaling Multi-task Pre-training with Task Prefix (2022.findings-emnlp)

Challenge:	Open-Domain Question Answering (ODQA) models typically include a retrieving module and a reading module.
Approach:	They propose a new open-domain question-answering framework that uses a knowledge-enhanced version of FiD to improve the approach.
Outcome:	The proposed model improves on ODQA benchmark datasets with less than 40% computation cost.

Copied to clipboard

Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng