Papers by Nan Duan

WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach (2021.findings-emnlp)

Challenge:	Adapting a model to a handful of personalized data is challenging, authors say . standard fine-tuning requires hundreds of millions of parameters for each user .
Approach:	They propose a lightweight method that clamps millions of parameters of a Transformer model and optimizes a tiny user-specific vector.
Outcome:	The proposed method improves accuracy on Yelp and IMDB datasets and reduces the number of parameters added for each user.

Copied to clipboard

Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, Nan Duan

Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy (2021.emnlp-main)

Challenge:	Pre-trained language models perform well on learning sentence semantics when fine-tuned with supervised data.
Approach:	They conduct a thorough examination of pretrained model based unsupervised sentence embeddings.
Outcome:	The proposed approach improves on whitening-based vector normalization with less than 10 lines of code.

Copied to clipboard

Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy

CULG: Commercial Universal Language Generation (2022.naacl-industry)

Challenge:	Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks.
Approach:	They propose an architecture-independent approach for leveraging syntactic hierarchies of source code . they use syntax trees to extract syntak hierarchical structures and integrate them into context window .
Outcome:	The proposed approach achieves state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark.

Copied to clipboard

Haonan Li, Yameng Huang, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval (2022.findings-acl)

Challenge:	Pre-trained language models have improved performance for many NLP tasks in finance and healthcare.
Approach:	They propose a large-scale commercial universal language generation model which is pre-trained on a corpus drawn from 10 markets across 7 languages.
Outcome:	The proposed model outperforms other models on commercial generation tasks and on other markets, languages, and tasks.

Copied to clipboard

Canwen Xu, Daya Guo, Nan Duan, Julian McAuley

Query Rewriting in Retrieval-Augmented Large Language Models (2023.emnlp-main)

Challenge:	Experimental results show that LaPraDoR is state-of-the-art compared with supervised dense retrieval models.
Approach:	They propose a pretrained dual-tower dense retriever that does not require supervised data for training.
Outcome:	The proposed method achieves state-of-the-art performance on 18 datasets of 9 zero-shot text retrieval tasks.

Copied to clipboard

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, Nan Duan

Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder (2020.acl-main)

Challenge:	Existing studies focus on adapting either the retriever or the reader, but this approach is more focused on adaptation of the query itself.
Approach:	They propose a new framework for retrieval-augmented Large Language Models . they propose rewrite-retrieve-read instead of retrieve-then-read .
Outcome:	The proposed framework improves performance on downstream tasks, open-domain QA and multiple-choice QA.

Copied to clipboard

Daya Guo, Duyu Tang, Nan Duan, Jian Yin, Daxin Jiang, Ming Zhou

Hashing based Efficient Inference for Image-Text Matching (2021.findings-acl)

Challenge:	Existing approaches for inferential text generation ignore context that is not explicitly provided . Existing models ignore background knowledge that provides crucial evidence to generate inferences .
Approach:	They propose an approach that automatically finds evidence for an event from a large text corpus and leverages it to guide the generation of inferential texts.
Outcome:	The proposed model generates inferential texts from a large text corpus and uses evidence to guide it.

Copied to clipboard

Rong-Cheng Tu, Lei Ji, Huaishao Luo, Botian Shi, Heyan Huang, Nan Duan, Xian-Ling Mao

CodeExp: Explanatory Code Document Generation (2022.findings-emnlp)

Challenge:	Recent work on image-text matching has focused on exploring interactions between images and sentences to improve performance without considering inference efficiency.
Approach:	They propose a hashing-based efficient inference module which can be plugged into existing frameworks to speed up inference step without reducing retrieval performance.
Outcome:	The proposed module can be plugged into existing framework to speed up inference step without reducing retrieval performance.

Copied to clipboard

Haotian Cui, Chenglong Wang, Junjie Huang, Jeevana Priya Inala, Todd Mytkowicz, Bo Wang, Jianfeng Gao, Nan Duan

Explainability & Interpretability in NLP

Challenge:	Existing code-to-text generation models produce only high-level code summaries that do not capture implementation-level choices essential for these scenarios.
Approach:	They propose a code explanation generation task that uses code docstrings to refine models.
Outcome:	The proposed model can generate well-structured long docstrings comparable to human-written ones.

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (D19-1)

Copied to clipboard

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Ming Zhou

CoSQA: 20,000+ Web Queries for Code Search and Question Answering (2021.acl-long)

Challenge:	Existing models that can handle cross-lingual tasks with limited or no training data are insensitive to different languages.
Approach:	They propose to use Unicoder to train models in one language and apply it to other languages.
Outcome:	Experiments show that Unicoder learns the mappings among different languages from more perspectives.

Copied to clipboard

Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, Nan Duan

Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching (D19-1)

Challenge:	Using deep neural networks to find codes is difficult . we present a dataset that includes 20,604 labels for natural language queries and codes .
Approach:	They introduce a contrastive learning method to enhance text-code matching . they find that CoSQA improves the accuracy of code question answering by 5.1% .
Outcome:	The proposed method improves the accuracy of code question answering by 5.1% and improves by 10.5% on a CodeBERT model.

Copied to clipboard

Bo Shao, Yeyun Gong, Weizhen Qi, Nan Duan, Xiaola Lin

Large Language Models Can Learn Representation in Natural Language (2024.findings-acl)

Challenge:	Recent work on text sequence matching tasks uses task specific supervised datasets, which are always limited to the amount due to the cost of annotation.
Approach:	They propose an aggregation method to combine Bidirectional Encoder Representations from Transformer (BERT) with a MatchLSTM layer for Sequence Matching.
Outcome:	The proposed model improves on two publicly available datasets, WikiQA and SNLI.

Copied to clipboard

Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan

Question Generation from SQL Queries Improves Neural Semantic Parsing (D18-1)

Challenge:	Large Language Models (LLMs) are unable to complete complex tasks involving multiple entities, such as tool APIs.
Approach:	They propose a method which uses natural language representations to refine entity descriptions for improved retrieval and LLM utilization.
Outcome:	The proposed method improves GPT-4's performance on classification tasks and API call tasks.

Copied to clipboard

Daya Guo, Yibo Sun, Duyu Tang, Nan Duan, Jian Yin, Hong Chi, James Cao, Peng Chen, Ming Zhou

ProQA: Structural Prompt-based Pre-training for Unified Question Answering (2022.naacl-main)

Challenge:	Using question generation, we learn a semantic parser with 30% of the supervised training data.
Approach:	They propose to use question generation to learn a semantic parser with less supervised training data.
Outcome:	The proposed method improves the state-of-the-art model with less training data.

Copied to clipboard

Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, Nan Duan

Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data (2023.emnlp-main)

Challenge:	Existing QA research on question answering is focused on specific question types, knowledge domains, or reasoning skills.
Approach:	They propose a unified QA paradigm that solves various tasks through a single model.
Outcome:	The proposed model improves QA-centric ability on 11 QA benchmarks.

Copied to clipboard

Canwen Xu, Daya Guo, Nan Duan, Julian McAuley

Dialogue Systems & Conversational Agents

Challenge:	Despite the promising potential of chat models, they are only accessible through restricted APIs, creating barriers for new research and progress in the field.
Approach:	They propose a pipeline that can automatically generate a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself.
Outcome:	The proposed pipeline generates a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself, simulating both user and AI responses.

ReACC: A Retrieval-Augmented Code Completion Framework (2022.acl-long)

Copied to clipboard

Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, Alexey Svyatkovskiy

GEM: A General Evaluation Benchmark for Multimodal Tasks (2021.findings-acl)

Challenge:	Recent work has shown that statistical language modeling with transformers can greatly improve the performance in code completion tasks.
Approach:	They propose a retrieval-augmented code completion framework that combines a source code retriever and an auto-regressive language model for programming language.
Outcome:	The proposed framework achieves state-of-the-art on CodeXGLUE benchmark.

Copied to clipboard

Lin Su, Nan Duan, Edward Cui, Lei Ji, Chenfei Wu, Huaishao Luo, Yongfei Liu, Ming Zhong, Taroon Bharti, Arun Sacheti

Asking Clarification Questions in Knowledge-Based Question Answering (D19-1)

Challenge:	Existing datasets that focus on natural language tasks are not considered as a general evaluation benchmark for multimodal tasks.
Approach:	They present a general evaluation benchmark for multimodal tasks, GEM 1 . they compare it with existing multimodal vision-language datasets .
Outcome:	The proposed model is compared with existing vision-language datasets focusing on natural language tasks . it is the largest vision-linguistic dataset covering image-language tasks and video-language task at the same time .

Copied to clipboard

Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, Xu Sun

Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis (2022.emnlp-main)

Challenge:	Existing clarification datasets with limited annotated examples do not address ambiguous phenomena.
Approach:	They propose a dataset that allows users to ask clarification questions using open-domain examples.
Outcome:	The proposed model achieves better performance than strong baselines and provides new challenges.

Copied to clipboard

Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jinsong Su, Hang Zhang, Yeyun Gong, JIan Guo, Nan Duan

Sentiment Analysis

Challenge:	Existing pre-trained language representation models (PLMs) capture sentiment information from word-level while under-considering sentence-level information.
Approach:	They propose a Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks that enhance the PLM’s knowledge about sentiment words.
Outcome:	The proposed model achieves state-of-the-art on various sentence-level and aspect-level sentiment classification benchmarks.

ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning (2023.acl-long)

Copied to clipboard

Xiao Xu, Bei Li, Chenfei Wu, Shao-Yen Tseng, Anahita Bhiwandiwalla, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan

Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension (2020.acl-main)

Challenge:	Two-Tower Vision-Language models suffer from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of unil-modal knowledge.
Approach:	They propose a model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels to facilitate more comprehensive cross-modal alignment and fusion.
Outcome:	The proposed model outperforms baselines with and without Vision-Language Pre-training (VLP) with 4M VLP data.

Copied to clipboard

Fei Yuan, Linjun Shou, Xuanyu Bai, Ming Gong, Yaobo Liang, Nan Duan, Yan Fu, Daxin Jiang

Multilinguality

Challenge:	Existing approaches to improve machine reading comprehension performance on low resource languages are limited due to the lack of sufficient training data.
Approach:	They propose to use a mixed MRC task to translate the question to other languages and build cross-lingual question-passage pairs.
Outcome:	The proposed task improves on two cross-lingual MRC datasets.

Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (2022.findings-acl)

Copied to clipboard

Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, Nan Duan

Responsible & Trustworthy NLP

Challenge:	Existing methods for logical reasoning of text focus on contextual semantics while struggling to explicitly model the logical inference process.
Approach:	They propose a logic-driven context extension framework and a data-driven augmentation algorithm that uses contrastive learning to better capture logical information.
Outcome:	The proposed framework outperforms existing methods on two benchmark datasets, ReClor and LogiQA.

Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations (2022.acl-long)

Copied to clipboard

Wei Chen, Yeyun Gong, Can Xu, Huang Hu, Bolun Yao, Zhongyu Wei, Zhihao Fan, Xiaowu Hu, Bartuer Zhou, Biao Cheng, Daxin Jiang, Nan Duan

Multi-level Alignment Pretraining for Multi-lingual Semantic Parsing (2020.coling-main)

Challenge:	Existing studies focus on coarse-grained response selection in retrieval-based dialogue systems.
Approach:	They propose a Contextual Fine-to-Coarse (CFC) distilled model for coarse-grained response selection in open-domain conversations.
Outcome:	The proposed model improves over baseline methods on two datasets based on the Reddit comments dump and Twitter corpus compared with baseline methods.

Copied to clipboard

Bo Shao, Yeyun Gong, Weizhen Qi, Nan Duan, Xiaola Lin

ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training (2020.findings-emnlp)

Challenge:	Existing methods for multilingual semantic parsing only handle monolingual parsers, while in real world applications such as Chatbot and search engine, we generally need to handle multi-lingual semanticparsing.
Approach:	They propose a multi-level alignment pretraining method in a unified architecture for multi-lingual semantic parsing.
Outcome:	The proposed method outperforms state-of-the-art methods on a publicly avail-able multi-lingual semantic parsing dataset and a newly constructed dataset.

Copied to clipboard

Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou

Learning to Plan by Updating Natural Language (2024.findings-emnlp)

Challenge:	Existing sequence-to-sequence models are optimized for future n-gram prediction and n stream self-attention mechanism.
Approach:	They propose a self-supervised objective called future n-gram prediction and the proposed n stream self-attention mechanism to optimize the model for sequence-to-sequence learning.
Outcome:	The proposed model achieves state-of-the-art on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks compared to the models using the same scale pre-training corpus.

Copied to clipboard

Yiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu, Dongyan Zhao, Nan Duan

Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast (2023.acl-long)

Challenge:	Large Language Models (LLMs) have shown remarkable performance in basic natural language tasks.
Approach:	They propose a method that iteratively updates the task plan with new steps and behavioral instructions to guide LLMs to generate the correct solutions step by step.
Outcome:	The proposed method improves performance on five reasoning type tasks and can be directly applied to other LLMs.

Copied to clipboard

Yiduo Guo, Yaobo Liang, Dongyan Zhao, Bing Liu, Nan Duan

No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension (2020.findings-emnlp)

Challenge:	Existing research shows that a multilingual pre-trained language model fine-tuned with one (source) language performs well on downstream tasks for non-source languages . However, there is a clear performance gap between the source and non-sourced languages - this gap can be reduced by reducing forgetting.
Approach:	They propose a method to fine-tune a multilingual pre-trained language model fine- tuned with one (source) language and four training policies to address the performance gap.
Outcome:	The proposed method outperforms baselines on the XNLI dataset by a clear margin.

Copied to clipboard

Xuguang Wang, Linjun Shou, Ming Gong, Nan Duan, Daxin Jiang

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion (2024.findings-emnlp)

Challenge:	Natural Questions (NQ) benchmark sets new challenges for machine reading comprehension.
Approach:	They propose a novel approach to handle all answer types systematically using a two-step training procedure.
Outcome:	The proposed approach achieved the top 1 on both long and short answer leaderboards with F1 scores of 77.2 and 64.1.

Copied to clipboard

Zekai Zhang, Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan

Robustness in NLP

Challenge:	Large Language Models (LLMs) are increasingly used for task completion in real-world situations.
Approach:	They propose a PowerPoint Task Completion-Robustness (PPTC-R) benchmark to measure LLMs’ robustness to the user PPT task instruction and software version (Powerpoint).
Outcome:	The proposed benchmark compares 3 closed-source and 4 open-source LLMs to the PowerPoint task instruction and software version (Powerpoint) .

Constructing Multilingual Code Search Dataset Using Neural Machine Translation (2023.acl-srw)

Copied to clipboard

Ryo Sekizawa, Nan Duan, Shuai Lu, Hitomi Yanaka

CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding (2023.acl-long)

Challenge:	Existing datasets for code search are monolingual, but their query data are only in English.
Approach:	They construct a multilingual code search dataset in four natural and four programming languages using a neural machine translation model and apply back-translation data filtering to it.
Outcome:	The proposed model pre-trained with all natural and programming language data performs best under almost all settings.

Copied to clipboard

Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, W.k. Chan, Chong-Wah Ngo, Mike Zheng Shou, Nan Duan

Low-code LLM: Graphical User Interface over Large Language Models (2024.naacl-demo)

Challenge:	Existing work on video temporal grounding for long videos is limited by existing datasets.
Approach:	They propose a query-guided window selection strategy and a coarse-to-fine mechanism to speed up inference for long videos.
Outcome:	The proposed framework accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keeping SOTA results.

Copied to clipboard

Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, WangYou WangYou, Ting Song, Yan Xia, Nan Duan, Furu Wei

Control Image Captioning Spatially and Temporally (2021.acl-long)

Challenge:	Low-code LLM is a visual programming interface that allows users to incorporate their ideas into the process without writing trivial prompts.
Approach:	They propose a human-LLM interaction framework that incorporates low-code visual programming interactions to achieve more controllable and stable responses.
Outcome:	The proposed framework enables users to incorporate ideas into the process without writing trivial prompts.

Copied to clipboard

Kun Yan, Lei Ji, Huaishao Luo, Ming Zhou, Nan Duan, Shuai Ma

Code Execution with Pre-trained Language Models (2023.findings-acl)

Challenge:	Existing methods to generate image captions with user intention are still under exploration.
Approach:	They propose a model that connects Contrastive constraints and Attention Guidance in a loop manner and engages explicit spatial and temporal constraints to the generating process.
Outcome:	The proposed model improves performance on a trace-controlled image captioning task.

Copied to clipboard

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, Nan Duan

Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decoding (2026.findings-acl)

Challenge:	Pre-trained code intelligence models ignore the execution trace and only rely on source code and syntactic structures to understand code execution.
Approach:	They develop a mutation-based data augmentation technique to create a Python dataset and task for code execution that challenges existing models.
Outcome:	The proposed model outperforms existing models on code execution and shows its potential for zero-shot code-to-code search and text-to code generation.

Copied to clipboard

Weixu Zhang, Fanghua Ye, Qiang Gao, Jian Li, Haolun Wu, Yuxing Tian, Sijing Duan, Nan Du, Xiaolong Li

Joint Generator-Ranker Learning for Natural Language Generation (2023.findings-acl)

Challenge:	Large language models produce content that contradicts or overlooks information provided in the input context, a phenomenon known as faithfulness hallucination.
Approach:	They propose a lightweight framework that boosts the generation probability of context-relevant tokens by boosting the generation of tokens.
Outcome:	The proposed framework improves faithfulness metrics with minimal generation overhead.

Copied to clipboard

Weizhou Shen, Yeyun Gong, Yelong Shen, Song Wang, Xiaojun Quan, Nan Duan, Weizhu Chen

Mask Attention Networks: Rethinking and Strengthen Transformer (2021.naacl-main)

Challenge:	Existing methods for text generation train the generator and ranker individually . existing methods neglect the mutual feedback that could enhance the quality of outputs .
Approach:	They propose a joint training algorithm that integrates the generator and ranker in a single framework.
Outcome:	The proposed algorithm surpasses existing methods on four public datasets across three common generation scenarios.

Copied to clipboard

Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, Xuanjing Huang

UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022.acl-long)

Challenge:	Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation.
Approach:	They propose to combine SAN and Feed-Forward Networks to create a dynamic mask attention network with a learnable mask matrix which can model localness adaptively.
Outcome:	The proposed model outperforms the original Transformer on translation and text summarization tasks.

Copied to clipboard

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, Jian Yin

CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)

Challenge:	Pre-trained models for programming languages have demonstrated great success on code intelligence . however, such pre-tried models are sub-optimal for auto-regressive tasks .
Approach:	They propose a unified cross-modal pre-trained model for programming language that leverages cross-module contents like AST and code comment to enhance code representation.
Outcome:	The proposed model achieves state-of-the-art on most code-related tasks and compares with existing models on zero-shot code-to-code search.

Copied to clipboard

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, Timothy Baldwin

Multi-Task Learning for Conversational Question Answering over a Large-Scale Knowledge Base (D19-1)

Challenge:	Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams.
Approach:	They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings.
Outcome:	The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models.

Copied to clipboard

Tao Shen, Xiubo Geng, Tao Qin, Daya Guo, Duyu Tang, Nan Duan, Guodong Long, Daxin Jiang

Reasoning Over Semantic-Level Graph for Fact Checking (2020.acl-main)

Challenge:	Recent approaches to handle large knowledge base decompose tasks into subtasks and solve them sequentially.
Approach:	They propose a multi-task learning framework that resolves coreference in conversations . they propose enabling shared supervisions and type-aware entity detection model .
Outcome:	The proposed framework improves overall F1 score from 67% to 79% on a large-scale conversational question answering dataset.

Copied to clipboard

Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)

Challenge:	Existing methods for fact checking use string concatenation or fusing features of isolated evidence sentences.
Approach:	They propose a method suitable for reasoning about the semantic-level structure of evidence . they use graph convolutional network and graph attention network to exploit the structure .
Outcome:	The proposed method improves claim verification accuracy and FEVER score on a benchmark dataset.

Copied to clipboard

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation (2022.findings-naacl)

Challenge:	Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold.
Approach:	They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives.
Outcome:	The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%).

Copied to clipboard

Yongfei Liu, Chenfei Wu, Shao-Yen Tseng, Vasudev Lal, Xuming He, Nan Duan

ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021.acl-demo)

Challenge:	Existing vision-and-language pretraining approaches rely on external object detectors to encode images in a multi-modal transformer framework.
Approach:	They propose an object-aware end-to-end VLP framework which feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly.
Outcome:	The proposed framework achieves competitive or superior performances on vision-language tasks.

Copied to clipboard

Weizhen Qi, Yeyun Gong, Yu Yan, Can Xu, Bolun Yao, Bartuer Zhou, Biao Cheng, Daxin Jiang, Jiusheng Chen, Ruofei Zhang, Houqiang Li, Nan Duan

SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval (2022.emnlp-industry)

Challenge:	Existing models for pre-training are not convenient for users to find and set them up.
Approach:	They propose to extend ProphetNet into other domains and languages by pre-training models . they pre-train a cross-lingual generation model ProphetNet-Multi and a Chinese generation model .
Outcome:	The proposed models achieve new state-of-the-art on 10 benchmarks.

Copied to clipboard

Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-rong Wen, Nan Duan

Multi-View Document Representation Learning for Open-Domain Dense Retrieval (2022.acl-long)

Challenge:	Existing methods for sapping negatives from large document pool suffer from the uninformative or false negative problem.
Approach:	They propose a method to sample negatives from a large document pool using a new sampling probability distribution.
Outcome:	The proposed method can be used to sample more ambiguous negatives on four public and one industry datasets.

Copied to clipboard

Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, Nan Duan

Representation Learning

Challenge:	Existing methods for dense retrieval are hard to match with multiple views.
Approach:	They propose a multi-view document representation learning framework to generate multiple embeddings through viewers to represent documents and enforce them to align with different queries.
Outcome:	The proposed method outperforms recent works and achieves state-of-the-art results.

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2021.findings-acl)

Copied to clipboard

Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, Ming Zhou

Selene: Pioneering Automated Proof in Software Verification (2024.acl-long)

Challenge:	Existing methods for injecting knowledge into pre-trained models are inconsistent and can flush out knowledge when multiple kinds of knowledge are injected.
Approach:	They propose a framework that retains the original parameters of pre-trained models fixed and supports the development of versatile knowledge-infused models.
Outcome:	The proposed framework retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused models.

Copied to clipboard

Lichen Zhang, Shuai Lu, Nan Duan

FastSeq: Make Sequence Generation Faster (2021.acl-demo)

Challenge:	Currently, software verification is resource-intensive and manpower-consuming.
Approach:	They propose a project-level automated proof benchmark based on the seL4 operating system . they propose augmentations to enhance the flexibility of the framework and lightweight verification environment .
Outcome:	The proposed framework provides a comprehensive framework for end-to-end proof generation and a lightweight verification environment.

Copied to clipboard

Yu Yan, Fei Hu, Jiusheng Chen, Nikhil Bhendawade, Ting Ye, Yeyun Gong, Nan Duan, Desheng Cui, Bingyu Chi, Ruofei Zhang

GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)

Challenge:	Transformer-based models have made tremendous impact in natural language generation, but inference speed is still a bottleneck due to large model size and intensive computing involved in auto-regressive decoding process.
Approach:	They propose an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel I/O to accelerate sequence generation without loss of accuracy.
Outcome:	The proposed framework can accelerate the sequence generation by 4x to 9x with a simple one-line code change for a set of widely used and diverse models.

Copied to clipboard

Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan

GRACE: Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis (2020.findings-emnlp)

Challenge:	Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models.
Approach:	They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks.
Outcome:	The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages.

Copied to clipboard

Huaishao Luo, Lei Ji, Tianrui Li, Daxin Jiang, Nan Duan

Sentiment Analysis

Challenge:	Existing studies ignore aspect terms interaction when labeling polarities . aspect terms extraction and aspect sentiment classification are two fundamental tasks .
Approach:	They propose a GRadient hArmonized and CascadEd labeling model to solve the imbalance issue . they extend the gradient harmonized mechanism used in object detection to aspect-based sentiment analysis .
Outcome:	The proposed model achieves consistency improvement on multiple benchmark datasets and generates state-of-the-art results.

RikiNet: Reading Wikipedia Pages for Natural Question Answering (2020.acl-main)

Copied to clipboard

Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Nan Duan

CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020.findings-emnlp)

Challenge:	Using Wikipedia pages to answer open-domain questions remains challenging in natural language understanding.
Approach:	They propose a model which reads Wikipedia pages for natural question answering . it uses a dynamic paragraph dual-attention reader and a cascaded answer predictor .
Outcome:	The proposed model outperforms the human model on the Natural Questions dataset . it achieves 74.3 F1 and 57.9 F1 on long-answer and short-answer tasks .

Copied to clipboard

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou

Semantic Parsing with Syntax- and Table-Aware SQL Generation (P18-1)

Challenge:	Large pre-trained models have improved performance on a variety of natural language processing tasks.
Approach:	They develop a bimodal pre-trained model for programming language (PL) and natural language (NL) it incorporates a hybrid objective function that detects replaced tokens from generators.
Outcome:	The proposed model performs better on two NL-PL applications by fine-tuning model parameters.

Copied to clipboard

Yibo Sun, Duyu Tang, Nan Duan, Jianshu Ji, Guihong Cao, Xiaocheng Feng, Bing Qin, Ting Liu, Ming Zhou

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)

Challenge:	Existing approaches generate a SQL query word-by-word but results are incorrect or not executable due to mismatch between question words and table contents.
Approach:	They propose a generative model to map natural language questions into SQL queries.
Outcome:	The proposed model significantly improves state-of-the-art execution accuracy from 69.0% to 74.4% on a large question- SQL dataset.

Copied to clipboard

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, Weizhu Chen

Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure (2022.acl-long)

Challenge:	Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation.
Approach:	They propose to have large language models actively involved in retrieval to guide retrieval with generation.
Outcome:	The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations.

Copied to clipboard

Yuan Chai, Yaobo Liang, Nan Duan

PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion (2024.findings-acl)

Challenge:	Multilingual pre-trained language models have shown impressive cross-lingual ability.
Approach:	They argue that cross-language ability comes from commonality between languages . they create an artificial language by modifying property in source language .
Outcome:	The proposed model can be implemented in multilingual and low-resource language scenarios without cross-lingual supervision or aligned data.

Copied to clipboard

Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, Nan Duan

Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning (2022.emnlp-main)

Challenge:	Recent evaluations of Large Language Models (LLMs) focus on their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs.
Approach:	They propose a PowerPoint Task Completion benchmark to assess LLMs’ ability to create and edit PPT files based on user instructions.
Outcome:	The proposed system outperforms open-source and closed LLMs with 75.1% accuracy in single-turn dialogue testing but only achieves 6% session accuracy.

Copied to clipboard

Xingwei He, Yeyun Gong, A-Long Jin, Weizhen Qi, Hang Zhang, Jian Jiao, Bartuer Zhou, Biao Cheng, Sm Yiu, Nan Duan

XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)

Challenge:	Existing work on commonsense generation requires models to have relational reasoning and compositional generalization capabilities.
Approach:	They propose a metric distillation rule to distill knowledge from a standard metric to a ranker and transfer it to re-ranking a retriever.
Outcome:	The proposed method surpasses the previous SOTA.

Copied to clipboard

Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou

Multilinguality

Challenge:	XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios .
Approach:	They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora.
Outcome:	The proposed dataset is labeled in English and includes only natural language understanding tasks.

An Enhanced Knowledge Injection Model for Commonsense Generation (2020.coling-main)

Copied to clipboard

Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang, Yameng Huang, Jian Jiao, Xuanjing Huang, Nan Duan, Ruofei Zhang

Syntax-Enhanced Pre-trained Model (2021.acl-long)

Challenge:	a recent study shows that digging the relationship of concepts from scratch is non-trivial for commonsense generation tasks.
Approach:	They use a retrieve-and-edit framework to retrieve a prototype with these concepts . they use qt and qq to generate commonsense questions at scale .
Outcome:	The proposed method significantly improves the performance on commonsense generation tasks.

Copied to clipboard

Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, Nan Duan

Syntactic Text Processing

Challenge:	Existing methods that use syntax of text in pre-training and fine-tuning suffer from discrepancy between the two stages.
Approach:	They propose a model that utilizes the syntactic structure of text in pre-training and fine-tuning stages.
Outcome:	The proposed model achieves state-of-the-art on six public benchmark datasets.

Intervention-Based Alignment of Code Search with Execution Feedback (2023.findings-emnlp)

Copied to clipboard

Hojae Han, Minsoo Kim, Seung-won Hwang, Nan Duan, Shuai Lu

Dense Procedure Captioning in Narrated Instructional Videos (P19-1)

Challenge:	Existing code search training datasets approximate text-code co-occurrences as positive execution feedback, but this approximation may misalign models’ retrieval decisions from ground-truth correctness.
Approach:	They propose a code intervention-based reinforcement learning approach that perturbs training code to result in misalignment, then tests models’ decisions and corrects them with the execution feedback by reinforcement learning.
Outcome:	The proposed method induces the execution feedback from perturbation, without actual execution, and then tests models’ decisions and corrects them with the execution input by reinforcement learning.

Copied to clipboard

Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, Ming Zhou

Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (P19-1)

Challenge:	Existing models for video dense captioning learn video segments and generate captions without considering transcripts.
Approach:	They propose a model to generate procedure captions from narrated instructional videos . they extract procedures by a cross-modality module and generate captions by encoding video frames and transcripts within each extracted procedure.
Outcome:	The proposed model can extract procedures from narrated instructional videos and generate procedure captions by encoding video frames and transcripts.

Copied to clipboard

Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, Jian Yin

Towards Compositional Generalization in Code Search (2022.emnlp-main)

Challenge:	a context-aware retrieval model and a meta-learning paradigm are used for context-dependent semantic parsing .
Approach:	They propose a retrieval model and a meta-learner to incorporate retrieved datapoints as context-dependent semantic parsing evidence.
Outcome:	The proposed approach performs better than retrieve-and-edit baselines on CONCODE and CSQA datasets.

Copied to clipboard

Hojae Han, Seung-won Hwang, Shuai Lu, Nan Duan, Seungtaek Choi

KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021.findings-emnlp)

Challenge:	Existing code search models that focus on code as an unstructured sequence fail to generalize when the lexical perturbation without changing structures and labels is applied in test codes.
Approach:	They propose a compositional generalization model that extracts structural elements and a code template that targets compositional genericization.
Outcome:	The proposed model is complementary to flow graphs in GraphCodeBERT, by enhancing structural context around variables.

Copied to clipboard

Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

Analytical Reasoning of Text (2022.findings-naacl)

Challenge:	Pre-trained language models have led to substantial gains over a broad range of NLP tasks, but have limitations for high-quality tasks such as commonsense generation and ad keyword generation.
Approach:	They propose a Knowledge Filtering and Contrastive learning Network which references external knowledge and achieves better generation performance.
Outcome:	The proposed model outperforms the current state of the art on the CommonGen benchmark by a large margin.

Copied to clipboard

Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, Nan Duan

Neural Deepfake Detection with Factual Structure of Text (2020.emnlp-main)

Challenge:	Existing models with implicit reasoning ability struggle to solve analytical reasoning of text.
Approach:	They propose an approach to analyze text and use it to perform reasoning over it.
Outcome:	The proposed approach outperforms pre-trained models on an analysis of the Law School Admission Test dataset.

Copied to clipboard

Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin

Structured Data in NLP

Challenge:	Existing approaches to deepfake detection typically represent documents with coarse-grained representations, but they struggle to capture factual structures of documents.
Approach:	They propose a graph-based model that captures factual structures of documents for deepfake detection.
Outcome:	The proposed model improves strong base models built with RoBERTa on two public deepfake datasets.

Joint Type Inference on Entities and Relations via Graph Convolutional Networks (P19-1)

Copied to clipboard

Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, Nan Duan

Structured Data in NLP

Challenge:	a novel graph convolutional network (GCN) is proposed for the task of joint entity relation extraction.
Approach:	They propose a graph convolutional network running on an entity-relation bipartite graph . they propose combining two different methods to perform joint entity relation extraction .
Outcome:	The proposed model outperforms existing joint models in entity performance and is competitive with the state-of-the-art in relation performance.

LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network (2020.acl-main)

Copied to clipboard

Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, Jian Yin

CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion (2023.emnlp-main)

Challenge:	Existing methods for fact checking textual statements are not yet available.
Approach:	They propose a neural network approach capable of leveraging logical operations for fact checking . they use a textual statement and semi-structured tables to generate a program from it .
Outcome:	The proposed approach achieves state-of-the-art performance on TABFACT dataset . it derives a program (a.k.a. logical form) of the statement in semantic parsing manner .

Copied to clipboard

Xingwei He, Yeyun Gong, A-Long Jin, Hang Zhang, Anlei Dong, Jian Jiao, Siu Yiu, Nan Duan

Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)

Challenge:	Experimental results show that dense retrieval models are better at obtaining query-informed representations.
Approach:	They propose a dual-encoder approach that computes latent representations of query and document independently, but inference replaces the real query with a generated one.
Outcome:	The proposed approach outperforms previous dense retrieval models on in-domain and out-of-domain datasets.

Copied to clipboard

Jiashuo Sun, Yi Luo, Yeyun Gong, Chen Lin, Yelong Shen, Jian Guo, Nan Duan

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)

Challenge:	Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference.
Approach:	They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task.
Outcome:	The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets.

Copied to clipboard

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan

CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search (2022.emnlp-main)

Challenge:	Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks.
Approach:	They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions.
Outcome:	The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam.

Copied to clipboard

Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, Nan Duan

Compare to The Knowledge: Graph Neural Fake News Detection with External Knowledge (2021.acl-long)

Challenge:	Existing code pre-training approaches often adopt (masked) language modeling as the training objective which targets on learning to predict (macked) tokens in a given code context.
Approach:	They propose a code-text contrastive learning model which learns function-level code semantic representations through large-scale code corpus.
Outcome:	The proposed model achieves new state-of-the-art with significant improvement over existing pre-trained models on eleven domain/language-specific code search tasks with six programming languages in different code granularity.

Copied to clipboard

Linmei Hu, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, Ming Zhou

Ethical NLP

Challenge:	Existing methods for fake news detection rely on linguistic and semantic features from news content and do not exploit external knowledge.
Approach:	They propose a graph neural model which compares news to knowledge base through entities for fake news detection.
Outcome:	The proposed model significantly outperforms state-of-the-art methods on two benchmark datasets.

Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation (2022.findings-emnlp)

Copied to clipboard

Xiaonan Li, Daya Guo, Yeyun Gong, Yun Lin, Yelong Shen, Xipeng Qiu, Daxin Jiang, Weizhu Chen, Nan Duan

NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation (2023.acl-long)

Challenge:	Existing methods for contrastive pre-training ignore the relevance between codes in large code corpus.
Approach:	They propose a Soft-labeled contrastive pre-training framework with positive sample construction methods to learn functional-level code representation.
Outcome:	The proposed framework can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation.

Copied to clipboard

Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Ming Gong, Lijuan Wang, Zicheng Liu, Houqiang Li, Nan Duan

Learning to Collaborate for Question Answering and Asking (N18-1)

Challenge:	Existing work generates long videos segment by segment sequentially, which is inefficient.
Approach:	They propose a Diffusion over Difference architecture for eXtremely Long video generation.
Outcome:	The proposed architecture reduces the average inference time from 7.55min to 26s (94.26%) and generates high-quality long videos with both global and local coherence.

Copied to clipboard

Duyu Tang, Nan Duan, Zhao Yan, Zhirui Zhang, Yibo Sun, Shujie Liu, Yuanhua Lv, Ming Zhou

Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension (2020.acl-main)

Challenge:	Question answering (QA) and question generation (QG) are closely related tasks.
Approach:	They propose a training algorithm that generalizes both Generative Adversarial Network and Generating Domain-Adaptive Nets under the question answering scenario.
Outcome:	The proposed training algorithm generalizes both Generative Adversarial Network (GAN) and Generating Domain-Adaptive Nets (GDAN) under the question answering scenario.

Copied to clipboard

Bo Zheng, Haoyang Wen, Yaobo Liang, Nan Duan, Wanxiang Che, Daxin Jiang, Ming Zhou, Ting Liu

Diverse, Controllable, and Keyphrase-Aware: A Corpus and Method for News Multi-Headline Generation (2020.emnlp-main)

Challenge:	Existing approaches to machine reading comprehension treat documents at their hierarchical nature, ignoring their dependencies.
Approach:	They propose a machine reading comprehension benchmark with two-grained answers . they use graph attention networks to model documents at their hierarchical nature .
Outcome:	The proposed framework outperforms existing systems at long and short answer criteria.

Copied to clipboard

Dayiheng Liu, Yeyun Gong, Yu Yan, Jie Fu, Bo Shao, Daxin Jiang, Jiancheng Lv, Nan Duan

Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection (2020.emnlp-main)

Challenge:	Existing methods for news headline generation focus on producing a single short sentence . et al., 2017; Gehrmann e.t., 2018; Zhong ee., 2019) focus on single-headline generation.
Approach:	They propose a method to generate multiple headlines with keyphrases of user interests . they propose generating multiple keyphrase-relevant headlines using a transformer decoder .
Outcome:	The proposed method achieves state-of-the-art in terms of quality and diversity.

Copied to clipboard

Ruize Wang, Duyu Tang, Nan Duan, Wanjun Zhong, Zhongyu Wei, Xuanjing Huang, Daxin Jiang, Ming Zhou

Challenge:	Existing methods for fine-grained propaganda detection are not based on input-output data, but instead use declarative knowledge to detect propagandistic text fragments.
Approach:	They propose a method to inject declarative knowledge of fine-grained propaganda techniques into training data to get better representations of propagandistic texts.
Outcome:	The proposed method achieves superior performance on a large dataset for propaganda detection.

Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA (2022.findings-emnlp)

Copied to clipboard

Junjie Huang, Wanjun Zhong, Qian Liu, Ming Gong, Daxin Jiang, Nan Duan

Competition-Level Problems are Effective LLM Evaluators (2024.findings-acl)

Challenge:	Existing methods to retrieve evidences from corpus are difficult due to table-text discrepancy and data sparsity problem.
Approach:	They propose an optimized OpenQA Table-Text Retriever to retrieve tabular and textual evidences from tabular resources.
Outcome:	The proposed OpenQA Table-Text Retriever significantly outperforms existing methods on QA tasks.

Copied to clipboard

Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen

Discovering Representation Sprachbund For Multilingual Pre-Training (2021.findings-emnlp)

Challenge:	Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about their capabilities and the potential data contamination problem.
Approach:	They propose to evaluate the reasoning capabilities of large language models in solving recent competition-level programming problems in Codeforces.
Outcome:	The proposed model has experienced a cliff-like decline in problems after September 2021, which shows the potential data contamination and the challenges for any existing LLM to solve unseen complex reasoning problems.

Copied to clipboard

Yimin Fan, Yaobo Liang, Alexandre Muzio, Hany Hassan, Houqiang Li, Ming Zhou, Nan Duan

Allies: Prompting Large Language Model with Beam Search (2023.findings-emnlp)

Challenge:	Existing models perform poorly on many languages and cross-lingual tasks due to typological differences and contradictions between some languages.
Approach:	They propose to pre-train multilingual pre-trained models to handle cross-lingual tasks in one model.
Outcome:	The proposed model improves performance on cross-lingual tasks compared to baselines on multiple languages .

Copied to clipboard

Hao Sun, Xiao Liu, Yeyun Gong, Yan Zhang, Daxin Jiang, Linjun Yang, Nan Duan

PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization (2024.lrec-main)

Challenge:	Existing methods to build LLMs with stacking are limited by their information coverage and low fault tolerance.
Approach:	They propose a method that leverages large language models to iteratively generate new queries from an input query.
Outcome:	The proposed method outperforms baselines on open-domain question answering benchmarks.

Copied to clipboard

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, Nan Duan

Summarization

Challenge:	Existing summarization strategies are abstractive and extractive, but are hard to control.
Approach:	They propose a PhRase-level cOpying Mechanism that enhances attention on n-grams and calculates an auxiliary loss for the copying prediction.
Outcome:	Empirical studies show that PROM improves copying accuracy and faithfulness on benchmarks.

Graph Neural News Recommendation with Unsupervised Preference Disentanglement (2020.acl-main)

Copied to clipboard

Linmei Hu, Siyong Xu, Chen Li, Cheng Yang, Chuan Shi, Nan Duan, Xing Xie, Ming Zhou

Responsible & Trustworthy NLP

Challenge:	Existing methods to learn informative user and news representations fail to consider high-order connectivity underlying the user-news interactions.
Approach:	They propose a novel Graph Neural News Recommendation model with Unsupervised Preference Disentanglement which can encode high-order relationships into user and news representations by information propagation along the graph.
Outcome:	The proposed model can encode high-order relationships into user and news representations by information propagation along the graph and disentangle latent preference factors by a neighborhood routing algorithm.

DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation (2022.acl-long)

Copied to clipboard

Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, Nan Duan

Dialogue Systems & Conversational Agents

Challenge:	Existing pre-trained dialog models shed light on various downstream tasks in natural language processing (NLP).
Approach:	They propose a dialog pre-training framework that introduces latent variables into the enhanced encoder-decoder pre-train framework to increase relevance and diversity of responses.
Outcome:	The proposed model achieves state-of-the-art on personaChat, DailyDialog, and DSTC7-AVSD datasets.

Machine Reasoning: Technology, Dilemma and Future (2020.emnlp-tutorials)

Copied to clipboard

Nan Duan, Duyu Tang, Ming Zhou

Responsible & Trustworthy NLP

Challenge:	Machine reasoning is a field of research that aims to build interpretable AI systems . Symbolic reasoning methods represent knowledge using symbolic logic, perform inference . probabilistic reasoning methods combine probability and symbolic logic into a unified model .
Approach:	This tutorial introduces machine reasoning frameworks and aims to define them . it will show how they can be used to build interpretable AI systems .
Outcome:	This tutorial aims to show how machine reasoning frameworks perform in real-world scenarios.

Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space (2020.emnlp-main)

Copied to clipboard

Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, Ming Zhou