Papers by Yelong Shen
Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing knowledge-grounded dialogue systems perform poorly on unseen topics due to limited topics covered in training data. |
| Approach: | They propose a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge-grounded dialogue generation tasks. |
| Outcome: | The proposed language model generalizes well across knowledge-grounded dialogue tasks. |
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability (2026.acl-long)
Copied to clipboard
Xiao Liang, Zhong-Zhi Li, Zhenghao Lin, Eric Hanchen Jiang, Hengyuan Zhang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Yeyun Gong, Weizhu Chen
| Challenge: | Large language models have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning, but their strictly sequential nature constrains test-time scalability. |
| Approach: | They propose an end-to-end reinforcement learning framework to enhance LLMs' DAC-style reasoning capacity by decomposing a problem into subproblems and solving them sequentially. |
| Outcome: | The proposed model surpasses CoT by 8.6% and 6.3% on competition-level benchmarks and is available at the [github.com/MasterVito/DAC-RL]. |
Language Models can be Deductive Solvers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge. |
| Approach: | They propose a novel language model that internalizes and emulates the reasoning processes of logical solvers and avoids parsing errors by learning strict adherence to solver syntax and grammar. |
| Outcome: | The proposed model outperforms state-of-the-art solver-augmented language models and few-shot prompting methods on public deductive reasoning benchmarks. |
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for generating paragraph descriptions for videos require a coherent paragraph and a higher level of coherence. |
| Approach: | They propose a new method that generates a summarized memory state from video segments and sentence history to help better predict the next sentence. |
| Outcome: | The proposed method generates more coherent and less repetitive paragraph captions while maintaining relevance to the input video events. |
Stochastic Answer Networks for Machine Reading Comprehension (P18-1)
Copied to clipboard
| Challenge: | Several recent MRC models employ multi-step reasoning . we show that the use of a stochastic prediction dropout improves robustness . |
| Approach: | They propose a stochastic answer network that simulates multi-step reasoning in machine reading comprehension. |
| Outcome: | The proposed model improves robustness and results competitive with state-of-the-art models on the Stanford Question Answering Dataset and Microsoft MAchine Reading COmprehension Dataset. |
Joint Generator-Ranker Learning for Natural Language Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text generation train the generator and ranker individually . existing methods neglect the mutual feedback that could enhance the quality of outputs . |
| Approach: | They propose a joint training algorithm that integrates the generator and ranker in a single framework. |
| Outcome: | The proposed algorithm surpasses existing methods on four public datasets across three common generation scenarios. |
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent few-shot learning models such as GPT3 are expensive and slow to deploy for real-world applications. |
| Approach: | They propose a prompt-based low-resource learning method for VL tasks with a few examples . they pre-train a sequence-to-sequence transformer model with prefix and masked language modeling . |
| Outcome: | The proposed method outperforms Frozen on vision-language tasks with prompt-based learning by 18.2% point. |
Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models show that they can fine-tune parameters but achieve good downstream performance. |
| Approach: | They find that a dominant winning ticket takes up 0.05% of the parameters and is transferable across different tasks. |
| Outcome: | The proposed model can achieve comparable performance with the full-parameter model, the authors show . the dominant winning ticket takes up 0.05% of the parameters, and the model is transferable across tasks, they show - the authors conclude . |
Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding is a novel method to expedite inference in autoregressive (large) language models. |
| Approach: | They propose to use a smaller model as a draft model to speculate a block of tokens, which the target model then evaluates for acceptance. |
| Outcome: | The proposed method can be used to accelerate inference in autoregressive (large) language models by using smaller models as draft models to speculate tokens for multiple inference steps. |
Memory-Efficient Differentiable Transformer Architecture Search (2021.findings-acl)
Copied to clipboard
| Challenge: | Current neural architecture search methods suffer from huge computational cost. |
| Approach: | They propose a reversible recursive backpropagation algorithm that uses the last layer to store the outputs of the network. |
| Outcome: | The proposed algorithm outperforms standard Transformers on three sequence-to-sequence datasets. |
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning (N19-1)
Copied to clipboard
| Challenge: | Existing methods for commonsense reasoning rely on human-crafted features and knowledge bases, but unsupervised learning is not feasible due to the lack of labeled training data or comprehensive knowledge bases. |
| Approach: | They propose two unsupervised models based on the Deep Structured Semantic Models framework to tackle two commonsense reasoning tasks: Winograd Schema Challenge (WSC) and Pronoun Disambiguation (PDP). |
| Outcome: | The proposed models capture contextual information in the sentence and co-reference information between pronouns and nouns, and achieve significant improvement over previous state-of-the-art approaches. |
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension (N19-1)
Copied to clipboard
| Challenge: | Existing models for Machine Reading Comprehension (MRC) are small, compared to their size, and there are many studies on using pre-trained word embeddings and back-translation approaches to improve model generalization. |
| Approach: | They propose a multi-task learning framework to learn a machine reading comprehension model that can be applied to a wide range of MRC tasks in different domains. |
| Outcome: | The proposed model can be applied to a wide range of MRC tasks in different domains. |
Controllable Natural Language Generation with Contrastive Prefixes (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work on controllable natural language generation has focused on fine-tuning existing models or using attribute discriminators. |
| Approach: | They propose a lightweight framework for controllable GPT2 generation that utilizes attribute-specific vectors to steer natural language generation. |
| Outcome: | The proposed framework can guide generation towards desired attributes while keeping high linguistic quality. |
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval (2022.emnlp-industry)
Copied to clipboard
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-rong Wen, Nan Duan
| Challenge: | Existing methods for sapping negatives from large document pool suffer from the uninformative or false negative problem. |
| Approach: | They propose a method to sample negatives from a large document pool using a new sampling probability distribution. |
| Outcome: | The proposed method can be used to sample more ambiguous negatives on four public and one industry datasets. |
UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work on open-domain question answering focuses on either extractive or generative readers exclusively. |
| Approach: | They propose a hybrid approach to extractive and generative readers that leverages both models. |
| Outcome: | The proposed approach outperforms state-of-the-art models on NaturalQuestions and TriviaQA respectively. |
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation. |
| Approach: | They propose to have large language models actively involved in retrieval to guide retrieval with generation. |
| Outcome: | The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations. |
Adapting LLM Agents with Universal Communication Feedback (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated potential for LLM agents. |
| Approach: | They propose a universal buffer and iterative pipeline to store feedback and itersative pipelines to enable LLM agents to explore and update their policy in an environment. |
| Outcome: | The proposed approach outperforms supervised instruction fine-tuning baselines on four datasets. |
Generation-Augmented Retrieval for Open-Domain Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to answer open-domain questions use sparse representations and sparsity. |
| Approach: | They propose a method which augments a query by generating relevant contexts from heuristically discovered contexts without external supervision. |
| Outcome: | The proposed approach outperforms state-of-the-art dense retrieval methods on natural questions and triviaQA datasets. |
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference. |
| Approach: | They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task. |
| Outcome: | The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets. |
Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructions (2026.acl-long)
Copied to clipboard
Boyan Duan, Xiao Liang, Shuai Lu, Yaoxiang Wang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Mao Yang, Weizhu Chen, Yeyun Gong
| Challenge: | Existing methods for geometry theorem proving in Euclidean geometry are challenging and require a neural network to perform. |
| Approach: | They propose a method for adding auxiliary points in geometry that runs on CPUs without relying on neural network-based inference. |
| Outcome: | The proposed method achieves silver-medal-level human performance on IMO-30 benchmark. |
CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search (2022.emnlp-main)
Copied to clipboard
Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, Nan Duan
| Challenge: | Existing code pre-training approaches often adopt (masked) language modeling as the training objective which targets on learning to predict (macked) tokens in a given code context. |
| Approach: | They propose a code-text contrastive learning model which learns function-level code semantic representations through large-scale code corpus. |
| Outcome: | The proposed model achieves new state-of-the-art with significant improvement over existing pre-trained models on eleven domain/language-specific code search tasks with six programming languages in different code granularity. |
Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation (2022.findings-emnlp)
Copied to clipboard
Xiaonan Li, Daya Guo, Yeyun Gong, Yun Lin, Yelong Shen, Xipeng Qiu, Daxin Jiang, Weizhu Chen, Nan Duan
| Challenge: | Existing methods for contrastive pre-training ignore the relevance between codes in large code corpus. |
| Approach: | They propose a Soft-labeled contrastive pre-training framework with positive sample construction methods to learn functional-level code representation. |
| Outcome: | The proposed framework can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation. |
Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine reading comprehension (MRC) on long texts typically chunk text into equally-spaced segments without considering information from other segments. |
| Approach: | They propose to let a model learn to chunk in a more flexible way via reinforcement learning. |
| Outcome: | The proposed model extracts a text span from document and query as answer . previous models can only take a fixed-length (e.g., 512) text as input . |
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing (2022.acl-long)
Copied to clipboard
| Challenge: | Existing work has resorted to sharing weights among models, but results are not affordable for real-world deployment. |
| Approach: | They propose a consistency-regularized ensemble learning approach based on perturbed models to retain ensemble benefits while maintaining a low memory cost. |
| Outcome: | The proposed approach outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0.7 with a significantly smaller model size. |
Competition-Level Problems are Effective LLM Evaluators (2024.findings-acl)
Copied to clipboard
Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen
| Challenge: | Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about their capabilities and the potential data contamination problem. |
| Approach: | They propose to evaluate the reasoning capabilities of large language models in solving recent competition-level programming problems in Codeforces. |
| Outcome: | The proposed model has experienced a cliff-like decline in problems after September 2021, which shows the potential data contamination and the challenges for any existing LLM to solve unseen complex reasoning problems. |
Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models (2024.naacl-long)
Copied to clipboard
Yi Luo, Zhenghao Lin, YuHao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong
| Challenge: | Guide-Align is a guideline-oriented approach to augment the safety and quality of Large Language Models. |
| Approach: | They propose a guideline-oriented method to augment the safety and quality of large language models. |
| Outcome: | The proposed method outperforms existing methods on three benchmarks and shows significant improvements in security and quality. |
Reader-Guided Passage Reranking for Open-Domain Question Answering (2021.findings-acl)
Copied to clipboard
| Challenge: | Current open-domain question answering systems follow a Retriever-Reader architecture . current systems do not use a reranker, which reranked passages based on top predictions of the reader . |
| Approach: | They propose a reader-guIDEd reranking method that reranked passages based on top predictions . they show that RIDER achieves 10 to 20 absolute gains in top-1 retrieval accuracy . |
| Outcome: | The proposed method achieves 10 to 20 gains in top-1 retrieval accuracy and 1 to 4 Exact Match gains without training. |