Papers by Yeyun Gong
Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for ensembling language models fail to address complex reasoning tasks. |
| Approach: | They propose a framework for process-level ensembling of large language models using Monte Carlo tree search. |
| Outcome: | The proposed framework outperforms both language model decoding and language model ensemble methods on five reasoning benchmarks. |
Too Long, Do Re-weighting for Efficient LLM Reasoning Compression (2026.acl-long)
Copied to clipboard
Zhong-Zhi Li, Xiao Liang, Zihao Tang, Lei Ji, Peijie Wang, Haotian Xu, Xing W, Haizhen Huang, Weiwei Deng, Yeyun Gong, Zhijiang Guo, Xiao Liu, Fei Yin, Cheng-Lin Liu
| Challenge: | Large Language Models (LLMs) have recently achieved remarkable progress on complex reasoning tasks by leveraging extended Chain-of-Thought (CoT) techniques. |
| Approach: | They propose a method that uses Extended Chain-of-Thought (EFT) to reduce the number of output tokens by nearly 40% while maintaining the accuracy of the reasoning. |
| Outcome: | The proposed method reduces the number of output tokens by nearly 40% while maintaining the accuracy of the reasoning. |
APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to generate reasoning programs that ignore the differences between facts treated all facts equally, leading to wrong punishment of programs that differed from the ground truth. |
| Approach: | They propose an optimized training framework for long-form numerical reasoning that incorporates a number-aware negative sampling strategy and consistency-based reinforcement learning to increase execution accuracy. |
| Outcome: | The proposed method improves the performance of long-form numerical reasoning on the FinQA and ConvFinQA leaderboards. |
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections (2026.findings-acl)
Copied to clipboard
| Challenge: | Several studies rely on additional models to optimize mixtures. |
| Approach: | They propose a method that dynamically optimizes instruction-tuning dataset mixtures by prior-scaled Boltzmann Exploration and a multi-armed bandit setup. |
| Outcome: | The proposed method improves the TÜLU-2-mixture and TÜLO-3-mixtures across 10 benchmarks while introducing minimal computational overhead over naive sampling. |
CULG: Commercial Universal Language Generation (2022.naacl-industry)
Copied to clipboard
| Challenge: | Pre-trained language models have improved performance for many NLP tasks in finance and healthcare. |
| Approach: | They propose a large-scale commercial universal language generation model which is pre-trained on a corpus drawn from 10 markets across 7 languages. |
| Outcome: | The proposed model outperforms other models on commercial generation tasks and on other markets, languages, and tasks. |
Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to optimise pretraining performance have not addressed the complexities of domain-adaptive continual pretraining. |
| Approach: | They propose a framework that dynamically assesses learning velocity and adjusts data proportions accordingly, favouring slower learning domains while de-emphasising faster learning ones. |
| Outcome: | The proposed framework achieves performance gains in math and code reasoning tasks and command-line generation benchmarks. |
Is Your Language Model Ready for Monetization Decisions? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on shopping-centric scenarios and user-facing data, overlooking intermediate decision stages and robustness considerations. |
| Approach: | They propose a multi-task benchmark to evaluate large language models in real-world monetization contexts. |
| Outcome: | The proposed benchmark covers intent understanding, commercial matching, and user behavior modeling. |
Query Rewriting in Retrieval-Augmented Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on adapting either the retriever or the reader, but this approach is more focused on adaptation of the query itself. |
| Approach: | They propose a new framework for retrieval-augmented Large Language Models . they propose rewrite-retrieve-read instead of retrieve-then-read . |
| Outcome: | The proposed framework improves performance on downstream tasks, open-domain QA and multiple-choice QA. |
Task Oriented In-Domain Data Augmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models suffer from two major issues: in-domain data are scarce compared with general domain-agnostic data. |
| Approach: | They propose a task-oriented in-domain data augmentation framework that uses in- domain data selection and task-orientated synthetic passage generation to adapt LLMs to two domains: advertisement and math. |
| Outcome: | The proposed framework improves LLM performance by 8% in the advertisement domain and 7.5% in the math domain. |
Noisy Pair Corrector for Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dense retrieval models assume that query-document pairs are exactly matched, resulting in mismatched-pair noise. |
| Approach: | They propose a novel approach to train an effective model with mismatched-pair noise. |
| Outcome: | The proposed model performs well on natural question and triviaQA, code-search benchmarks and SO-DS. |
Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching (D19-1)
Copied to clipboard
| Challenge: | Recent work on text sequence matching tasks uses task specific supervised datasets, which are always limited to the amount due to the cost of annotation. |
| Approach: | They propose an aggregation method to combine Bidirectional Encoder Representations from Transformer (BERT) with a MatchLSTM layer for Sequence Matching. |
| Outcome: | The proposed model improves on two publicly available datasets, WikiQA and SNLI. |
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability (2026.acl-long)
Copied to clipboard
Xiao Liang, Zhong-Zhi Li, Zhenghao Lin, Eric Hanchen Jiang, Hengyuan Zhang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Yeyun Gong, Weizhu Chen
| Challenge: | Large language models have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning, but their strictly sequential nature constrains test-time scalability. |
| Approach: | They propose an end-to-end reinforcement learning framework to enhance LLMs' DAC-style reasoning capacity by decomposing a problem into subproblems and solving them sequentially. |
| Outcome: | The proposed model surpasses CoT by 8.6% and 6.3% on competition-level benchmarks and is available at the [github.com/MasterVito/DAC-RL]. |
Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for reweighting data mixtures rely on manual designation with certain heuristics based on intuition or empirical results. |
| Approach: | They propose a model-based framework that learns to re-weight domains by reinforcement learning on large quantities of data mixing trajectories with corresponding feedback from an evaluation environment. |
| Outcome: | The proposed framework outperforms baselines in achieving balanced performance across source and target fields and domain spaces without retraining. |
Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis (2022.emnlp-main)
Copied to clipboard
Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jinsong Su, Hang Zhang, Yeyun Gong, JIan Guo, Nan Duan
| Challenge: | Existing pre-trained language representation models (PLMs) capture sentiment information from word-level while under-considering sentence-level information. |
| Approach: | They propose a Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks that enhance the PLM’s knowledge about sentiment words. |
| Outcome: | The proposed model achieves state-of-the-art on various sentence-level and aspect-level sentiment classification benchmarks. |
Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations (2022.acl-long)
Copied to clipboard
Wei Chen, Yeyun Gong, Can Xu, Huang Hu, Bolun Yao, Zhongyu Wei, Zhihao Fan, Xiaowu Hu, Bartuer Zhou, Biao Cheng, Daxin Jiang, Nan Duan
| Challenge: | Existing studies focus on coarse-grained response selection in retrieval-based dialogue systems. |
| Approach: | They propose a Contextual Fine-to-Coarse (CFC) distilled model for coarse-grained response selection in open-domain conversations. |
| Outcome: | The proposed model improves over baseline methods on two datasets based on the Reddit comments dump and Twitter corpus compared with baseline methods. |
Multi-level Alignment Pretraining for Multi-lingual Semantic Parsing (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for multilingual semantic parsing only handle monolingual parsers, while in real world applications such as Chatbot and search engine, we generally need to handle multi-lingual semanticparsing. |
| Approach: | They propose a multi-level alignment pretraining method in a unified architecture for multi-lingual semantic parsing. |
| Outcome: | The proposed method outperforms state-of-the-art methods on a publicly avail-able multi-lingual semantic parsing dataset and a newly constructed dataset. |
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing sequence-to-sequence models are optimized for future n-gram prediction and n stream self-attention mechanism. |
| Approach: | They propose a self-supervised objective called future n-gram prediction and the proposed n stream self-attention mechanism to optimize the model for sequence-to-sequence learning. |
| Outcome: | The proposed model achieves state-of-the-art on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks compared to the models using the same scale pre-training corpus. |
Joint Generator-Ranker Learning for Natural Language Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text generation train the generator and ranker individually . existing methods neglect the mutual feedback that could enhance the quality of outputs . |
| Approach: | They propose a joint training algorithm that integrates the generator and ranker in a single framework. |
| Outcome: | The proposed algorithm surpasses existing methods on four public datasets across three common generation scenarios. |
Mask Attention Networks: Rethinking and Strengthen Transformer (2021.naacl-main)
Copied to clipboard
Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, Xuanjing Huang
| Challenge: | Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. |
| Approach: | They propose to combine SAN and Feed-Forward Networks to create a dynamic mask attention network with a learnable mask matrix which can model localness adaptively. |
| Outcome: | The proposed model outperforms the original Transformer on translation and text summarization tasks. |
CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. |
| Approach: | They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings. |
| Outcome: | The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models. |
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021.acl-demo)
Copied to clipboard
Weizhen Qi, Yeyun Gong, Yu Yan, Can Xu, Bolun Yao, Bartuer Zhou, Biao Cheng, Daxin Jiang, Jiusheng Chen, Ruofei Zhang, Houqiang Li, Nan Duan
| Challenge: | Existing models for pre-training are not convenient for users to find and set them up. |
| Approach: | They propose to extend ProphetNet into other domains and languages by pre-training models . they pre-train a cross-lingual generation model ProphetNet-Multi and a Chinese generation model . |
| Outcome: | The proposed models achieve new state-of-the-art on 10 benchmarks. |
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval (2022.emnlp-industry)
Copied to clipboard
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-rong Wen, Nan Duan
| Challenge: | Existing methods for sapping negatives from large document pool suffer from the uninformative or false negative problem. |
| Approach: | They propose a method to sample negatives from a large document pool using a new sampling probability distribution. |
| Outcome: | The proposed method can be used to sample more ambiguous negatives on four public and one industry datasets. |
FastSeq: Make Sequence Generation Faster (2021.acl-demo)
Copied to clipboard
Yu Yan, Fei Hu, Jiusheng Chen, Nikhil Bhendawade, Ting Ye, Yeyun Gong, Nan Duan, Desheng Cui, Bingyu Chi, Ruofei Zhang
| Challenge: | Transformer-based models have made tremendous impact in natural language generation, but inference speed is still a bottleneck due to large model size and intensive computing involved in auto-regressive decoding process. |
| Approach: | They propose an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel I/O to accelerate sequence generation without loss of accuracy. |
| Outcome: | The proposed framework can accelerate the sequence generation by 4x to 9x with a simple one-line code change for a set of widely used and diverse models. |
GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)
Copied to clipboard
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan
| Challenge: | Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models. |
| Approach: | They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. |
| Outcome: | The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages. |
RikiNet: Reading Wikipedia Pages for Natural Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Using Wikipedia pages to answer open-domain questions remains challenging in natural language understanding. |
| Approach: | They propose a model which reads Wikipedia pages for natural question answering . it uses a dynamic paragraph dual-attention reader and a cascaded answer predictor . |
| Outcome: | The proposed model outperforms the human model on the Natural Questions dataset . it achieves 74.3 F1 and 57.9 F1 on long-answer and short-answer tasks . |
Generative Prompt Internalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Prompts used in large language model based applications are often fixed and lengthy, leading to significant computational overhead. |
| Approach: | They propose a method that internalizes complex prompts using a joint training approach and a data synthesis technique that auto-collects conversational datasets by swapping roles of agent and environment. |
| Outcome: | The proposed method internalizes complex prompts across agent-based applications and generates the content along with reasons for why it should change accordingly. |
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation. |
| Approach: | They propose to have large language models actively involved in retrieval to guide retrieval with generation. |
| Outcome: | The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations. |
Knowledge Enhanced Pre-training for Cross-lingual Dense Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing mPLMs neglect the importance of knowledge in cross-lingual dense retrieval. |
| Approach: | They propose a novel mPLM that leverages knowledge to learn language-agnostic semantic representations from a multilingual knowledge base and an annotation of Wiki. |
| Outcome: | The proposed model achieves strong multilingual and cross-lingual retrieval performance with significant improvements over existing mPLMs. |
Adapting LLM Agents with Universal Communication Feedback (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated potential for LLM agents. |
| Approach: | They propose a universal buffer and iterative pipeline to store feedback and itersative pipelines to enable LLM agents to explore and update their policy in an environment. |
| Outcome: | The proposed approach outperforms supervised instruction fine-tuning baselines on four datasets. |
LayerNorm Induces Recency Bias in Transformer Decoders (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that stacking causal self-attention layers alone induces a positional bias in attention scores toward earlier tokens, but this differs from the bias toward later tokens observed in Transformer decoders, known as recency bias. |
| Approach: | They propose to stack causal self-attention layers and layer norm to induce recency bias in Transformer decoders by analyzing the interaction between causal self and other architectural components. |
| Outcome: | The proposed method provides new theoretical insights into how positional information interacts with architectural components and suggests improvements in positional encoding strategies. |
Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning (2022.emnlp-main)
Copied to clipboard
Xingwei He, Yeyun Gong, A-Long Jin, Weizhen Qi, Hang Zhang, Jian Jiao, Bartuer Zhou, Biao Cheng, Sm Yiu, Nan Duan
| Challenge: | Existing work on commonsense generation requires models to have relational reasoning and compositional generalization capabilities. |
| Approach: | They propose a metric distillation rule to distill knowledge from a standard metric to a ranker and transfer it to re-ranking a retriever. |
| Outcome: | The proposed method surpasses the previous SOTA. |
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)
Copied to clipboard
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou
| Challenge: | XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios . |
| Approach: | They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora. |
| Outcome: | The proposed dataset is labeled in English and includes only natural language understanding tasks. |
An Enhanced Knowledge Injection Model for Commonsense Generation (2020.coling-main)
Copied to clipboard
Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang, Yameng Huang, Jian Jiao, Xuanjing Huang, Nan Duan, Ruofei Zhang
| Challenge: | a recent study shows that digging the relationship of concepts from scratch is non-trivial for commonsense generation tasks. |
| Approach: | They use a retrieve-and-edit framework to retrieve a prototype with these concepts . they use qt and qq to generate commonsense questions at scale . |
| Outcome: | The proposed method significantly improves the performance on commonsense generation tasks. |
KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have led to substantial gains over a broad range of NLP tasks, but have limitations for high-quality tasks such as commonsense generation and ad keyword generation. |
| Approach: | They propose a Knowledge Filtering and Contrastive learning Network which references external knowledge and achieves better generation performance. |
| Outcome: | The proposed model outperforms the current state of the art on the CommonGen benchmark by a large margin. |
Joint Type Inference on Entities and Relations via Graph Convolutional Networks (P19-1)
Copied to clipboard
| Challenge: | a novel graph convolutional network (GCN) is proposed for the task of joint entity relation extraction. |
| Approach: | They propose a graph convolutional network running on an entity-relation bipartite graph . they propose combining two different methods to perform joint entity relation extraction . |
| Outcome: | The proposed model outperforms existing joint models in entity performance and is competitive with the state-of-the-art in relation performance. |
CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that dense retrieval models are better at obtaining query-informed representations. |
| Approach: | They propose a dual-encoder approach that computes latent representations of query and document independently, but inference replaces the real query with a generated one. |
| Outcome: | The proposed approach outperforms previous dense retrieval models on in-domain and out-of-domain datasets. |
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference. |
| Approach: | They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task. |
| Outcome: | The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets. |
Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructions (2026.acl-long)
Copied to clipboard
Boyan Duan, Xiao Liang, Shuai Lu, Yaoxiang Wang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Mao Yang, Weizhu Chen, Yeyun Gong
| Challenge: | Existing methods for geometry theorem proving in Euclidean geometry are challenging and require a neural network to perform. |
| Approach: | They propose a method for adding auxiliary points in geometry that runs on CPUs without relying on neural network-based inference. |
| Outcome: | The proposed method achieves silver-medal-level human performance on IMO-30 benchmark. |
CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search (2022.emnlp-main)
Copied to clipboard
Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, Nan Duan
| Challenge: | Existing code pre-training approaches often adopt (masked) language modeling as the training objective which targets on learning to predict (macked) tokens in a given code context. |
| Approach: | They propose a code-text contrastive learning model which learns function-level code semantic representations through large-scale code corpus. |
| Outcome: | The proposed model achieves new state-of-the-art with significant improvement over existing pre-trained models on eleven domain/language-specific code search tasks with six programming languages in different code granularity. |
P3LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing autoregressive left-to-right (L2R) models are limited to unidirectional information and constrained on strong local dependencies. |
| Approach: | They propose a probabilistically permuted prophet language model which strengthens the modeling of bidirectional information and long token dependencies for sequence generation. |
| Outcome: | Experiments on GLGE dataset show that P3LM improves on natural language generation tasks. |
Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation (2022.findings-emnlp)
Copied to clipboard
Xiaonan Li, Daya Guo, Yeyun Gong, Yun Lin, Yelong Shen, Xipeng Qiu, Daxin Jiang, Weizhu Chen, Nan Duan
| Challenge: | Existing methods for contrastive pre-training ignore the relevance between codes in large code corpus. |
| Approach: | They propose a Soft-labeled contrastive pre-training framework with positive sample construction methods to learn functional-level code representation. |
| Outcome: | The proposed framework can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation. |
Diverse, Controllable, and Keyphrase-Aware: A Corpus and Method for News Multi-Headline Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for news headline generation focus on producing a single short sentence . et al., 2017; Gehrmann e.t., 2018; Zhong ee., 2019) focus on single-headline generation. |
| Approach: | They propose a method to generate multiple headlines with keyphrases of user interests . they propose generating multiple keyphrase-relevant headlines using a transformer decoder . |
| Outcome: | The proposed method achieves state-of-the-art in terms of quality and diversity. |
Uncertainty-Aware Label Refinement for Sequence Labeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Conditional random fields (CRF) for label decoding have been a problem for many tasks. |
| Approach: | They propose a two-stage label decoding framework that model long-term label dependencies while being much more computationally efficient. |
| Outcome: | The proposed method outperforms the CRF-based methods and greatly accelerates the inference process. |
Competition-Level Problems are Effective LLM Evaluators (2024.findings-acl)
Copied to clipboard
Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen
| Challenge: | Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about their capabilities and the potential data contamination problem. |
| Approach: | They propose to evaluate the reasoning capabilities of large language models in solving recent competition-level programming problems in Codeforces. |
| Outcome: | The proposed model has experienced a cliff-like decline in problems after September 2021, which shows the potential data contamination and the challenges for any existing LLM to solve unseen complex reasoning problems. |
Allies: Prompting Large Language Model with Beam Search (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to build LLMs with stacking are limited by their information coverage and low fault tolerance. |
| Approach: | They propose a method that leverages large language models to iteratively generate new queries from an input query. |
| Outcome: | The proposed method outperforms baselines on open-domain question answering benchmarks. |
Process-based Self-Rewarding Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reward LLMs' outputs are not effective in mathematical reasoning scenarios and may lead to a decline in performance. |
| Approach: | They propose a process-based self-rewarding pipeline that integrates long-thought reasoning, step-wise LLM-as-a-Judge, and step- wise preference optimization within the existing paradigm. |
| Outcome: | The proposed model improves the performance of Large Language Models on multiple mathematical reasoning benchmarks and shows that it can surpass human capabilities. |
PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing summarization strategies are abstractive and extractive, but are hard to control. |
| Approach: | They propose a PhRase-level cOpying Mechanism that enhances attention on n-grams and calculates an auxiliary loss for the copying prediction. |
| Outcome: | Empirical studies show that PROM improves copying accuracy and faithfulness on benchmarks. |
DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation (2022.acl-long)
Copied to clipboard
Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, Nan Duan
| Challenge: | Existing pre-trained dialog models shed light on various downstream tasks in natural language processing (NLP). |
| Approach: | They propose a dialog pre-training framework that introduces latent variables into the enhanced encoder-decoder pre-train framework to increase relevance and diversity of responses. |
| Outcome: | The proposed model achieves state-of-the-art on personaChat, DailyDialog, and DSTC7-AVSD datasets. |
Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models (2024.naacl-long)
Copied to clipboard
Yi Luo, Zhenghao Lin, YuHao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong
| Challenge: | Guide-Align is a guideline-oriented approach to augment the safety and quality of Large Language Models. |
| Approach: | They propose a guideline-oriented method to augment the safety and quality of large language models. |
| Outcome: | The proposed method outperforms existing methods on three benchmarks and shows significant improvements in security and quality. |
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing data augmentation techniques for natural language processing tasks are difficult to design. |
| Approach: | They propose a controllable rewriting based question data augmentation method for machine reading comprehension, question generation and question-answering natural language inference tasks. |
| Outcome: | The proposed method generates high-quality, high-quality question data samples on machine reading comprehension, question generation, and question-answering natural language inference tasks. |