Papers by Tao Ge

Parallel Data Augmentation for Formality Style Transfer (2020.acl-main)

Challenge:	DisCo-Speech is a zero-shot controllable text-to-speech framework . standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs.
Approach:	They propose a disentangled speech codec and an LM-based generator to solve this problem . they propose fusion and reconstruction that merges content and prosody into unified tokens .
Outcome:	DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control.

Copied to clipboard

Yi Zhang, Tao Ge, Xu Sun

Low-Resource NLP

Challenge:	Formality style transfer is a task of automatically transforming text in one particular formality style into another.
Approach:	They propose to augment parallel data with three specific data augmentation methods to improve the model's generalization ability and reduce the overfitting risk.
Outcome:	The proposed methods significantly improve performance when used to pre-train the model and lead to the state-of-the-art results in the GYAFC benchmark dataset.

EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation (2022.emnlp-main)

Copied to clipboard

Tao Ge, Si-Qing Chen, Furu Wei

Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021.emnlp-main)

Challenge:	Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints.
Approach:	They propose a parameter-efficient Transformer for on-device seq2seq generation that uses two novel principles for cost-effective parameterization.
Outcome:	Extensive experiments show that EdgeFormer outperforms the previous parameter-efficient Transformers and achieves competitive results under both the computation and memory constraints.

Copied to clipboard

Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei

Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge (2021.naacl-main)

Challenge:	Recent studies on compression of pretrained language models usually use preserved accuracy as the metric for evaluation.
Approach:	They propose two new metrics that measure how closely a compressed model mimics the original model.
Outcome:	The proposed metrics measure how closely a compressed model (i.e., student) mimics the original model (e.g., teacher).

Copied to clipboard

Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling (2024.acl-long)

Challenge:	Cant is important for understanding advertising, comedies and dogwhistle politics . currently, there are very few resources available for the research of cant .
Approach:	They propose a large and diverse dataset for creating and understanding cant from a computational linguistics perspective.
Outcome:	The proposed dataset can be used to test word embedding similarity and pretrained language models.

Copied to clipboard

Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu

Reviews Meet Graphs: Enhancing User and Item Representations for Recommendation with Hierarchical Attentive Graph Neural Network (D19-1)

Challenge:	Existing language models that use discrete representations for unified processing of various modalities are limited to text generation and do not include multimodal output.
Approach:	They propose a multimodal language model that utilizes discrete representations for unified processing of various modalities.
Outcome:	The proposed model can be trained stably without any alterations to existing models or training paradigms.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Tao Qi, Suyu Ge, Yongfeng Huang, Xing Xie

Representation Learning

Challenge:	Existing methods to learn user and item representations from review texts do not take into account the user-user and item-item relatedness of the user.
Approach:	They propose to use review content and user-item graphs to integrate them as different views.
Outcome:	The proposed approach can learn user and item representations from review content and user-item graphs.

Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)

Copied to clipboard

Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang

Low-code LLM: Graphical User Interface over Large Language Models (2024.naacl-demo)

Challenge:	Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks.
Approach:	They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially.
Outcome:	The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks.

Copied to clipboard

Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, WangYou WangYou, Ting Song, Yan Xia, Nan Duan, Furu Wei

MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation (2025.findings-emnlp)

Challenge:	Low-code LLM is a visual programming interface that allows users to incorporate their ideas into the process without writing trivial prompts.
Approach:	They propose a human-LLM interaction framework that incorporates low-code visual programming interactions to achieve more controllable and stable responses.
Outcome:	The proposed framework enables users to incorporate ideas into the process without writing trivial prompts.

Copied to clipboard

Chenghao Yang, Yinbo Luo, Zhoufutu Wen, Qi Chu, Tao Gong, Longxiang Liu, Kaiyuan Zhang, Jianpeng Jiao, Ge Zhang, Wenhao Huang, Nenghai Yu

Dialogue Systems & Conversational Agents

Challenge:	Large Language Models (LLMs) have been widely adopted in real-world dialogue applications, but their robustness is criticized all along.
Approach:	They propose to use play-by-play text commentary to build a multi-turn athletic real-world scenario dialogue benchmark to evaluate three critical aspects of multi-turned conversations: ultra multi- turn, interactive multi-twist, and cross-turn tasks.
Outcome:	The proposed benchmarks outperform open-source LLMs on three critical aspects of multi-turn conversations: ultra multi-turned, interactive multi- turn, and cross-turn tasks.

Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation (2024.emnlp-main)

Copied to clipboard

Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, JingBo Zhu

Low-Bit Quantization Favors Undertrained LLMs (2025.acl-long)

Challenge:	Existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset.
Approach:	They propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR).
Outcome:	The proposed method outperforms Alpaca's existing methods by 32.1% in GPT-4 evaluations.

Copied to clipboard

Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, Dong Yu

Challenge:	Larger models or those trained on fewer tokens exhibit less quantization-induced degradation (QiD), while smaller, well-trained models face significant performance losses.
Approach:	They propose to use QiD to measure an LLM’s training levels and determine the number of training tokens required for fully training LLMs of various sizes.
Outcome:	The proposed scaling laws can predict the quantization performance of different-sized LLMs trained with tokens.

Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021.acl-long)

Copied to clipboard

Xin Sun, Tao Ge, Furu Wei, Houfeng Wang

Arxiv Copilot: A Self-Evolving and Efficient LLM System for Personalized Academic Assistance (2024.emnlp-demo)

Challenge:	Existing approaches to improve online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC) are sequenceto-sequence (seq2sequ) and sequenceto sequence (saq2eq)
Approach:	They propose a novel approach to improve the online inference efficiency of the Transformer model for instantaneous Grammatical Error Correction (GEC) it aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism.
Outcome:	The proposed approach can achieve state-of-the-art results in English and Chinese benchmarks with 10x speedup over the Transformer-big model.

Copied to clipboard

Guanyu Lin, Tao Feng, Pengrui Han, Ge Liu, Jiaxuan You

Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020.emnlp-main)

Challenge:	Existing solutions for document QA fail to provide personalized and up-to-date information efficiently.
Approach:	They propose to deploy a self-evolving, efficient LLM system that can offer personalized research services, maintaining a real-time updated database.
Outcome:	The proposed system saves 69.92% of time after efficient deployment.

Copied to clipboard

Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, Ming Zhou

Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2024.naacl-long)

Challenge:	Existing methods to improve the efficiency of GEC are not efficient enough for GEC.
Approach:	They propose a language-independent approach to improve the efficiency of GEC by dividing the task into two subtasks: ESD and ESC.
Outcome:	The proposed approach performs comparably to conventional seq2seq approaches in English and Chinese GEC benchmarks with less than 50% time cost for inference.

Copied to clipboard

Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, Heng Ji

Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction (2024.findings-acl)

Challenge:	Existing work on LLMs that only enhance reasoning abilities, but which lack factual hallucination and slow-thinking capabilities, argues that SPP is a cognitive synergist.
Approach:	They propose a Solo Performance Prompting (SPP) that transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas.
Outcome:	The proposed model reduces factual hallucination and maintains strong reasoning abilities on three challenging tasks .

Copied to clipboard

Dingyao Yu, Yang An, Wei Ye, Xiongfeng Xiao, Shaoguang Mao, Tao Ge, Shikun Zhang

MdEval: Massively Multilingual Code Debugging (2026.findings-acl)

Challenge:	Chinese Spelling Correction (CSC) lacks large-scale high-quality corpora due to labor-intensive labeling of spelling errors in real-life writing or typing scenarios.
Approach:	They propose to use OCR/ASR-based generation to refine Chinese Spelling Correction models on random replacement-based corpora and filter them based on prediction confidence.
Outcome:	The proposed model outperforms existing models on three widely-used benchmarks while significantly alleviating over-correction.

Copied to clipboard

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Jin Ke, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, Zhoujun Li

Benchmarking Long-Context Language Models on Long Code Understanding (2025.acl-long)

Challenge:	Existing benchmarks primarily focus on Python and are limited in terms of language diversity.
Approach:	They propose a multilingual debugging benchmark that includes 3.9K test samples of 20 programming languages and introduces the debug instruction corpora MdEval-Instruct by injecting bugs into the correct multilingual queries and solutions.
Outcome:	The proposed benchmark includes 3.9K test samples of 20 programming languages and covers the automated program repair task, bug localization task, and bug identification task.

Copied to clipboard

Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, Zhi Jin

ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)

Challenge:	Currently, long-context language models are limited by the lack of a rigorous evaluation framework for long code understanding.
Approach:	They propose to use a long code understanding benchmark LongCodeU to evaluate LCLMs' long code comprehension ability for practical applications.
Outcome:	The proposed benchmarks show that current LCLMs are limited in their long code understanding ability, particularly when the long code length is greater than 32K, falling far short of their claimed 128K to 1M context windows.

Copied to clipboard

Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo

Characterizing the Impacts of Instances on Robustness (2023.findings-acl)

Challenge:	Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task.
Approach:	They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5.
Outcome:	The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers.

Copied to clipboard

Rui Zheng, Zhiheng Xi, Qin Liu, Wenbin Lai, Tao Gui, Qi Zhang, Xuanjing Huang, Jin Ma, Ying Shan, Weifeng Ge

Robustness in NLP

Challenge:	Existing defense approaches focus on developing new model structures or training algorithms, but they do little to tap the potential of training instances.
Approach:	They propose a method that can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training.
Outcome:	The proposed method can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training.

Improving Grammatical Error Correction with Machine Translation Pairs (2020.findings-emnlp)

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, Ming Zhou

Machine Translation

Challenge:	Existing methods to generate error-corrected sentence pairs for improving grammatical error correction are not available.
Approach:	They propose a method to generate error-corrected sentence pairs for improving grammatical error correction based on machine translation models of different qualities .
Outcome:	The proposed method can generate multiple error-corrected sentence pairs from Chinese to English text.

LEDGER: Scaling Agentic Document Editing with Dependency-aware Graph Retrieval (2026.findings-acl)

Copied to clipboard

Hang Wang, Utkarsh Garg, Reza Davari, Huitian Jiao, Hao Cheng, Baolin Peng, Si-Qing Chen, Tao Ge

Smart Word Suggestions for Writing Assistance (2023.findings-acl)

Challenge:	Document editing requires full-context awareness of dependencies, but processing entire documents for each edit incurs prohibitive token costs and latency.
Approach:	a framework that constructs lightweight dependency graphs captures semantic relationships and structural hierarchies across document elements is proposed for agentic document editing . a scaLing agentic agentic framework is based on a dependency graph framework that captures dependencies and refactors function dependencies.
Outcome:	a new framework achieves 76 consistency versus 56 baseline while reducing token usage by 85 . the framework is based on a framework that captures semantic relationships and structural hierarchies across document elements . it can be used to improve document consistency, but it also reduces token costs and latency .

Copied to clipboard

Chenshuo Wang, Shaoguang Mao, Tao Ge, Wenshan Wu, Xun Wang, Yan Xia, Jonathan Tien, Dongyan Zhao

Challenge:	Using word suggestions, writing assistance is a widely used application of natural language processing (NLP) . a task is performed to identify words or phrases that require improvement and provide substitution suggestions for each improvable target.
Approach:	They propose a task and benchmark to help writers improve word usage . they use human-labeled data and a distantly supervised dataset for testing .
Outcome:	The proposed task and benchmark aims to improve word usage in writing aids.

Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory (2025.acl-long)

Copied to clipboard

Guangyue Peng, Tao Ge, Wen Luo, Wei Li, Houfeng Wang

EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning for LLMs (2026.findings-acl)

Challenge:	Semiparametric language models (LMs) use static storage, which lacks learning capability and is disconnected from the internal information flow of the parametric models.
Approach:	They reconceptualize the non-parametric memory represented by kNN-LM as a learnable Mixture-of-Neighbors Induction Memory (MoNIM) this synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks .
Outcome:	The proposed model is a learnable Mixture-of-neighbors induction memory (MoNIM) it synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks (FFNs).

Copied to clipboard

Huanyu Liu, Jia Li, Yihong Dong, Chang Yu, Taozhi Chen, Lecheng Wang, Yongding Tao, Bin Gu, Ge Li

Reasoning

Challenge:	Existing approaches to reinforcement learning with verifiable reward (RLVR) are limited by difficulty or lack of exploration.
Approach:	They propose a self-evolving curriculum learning framework based on chain-of-thought reasoning optimization that constrains exploration space by self-generating and verifying CoT trajectories.
Outcome:	The proposed framework enables LLMs to solve previously unsolved problems without external supervision and is compatible with various RL fine-tuning methods.

BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020.emnlp-main)

Copied to clipboard

Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou

Automatic Grammatical Error Correction for Sequence-to-sequence Text Generation: An Empirical Study (P19-1)

Challenge:	a novel approach to compress neural networks by progressive module replacement is proposed . a number of techniques have been proposed to compress pretraining and fine-tuning models .
Approach:	They propose a model compression approach that divides BERT into modules and builds their compact substitutes.
Outcome:	The proposed approach outperforms existing knowledge distillation approaches on GLUE benchmark . it is based on a model that divides the original BERT into several modules and builds their substitutes .

Copied to clipboard

Tao Ge, Xingxing Zhang, Furu Wei, Ming Zhou

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting (2021.emnlp-main)

Challenge:	Sequence-to-sequence (seq2sequ) models have a weakness: they cannot always generate sentences without grammatical errors.
Approach:	They propose to use automatic grammatical error correction to improve seq2seq models . they conduct experiments on machine translation, formality style transfer, sentence compression and simplification .
Outcome:	The proposed system can improve grammaticality of generated text and improve formal style tasks.

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, Furu Wei

Text Generation

Challenge:	Existing text infilling objectives for pretrained language models require self-supervision by masking out tokens or spans in text.
Approach:	They propose to extend text infilling to a self-supervised sequence-to-sequence (Seq2Sequen) task.
Outcome:	The proposed task improves the model's performance on various natural language generation tasks.

Plug and Play Knowledge Distillation for kNN-LM with External Logits (2022.aacl-short)

Copied to clipboard

Xuyang Jin, Tao Ge, Furu Wei

ALYMPICS: LLM Agents Meet Game Theory (2025.coling-main)

Challenge:	Despite the promising evaluation results by knowledge distillation (KD) in natural language understanding (NLU) and sequence-to-sequence (seq2sequ) tasks, KD for causal language modeling (LM) remains a challenge.
Approach:	They propose to use external logits to improve a student's kNN-LM by leveraging teacher's knowledge at test time.
Outcome:	The proposed method improves a student's kNN-LM in multiple language modeling datasets and improves perplexity.

Copied to clipboard

Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Qiang Guan, Tao Ge, Furu Wei

Linguistics & Cognitive NLP

Challenge:	Alympics provides a framework for simulating human-like strategic interactions with Large Language Model (LLM) agents.
Approach:	They propose a framework utilizing Large Language Models (LLM) agents for empirical game theory research.
Outcome:	The proposed framework can be used to study human-like strategic interactions with large language model (LLM) agents in a game on the multi-round auction of scarce survival resources.

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)

Copied to clipboard

Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui

Neural News Recommendation with Multi-Head Self-Attention (D19-1)

Challenge:	Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding.
Approach:	They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding .
Outcome:	The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, Xing Xie

SCALE: Synergized Collaboration of Asymmetric Language Translation Engines (2024.findings-acl)

Challenge:	Precisely modeling news and users is critical for news recommendation, and capturing the contexts of words and news is important to learn news and user representations.
Approach:	They propose a neural news recommendation approach with multi-head self-attention to model the interactions between words and news and use multi-headed self- attention to capture relatedness between the news.
Outcome:	The proposed approach can learn representations from news titles by modeling the interactions between words and users and capture relatedness between the news.

Copied to clipboard

Xin Cheng, Xun Wang, Tao Ge, Si-Qing Chen, Furu Wei, Dongyan Zhao, Rui Yan

Multilinguality

Challenge:	In this paper, we introduce SCALE, a collaborative framework that connects a compact Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine.
Approach:	They propose a collaborative framework that connects a Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine.
Outcome:	The proposed framework outperforms both LLMs and supervised models in high-resource or challenging low-resourced settings.

K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning (2025.naacl-long)

Copied to clipboard

Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, Furu Wei

Scheduled DropHead: A Regularization Method for Transformer Models (2020.findings-emnlp)

Challenge:	Strategic reasoning requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent environments.
Approach:	They propose a framework that enables Large Language Models to achieve varying levels of strategic depth by recursive mechanisms that allow agents to form higher order beliefs about others' beliefs.
Outcome:	The proposed framework enables LLMs to achieve varying levels of strategic depth, allowing agents to form higher order beliefs—beliefs about others’ beliefs.

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, Ke Xu

EventWiki: A Knowledge Base of Major Events (L18-1)

Challenge:	DropHead is a structured dropout method for regularizing multi-head attention . DropHed drops entire attention heads during training to prevent overfitting .
Approach:	They propose a structured dropout method specifically designed for regularizing multi-head attention mechanism . DropHead drops entire attention heads during training to prevent overfitting .
Outcome:	The proposed method can improve transformer models by 0.9 BLEU score on translation task and around 1.0 accuracy for various text classification tasks.

Copied to clipboard

Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou

Knowledge Representation

Challenge:	Existing knowledge bases focus on static entities such as people, locations and organizations.
Approach:	They propose a new knowledge base resource called EventWiki which concentrates on major events . they show that EventWiki is a very useful resource for information extraction regarding events in NLP .
Outcome:	The proposed resource is the first knowledge base resource of major events.

Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition (D18-1)

Copied to clipboard

Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou

Multilinguality

Challenge:	a novel information network decipherment paradigm is proposed for fine-grained coordinated cross-lingual text stream alignment.
Approach:	They propose to use Burst Information Networks as media to represent text streams . they propose a simple yet effective information network decipherment algorithm with diverse clues .
Outcome:	The proposed approach outperforms existing approaches on bilingual lexicon extraction from coordinated text streams and can harvest high-quality alignments from large amounts of streaming data for endless language knowledge mining.

Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoning (2024.emnlp-main)

Copied to clipboard

Zhihan Zhang, Tao Ge, Zhenwen Liang, Wenhao Yu, Dian Yu, Mengzhao Jia, Dong Yu, Meng Jiang

Reasoning

Challenge:	Existing studies focus on broadening the training set with data augmentation techniques to maximize such benefits.
Approach:	They propose a method that embeds problem reflection into each training instance.
Outcome:	The proposed method enhances performance in standard and complex scenarios that require reflective thinking.

Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023.findings-emnlp)

Copied to clipboard

Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, Zhifang Sui

UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database (2020.aacl-main)

Challenge:	Experimental results show draft-then-verify paradigm can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding.
Approach:	They propose to use Spec-Drafter and Spec Verification to accelerate autoregressive (AR) decoding by combining a model optimized for efficient and accurate drafting and a reliable method for verifying the drafted tokens efficiently.
Outcome:	The proposed method achieves 5x speedup on seq2seq tasks with comparable generation quality to beam search decoding, refreshing the impression that draft-then-verify paradigm introduces only 1.4x2x speed up.

Copied to clipboard

Canwen Xu, Tao Ge, Chenliang Li, Furu Wei

Router-Tuning: A Simple and Effective Approach for Dynamic Depth (2025.emnlp-main)

Challenge:	Chinese and Japanese share many characters with similar surface morphology.
Approach:	They propose a Chinese-Japanese pretrained masked language model with a coarse-to-fine training approach to exploit the shared knowledge across the languages.
Outcome:	The proposed model is effective on mono- and cross-lingual Chinese and Japanese tasks.

Copied to clipboard

Shwai He, Tao Ge, Guoheng Sun, Bowei Tian, Xiaoyang Wang, Dong Yu

RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization (2026.acl-long)

Challenge:	Existing methods to improve computational efficiency are under-explored and face several critical challenges.
Approach:	They propose a method that selectively activates only a subset of the model's layers, skipping those deemed less important.
Outcome:	The proposed method significantly improves performance on Attention layers and MoE layers while reducing redundant computation and memory usage.

Copied to clipboard

Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei MA, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li

Reasoning

Challenge:	Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs).
Approach:	They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities.
Outcome:	The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks.

Fluency Boost Learning and Inference for Neural Grammatical Error Correction (P18-1)

Copied to clipboard

Tao Ge, Furu Wei, Ming Zhou

Syntactic Text Processing

Challenge:	Seq2seq models for grammatical error correction (GEC) have two limitations: (1) a seq2q model may not be well generalized with only limited error-corrected data; (2) a model may fail to completely correct a sentence with multiple errors through normal seq1sequeq inference.
Approach:	They propose a fluency boost learning and inference mechanism to improve the performance of seq2seq models for grammatical error correction (GEC) by generating fluency-boost sentence pairs during training.
Outcome:	Experiments show that the proposed model improves on both CoNLL-2014 and JFLEG benchmark datasets.

BERT-based Lexical Substitution (P19-1)

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou

Navigating the OverKill in Large Language Models (2024.acl-long)

Challenge:	Existing approaches to lexical substitution tend to overlook good substitute candidates that are not the synonyms of the target words in the lexicals and fail to take into account the substitution’s influence on the global context of the sentence.
Approach:	They propose an end-to-end BERT-based lexical substitution approach which proposes and validates substitute candidates without using annotated data or manually curated resources.
Outcome:	The proposed approach performs well in proposing and ranking substitute candidates, achieving the state-of-the-art results in both LS07 and LS14 benchmarks.

Copied to clipboard

Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Xun Zhao, Dahua Lin

Pseudo-Bidirectional Decoding for Local Sequence Transduction (2020.findings-emnlp)

Challenge:	Recent studies have highlighted a tendency among large language models to refuse to answer benign queries.
Approach:	They propose a model-agnostic approach to reduce excessive attention to harmful words like ‘kill’ and a method to decode the next-token predictions by contrastive decoding.
Outcome:	The proposed approach reduces the refusal rate by 20% while having little impact on safety.

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Ke Xu