Papers by Shaohan Huang

Context-DPO: Aligning Language Models for Context-Faithfulness (2025.findings-acl)

Challenge:	ELECTRA-style tasks are used to pretrain cross-lingual models for NLP tasks . masked language modeling tasks require massive computation resources, rendering such models quite expensive .
Approach:	They propose to use ELECTRA-style tasks to pre-train a cross-lingual language model . they propose to pretrain the model on multilingual and parallel corpora .
Outcome:	The proposed model outperforms baseline models on cross-lingual understanding tasks with much less computation cost.

Copied to clipboard

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu

TableBank: Table Benchmark for Image-based Table Detection and Recognition (2020.lrec-1)

Challenge:	Context-DPO is the first alignment method specifically designed to enhance contextfaithfulness for large language models.
Approach:	They propose a benchmark that simulates Retrieval-Augmented Generation scenarios with knowledge conflicts to evaluate context-faithfulness.
Outcome:	The proposed method improves LLMs' context-faithfulness by 35% to 280% over open-source models.

Copied to clipboard

Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li

Multimodality

Challenge:	Existing techniques for table detection and recognition are limited to document types and layouts.
Approach:	They propose to build a table detection and recognition dataset with weak supervision from Word and Latex documents on the internet.
Outcome:	The proposed dataset contains 417K high quality labeled tables and is publicly available.

Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training (2021.emnlp-main)

Copied to clipboard

Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

Scaling Sentence Embeddings with Large Language Models (2024.findings-emnlp)

Challenge:	Existing models require a more expressive vocabulary to represent all languages . however, increasing the vocabulary size significantly slows down the pre-training speed .
Approach:	They propose an algorithm VoCap to determine the desired vocabulary capacity of each language.
Outcome:	The proposed algorithm improves cross-lingual model pre-training while reducing side effects of increasing vocabulary size.

Copied to clipboard

Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang

HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition (2024.acl-long)

Challenge:	Current methods based on contrastive learning have generated high-quality sentence embeddings.
Approach:	They propose a method to enhance LLM performance on sentence embeddings with a one-word limitation.
Outcome:	The proposed method outperforms contrastive learning methods on sentence embeddings without fine-tuning and with fine-untun.

Copied to clipboard

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks (2026.acl-long)

Challenge:	Large language models (LLMs) are a promising alternative to expensive human evaluations.
Approach:	They propose a framework that iteratively aligns LLM-based evaluators with human preference . they decompose a given evaluation task into finer-grained criteria .
Outcome:	The proposed framework iteratively aligns LLM-based evaluators with human preference . it decomposes a given evaluation task into finer-grained criteria . the framework is efficient to train and more explainable than relying solely on prompts .

Copied to clipboard

Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, Guanhua Chen

Responsible & Trustworthy NLP

Challenge:	Proximal Policy Optimization (PPO) is central to aligning Large Language Models with verifiable rewards.
Approach:	They propose a scalable algorithm that harmonizes sample efficiency with stability of outcome-based updates.
Outcome:	The proposed algorithm outperforms standard PPO and matches the performance of computation-heavy group-based methods.

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021.emnlp-main)

Copied to clipboard

Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, Furu Wei

Multilinguality

Challenge:	Multilingual T5 pretrains a sequence-to-sequence model on monolingual texts, but it has shown promising results on many cross-lingual tasks.
Approach:	They propose a partially non-autoregressive objective for text-to-text pre-training and propose mT6 to improve cross-lingual transferability over multilingual T5.
Outcome:	The proposed model improves cross-lingual transferability over existing models.

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator (2023.acl-long)

Copied to clipboard

Jian Yang, Shuming Ma, Li Dong, Shaohan Huang, Haoyang Huang, Yuwei Yin, Dongdong Zhang, Liqun Yang, Furu Wei, Zhoujun Li

Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs (2026.findings-acl)

Challenge:	Existing pre-training methods underutilize the benefits of language understanding for generation.
Approach:	They propose a GAN-style model for encoder-decoder pre-training with an auxiliary discriminator.
Outcome:	The proposed model outperforms existing pre-trained models and achieves state-of-the-art performance.

Copied to clipboard

Qibin Wang, Pu Zhao, Shaohan Huang, Fangkai Yang, Lu Wang, Furu Wei, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Se2: Sequential Example Selection for In-Context Learning (2024.findings-acl)

Challenge:	Existing approaches to test-time scaling are limited due to the quality of candidate responses.
Approach:	They propose a new metric to quantify the relative improvement of self-refinement beyond majority voting.
Outcome:	The proposed method achieves state-of-the-art performance across five benchmarks over other methods.

Copied to clipboard

Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, Qi Zhang

GeAR: Generation Augmented Retrieval (2025.findings-acl)

Challenge:	Prior work has explored the selection of examples for in-context learning, neglecting the internal relationships between examples and exist an inconsistency between training and inference.
Approach:	They propose a sequential-aware method that leverages the LLM’s feedback on varying context, aiding in capturing inter-relationships and sequential information among examples.
Outcome:	Experiments on 23 NLP tasks show that Se2 surpasses baselines and achieves 42% relative improvement over random selection.

Copied to clipboard

Haoyu Liu, Shaohan Huang, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Furu Wei, Qi Zhang

Information Retrieval

Challenge:	Document retrieval techniques are used to compute semantic similarity between a query and documents, but the scalar similarity fails to reflect enough information, hindering the interpretation of retrieval results.
Approach:	They propose a method which improves the global document-query similarity through contrastive learning and integrates well-designed fusion and decoding modules.
Outcome:	The proposed method improves the global document-query similarity through contrastive learning and integrates well-designed fusion and decoding modules.

Democratizing Reasoning Ability: Tailored Learning from Large Language Model (2023.emnlp-main)

Copied to clipboard

Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

Textual Aesthetics in Large Language Models (2025.emnlp-main)

Challenge:	Large language models (LLMs) exhibit impressive emergent abilities in natural language processing, but their democratization is hindered due to huge computation requirements and closed-source nature.
Approach:	They propose a tailored learning approach to distill the exclusive reasoning ability to smaller LMs to facilitate democratization.
Outcome:	The proposed approach enables the democratization of the exclusive reasoning ability by leveraging the black-box model as a reasoning teacher.

Copied to clipboard

Lingjie Jiang, Shaohan Huang, Xun Wu, Furu Wei

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation (2026.acl-long)

Challenge:	Existing studies on image aesthetics have focused on content correctness and helpfulness of responses.
Approach:	They propose a textual aesthetics-powered fine-tuning method that leverages textual visual aesthetics without compromising content correctness.
Outcome:	The proposed method improves aesthetic scores and performs well on general evaluation datasets.

Copied to clipboard

Yixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang, Lingjie Jiang, Shaohan Huang, Yun Chen, Guanhua Chen, Furu Wei

Multimodality

Challenge:	Multimodal large language models have advanced rapidly, yet most remain English-centric . scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of non-English image–text supervision.
Approach:	They propose a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone.
Outcome:	The proposed framework achieves competitive performance with a fully multimodally trained model using less than 2% of the text data.

Pre-training Language Model as a Multi-perspective Course Learner (2023.findings-acl)

Copied to clipboard

Beiduo Chen, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang

DocBank: A Benchmark Dataset for Document Layout Analysis (2020.coling-main)

Challenge:	Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks.
Approach:	They propose a multi-perspective course learning method to fetch many degrees and visual angles for sample-efficient pre-training and to fully leverage the relationship between generator and discriminator.
Outcome:	The proposed method improves ELECTRA's performance on GLUE and SQuAD 2.0 benchmarks and overshadows recent advanced ELECL-style models under the same settings.

Copied to clipboard

Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou

Challenge:	Existing approaches for document layout analysis are based on rule-based or machine learning methods that ignore textual information.
Approach:	They present a benchmark document layout analysis dataset using a computer vision model . they build strong baselines and manually split train/dev/test sets for evaluation .
Outcome:	The proposed model trains on DocBank accurately recognize layout information for a variety of documents.

NL2Lean: Translating Natural Language into Lean 4 through Multi-Aspect Reinforcement Learning (2025.emnlp-main)

Copied to clipboard

Yue Fang, Shaohan Huang, Xin Yu, Haizhen Huang, Zihan Zhang, Weiwei Deng, Furu Wei, Feng Sun, Qi Zhang, Zhi Jin

Machine Translation

Challenge:	Existing formal proof assistants rely on instruction tuning and lack fine-grained structural and semantic alignment.
Approach:	They propose a reinforcement learning framework that enables LLMs to translate natural language into formal language such as Lean 4 . they use a model with basic translation ability to refine the model's reinforcement learning .
Outcome:	The proposed method outperforms baseline models on NL-to-Lean 4 tasks.

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption (2022.findings-acl)

Copied to clipboard

Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei

PromptBERT: Improving BERT Sentence Embeddings with Prompts (2022.emnlp-main)

Challenge:	enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks.
Approach:	They propose an approximation approach for transformers which enables inference on ciphertext data.
Outcome:	The proposed approach can infer pre-trained models on encrypted data with negligible performance drop but enjoy theory-guaranteed privacy-preserving advantage.

Copied to clipboard

Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang

Dual-Alignment Pre-training for Cross-lingual Sentence Embedding (2023.acl-long)

Challenge:	Existing research shows that BERT and RoBERTa are poor in sentence embeddings due to static token embeddable bias and ineffective BERT layers.
Approach:	They propose a novel contrastive learning method for better sentence embeddings by using a template denoising technique.
Outcome:	The proposed method achieves 2.29 and 2.58 points of improvement compared to SimCSE and RoBERTa in the unsupervised setting.

Copied to clipboard

Ziheng Li, Shaohan Huang, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, Qi Zhang

Unsupervised Fine-tuning for Text Clustering (2020.coling-main)

Challenge:	Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding.
Approach:	They propose a dual-alignment pre-training framework that incorporates both sentence-level and token-level alignment.
Outcome:	The proposed framework improves cross-lingual sentence embedding on three cross-linguistic benchmarks.

Copied to clipboard

Shaohan Huang, Furu Wei, Lei Cui, Xingxing Zhang, Ming Zhou

Low-Resource NLP

Challenge:	Existing approaches to text clustering fine-tune pre-trained models have been limited.
Approach:	They propose a method to fine-tune pre-trained models unsupervisedly for text clustering by learning text representations and cluster assignments using a clustering oriented loss.
Outcome:	The proposed model outperforms baseline methods and achieves state-of-the-art results on three text clustering datasets.

Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph (2020.emnlp-main)

Copied to clipboard

Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, Minlie Huang

ResLoRA: Identity Residual Mapping in Low-Rank Adaption (2024.findings-acl)

Challenge:	Existing approaches that integrate commonsense knowledge into pre-trained language models simply transfer relational knowledge while ignoring rich connections within the knowledge graph.
Approach:	They propose a method that leverages structural and semantic information of the knowledge graph to generate commonsense-aware text.
Outcome:	The proposed method outperforms baseline models on three text generation tasks that require reasoning over commonsense knowledge.

Copied to clipboard

Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

Challenge:	Low-rank adaptation (LoRA) is one of the most popular parameter-efficient fine-tuning methods.
Approach:	They propose a low-rank adaptation method that adds residual paths during training and merges them together during inference to achieve better results.
Outcome:	The proposed method achieves 2.5x faster convergence speed and improves performance by 14.3% on NLG, NLU, and text-to-image tasks.

Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths (2020.aacl-main)

Copied to clipboard

Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Minlie Huang

Shorten After You’re Right: Lazy Length Penalties for Reasoning RL (2026.findings-acl)

Challenge:	Existing tasks that use commonsense reasoning as multi-choice reading comprehension lack direct assessment to machine commonsence and impede its practicability to realistic scenarios.
Approach:	They propose a method that first extracts the underlying concepts which are served as bridges in the reasoning chain and then integrates these concepts to generate the final explanation.
Outcome:	The proposed model outperforms the state-of-the-art models in automatic and human evaluation.

Copied to clipboard

Danlong Yuan, Tian Xie, Shaohan Huang, Huishuai Zhang, Zhuocheng Gong, Chong Luo, Furu Wei, Dongyan Zhao

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts (2026.acl-long)

Challenge:	Existing shortening methods for long reasoning models rely on additional supervision or multi-stage post-training.
Approach:	They propose a lazy length penalty that imposes length pressure on models without extra training stages.
Outcome:	The proposed method significantly reduces response length without extra training stages while maintaining or improving performance.

Copied to clipboard

Di Zhang, Xun Wu, Shaohan Huang, Lingjie Jiang, Yaru Hao, Li Dong, Zewen Chi, Zhifang Sui, Furu Wei

Instruction Pre-Training: Language Models are Supervised Multitask Learners (2024.emnlp-main)

Challenge:	Reinforcement learning with verifiable rewards (RLVR) training with Mixture-of-Experts policies remains fragile and prone to reward collapse.
Approach:	They propose a router shift-based policy optimization method that computes a per-token router-shift ratio conditioned on the previously activated experts and applies stop-gradient and a lower-bound floor.
Outcome:	The proposed method achieves better performance and greater stability than previous methods.

Copied to clipboard

Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei

On Domain-Adaptive Post-Training for Multimodal Large Language Models (2025.findings-emnlp)

Challenge:	Unsupervised multitask pre-training has been the key to the success of language models (LMs) however, scaling it in the post-training stage trends towards better generalization.
Approach:	They propose a framework that augments massive raw corpora with instruction-response pairs to pre-train LMs.
Outcome:	The proposed framework augments massive raw corpora with instruction-response pairs to pre-train LMs.

Copied to clipboard

Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Xin Zhao, Zhongzhi Luan, Bo Dai, Zhenliang Zhang

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning (2023.acl-long)

Challenge:	Adapting general multimodal large language models to specific domains is important for practical applications.
Approach:	They investigate domain adaptation of multimodal large language models via post-training . they develop a generate-then-filter pipeline that curates diverse visual instruction tasks .
Outcome:	The proposed model outperforms existing models in domain adaptation by combining data from open-source models with training pipelines.

Copied to clipboard

Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song

A Length-Extrapolatable Transformer (2023.acl-long)

Challenge:	XY-LENT: X-Y bitext enhanced Language ENcodings achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands.
Approach:	They propose a method for building multilingual representation models that are competitive with existing models and more parameter efficient.
Outcome:	The proposed model outperforms XLM-R XXL and is 5x and 6x smaller respectively.

Copied to clipboard

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei

Neural Document Summarization by Jointly Learning to Score and Select Sentences (P18-1)

Challenge:	Existing Transformers can only deal with the in-distribution size of inputs.
Approach:	They propose a relative position embedding to explicitly maximize attention resolution . they also use blockwise causal attention during inference for better resolution a .
Outcome:	The proposed model achieves strong performance in interpolation and extrapolation settings.

Copied to clipboard

Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, Tiejun Zhao

Information Extraction & Text Mining

Challenge:	Sentence scoring and sentence selection are two main steps in extractive document summarization systems.
Approach:	They propose an end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences.
Outcome:	The proposed framework outperforms the state-of-the-art summarization models on the CNN/Daily Mail dataset.

Consistency Regularization for Cross-Lingual Fine-Tuning (2021.acl-long)

Copied to clipboard

Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

Multilinguality

Challenge:	Experimental results show that consistency regularization improves cross-lingual fine-tuning . pre-trained cross-linguistic models can transfer task-specific supervision from one language to the other .
Approach:	They propose to improve cross-lingual fine-tuning with consistency regularization . they use example consistency regularized to penalize prediction sensitivity to four types of data augmentations .
Outcome:	The proposed method improves cross-lingual fine-tuning across tasks . it can be generalized to other target languages without additional training .

Calibrating LLM-Based Evaluator (2024.lrec-main)

Copied to clipboard

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment (2021.acl-long)

Challenge:	Existing models for large language models lack the ability to calibrate their outputs towards human preference.
Approach:	They propose a multi-stage, gradient-free approach to calibrate an LLM-based evaluator toward human preference.
Outcome:	The proposed approach improves correlation with expert evaluation on multiple text quality evaluation datasets.

Copied to clipboard

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation (2023.emnlp-main)

Challenge:	Experimental results show that denoising word alignment improves cross-lingual transferability . most applications and resources are still English-centric, making non-English users hard to access.
Approach:	They propose to denoise word alignment as a cross-lingual pre-training task . they first self-label word alignments for parallel sentences and then mask tokens .
Outcome:	The proposed model improves cross-lingual transferability on token-level tasks, especially on question answering, and structured prediction.

Copied to clipboard

Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, Qi Zhang

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2021.findings-acl)

Challenge:	Large Language Models (LLMs) have impressive capabilities but need for task-specific prompt engineering can hinder their generalization.
Approach:	They propose a lightweight and versatile retriever that automatically retrieves prompts for a given zero-shot task input.
Outcome:	The proposed model is universally applicable across tasks and models . it mitigates hallucination problem in chatGPT, and it improves even the strongest LLMs.

Copied to clipboard

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)

Challenge:	Existing work on deep self-attention distillation for natural language processing tasks is limited by computational resources and latency.
Approach:	They generalize deep self-attention distillation in MINILM by using only self- attention relation distillation for taskagnostic compression of pretrained Transformers.
Outcome:	The proposed model outperforms the state-of-the-art in a multilingual and multilingual teacher model.

Copied to clipboard

Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei