Papers by Pei Zhang

ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch (2026.acl-long)

Challenge:	Iterative non-autoregressive (NAR) models have recently demonstrated impressive performance in varied generation tasks, surpassing the autoregressive Transformer.
Approach:	They propose a strategy to conduct efficient refinements without performance declines by using two simple metrics to identify potential problems existing in current refinement processes.
Outcome:	The proposed model outperforms the autoregressive Transformer by around one BLEU on average.

Copied to clipboard

Zheng Liu, Honglin Lin, Xiaoyang Wang, Xin Gao, Yu Li, Mengzhang Cai, Yun Zhu, Zhanping Zhong, Qizhi Pei, Zhuoshi Pan, Xiaoran Shang, Conghui He, Bin Cui, Wentao Zhang, Lijun Wu

Aligning Large Language Models with Implicit Preferences from User-Generated Content (2025.acl-long)

Challenge:	Existing open-source vision language models lack high-quality training data for chart reasoning . current models are simplistic and repetitive, while associated QA pairs are prone to hallucinations .
Approach:	They propose a framework to synthesize complex charts and reliable reasoning data from scratch.
Outcome:	Experimental results show that ChartVerse-8B surpasses existing models in QA and difficulty . lack of high-quality training data hampers development of open-source models .

Copied to clipboard

Zhaoxuan Tan, Zheng Li, Tianyi Liu, Haodong Wang, Hyokun Yun, Ming Zeng, Pei Chen, Zhihan Zhang, Yifan Gao, Ruijie Wang, Priyanka Nigam, Bing Yin, Meng Jiang

Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewards (2026.acl-long)

Challenge:	Existing preference learning methods rely heavily on curated data from humans or advanced LLMs, which is costly and difficult to scale.
Approach:	They propose a framework that leverages implicit preferences in unlabeled user-generated content to generate preference data.
Outcome:	The proposed framework transforms user-generated content into user queries and generates responses from the policy model.

Copied to clipboard

Ming Li, Pei Chen, Zhenhao Zhang, Tao Yang, Xinyang Zhang, Han Li, Tianyu Cao, Ming Zeng, Zhuofeng Wu, Meng Jiang, Huasheng Li, Lihong Li, Bing Yin

Dialogue Systems & Conversational Agents

Challenge:	Large Language Models exhibit strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC) when instructions are revealed progressively in multi-turn settings, models get "Lost in Conversation"
Approach:	They propose a framework that encourages models to generate correct answers and judge solvability in multi-turn conversations.
Outcome:	The proposed framework improves models' ability to balance problem-solving with abstention . it reduces premature answering behaviors that cause lost-in-conversation (LiC)

One Battle After Another: Probing LLMs’ Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework (2026.acl-long)

Copied to clipboard

Qi Jia, Ye Shen, Xiujie Song, Kaiwei Zhang, Shibo Wang, Dun Pei, Xiangyang Zhu, Guangtao Zhai

Dialogue Systems & Conversational Agents

Challenge:	Existing benchmarks for instruction-following in multi-topic dialogues are limited to a fixed number of turns, susceptible to saturation and failing to account for users’ interactive experience.
Approach:	They propose a framework featuring a three-layer tracking mechanism and a query synthesis agent to mimic sequential user behaviors.
Outcome:	The proposed framework outperforms existing benchmarks in the evaluation of instruction following in multi-topic dialogues and demonstrates deficiencies in failure recovery and fine-grained instruction following.

ALERT: An LLM-powered Benchmark for Automatic Evaluation of Recommendation Explanations (2025.naacl-long)

Copied to clipboard

Yichuan Li, Xinyang Zhang, Chenwei Zhang, Mao Li, Tianyi Liu, Pei Chen, Yifan Gao, Kyumin Lee, Kaize Ding, Zhengyang Wang, Zhihan Zhang, Jingbo Shang, Xian Li, Trishul Chilimbi

ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors (2024.findings-emnlp)

Challenge:	Existing benchmarks for recommendation explanation evaluation lack item diversity and user preferences data.
Approach:	They propose a model-agnostic recommendation explanation evaluation benchmark based on Amazon e-commerce categories with implicit preferences . they propose two novel automatic evaluators that enable scalable and human-preference aligned evaluation of explanations .
Outcome:	The proposed model-agnostic evaluation benchmark outperforms existing methods in a variety of domains.

Copied to clipboard

Zhexin Zhang, Yida Lu, Jingyuan Ma, Di Zhang, Rui Li, Pei Ke, Hao Sun, Lei Sha, Zhifang Sui, Hongning Wang, Minlie Huang

Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training (2025.naacl-long)

Challenge:	Existing tools for detecting safety issues in LLMs are expensive and inefficient.
Approach:	They propose an LLM-based safety detector which annotates the safety of queries and provides explanations for its decisions.
Outcome:	The proposed detector outperforms baselines on four sets of query-response pairs and is effective as a safety evaluator for advanced LLMs.

Copied to clipboard

Yuchen Zhuang, Jingfeng Yang, Haoming Jiang, Xin Liu, Kewei Cheng, Sanket Lokegaonkar, Yifan Gao, Qing Ping, Tianyi Liu, Binxuan Huang, Zheng Li, Zhengyang Wang, Pei Chen, Ruijie Wang, Rongzhi Zhang, Nasser Zalmout, Priyanka Nigam, Bing Yin, Chao Zhang

MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing (2026.acl-industry)

Challenge:	Existing LLMs often rely on complex prompting or extensive fine-tuning to introduce new capabilities while preserving strong generalizability.
Approach:	They propose a large-scale pre-training corpus to enhance LLM agents' capabilities . they use 103B agent-specific data encompassing 76,537 APIs .
Outcome:	The proposed training corpus outperforms open-source LLMs and commercial LLM agents on three agent benchmarks.

Copied to clipboard

Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Zirui Tang, Boyu Niu, Yuanhong Zheng, Dongsheng Ma, Ziyang Miao, Hejun Dong, Siyi Qian, Junyuan Zhang, Fangdong Wang, Jingzhou Chen, Xiaomeng Zhao, Liqun Wei, Wei Li, Shasha Wang, RuiLiang Xu, Yuanyuan Cao, Lu Chen, Qianqian Wu, Huaiyu Gu, Lindong Lu, Dechen Lin, null Shenguanlin, Xuanhe Zhou, Linfeng Zhang, Yuhang Zang, Xiaoyi Dong, Jiaqi Wang, Bo Zhang, Lei Bai, Pei Chu, Weijia Li, Jiang Wu, Lijun Wu, Zhenxiang Li, Guangyu Wang, Zhongying Tu, Chao Xu, Kai Chen, Bowen Zhou, Dahua Lin, Wentao Zhang, Conghui He

Green & Sustainable NLP

Challenge:	Document images are characterized by higher resolutions, denser content, and more complex structural layouts.
Approach:	They propose a 1.2B-parameter document parsing vision-language model that decouples layout analysis from local content recognition.
Outcome:	The proposed model surpasses general-purpose and domain-specific models on multiple benchmarks while maintaining significantly lower computational overhead.

Analyze Like a Venture Capitalist: Information-Gain and Knowledge Enhanced Graph Reasoning for Startup Success Prediction (2026.findings-acl)

Copied to clipboard

Haoyu Pei, Zhongyang Liu, Xiangyi Xiao, Xiaocong Du, Suting Hong, Kunpeng Zhang, Haipeng Zhang

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models (2024.findings-acl)

Challenge:	Most venture capital investments fail, while a few deliver outsized returns.
Approach:	They propose a framework that synthesizes relational evidence across sources . they propose combining information-gain-driven retriever and knowledge base to ground reasoning .
Outcome:	The proposed framework achieves +5.9% F1 and +22.1% Precision@5 over state-of-the-art baselines.

Copied to clipboard

Yiming Wang, Zhuosheng Zhang, Pei Zhang, Baosong Yang, Rui Wang

Exploring Reversal Mathematical Reasoning Ability for Large Language Models (2024.findings-acl)

Challenge:	Existing methods rely on syntactically mapping natural languages to complete formal languages like Python and SQL.
Approach:	They propose to deconstruct reasoning-independent semantic information into generic symbolic representations, thereby efficiently capturing more generalized reasoning knowledge.
Outcome:	The proposed method improves in-context reasoning accuracy, learning efficiency, out-of-domain generalization, and output stability compared to the Chain-of thought technique.

Copied to clipboard

Pei Guo, WangJie You, Juntao Li, Yan Bowen, Min Zhang

Reasoning

Challenge:	Large language models (LLMs) have been a success in the wide range of natural language understanding and reasoning tasks.
Approach:	They propose a training method to improve general and reversal reasoning abilities by using a reversed dataset.
Outcome:	The proposed method improves general and reversal reasoning abilities and alleviates the reverse curse.

Instant Personalized Large Language Model Adaptation via Hypernetwork (2026.acl-long)

Copied to clipboard

Zhaoxuan Tan, Zixuan Zhang, Haoyang Wen, Zheng Li, Rongzhi Zhang, Pei Chen, Fengran Mo, Zheyuan Liu, Qingkai Zeng, Qingyu Yin, Meng Jiang

BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations (2023.emnlp-main)

Challenge:	Existing parameter-efficient fine-tuning methods require training a separate adapter for each user, making them computationally expensive and impractical for real-time updates.
Approach:	They propose a scalable framework that maps a user's profile directly to a full set of adapter parameters.
Outcome:	The proposed framework outperforms prompt-based personalization and OPPU while using substantially fewer computational resources at deployment.

Copied to clipboard

Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, Rui Yan

Efficient Domain Adaptation for Non-Autoregressive Machine Translation (2024.findings-acl)

Challenge:	et al., 2022) argue that the current models for drug discovery lack the ability to integrate molecules, proteins, and natural language.
Approach:	They propose a framework that integrates biological knowledge with chemical knowledge and natural language associations.
Outcome:	The proposed framework shows superior performance across a wide range of tasks.

Copied to clipboard

WangJie You, Pei Guo, Juntao Li, Kehai Chen, Min Zhang

Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use (2026.findings-acl)

Challenge:	Existing non-parametric approaches like nearest neighbor machine translation have made small Autoregressive translation models less efficient . despite their impressive generalization and task performance, LLMs suffer from prohibitive inference cost when confronted with specific domains.
Approach:	They propose a domain adaptation approach that tailors a k-nearest-neighbor algorithm for NAT models that incorporates the parallel nature of NAT.
Outcome:	The proposed approach achieves significant improvements over the Base-NAT model and exhibits enhanced efficiency.

Copied to clipboard

Wuyang Zhang, Shichao Pei

Robustness in NLP

Challenge:	Existing studies have not examined how backdoored agents can influence tool-use sequences to perform harmful actions.
Approach:	They propose a backdoor attack framework that embeds semantic triggers into fine-tuned LLM agents.
Outcome:	The proposed framework embeds semantic triggers into fine-tuned LLM agents . when triggered, the backdoored agent invokes memory-access tool calls to retrieve stored user context and exfiltrates it via disguised retrieval tool calls.

AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024.acl-long)

Copied to clipboard

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, Jie Tang

Lattice Transformer for Speech Translation (P19-1)

Challenge:	Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment.
Approach:	They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references.
Outcome:	The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references.

Copied to clipboard

Pei Zhang, Niyu Ge, Boxing Chen, Kai Fan

Text Generation

Challenge:	Recent advances in sequence modeling have highlighted the strengths of the transformer architecture.
Approach:	They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations.
Outcome:	The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task.

STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework (2025.findings-acl)

Copied to clipboard

Wenhao Liu, Zhenyi Lu, Xinyu Hu, Jerry Zhang, Dailin Li, Jiacheng Cen, Huilin Cao, Haiteng Wang, Yuhan Li, Xie Kun, Dandan Li, Pei Zhang, Chengbo Zhang, Yuxiang Ren, Xiaohong Huang, Yan Ma

Reasoning

Challenge:	Existing datasets suffer from outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation.
Approach:	They propose a human-in-the-loop, multi-agent data generation framework that integrates reasoning-dense filters, multiagent collaboration, and human mathematicians’ evaluations to ensure the reliability and quality of the dataset.
Outcome:	The proposed framework improves accuracy and quality of the 2,000-synthesized datasets by integrating reasoning-dense filters, multi-agent collaboration, and human mathematicians’ evaluations.

LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)

Copied to clipboard

Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang

Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning (2024.findings-emnlp)

Challenge:	LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases .
Approach:	They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website .
Outcome:	The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks.

Copied to clipboard

Tianxiang Hu, Pei Zhang, Baosong Yang, Jun Xie, Derek Wong, Rui Wang

AnyTrans: Translate AnyText in the Image with Large Scale Models (2024.findings-emnlp)

Challenge:	Achieving consistent high-quality machine translation across diverse domains remains a challenge due to limited and imbalanced parallel training data available in various domains.
Approach:	They propose a domain Chain of Thought technique that uses the multi-domain intelligence of LLMs to improve translation performance.
Outcome:	The proposed method achieves significant improvements in translation accuracy and domain robustness over traditional fine-tuning on a small dataset of four domains.

Copied to clipboard

Zhipeng Qian, Pei Zhang, Baosong Yang, Kai Fan, Yiwei Ma, Derek Wong, Xiaoshuai Sun, Rongrong Ji

Multilinguality

Challenge:	Recent advances in natural language processing and computer vision have made it possible to translate images with text in one language into equivalent images displaying that text translated into another language.
Approach:	They propose an all-encompassing framework for the task–In-Image Machine Translation (IIMT) that incorporates contextual cues from both textual and visual elements during translation.
Outcome:	The proposed framework can be constructed using open-source models and requires no training, making it highly accessible and expandable.

ThinkAnswer Loss: Balancing Semantic Similarity and Exact Matching for LLM Reasoning Enhancement (2025.findings-emnlp)

Copied to clipboard

Shan Yang, Kun Wu, Zeju Li, Linlin Zhang, Xiangyu Pei, Leike An, Yu Liu

Reasoning

Challenge:	Existing methods for knowledge distillation use Chain-of-Thought (CoT) and answer pairs, but they lack appropriate supervision signals.
Approach:	They propose a framework that decouples CoT and answer supervision . the framework applies semantic similarity constraints while maintaining strict literal matching for the answer .
Outcome:	The proposed framework decouples CoT and answer supervision while maintaining strict literal matching for the answer.

Beyond Task-Level Context: Class-Conditional Context Vectors for Implicit In-Context Learning (2026.findings-acl)

Copied to clipboard

Jianxin Zhang, Yilu Hu, Teng Liu, Pei Guo, Juntao Li

Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents (2026.acl-long)

Challenge:	Existing approaches aggregate demonstrations from all classes into a shared, task-level context vector, capturing global task information but without explicitly preserving fine-grained, class-conditional semantic distinctions.
Approach:	They propose a class-conditional context vector extension to implicit in-context learning that explicitly models class-specific contextual information by constructing separate context vectors for each class.
Outcome:	The proposed extension outperforms task-level context vector baselines and achieves higher average accuracy than conventional few-shot learning on most models.

Copied to clipboard

Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Pei Chen, Ziwei Dong, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Qun Liu, Yisi Sang, Hanqing Lu, Manling Li, Jin Lai, Dakuo Wang

CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis (2025.findings-emnlp)

Challenge:	Tool-calling agents are increasingly deployed in real-world customer-facing workflows . but most studies on tool-callers focus on idealized settings with general, fixed, and well-specified tasks.
Approach:	They propose a tool-calling agent-based data pipeline that converts trajectories into user-facing tasks with controlled intent adaptations.
Outcome:	The proposed pipeline can be used to study tool use under three scenarios.

Copied to clipboard

Xinyu Zhang, Pei Zhang, Shuang Luo, Jialong Tang, Yu Wan, Baosong Yang, Fei Huang

Information Retrieval

Challenge:	Cultural competence is defined as the ability to understand and adapt to multicultural contexts.
Approach:	They propose a framework that uses a hierarchical multilingual taxonomy and a Retrieval-Augmented Generation to synthesize culturally relevant question-answer pairs.
Outcome:	The proposed framework contains a hierarchical multilingual taxonomy covering 12 primary and 130 secondary topics and a Retrieval-Augmented Generation (RAG)-based methodology leveraging factual knowledge to synthesize culturally relevant question-answer pairs.

A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese (2026.tacl-1)

Copied to clipboard

Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, Hai Hu

Training Language Model to Critique for Better Refinement (2025.findings-acl)

Challenge:	Using sub-linear length normalized log-probabilities (SLLN-LP), we find unequal lengths of sentences in minimal pairs difficult for LMs even up to 32B parameters.
Approach:	They propose to use ZhoBLiMP as a linguistic minimal pair benchmark for Chinese language models to mitigate biases.
Outcome:	The proposed metric mitigates biases in Chinese language models with over 100 paradigms . Anaphor, Quantifiers, and Ellipsis are difficult for LMs even up to 32B parameters .

Copied to clipboard

Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Large language models (LLMs) have remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks.
Approach:	They propose a framework to train critic models using refinement signals to generate feedback loops where critiques guide the model in refining its responses.
Outcome:	The proposed framework outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes.

Copied to clipboard

Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei

DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning (2024.acl-long)

Challenge:	Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible.
Approach:	They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times .
Outcome:	The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.

Copied to clipboard

Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Weiran Xu, Jingang Wang, Mengdi Zhang, Xunliang Cai

A Graph Representation of Semi-structured Data for Web Question Answering (2020.coling-main)

Challenge:	Numerous code large language models (LLMs) have been proposed to enhance code generation performance.
Approach:	They propose a diverse instruction model DolphCoder with self-evaluating for code generation that learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
Outcome:	The proposed model achieves superior performance on the HumanEval and MBPP benchmarks, demonstrating new insights for future code instruction tuning work.

Copied to clipboard

Xingyao Zhang, Linjun Shou, Jian Pei, Ming Gong, Lijie Wen, Daxin Jiang

Structured Data in NLP

Challenge:	Existing studies treat semi-structured data as flat documents with pieces of text . semi-structural data is more effective to represent rich relational information . question answering is an important feature in most search engines .
Approach:	They propose a graph representation of Web tables and lists based on categorization of components and their relations . they also develop reasoning techniques on the graph model for the question answering task .
Outcome:	The proposed graph improves F1 score by 3.90 points over the state-of-the-art baselines on real datasets.

CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving (2024.findings-naacl)

Copied to clipboard

Pei Chen, Shuai Zhang, Boran Han

Low-Resource NLP

Challenge:	Large Language Models (LLMs) have shown great ability in solving traditional natural language tasks and elementary reasoning tasks with appropriate prompting techniques.
Approach:	They propose a collaborative multi-agent, multi-reasoning-path prompting framework that prompts LLMs to play different roles in a problem-solving team and encourages different role-play agents to collaboratively solve the target task.
Outcome:	The proposed framework is applied to two college-level science problems over competitive baselines.

SynthFix: Adaptive Neuro-Symbolic Code Vulnerability Repair (2026.findings-acl)

Copied to clipboard

Yifan Zhang, Jieyu Li, Kexin Pei, Yu Huang, Kevin Leach

Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation (2020.emnlp-main)

Challenge:	Large Language Models (LLMs) struggle with complex semantic and structural correctness required for automated code repair.
Approach:	They propose a hybrid neural-symbolic framework that unifies code synthesis with compiler-informed symbolic feedback to improve LLM-based vulnerability repair.
Outcome:	The proposed framework improves code repair accuracy and efficiency over strong SFT and RFT training strategies on the FixJS and CodeFlaws benchmarks.

Copied to clipboard

Pei Zhang, Boxing Chen, Niyu Ge, Kai Fan

ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods (2024.emnlp-main)

Challenge:	Recent advances in deep learning have led to significant improvement of document-level neural machine translation (NMT).
Approach:	They propose a long-short term masking self-attention on top of the standard transformer to capture the long-range dependence and reduce the propagation of errors.
Outcome:	The proposed model captures the long-range dependence and reduces errors on two publicly available document-level datasets.

Copied to clipboard

Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, Bhuwan Dhingra

Improving Multilingual Sign Language Translation with Automatically Clustered Language Family Information (2025.coling-main)

Challenge:	ReCaLL (Relative Conditional Log-Likelihood) is a membership inference attack that can detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Approach:	They propose a membership inference attack to detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Outcome:	The proposed model achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach.

Copied to clipboard

Ruiquan Zhang, Cong Hu, Pei Yu, Yidong Chen

Multilinguality

Challenge:	Recent research has focused on bilingual translation models, but multilingual sign language translation presents unique challenges due to the diversity of sign languages across nations.
Approach:	They propose a method that leverages sign language families to improve MSLT performance.
Outcome:	The proposed approach can achieve balance between translation accuracy and computational cost by regulating the number of language families.

Few-shot Low-resource Knowledge Graph Completion with Reinforced Task Generation (2023.findings-acl)

Copied to clipboard

Shichao Pei, Qiannan Zhang, Xiangliang Zhang

IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation (2026.acl-long)

Challenge:	Existing few-shot learning-based models have difficulty alleviating the long-tail issue on low-resource KGs because of the lack of training tasks.
Approach:	They propose a few-shot low-resource knowledge graph completion framework that generates and selects beneficial few- shot tasks that complement current tasks.
Outcome:	The proposed framework is based on several real-world knowledge graphs and validates on multiple domains.

Copied to clipboard

Bosi Wen, Yilin Niu, Cunxiang Wang, Pei Ke, Xiaoying Ling, Ying Zhang, Aohan Zeng, Hongning Wang, Minlie Huang

Challenge:	Existing evaluation models for instruction-following have many shortcomings, such as substantial costs and unreliable assessments.
Approach:	They propose an LLM critic for fine-grained instruction-following evaluation using a checklist generator and a constraint-level preference optimization method.
Outcome:	The proposed model beats strong LLM-as-a-Judge baselines in evaluations under lower computational overhead compared to baselines.

RAt: Injecting Implicit Bias for Text-To-Image Prompt Refinement Models (2024.emnlp-main)

Copied to clipboard

Ziyi Kou, Shichao Pei, Meng Jiang, Xiangliang Zhang

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation (2026.acl-long)

Challenge:	Text-to-image prompt refinement (T2I-Refine) aims to rephrase or extend an input prompt with more descriptive details that can be leveraged to generate images with higher quality.
Approach:	They develop an adversarial prompt attacking framework that implicitly attacks input prompts with intentional adversarials to generate images with higher quality.
Outcome:	The proposed framework can implicitly attack input prompts with implicit concept biases to generate images with higher quality and explicit visual bias towards the target group.

Copied to clipboard

Bosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling, Ying Zhang, Pei Ke, Hongning Wang, Minlie Huang

Challenge:	Existing benchmarks for instruction-following lack data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios.
Approach:	They propose a meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types and a preference graph for each instruction.
Outcome:	Extensive experiments on IF-RewardBench show that the proposed benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks.

Modeling Event-Pair Relations in External Knowledge Graphs for Script Reasoning (2021.findings-acl)

Copied to clipboard

Yucheng Zhou, Xiubo Geng, Tao Shen, Jian Pei, Wenqiang Zhang, Daxin Jiang

Knowledge Representation

Challenge:	Existing methods focus on graph triples with event overlap, but ignore more supportive triples . Script reasoning relies on understanding the relationship between two events .
Approach:	They propose a model to learn the inferential relations between events from the whole eventuality KG . they propose 'script adapter' to extend the model to infer the associated relations between an event chain and a subsequent event candidate.
Outcome:	The proposed model is compared with baselines using external KG or not on a script reasoning task.

Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese (2025.emnlp-main)

Copied to clipboard

Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu

Machine Translation

Challenge:	Translationese is a linguistic property that is often introduced in the translation process that is different from those of original texts.
Approach:	They propose to use synthesized translations and translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments.
Outcome:	The proposed measure can generalize to unseen genres, authors, and language pairs.

DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models (2025.naacl-long)

Copied to clipboard

Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, Xingwei Qu, Xiao Zhang, Pei Wang, Yanan Wu, Jihao Gu, Yangguang Li, Jianke Zhu

Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? (2022.emnlp-main)

Challenge:	Multimodal Large Language Models (MLLMs) pose unique safety challenges due to their integration of visual and textual data.
Approach:	They propose a method to disentangle risks through step-by-step reasoning within multimodal inputs.
Outcome:	The proposed approach improves safety alignment in MLLMs by fine-tuning and iterative Reinforcement Learning from AI feedback.

Copied to clipboard

Pei Zhang, Baosong Yang, Hao-Ran Wei, Dayiheng Liu, Kai Fan, Luo Si, Jun Xie

Machine Translation

Challenge:	Neural machine translation models are often criticized for failures that happen without competency awareness.
Approach:	They propose a method that extends conventional NMT with a self-estimator to translate a source sentence and estimate its competency.
Outcome:	The proposed method performs on translation tasks intact and on quality estimation tasks better than existing methods.

Enhancing Machine Translation with Self-Supervised Preference Data (2025.acl-long)

Copied to clipboard

Haoxiang Sun, Ruize Gao, Pei Zhang, Baosong Yang, Rui Wang

Machine Translation

Challenge:	Current approaches to constructing preference data rely on human annotations.
Approach:	They propose a framework which efficiently constructs translation preference data for iterative training.
Outcome:	The proposed framework improves translation preference data on large language models.

Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA (2026.acl-long)

Copied to clipboard

Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, Zhenbo Luo, Jian Luan, Wei Chen, Xiang Bai

Unveiling the Implicit Toxicity in Large Language Models (2023.emnlp-main)

Challenge:	Existing OCR-free approaches to document visual question answering are brittle and passive.
Approach:	They propose an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation.
Outcome:	The proposed framework outperforms open-source and proprietary models in five benchmarks and improves out-of-domain performance by 47.9% over baseline.

Copied to clipboard

Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, Minlie Huang

Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization (2024.acl-long)

Challenge:	Recent studies focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, but LLMs can generate diverse implicit toxic output that are difficult to detect via simply zero-shot prompting.
Approach:	They propose a reinforcement learning based attacking method to induce the implicit toxic outputs in large language models by fine-tuning toxicity classifiers.
Outcome:	The proposed method generates implicit toxic outputs that are difficult to detect via zero-shot prompting on five widely-adopted toxicity classifiers.

Copied to clipboard

Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, Minlie Huang

Robustness in NLP

Challenge:	Despite the success of jailbreaking attacks, there is a lack of effort in defending against them.
Approach:	They propose to integrate goal prioritization at both training and inference stages to counteract this conflict between the goals of being helpful and ensuring safety.
Outcome:	The proposed approach reduces the Attack Success Rate (ASR) of jailbreaking from 66.4% to 3.6% for ChatGPT and reduces it from 71.0% to 6.6% for Llama2-13B.

Crossroads, Buildings and Neighborhoods: A Dataset for Fine-grained Location Recognition (2022.naacl-main)

Copied to clipboard

Pei Chen, Haotian Xu, Cheng Zhang, Ruihong Huang

Information Extraction & Text Mining

Challenge:	Named Entity Recognition (NER) datasets annotate coarse-grained entities such as a continent, a country, or a city.
Approach:	They propose a dataset HarveyNER with fine-grained locations annotated in tweets that characterizes many complex and long location mentions in informal descriptions.
Outcome:	The proposed dataset outperforms existing systems on hard cases and improves on the heuristic curricula.

CharacterGLM: Customizing Social Characters with Large Language Models (2024.emnlp-industry)

Copied to clipboard

Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Pei Ke, Guanqun Bi, Libiao Peng, JiaMing Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang

Isotropy-Enhanced Conditional Masked Language Models (2023.findings-emnlp)

Challenge:	Character-based dialogue systems (CharacterDial) allow users to customize social characters for social interactions.
Approach:	They will collect a large-scale Chinese corpus of characters with diverse categories and behaviors and develop CharacterGLM models to address these challenges.
Outcome:	Experiments show that CharacterGLM outperforms most popular open- and closed-source LLMs and performs comparable to GPT-4.

Copied to clipboard

Pei Guo, Yisheng Xiao, Juntao Li, Yixin Ji, Min Zhang

Causally Modeling the Linguistic and Social Factors that Predict Email Response (2025.naacl-long)

Challenge:	Existing non-autoregressive models with auto-regressing decoding paradigms have been used for various text generation tasks to accelerate inference but at the cost of generation quality to some extent.
Approach:	They propose to use Look Neighbors strategy to enhance learning of target token representations during training to achieve a good balance between inference speedup and generation quality.
Outcome:	The proposed models outperform current models on 4 WMT datasets and outperformed the current SoTA results.

Copied to clipboard

Yinuo Xu, Hong Chen, Sushrita Rakshit, Aparna Ananthasubramaniam, Omkar Yadav, Mingqian Zheng, Michael Jiang, Lechen Zhang, Bowen Yi, Kenan Alkiek, Abraham Israeli, Bangzhao Shu, Hua Shen, Jiaxin Pei, Haotian Zhang, Miriam Schirmer, David Jurgens

Challenge:	a key intent behind many emails is to get a reply from the recipient.
Approach:	They propose to model the intents, expectations, and responsiveness in email exchanges by using a dataset containing 1800 emails annotated with nuanced types of intents and expectations.
Outcome:	The proposed model is based on 1800 emails annotated with nuanced types of intents and expectations . it shows that social status, argumentation, and strength of social connection influence email response rates .

Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning (2023.findings-acl)

Copied to clipboard

Chujie Zheng, Pei Ke, Zheng Zhang, Minlie Huang

WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback (2026.acl-long)

Challenge:	Current language models have shown impressive capability of generating fluent and grammatical text, but they often produce behaviors misaligned with human expectations.
Approach:	They propose a new language model called Leo for controllable text generation which employs a contrastive loss on sequence likelihood which fundamentally decreases the generation probability of negative samples.
Outcome:	The proposed model outperforms baselines on language detoxification, sentiment steering, and repetition reduction tasks.

Copied to clipboard

Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song, Jennifer Neville

Challenge:	Traditional alignment methods rely on human annotations and are subjective and misalignment with real-world user preferences.
Approach:	They propose a framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically.
Outcome:	The proposed framework identifies and classifies user feedback to LLM responses between conversation turns and creates examples of preferred and dispreferred responses according to user preferences.

Assessing “Implicit” Retrieval Robustness of Large Language Models (2024.emnlp-main)

Copied to clipboard

Xiaoyu Shen, Rexhina Blloshmi, Dawei Zhu, Jiahuan Pei, Wei Zhang

Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling (2025.emnlp-main)

Challenge:	Retrieval-augmented generation (RAG) is a framework to enhance large language models with external knowledge, but its effectiveness is constrained by the retrieval robustness of the model.
Approach:	They propose to use gold and distracting context to fine-tune models to handle relevant or irrelevant retrieved context in an end-to-end manner.
Outcome:	The proposed model performs better when gold and distracting context are used, while still extracting correct answers when retrieval is accurate.

Copied to clipboard

Minghui Li, Hao Zhang, Yechao Zhang, Wei Wan, Shengshan Hu, Pei Xiaobing, Jing Wang