Papers by Kai Zheng
Bridge Video and Text with Cascade Syntactic Structure (C18-1)
Copied to clipboard
| Challenge: | Using LSTM-CSS, we construct basic syntactic structure by completing syntastic structure. |
| Approach: | They propose a video captioning approach that progressively completes syntactic structure by a conditional random field to construct basic syntaktic structure. |
| Outcome: | The proposed method produces natural sentences with 42.3% and 28.5% accuracy compared to state-of-the-art methods. |
FastMCTS: A Simple Sampling Strategy for Data Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating multi-step reasoning data rely on rejection sampling, which generates trajectories independently and suffers from inefficiency and imbalanced sampling across problems of varying difficulty levels. |
| Approach: | They propose a data synthesis strategy inspired by Monte Carlo Tree Search . it offers step-level evaluation signals and promotes balanced sampling . |
| Outcome: | Experiments show that FastMCTS generates 30% more correct reasoning paths than rejection sampling. |
ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Causal chain reasoning models suffer from two main transitive problems: threshold effect and scene drift. |
| Approach: | They propose a framework that uses exogenous variables to represent causal pairs and estimates the threshold and scene contradictions using structural causal recurrent neural networks. |
| Outcome: | The proposed framework outperforms baselines on Chinese and English CCR datasets. |
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence. |
| Approach: | They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes. |
| Outcome: | The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models. |
NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing backdoor attacks against prompt-based learning involve injecting back doors into embedding layers or word embedders. |
| Approach: | They propose a backdoor attack against prompt-based learning that injects backdoors into embedding layers or word embeddable vectors. |
| Outcome: | The proposed backdoor attack outperforms two state-of-the-art models on six NLP tasks and three prompting strategies. |
RubricBench: Aligning Model-Generated Rubrics with Human Standards (2026.acl-long)
Copied to clipboard
Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, Chen Ma
| Challenge: | Existing benchmarks lack discriminative complexity and ground-truth rubric annotations required for rigorous evaluation. |
| Approach: | They propose a curated benchmark with 1,147 pairwise comparisons to assess the reliability of rubric-based evaluation. |
| Outcome: | The proposed benchmarks show that they support diverse domains, exhibit discriminative ability, provide high-quality annotations, and include human-authored rubrics. |
Wanda++: Pruning Large Language Models via Regional Gradients (2025.findings-acl)
Copied to clipboard
Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, Nathan Susanj, Zheng Zhang, Jack FitzGerald, Abhishek Kumar
| Challenge: | Existing pruning methods suffer from accuracy degradation without full-model sparsity-aware fine-tuning. |
| Approach: | They propose a pruning framework that uses decoder-block-level regional gradients to improve pruning accuracy. |
| Outcome: | The proposed pruning framework outperforms the state-of-the-art pruning frameworks by utilizing decoder-block-level regional gradients. |
Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs (2026.acl-long)
Copied to clipboard
Shenglai Zeng, Tianqi Zheng, Chuan Tian, Dante Everaert, Yau-Shian Wang, Yupin Huang, Michael J. Morais, Rohit Patki, Jinjin Tian, Xinnan Dai, Kai Guo, Monica Xiao Cheng, Hui Liu
| Challenge: | Existing approaches to personalize large language models (LLMs) rely on heuristic methods to compress user profiles but they ignore how LLMs process and prioritize different profile components. |
| Approach: | They propose an attention-guided context compression framework that leverages attention feedback from a marking model to mark important personalization sentences and guides a compression model to generate task-relevant compressed user contexts. |
| Outcome: | The proposed framework outperforms baselines across tasks, token limits, and settings while reducing token usage by 50 times. |
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance across NLP tasks. |
| Approach: | They propose to use long-context SFT to improve short-contemporary performance . they also decouple and analyze two key components, Multi-Head Attention and Feed-Forward Network . |
| Outcome: | The proposed model improves short-context performance, contrary to pretraining. |
CSS: A Large-scale Cross-schema Chinese Text-to-SQL Medical Dataset (2023.findings-acl)
Copied to clipboard
| Challenge: | a cross-domain text-to-SQL task aims to parse user questions into SQL on complete unseen databases . a single-domain task evaluates the performance on identical databases based on the same domain . |
| Approach: | They propose a cross-domain text-to-SQL task that parses user questions into SQL on unseen databases. |
| Outcome: | The proposed system can parse user questions into SQL on complete unseen databases. |
CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space (2025.emnlp-main)
Copied to clipboard
Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, Jincai Huang
| Challenge: | Embodied Question Answering (EQA) tasks are primarily focused on indoor environments, leaving the complexities of urban settings unexplored. |
| Approach: | They propose a task where an embodied agent answers open-vocabulary questions in dynamic city spaces. |
| Outcome: | The proposed agent achieves 60.7% of human-level answering accuracy compared to baselines . the proposed agent outperforms existing agents in open-ended city spaces . |
MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing (2026.acl-industry)
Copied to clipboard
Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Zirui Tang, Boyu Niu, Yuanhong Zheng, Dongsheng Ma, Ziyang Miao, Hejun Dong, Siyi Qian, Junyuan Zhang, Fangdong Wang, Jingzhou Chen, Xiaomeng Zhao, Liqun Wei, Wei Li, Shasha Wang, RuiLiang Xu, Yuanyuan Cao, Lu Chen, Qianqian Wu, Huaiyu Gu, Lindong Lu, Dechen Lin, null Shenguanlin, Xuanhe Zhou, Linfeng Zhang, Yuhang Zang, Xiaoyi Dong, Jiaqi Wang, Bo Zhang, Lei Bai, Pei Chu, Weijia Li, Jiang Wu, Lijun Wu, Zhenxiang Li, Guangyu Wang, Zhongying Tu, Chao Xu, Kai Chen, Bowen Zhou, Dahua Lin, Wentao Zhang, Conghui He
| Challenge: | Document images are characterized by higher resolutions, denser content, and more complex structural layouts. |
| Approach: | They propose a 1.2B-parameter document parsing vision-language model that decouples layout analysis from local content recognition. |
| Outcome: | The proposed model surpasses general-purpose and domain-specific models on multiple benchmarks while maintaining significantly lower computational overhead. |
Knowledge Stimulated Contrastive Prompting for Low-Resource Stance Detection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Stance Detection Tasks require background knowledge especially when there is no explicit target mentioned in text. |
| Approach: | They propose a masked language prompt joint contrastive learning approach to stimulate the knowledge inherit from pre-trained models. |
| Outcome: | The proposed model is effective in stance detection on three benchmarks. |
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)
Copied to clipboard
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang
| Challenge: | Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference. |
| Approach: | They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes. |
| Outcome: | The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training. |
WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on weak supervision for NLU focus on a specific task or simulate weak supervision signals from ground-truth labels. |
| Approach: | They propose a benchmark to advocate and facilitate research on weak supervision for NLU . they use document-level and token-level prediction tasks as examples . |
| Outcome: | The proposed benchmark advocates and facilitates research on weak supervision for NLU tasks. |
SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent large reasoning models have shown exceptional performance on various tasks, but they consume excessive tokens even for simple queries, leading to resource waste and prolonged user latency. |
| Approach: | They propose a self-adaptive reasoning strategy that automatically allocates budgets according to problem complexity and introduces GRPO for reinforcement learning to reduce output length. |
| Outcome: | The proposed model achieves an average response length compression of 61% on math reasoning tasks while maintaining accuracy. |
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)
Copied to clipboard
Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu
| Challenge: | Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. |
| Approach: | They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning . |
| Outcome: | The proposed model can encode words into fine-grained representations without modification of production pipelines. |
Reinforcement Learning on Pre-Training Data (2026.acl-long)
Copied to clipboard
Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Kun Li, Haoyuan Wu, null Wujiajia, Zihao Zheng, Chenchen Zhang, Kun Shi, Xue Gong, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang xu, Zheng Fang, Shuai LI, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bochao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Bo Zhou, Di Wang
| Challenge: | Recent progress in large language models is driven by scaling of training compute through pre-training with nexttoken prediction (NTP) or post-training (RL) Pre-training using NTP enables models to acquire extensive knowledge and skills from general data, but it suffers from data inefficiency and catastrophic forgetting in continual learning settings. |
| Approach: | They propose to scale training compute through pre-training with next-token prediction (NTP) or post-training by scaling reinforcement learning (RL) to improve learning from general data. |
| Outcome: | Experiments on multiple benchmarks and models show that the proposed approach improves continual pre-training and provides a strong foundation for post-training on Qwen3-8B-Base. |
X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding (2025.findings-emnlp)
Copied to clipboard
Wenqi Zhou, Kai Cao, Hao Zheng, Yunze Liu, Xinyi Zheng, Miao Liu, Per Ola Kristensson, Walterio W. Mayol-Cuevas, Fan Zhang, Weizhe Lin, Junxiao Shen
| Challenge: | Existing benchmark datasets focus on short to moderately long videos, leaving a substantial gap in evaluating extensive, ultra-long egocentric video recordings. |
| Approach: | X-LeBench is a benchmark dataset designed to evaluate long egocentric video recordings . it uses a life-logging pipeline to produce realistic, coherent daily plans . |
| Outcome: | X-LeBench is a new benchmark dataset designed to evaluate long-form egocentric video understanding . the approach produces realistic, coherent daily plans aligned with real-world video data . |
Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts. |
| Approach: | They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models. |
| Outcome: | The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks. |
Multimodal Dialogue Response Generation (2022.acl-long)
Copied to clipboard
Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yaming Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, Daxin Jiang
| Challenge: | Existing studies focus on multimodal dialogue models but neglect generation methods. |
| Approach: | They propose a multimodal dialogue response generation task which requires multimodal dialogs containing both texts and images which are difficult to obtain. |
| Outcome: | Experiments show that the proposed model can generate informative text and high-resolution image responses. |
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to expand query use pseudo relevance feedback (PRF) but they are under-equipped to evaluate the relevance of information pieces used for expansion. |
| Approach: | They propose a query expansion model that leverages the BERT model to select relevant document chunks for expansion. |
| Outcome: | The proposed model significantly outperforms existing models on the TREC Robust04 and GOV2 test collections. |
MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (2024.findings-acl)
Copied to clipboard
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, Kai Chen
| Challenge: | Recent advances in large language models have showcased significant improvements in mathematics, but traditional benchmarks like GSM8k offer a unidimensional perspective. |
| Approach: | MathBench is a benchmark that rigorously assesses the mathematical capabilities of large language models. |
| Outcome: | MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. |
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2025.acl-long)
Copied to clipboard
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig
| Challenge: | Recent advances in multimodal large language models have led to progress in tackling complex reasoning tasks that combine textual and visual information. |
| Approach: | They introduce a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. |
| Outcome: | The proposed model performs lower on MMMU-Pro than on the previous benchmark, ranging from 16.8% to 26.9%. |
Training-Free Test-Time Contrastive Learning for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training-free alternatives to training-based models are static or depend on external guidance. |
| Approach: | They propose a training-free adaptation framework that enables a frozen LLM to improve online by distilling supervision from its own inference experiences. |
| Outcome: | The proposed framework outperforms existing test-time adaptation methods under online evaluation. |
Everyone is unique: Towards Behaviorally Heterogeneous Negotiation Dialogue Systems for Debt Collection (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models that assume users to be static, rational agents with fixed preferences fail to capture rich behavioral heterogeneity in real-world debt collection scenarios. |
| Approach: | They propose a public persona-enriched debt collection benchmark that highlights behavioral heterogeneity in negotiation. |
| Outcome: | The proposed benchmark outperforms existing models in realistic scenarios using 16 state-of-the-art LLMs. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark (2026.acl-long)
Copied to clipboard
Kai Zou, Ziqi Huang, Yuhao Dong, Shulin Tian, Dian Zheng, Hongbo Liu, Jingwen He, Bin Liu, Yu Qiao, Ziwei Liu
| Challenge: | Existing evaluations treat visual understanding and generation in isolation or overlook tasks that inherently couple them. |
| Approach: | They propose a benchmark that examines the bidirectional synergy between generation and understanding across eight reasoning-centric domains. |
| Outcome: | The proposed model systematically unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence. |
| Approach: | They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function . |
| Outcome: | The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes. |
The Strength of the Weakest Supervision: Topic Classification Using Class Labels (N19-3)
Copied to clipboard
| Challenge: | a topic classifier can understand only class labels when training for tasks that require a large amount of labeled documents. |
| Approach: | They propose an algorithm that can initialize a topic classifier using only class labels . they propose a method that combines word embedding and naive Bayes classification . |
| Outcome: | The proposed approach saves significant initial labeling effort by providing a "warm start" the proposed approach can be fine-tuned with more labeled documents to reach a certain performance level. |
Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Generative Reward Models have demonstrated that scaling the length of Chain-of-Thought reasoning enhances reliability of evaluation. |
| Approach: | They propose a framework that reconfigures raw rationales into structured Breadth-CoT and Depth-Co T through a modular synthesis pipeline. |
| Outcome: | The proposed framework surpasses open-source RMs by an average of 8.2%. |
Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Ryan Solgi, Kai Zhen, Rupak Vignesh Swaminathan, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang
| Challenge: | Low-rank tensor compression techniques are used for over-parameterized neural networks, but their applications to compress pre-trained LLMs for downstream tasks remain challenging due to the high-rank nature of pre-training data. |
| Approach: | They propose sparse augmented tensor networks to enhance low-rank tenorized LLMs . they also propose a framework that enables full model compression . |
| Outcome: | The proposed framework improves accuracy and efficiency in tensorized language models. |
MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang
| Challenge: | Large language models (LLMs) have demonstrated exceptional capabilities across diverse tasks, but their fine-tuning requires significant memory, posing challenges for resource-constrained environments. |
| Approach: | They propose a ZO-based framework that eliminates the need for backpropagation and provides a memory-efficient alternative to backprograming. |
| Outcome: | The proposed framework surpasses first-order methods in performance and accuracy. |
Social-aware Sparse Attention Network for Session-based Social Recommendation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for predicting the next item for an anonymous session do not capture user preferences and noisy irrelevant interactions. |
| Approach: | They propose to use social networks and historical sessions to provide personalized recommendations for the current session. |
| Outcome: | The proposed model outperforms existing models on two benchmark datasets. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step (2024.acl-long)
Copied to clipboard
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, Feng Zhao
| Challenge: | Existing studies evaluate the tool utilization ability of large language models based on the final output or only consider the single-step tool calling. |
| Approach: | They propose a new approach to evaluate the tool utilization capability of large language models (LLMs) they decompose the tool usage into multiple sub-processes, including instruction following, planning, reasoning, retrieval, understanding, and review. |
| Outcome: | The proposed model exhibits consistency with the outcome-oriented evaluation and provides a more fine-grained analysis of the capabilities of LLMs. |