Papers by Pengfei He

VALU: A Benchmark for Video Anomaly Temporal Localization and Understanding at Multiple Semantic Levels (2026.acl-long)

Challenge:	Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets.
Approach:	They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved.
Outcome:	The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance.

Copied to clipboard

Yixiao He, Menghao Zhang, Haifeng Sun, Jing Wang, Kangheng Lin, Jinghan Wang, Chenye Xu, Pengfei Ren, Qi Qi, Jingyu Wang

Multimodality

Challenge:	Recent advances in Video Large Language Models (Video-LLMs) enhance the ability of VAU models to describe and interpret anomalies.
Approach:	They propose a benchmark that explicitly defines anomalies across five semantic levels and provides detailed temporal boundaries and detailed textual descriptions for each.
Outcome:	The proposed benchmark defines anomalies across five semantic levels and provides detailed descriptions for each.

LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey (2026.findings-acl)

Copied to clipboard

Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Jizhou Guo, Yankai Chen, Chunyu Miao, Hoang H Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Hanrong Zhang, Fangxin Wang, Pengfei Zhang, Langzhou He, Yangning Li, Dongyuan Li, Renhe Jiang, Philip S. Yu

Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty? (2025.findings-emnlp)

Challenge:	Recent advances in large language models (LLMs) have sparked growing interest in building fully autonomous agents.
Approach:	They propose to integrate human-provided information, feedback, or control into the agent system to enhance system performance, reliability, and safety.
Outcome:	The proposed systems improve system performance, reliability, and safety by integrating human-provided information, feedback, or control into the agent system.

Copied to clipboard

Yang Nan, Pengfei He, Ravi Tandon, Han Xu

Challenge:	Large language models can produce unreliable or misleading outputs, posing challenges for real-world applications.
Approach:	They employ an auxiliary LLM to analyze the patterns of disagreement among LLMs . they validate their framework on AmbigQA, OpenBookQA, and MMLU-Pro .
Outcome:	The proposed model can be used to diagnose uncertainty sources in a model with an auxiliary model.

LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)

Copied to clipboard

Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang

Multimodality

Challenge:	Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics.
Approach:	They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks.
Outcome:	The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring.

Advancing Reasoning with Off-the-Shelf LLMs: A Semantic Structure Perspective (2025.findings-emnlp)

Copied to clipboard

Pengfei He, Zitao Li, Yue Xing, Yaliang Li, Jiliang Tang, Bolin Ding

Low-Resource NLP

Challenge:	Existing reasoning models suffer from hallucinations and unfaithfulness, whereas general LLMs perform suboptimal on complex tasks.
Approach:	They propose a structure analysis method that helps LLMs better understand the question structure and guide the problem-solving process.
Outcome:	The proposed method improves zero-shot performance on knowledge-intensive and mathematical tasks while demonstrating strong robustness against corrupted reasoning paths.

Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models (2025.findings-acl)

Copied to clipboard

Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, Suhang Wang, Yue Xing, Jiliang Tang, Qi He

Exploring Memorization in Fine-tuned Language Models (2024.acl-long)

Challenge:	Chain-of-Thought (CoT) reasoning has improved the performance of large language models (LLMs) however, the detailed reasoning process in CoT often incurs long generation times and high computational costs due to the inclusion of unnecessary steps.
Approach:	They propose a method to identify critical reasoning steps using perplexity as a measure of their importance.
Outcome:	The proposed method achieves a better balance between reasoning accuracy and efficiency of CoT.

Copied to clipboard

Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, Dawei Yin

The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) (2024.findings-acl)

Challenge:	Existing studies have shown that pre-trained langauge models tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately.
Approach:	They conduct the first comprehensive analysis to explore language models’ memorization during fine-tuning across tasks.
Outcome:	The proposed analysis shows that memorization presents a strong disparity among different fine-tuning tasks.

Copied to clipboard

Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, Jiliang Tang

Data Poisoning for In-context Learning (2025.findings-naacl)

Challenge:	Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model generation with proprietary and private data, where data privacy is . a privacy issue that is currently under-explored, is posed by RAG.
Approach:	They propose to use retrieval-augmented generation (RAG) to facilitate language model generation with proprietary and private data where data privacy is a pivotal concern.
Outcome:	The proposed attack methods demonstrate that RAG can mitigate the old risks, i.e., leakage of the LLMs’ training data.

Copied to clipboard

Pengfei He, Han Xu, Yue Xing, Hui Liu, Makoto Yamada, Jiliang Tang

Reformatted Alignment (2024.findings-emnlp)

Challenge:	In-context learning (ICL) has emerged as a capability of large language models (LLMs) but there is limited understanding of its vulnerability against data poisoning attacks.
Approach:	They propose an attack method that exploits ICL’s unique learning mechanisms by identifying discrete text perturbations that influence LLM hidden states.
Outcome:	The proposed attack method exploits ICL’s learning mechanisms by identifying discrete text perturbations that influence LLM hidden states.

Copied to clipboard

Run-Ze Fan, Xuefeng Li, Haoyang Zou, Junlong Li, Shwai He, Ethan Chern, Jiewen Hu, Pengfei Liu

Reasoning

Challenge:	Current methods to improve data quality are labor-intensive or prone to factual errors caused by LLM hallucinations.
Approach:	They propose a method which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence.
Outcome:	The proposed approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques.

CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs (2026.findings-acl)

Copied to clipboard

Pengfei He, Shaowei Wang, Tse-Hsun Chen

How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior (2026.acl-long)

Challenge:	Existing prompt compression techniques for natural language lack fine-grained control over compression ratios.
Approach:	They propose a code-aware prompt compression framework for RAG that enables precise length control while preserving critical information.
Outcome:	The proposed framework outperforms baselines on three code-related tasks while maintaining the most informative tokens.

Copied to clipboard

Zidi Xiong, Yuping Lin, Wenya Xie, Pengfei He, Zirui Liu, Jiliang Tang, Himabindu Lakkaraju, Zhen Xiang

Information Retrieval

Challenge:	In practice, memory designs vary widely across agents due to their diverse objectives and functionalities.
Approach:	They conduct an empirical study on how memory management choices impact the LLM agents’ behavior, especially their long-term performance.
Outcome:	The proposed methods show that LLM agents display an experience-following property, which results in highly similar agent outputs.

Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation Networks (2021.emnlp-main)

Copied to clipboard

Qingbin Liu, Pengfei Cao, Cao Liu, Jiansong Chen, Xunliang Cai, Fan Yang, Shizhu He, Kang Liu, Jun Zhao

Dialogue Systems & Conversational Agents

Challenge:	Existing offline DST models require a fixed dataset to train . Existing domain-lifelong learning methods are impractical in real-world applications .
Approach:	They propose a domain-lifelong learning method to continuously train a DST model on new data to learn incessantly emerging new domains while avoiding catastrophically forgetting old learned domains.
Outcome:	The proposed method outperforms state-of-the-art lifelong learning methods by 4.25% and 8.27% on the MultiWOZ and the SGD benchmarks.

On the Generalization of Training-based ChatGPT Detection Methods (2024.findings-emnlp)

Copied to clipboard

Han Xu, Jie Ren, Pengfei He, Shenglai Zeng, Yingqian Cui, Amy Liu, Hui Liu, Jiliang Tang

Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)

Challenge:	Existing studies show that training-based methods are ineffective to detect LLM generated texts from unseen tasks or topics which are not collected during training.
Approach:	They propose to train classification models to distinguish LLMs from human texts by a distribution shift caused by prompts, text lengths, topics, and language tasks.
Outcome:	The proposed methods can detect LLMs from black-box models, but they suffer from distribution shifts due to a wide range of factors, including prompts, text lengths, topics, and language tasks.

Copied to clipboard

Shenglai Zeng, Pengfei He, Kai Guo, Tianqi Zheng, Hanqing Lu, Yue Xing, Hui Liu

PEAR: Planner-Executor Agent Robustness Benchmark (2026.findings-eacl)

Challenge:	Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence.
Approach:	They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function .
Outcome:	The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes.

Copied to clipboard

Shen Dong, Mingxuan Zhang, Pengfei He, Li Ma, Bhavani Thuraisingham, Hui Liu, Yue Xing

Robustness in NLP

Challenge:	Existing studies examine isolated attack surfaces or specific scenarios, leaving a lack of holistic understanding of MAS vulnerabilities.
Approach:	They propose a benchmark to evaluate the utility and vulnerability of planner–executor MAS.
Outcome:	The proposed benchmark evaluates planner–executor MAS on a widely adopted design.

Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis (2024.emnlp-main)

Copied to clipboard

Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, Jiliang Tang

Zero-Shot Cross-Lingual Document-Level Event Causality Identification with Heterogeneous Graph Contrastive Transfer Learning (2024.lrec-main)

Challenge:	Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents.
Approach:	They propose to leverage hidden representations into existing jailbreak targets to move the attacks along the acceptance direction.
Outcome:	The proposed methods are validated using the objective of existing jailbreak attacks.

Copied to clipboard

Zhitao He, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, Zhiqiang Zhang, Mengshu Sun, Jun Zhao

Retrieval Heads are Dynamic (2026.acl-long)

Challenge:	Existing studies focus on sentence-level ECI with high-resource languages, leaving document-level DECI with low-resourced languages under-explored.
Approach:	They propose a Heterogeneous Graph Interaction Model with Multi-granularity Contrastive Transfer Learning for zero-shot cross-lingual ECI.
Outcome:	The proposed model outperforms the state-of-the-art model on monolingual and multilingual scenarios by 9.4% and 8.2% of average F1 score.

Copied to clipboard

Yuping Lin, Zitao Li, Yue Xing, Pengfei He, Yingqian Cui, Yaliang Li, Bolin Ding, Jingren Zhou, Jiliang Tang

Information Retrieval

Challenge:	Recent studies have identified "retrieval heads" in Large Language Models responsible for extracting information from input contexts.
Approach:	They propose to examine retrieval heads from a dynamic perspective . they establish that retrieval head activation is highly dynamic and functionally irreplaceable .
Outcome:	The proposed model's hidden state encodes a predictive signal for future retrieval head patterns, indicating an internal planning mechanism.

Unveiling Privacy Risks in LLM Agent Memory (2025.acl-long)

Copied to clipboard

Bo Wang, Weiyi He, Shenglai Zeng, Zhen Xiang, Yue Xing, Jiliang Tang, Pengfei He

Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data (2025.emnlp-main)

Challenge:	Large Language Model (LLM) agents store private user-agent interactions in memory for demonstrations, introducing new privacy risks for LLM agents.
Approach:	They propose an attack that extracts private information from memory under a black-box setting and propose a method that can be used to attack the agent.
Outcome:	The proposed attack is effective under a black-box setting and it is demonstrated on two representative agents.

Copied to clipboard

Shenglai Zeng, Jiankun Zhang, Pengfei He, Jie Ren, Tianqi Zheng, Hanqing Lu, Han Xu, Hui Liu, Yue Xing, Jiliang Tang

Target-Guided Structured Attention Network for Target-Dependent Sentiment Analysis (2020.tacl-1)

Challenge:	Existing literature suggests that RAG systems may face privacy issues when the retrieval process involves private data.
Approach:	They propose a two-stage synthetic data generation paradigm that uses attributes to preserve contextual information from the original data.
Outcome:	The proposed approach preserves key contextual information from the original data while reducing privacy risks.

Copied to clipboard

Ji Zhang, Chengyao Chen, Pengfei Liu, Chao He, Cane Wing-Ki Leung

Sentiment Analysis

Challenge:	TDSA aims to classify the sentiment of a text towards a given target.
Approach:	They propose a novel Target-Guided Structured Attention Network (TG-SAN) which captures target-related contexts for TDSA in a fine-to-coarse manner.
Outcome:	The proposed network outperforms the state-of-the-art in terms of accuracy and Marco-F1 on three benchmarks with three major findings.

Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems (2023.findings-acl)

Copied to clipboard

Chiyu Song, Hongliang He, Haofei Yu, Pengfei Fang, Leyang Cui, Zhenzhong Lan

Dialogue Systems & Conversational Agents

Challenge:	Existing approaches to decode a given context-candidate pair are expensive and time-consuming.
Approach:	They propose a new paradigm that keeps full attention over each pair while only encoding the context once.
Outcome:	The proposed paradigm achieves new state-of-the-art results on four benchmark datasets with high computational efficiency.

Alignment Precedes Fusion: Open-Vocabulary Named Entity Recognition as Context-Type Semantic Matching (2023.findings-emnlp)

Copied to clipboard

Zhuoran Jin, Pengfei Cao, Zhitao He, Yubo Chen, Kang Liu, Jun Zhao

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022.emnlp-main)

Challenge:	Continual learning and zero-shot learning approaches have not been adopted to scale to novel-emerging types.
Approach:	They propose a method to recognize entities in novel types by their textual names or descriptions.
Outcome:	The proposed method outperforms the state-of-the-art methods on three challenging OVNER benchmarks by 9.7%, 9.5%, and 1.8% F1-score of novel types.

Copied to clipboard

Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, Yejin Choi

Question Answering

Challenge:	Recent research shows that relevant knowledge can provide useful context for commonsense tasks.
Approach:	They propose a method that learns to generate contextually relevant knowledge in response to given questions.
Outcome:	The proposed method shows consistent gains over 9 commonsense benchmarks.

A Reward-Guided Dual-Phase Framework for Adaptive Inference-Time Reasoning (2026.findings-acl)

Copied to clipboard

Yingqian Cui, Zhenwei Dai, Pengfei He, Bing He, Hui Liu, Zhan Shi, Xianfeng Tang, Jingying Zeng, Suhang Wang, Yue Xing, Jiliang Tang, Benoit Dumoulin

Reasoning

Challenge:	Large Language Models (LLMs) have made strong progress in reasoning.
Approach:	They propose a dual-phase test-time scaling framework that separates planning and execution and performs search over each phase independently.
Outcome:	Experiments on math reasoning and code generation benchmarks show that the proposed approach improves accuracy while reducing redundant computation.

AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation (2024.findings-emnlp)

Copied to clipboard

Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Kang Liu, Jun Zhao

SLICEFORMER: Static Program Slicing Using Language Models With Dataflow-Aware Pretraining and Constrained Decoding (2026.acl-long)

Challenge:	Recent advances in deep learning have significantly impacted the legal domain.
Approach:	They propose a multi-agent framework for judicial decision-making that simulates the court trial process . they propose 420 Chinese judgment documents to support their framework and build a large-scale legal knowledge base .
Outcome:	The proposed framework outperforms existing methods in various aspects, especially in generating legal articles.

Copied to clipboard

Pengfei He, Shaowei Wang, Tse-Hsun Chen, Muhammad Asaduzzaman

Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? (2025.naacl-long)

Challenge:	Static program slicing is a software engineering technique for isolating code relevant to specific variables.
Approach:	They propose a new approach that reformulates static program slicing as a sequence-to-sequence task using small language models such as CodeT5+.
Outcome:	The proposed approach improves on Java and Python program slicing benchmarks with up to 22% gain in ExactMatch.

Copied to clipboard

Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, Jing Wang

Multimodality

Challenge:	LVLMs often mistakenly determine objects as present in images where they do not exist . authors propose a new benchmark to evaluate object hallucinations by removing objects from images and asking the model whether it can still see the removed objects.
Approach:	They propose a benchmark to evaluate object hallucinations by removing objects from images . they propose oDPO, a direct preference optimization objective based on visual objects .
Outcome:	The proposed benchmark reduces the likelihood of object hallucinations by removing objects from images and asking the model whether it can still see the removed objects.

LASS: A Novel and Economical Data Augmentation Framework Based on Language Models for Debiasing Opinion Summarization (2025.coling-main)

Copied to clipboard

Yanyue Zhang, Pengfei Li, Yilong Lai, Yulan He, Deyu Zhou