Papers by Pengfei Yang
GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs (2026.acl-long)
Copied to clipboard
Weidong Tang, Jierui Li, Yueling Hou, Zihan Mei, Can Zhang, Xinyan Wan, Zhiyuan Liang, Pengfei Zhou, Yang You, Wangbo Zhao
| Challenge: | Existing models for general intelligence fail to model how mental states interact and crystallize into group-level outcomes. |
| Approach: | They propose a multimodal benchmark for group-level Theory of Mind (ToM) to probe nonlinear collective behavior. |
| Outcome: | The proposed model performs significantly below human levels, exposing blind spots in modeling social structures and nonlinear collective behavior. |
Enhancing Self-Attention with Knowledge-Assisted Attention Maps (2022.naacl-main)
Copied to clipboard
Jiangang Bai, Yujing Wang, Hong Sun, Ruonan Wu, Tianmeng Yang, Pengfei Tang, Defu Cao, Mingliang Zhang1, Yunhai Tong, Yaming Yang, Jing Bai, Ruofei Zhang, Hao Sun, Wei Shen
| Challenge: | Existing works of knowledge infusion depend on multi-task learning frameworks, which are inefficient and require large-scale retraining when new knowledge is considered. |
| Approach: | They propose a method which integrates knowledge-generated attention maps into the self-attention mechanism and integrates it into the model. |
| Outcome: | The proposed model outperforms existing methods on academic datasets and industry-scale ad relevance applications. |
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022.emnlp-main)
Copied to clipboard
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics. |
| Approach: | They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer. |
| Outcome: | The proposed evaluator improves on three typical NLG tasks and improves with external knowledge. |
Uncertain Local-to-Global Networks for Document-Level Event Factuality Identification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on identifying event factuality at sentence level, which leads to conflicts between different mentions of the same event. |
| Approach: | They propose a document-level event factuality identification model that uses local uncertainty and global structure to model event factuality. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets. |
Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction (2023.acl-long)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) aims to extract semantic relations between entities in a document. |
| Approach: | They propose a Document-level distant relation extraction framework with unreliable pseudo labels to denoise DS data. |
| Outcome: | The proposed framework outperforms strong baselines on two public datasets. |
ExplainaBoard: An Explainable Leaderboard for NLP (2021.acl-demo)
Copied to clipboard
Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Graham Neubig
| Challenge: | Using leaderboards, researchers can track the performance of various systems on various NLP tasks. |
| Approach: | They propose a new conceptualization and implementation of NLP evaluation using a leaderboard. |
| Outcome: | The ExplainaBoard is an evaluation tool for natural language processing (NLP) it covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. |
Unveiling Internal Reasoning Modes in LLMs: A Deep Dive into Latent Reasoning vs. Factual Shortcuts with Attribute Rate Ratio (2025.emnlp-main)
Copied to clipboard
Yiran Yang, Haifeng Sun, Jingyu Wang, Qi Qi, Zirui Zhuang, Huazheng Wang, Pengfei Ren, Jing Wang, Jianxin Liao
| Challenge: | Existing research in multi-hop questions has identified two reasoning modes, but has not investigated how these modes differ during inference. |
| Approach: | They propose a classification metric that compares latent reasoning and factual shortcuts in multi-hop questions. |
| Outcome: | The proposed metric achieves 90% accuracy on the proposed datasets and demonstrates effectiveness in RAG conflict scenarios. |
Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can produce unreliable or misleading outputs, posing challenges for real-world applications. |
| Approach: | They employ an auxiliary LLM to analyze the patterns of disagreement among LLMs . they validate their framework on AmbigQA, OpenBookQA, and MMLU-Pro . |
| Outcome: | The proposed model can be used to diagnose uncertainty sources in a model with an auxiliary model. |
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)
Copied to clipboard
Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang
| Challenge: | Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics. |
| Approach: | They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks. |
| Outcome: | The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring. |
Weak-to-Strong Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to supervise large language models (LLMs) exceed human capabilities, but the effectiveness of this approach is still unexplored. |
| Approach: | They propose a weak-to-strong reasoning framework that enables strong models to refine training data . they use supervised fine-tuning and preference optimization to optimize weak models . |
| Outcome: | The proposed framework improves the reasoning capabilities of a language model using three weak models. |
Human-in-the-loop Robotic Grasping Using BERT Scene Representation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches for robotic grasping in cluttered scenes are expensive and lack structure information. |
| Approach: | They propose a human-in-the-loop framework for robotic grasping in cluttered scenes . they substitute scene-graph representation with a text representation of the scene using BERT . |
| Outcome: | The proposed framework outperforms object-agnostic and scene-graph based methods on robots and physical robots. |
Improving Relation Extraction with Knowledge-attention (D19-1)
Copied to clipboard
| Challenge: | Existing attention mechanisms are data-driven, but most are data driven. |
| Approach: | They propose a knowledge-attention encoder which integrates prior knowledge from external lexical resources into deep neural networks for relation extraction task. |
| Outcome: | The proposed system outperforms existing CNN, RNN, and self-attention based models on a large-scale relation extraction dataset. |
Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. |
| Approach: | They examine the distinguishability of benchmark datasets when comparing different systems . they find that existing benchmark dataset contribute little to discriminating top-scoring systems - whereas those less used datasets exhibit impressive discriminative power. |
| Outcome: | The proposed datasets are released on DataLab. |
Knowledge-Centric Hallucination Detection (2024.emnlp-main)
Copied to clipboard
Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, Zheng Zhang
| Challenge: | Large Language Models (LLMs) have shown impressive capabilities but a tendency to hallucinate. |
| Approach: | They propose a framework that introduces claim-triplets to represent claims in LLM responses and evaluates them against a reference. |
| Outcome: | The proposed framework outperforms prior methods by 18.2 to 27.2 points on a benchmark spanning various NLP tasks and annotated 11k claim-triplets from 2.1k responses by seven LLMs. |
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction (2025.emnlp-main)
Copied to clipboard
Yuchi Wang, Yishuo Cai, Shuhuai Ren, Sihan Yang, Linli Yao, Yuanxin Liu, Yuanxing Zhang, Pengfei Wan, Xu Sun
| Challenge: | Existing recaptioning methods suffer from inaccuracies due to missing fine-grained details. |
| Approach: | They propose a framework that refines captions through visual reconstruction using a text-to-image model and a visual reconstruction framework. |
| Outcome: | The proposed framework outperforms baselines on CapsBench and CompreCap by 10%. |
Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation Networks (2021.emnlp-main)
Copied to clipboard
Qingbin Liu, Pengfei Cao, Cao Liu, Jiansong Chen, Xunliang Cai, Fan Yang, Shizhu He, Kang Liu, Jun Zhao
| Challenge: | Existing offline DST models require a fixed dataset to train . Existing domain-lifelong learning methods are impractical in real-world applications . |
| Approach: | They propose a domain-lifelong learning method to continuously train a DST model on new data to learn incessantly emerging new domains while avoiding catastrophically forgetting old learned domains. |
| Outcome: | The proposed method outperforms state-of-the-art lifelong learning methods by 4.25% and 8.27% on the MultiWOZ and the SGD benchmarks. |
OpenResearcher: Unleashing AI for Accelerated Scientific Research (2024.emnlp-demo)
Copied to clipboard
Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, Yang Xu, Qingkai Min, Zizhao Zhang, Yiwen Wang, Wenjie Li, Pengfei Liu
| Challenge: | Global scientific publications are growing annually by about 4%-5% (Pinedo et al., 2024). |
| Approach: | They introduce an AI-assisted platform that answers diverse questions from researchers using Retrieval-Augmented Generation (RAG) they develop various tools to understand queries, search from the scientific literature, filter retrieved information, provide accurate and comprehensive answers, and self-refine answers. |
| Outcome: | OpenResearcher is built on Retrieval-Augmented Generation (RAG) to integrate Large Language Models (LLMs) with up-to-date, domain-specific knowledge. |
AscendKernelGen: LLM-Driven Kernel Generation for NPUs (2026.findings-acl)
Copied to clipboard
Xinzi Cao, Jianyang Zhai, Pengfei Li, Zhiheng Hu, Cen Yan, null Mubingxu, Guanghuan Fang, Bin She, Jiayu Li, Yihan Su, Dongyang Tao, Feidiao Yang, Chang-Dong Wang, Yutong Lu, Weicheng Xue, Bin Zhou, Yonghong Tian
| Challenge: | Neural Processing Units (NPUs) are critical for AI infrastructure, but their development remains a bottleneck due to vendor-specific Domain-Specific Languages (DSLs). |
| Approach: | They propose a framework for NPU kernel development that bridges the gap in hardware-specific coding . compiler success on complex Level-2 kernels improves from 0% to 95.5%, they say . |
| Outcome: | The proposed framework bridges the gap in hardware-specific coding, showing a near-zero success rate on complex kernels. |
Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge (2024.emnlp-main)
Copied to clipboard
| Challenge: | despite advances in multimodal large language models, the challenge of interpreting long-form videos remains a challenge . despite advancements in video-language benchmarks, the inefficiency in temporal grounding and limited pre-trained context window size remains . |
| Approach: | They propose a framework that bootstraps MLLMs with advanced temporal grounding capabilities and broadens their contextual scope. |
| Outcome: | The proposed framework significantly enhances the temporal capabilities of existing MLLMs. |
M3TQA: Massively Multilingual Multitask Table Question Answering (2026.findings-acl)
Copied to clipboard
Daixin Shu, Jian Yang, Zhenhe Wu, Xianjie Wu, Xianfu Cheng, Guan Xiangyuan, Yanghai Wang, Pengfei Wu, Tingyang Yang, Hualei Zhu, Wei Zhang, Ge Zhang, Jiaheng Liu, Zhoujun Li
| Challenge: | Existing multilingual table benchmarks suffer from geolinguistic imbalance - overrepresenting certain languages and lacking sufficient scale for rigorous cross-lingual analysis. |
| Approach: | They propose a framework for massively multilingual table question answering that includes tables expanded to 97 languages from Chinese and English sources. |
| Outcome: | Experiments on state-of-the-art LLMs show that synthetically generated training data significantly boosts performance, especially for low-resource languages. |
Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks assess basic Theory of Mind abilities but neglect temporal evolution of mental states in real-world social contexts. |
| Approach: | They propose a benchmark specifically designed to evaluate Large Language Models' ability to understand and track the temporal progression of mental states across interconnected scenarios. |
| Outcome: | The proposed benchmarks underperform humans by 44.7% and show that they can model the dynamic nature of human mental states better than existing models. |
DataLab: A Platform for Data Analysis and Intervention (2022.acl-demo)
Copied to clipboard
Yang Xiao, Jinlan Fu, Weizhe Yuan, Vijay Viswanathan, Zhoumianze Liu, Yixin Liu, Graham Neubig, Pengfei Liu
| Challenge: | Existing tools and research focus on how to interpret and manipulate data, despite its crucial role in machine learning, . existing tools and researchers focus on systems on top of existing data, rather than how to use it. |
| Approach: | They propose a unified data-oriented platform that allows users to interactively analyze the characteristics of data and provides a standard interface for many data processing operations. |
| Outcome: | The proposed platform allows users to analyze the characteristics of data and provides a standardized interface so that many data processing operations can be provided within a single interface. |
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts (2026.acl-long)
Copied to clipboard
Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, Pengfei Liu
| Challenge: | Existing benchmarks focus on single agentic capability, failing to capture long-horizon real-world scenarios. |
| Approach: | They propose a benchmark that evaluates 6 agentic capabilities across 32 real-world scenarios. |
| Outcome: | Experiments show that closed-source models outperform open-source model (48.4% vs 32.1%) integrating models with advanced scaffolds to form autonomous agents is a paradigm shift. |
M2Edit: Locate and Edit Multi-Granularity Knowledge in Multimodal Large Language Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge editing methods for MLLMs lack multi-granularity knowledge . existing knowledge editing approaches lack multimodality knowledge and generalize to multimodal data. |
| Approach: | They propose a multimodal knowledge editing method which integrates key knowledge layers within MLLMs and collaboratively edits them. |
| Outcome: | The proposed method improves visual generality performance on knowledge data of different granularities. |
SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs (2026.findings-acl)
Copied to clipboard
Jie Sun, Yu Liu, Lu Han, Qiwen Deng, Xiang Shu, Yang Xiao, Lintao Ma, Xingyu Lu, Jun Zhou, Pengfei Liu, Jiancan Wu, Xiang Wang
| Challenge: | Existing large-scale large-context models suffer from performance degradation when processing long numerical sequences. |
| Approach: | They propose a framework to mitigate attention dispersion by strategically inserting separator tokens into the model to recalibrat attention to local segments while preserving global context. |
| Outcome: | The proposed framework improves accuracy and reduces inference token consumption by 16.4% on 9 widely-adopted LLMs. |
DavIR: Data Selection via Implicit Reward for Large Language Models (2025.acl-long)
Copied to clipboard
Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, Hongxia Yang
| Challenge: | 6% of Alpaca dataset selected with DavIR can steer both LLaMA and Gemma models to produce superior performance compared to the same models trained on the full 52K dataset. |
| Approach: | They propose a model-based data selection method for post-training Large Language Models . they generalize Reducible Holdout Loss to core-set selection problem of causal language modeling . |
| Outcome: | The proposed method can steer both LLaMA and Gemma models to superior performance compared to the same models trained on the full 52K dataset. |
Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Prompt (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies require massive labeled data to train models for multimodal data analysis. |
| Approach: | They propose a novel multimodal prompt model that captures specific aspect terms in a few-shot scenario. |
| Outcome: | The proposed model outperforms baselines on two MABSA-related tasks on a few-shot dataset. |
RLShield: Dynamic Jailbreak Detection for LLMs via Reinforced Adaptive Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to detect jailbreak prompts rely on static model components or fixed decision thresholds. |
| Approach: | They propose a dynamic jailbreak detection framework that employs reinforcement learning for adaptive threshold selection. |
| Outcome: | Experimental results show that the framework outperforms baselines in detection performance while maintaining high computational efficiency. |
Enhancing Agentic Textual Graph Retrieval with Synthetic Stepwise Supervision (2026.acl-long)
Copied to clipboard
Ge Chang, Jinbo Su, Jiacheng Liu, Pengfei Yang, Yuhao Shang, Huiwen Zheng, Hongli Ma, Yan Liang, Yuanchun Li, Yunxin Liu
| Challenge: | Existing methods for integrating textual graphs with LLMs are limited by symbolic inference and high annotation costs. |
| Approach: | They propose a textual graph reasoning framework that integrates textual diagrams with large language models. |
| Outcome: | The proposed approach achieves 15.6% accuracy and 17.2% in F1 score on three common datasets. |