Papers by Peng Qiu
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models (2025.acl-long)
Copied to clipboard
Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Qiu Jiantao, Chi Zhang, Ying Qian, Conghui He
| Challenge: | composition of pre-training datasets for large language models remains undisclosed . current methods for evaluating data quality are limited by single-dimensional evaluation or redundancy-focused strategies. |
| Approach: | They propose a multi-dimensional data selection method that integrates dimensions with existing quality metrics through learned optimal weightings. |
| Outcome: | The proposed method doubles convergence speed for 1.3B model models and improves downstream task performance by 3.23%. |
SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on identifying entities' relations from the semantics of dialogues-they utilize either the attention mechanism or a refined token graph to locate informative words. |
| Approach: | They propose a sequential structure prediction task to incrementally parse SocAoG for dynamic inference upon any incoming utterance. |
| Outcome: | Empirical results show that the proposed model infers social relations more accurately than the state-of-the-art methods. |
Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained evaluation metrics can perpetuate and amplify biases, causing inability to differentiate between biased and unbiased generations. |
| Approach: | They conduct a systematic study of gender biases in image captioning tasks . they show that pretrained models perpetuate and amplify biase . |
| Outcome: | The proposed model-based evaluation metrics have shown good correlations with human judgments in language generation tasks. |
Firewall Routing: Blocking Leads to Better Hybrid Inference for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have significantly enhanced performance across various NLP tasks . high computational costs and latency associated with deploying such models pose bottlenecks . |
| Approach: | They propose a dynamic hybrid inference framework that efficiently selects between a strong and a weak LLM based on the complexity of the query. |
| Outcome: | The proposed method outperforms existing routing strategies by up to 5.29% in APGR . large models often introduce higher latency, making them less suitable for real-time or resource-constrained applications. |
Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration (2025.acl-long)
Copied to clipboard
Tianyi Bai, Ling Yang, Zhen Hao Wong, Fupeng Sun, Xinlin Zhuang, Jiahui Peng, Chi Zhang, Lijun Wu, Qiu Jiantao, Wentao Zhang, Binhang Yuan, Conghui He
| Challenge: | Efficient data selection is crucial to accelerate the pretraining of language models . limited research has addressed the inherent conflicts between data selection methods . |
| Approach: | They propose a multi-actor collaborative data selection mechanism that prioritizes data based on its specific criterion and updates prioritization rules using the current state of the model. |
| Outcome: | The proposed model accelerates convergence in LM pretraining and achieves an average relative performance gain of 10.5% across multiple language model benchmarks. |
VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on object hallucinations, focusing on object outputs . current evaluation methods struggle to address subtle semantic distinctions between outputs and reference data . |
| Approach: | They propose a multi-dimensional benchmark covering objects, attributes, and relations . they propose metric that generalizes CHAIR metric and incorporates faithfulness and coverage . |
| Outcome: | The proposed evaluation framework is more comprehensive and better correlated with humans than existing evaluation methods. |
Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current LLMs are achieving better performance on various benchmarks, but their performance in practical applications does not always match their benchmark results. |
| Approach: | They propose to detect and rewrite leaked benchmarks without altering their difficulties by using Inference-Time Decontamination (ITD) to mitigate performance inflation caused by memorizing leaked samples. |
| Outcome: | The proposed method reduces inflated accuracy by 22.9% on GSM8K and 19.0% on MMLU. |
Evaluating Cultural and Social Awareness of LLM Web Agents (2025.findings-naacl)
Copied to clipboard
Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu
| Challenge: | Existing benchmarks often overlook cultural and social awareness . current evaluations focus on task completion, often ignoring the diverse cultural and socio-cultural backgrounds. |
| Approach: | They propose a benchmark to assess LLM agents’ sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums. |
| Outcome: | The proposed framework evaluates LLM agents’ ability to detect and appropriately respond to norm-violating user queries and observations across two web-based tasks. |
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Kyle Lam, Bang Zheng, Lin Li, Jianing Qiu
| Challenge: | Existing benchmarks for multimodal large language models do not capture real-world clinical complexity. |
| Approach: | They evaluate multilingual, multimodal multimodal models of clinical cases with up to 7 distinct visual clinical evidence types per case. |
| Outcome: | The proposed model outperforms human models on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. |
GRNFormer: A Biologically-Guided Framework for Integrating Gene Regulatory Networks into RNA Foundation Models (2025.findings-acl)
Copied to clipboard
Mufan Qiu, Xinyu Hu, Fengwei Zhan, Sukwon Yun, Jie Peng, Ruichen Zhang, Bhavya Kailkhura, Jiekun Yang, Tianlong Chen
| Challenge: | Foundation models for single-cell RNA sequencing ignore biological prior knowledge encoded in gene regulatory relationships and fail to leverage multi-omics signals. |
| Approach: | They propose a framework that integrates multi-scale gene regulatory networks into RNA foundation model training. |
| Outcome: | The proposed framework improves on state-of-the-art models on three downstream tasks . it integrates multi-scale gene regulatory networks (GRNs) from multi-omics data into training . |
Revisiting the Knowledge Injection Frameworks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Injecting unaligned knowledge tuple into large language models achieves comparable (and sometimes better) results than aligned knowledge. |
| Approach: | They propose a technique to inject random knowledge into large language models to improve performance. |
| Outcome: | The proposed technique overcomes the sanity problem and pushes the performance limit. |
MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application (2026.acl-long)
Copied to clipboard
Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Vincent Jim Zhang, Yuqing Guo, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Zhiyang Deng, Heming Fu, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Guojun Xiong, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, Qianqian Xie
| Challenge: | Existing evaluations of LLMs in finance are text-only, monolingual, and largely saturated by current models. |
| Approach: | They propose a multilingual and multimodal benchmark for evaluating LLMs in real financial contexts. |
| Outcome: | The first expert-annotated multilingual and multimodal benchmark is released . it evaluates 21 leading LLMs and shows they perform better in multilingual settings . |
NumNet: Machine Reading Comprehension with Numerical Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing numerical MRC models are weak in numerical reasoning, such as addition, subtraction, sorting and counting. |
| Approach: | They propose a numerical MRC model that integrates numerical reasoning into existing MRC models and achieves an EM-score of 64.56% on the DROP dataset. |
| Outcome: | The proposed model outperforms all existing machine reading comprehension models by considering the numerical relations among numbers on the DROP dataset. |
HistLens: Mapping Idea Change across Concepts and Corpora (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to diachronic semantics and discourse analysis focus on a single concept or corpus, argues a new paper. |
| Approach: | They propose a framework for multi-concept, multi-corpus conceptual-history analysis that decomposes concept representations into interpretable features and tracks activation dynamics over time and across sources. |
| Outcome: | The proposed framework decomposes concept representations into interpretable features and tracks their activation dynamics over time and across sources. |
Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction (2021.findings-acl)
Copied to clipboard
Tianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
| Challenge: | Distantly supervised relation extraction (RE) has attracted much attention in the past few years . previous methods to evaluate models manually or directly on autolabeled data have produced inaccurate evaluations . |
| Approach: | They propose to use distant supervision to generate large-scale autolabeled data . they build manually-annotated test sets for two DS-RE datasets and evaluate models . |
| Outcome: | The proposed method produces 53% wrong labels at the entity pair level in the popular NYT10 dataset. |
Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models suffer from multi-modality problem . despite their autoregressivity, most NMT models suffer with slow decoding speed . |
| Approach: | They propose a semi-autoregressive model which generates a translation as a sequence of segments while each segment is predicted token-by-token. |
| Outcome: | The proposed model can achieve 4 times speedup while maintaining comparable performance. |
AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating factual consistency of abstractive summarization lack coherence or error-type coverage. |
| Approach: | They propose a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) they use a selection module NegFilter to ensure the quality of the generated negative examples . |
| Outcome: | The proposed framework outperforms existing systems on the AggreFact-SOTA benchmark and provides high error-type coverage. |
Calibrating the Confidence of Large Language Models by Eliciting Fidelity (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models with RLHF and RLAIF have good alignment but exhibit overconfidence post-alignment. |
| Approach: | They propose a plug-and-play method to estimate the confidence of large language models. |
| Outcome: | The proposed method has shown good calibration performance on 6 RLHF-LMs on four MCQA datasets. |
BASES: Large-scale Web Search User Simulation with Large Language Model based Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on web search rely on real-user experiments, which can be costly to scale up. |
| Approach: | They propose a user simulation framework with LLM-based agents that can generate unique user profiles at scale. |
| Outcome: | The proposed framework can generate unique user profiles at scale, leading to diverse search behaviors. |
Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design (2025.naacl-long)
Copied to clipboard
| Challenge: | Using Mixture-of-Experts, researchers have found that efficient MoE is difficult to achieve due to two key reasons: imbalanced expert activation and massive communication overhead. |
| Approach: | They propose a collaboration-constrained routing strategy that encourages more specialized expert groups and leverages expert specialization. |
| Outcome: | The proposed approach achieves an average performance improvement of 0.51% and 0.33% on LLaMA-MoE and Qwen-MaE respectively. |
TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing (2021.acl-demo)
Copied to clipboard
Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, Xuanjing Huang
| Challenge: | Existing approaches to textual robustness evaluation focus on slightly modifying the input data, which maintains the original meaning and results in a different prediction. |
| Approach: | They propose a multilingual robustness evaluation toolkit for NLP that integrates universal text transformations, task-specific transformations and adversarial attack. |
| Outcome: | The toolkit includes universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analyses. |
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models pre-trained on massive corpora have shown impressive few-shot learning ability on many NLP tasks. |
| Approach: | They propose to recast structured output in the form of code instead of natural language and use generative LLMs of code to perform IE tasks. |
| Outcome: | The proposed method outperforms fine-tuning moderate-size pre-trained models and prompting NL-LLMs under few-shot settings. |