Papers by Simin Niu
RoleCDE: Benchmarking and Mitigating Role–Alignment Trade-offs in Role-Playing Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for role-playing agents only evaluate surface-level fidelity and provide limited insight into decision making under role–alignment value conflicts. |
| Approach: | They propose a benchmark to evaluate RPAs under role–alignment value conflicts . they use 8k diverse role profiles and 240k dilemma instances to evaluate role-aware decision making . |
| Outcome: | The proposed benchmark covers 8k diverse role profiles and scenarios and nearly 240k dilemma instances across three difficulty levels and eight role categories. |
UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation (2024.acl-long)
Copied to clipboard
Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Yezhaohui Wang, Dawei He, Cheng Peng, Zhonghao Wang, Haiying Deng
| Challenge: | Large language models (LLMs) produce hallucinated text, compromising their practical utility in professional contexts. |
| Approach: | They have developed an unconstrained hallucination generation evaluation benchmark that contains hallucines generated by large language models with minimal restrictions. |
| Outcome: | The proposed benchmarks are based on a Chinese-language dataset that is lacking in the field. |
MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text chunking are limited by text chunks and lack of domain-specific knowledge. |
| Approach: | They propose a dual-metric evaluation method to quantify text chunking quality . they aim to generate a structured list of chunking regular expressions . |
| Outcome: | The proposed method enables direct quantification of chunking quality . it substantiates the need to integrate LLMs into chunking process . |
PibE-MPP: A Play-it-by-Ear Masking Performance Plug-in for LLMs (2026.findings-acl)
Copied to clipboard
Mengwei Wang, Simin Niu, Xun Liang, Yuefeng Ma, Sensen Zhang, Jiawei Yang, Shichao Song, Hanyu Wang, Huayi Lai
| Challenge: | Random masking is a widely adopted classic baseline in large language models (LLMs). |
| Approach: | They propose a play-it-by-ear masking performance plug-in which enables LLMs to adaptively select masking target combinations for each task. |
| Outcome: | The proposed performance plug-in retains the advantages and mitigates the drawbacks of random masking in large language models. |
QAEncoder: Towards Aligned Representation Learning in Question Answering Systems (2025.acl-long)
Copied to clipboard
Zhengren Wang, Qinhan Yu, Shida Wei, Zhiyu Li, Feiyu Xiong, Xiaoxing Wang, Simin Niu, Hao Liang, Wentao Zhang
| Challenge: | Modern QA systems entail retrieval-augmented generation (RAG) for accurate and trustworthy responses, but the inherent gap between user queries and relevant documents hinders precise matching. |
| Approach: | They propose a retrieval-augmented generation (RAG)-based approach to bridge this gap by attaching document fingerprints to the embedding to estimate the expectation of potential queries. |
| Outcome: | Experiments across diverse datasets, languages, and embedding models confirm the proposed solution is simple-yet-effective with zero additional index storage, retrieval latency, training costs, or catastrophic forgetting and hallucination issues. |
GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for large language models rely on static benchmarks and standardized evaluation protocols. |
| Approach: | They propose an adaptive evaluation framework that integrates dynamic domain knowledge modeling with progressive reasoning assessment to improve evaluation fidelity. |
| Outcome: | Empirical results show that the framework distinguishes LLMs in terms of domain knowledge coverage and reasoning chain completeness. |
SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model (2025.acl-long)
Copied to clipboard
Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Zhaoxin Fan, Bo Tang, Jihao Zhao, Jiawei Yang, Shichao Song, Mengwei Wang
| Challenge: | Existing approaches to integrating external knowledge into large language models (LLMs) however, the incorporation of external knowledge increases the vulnerability of LLMs . |
| Approach: | They propose a benchmark to evaluate the RAG security using a dataset . they classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service . |
| Outcome: | The proposed benchmark evaluates the security of RAG against 14 representative RAG components. |
UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning (2026.findings-acl)
Copied to clipboard
Xiaolong Wei, Zerun Zhu, Simin Niu, Xingyu Zhang, Peiying Yu, Changxuan Xiao, Yuchen Li, Jicheng Yang, Zhejun Zhao, Chong Meng, Long Xia, Daiting Shi
| Challenge: | Existing alignment paradigms for creative writing use static reward signals and supervised data. |
| Approach: | They propose a constraint-aware reward model that synthesizes query-specific criteria to provide fine-grained preference judgments. |
| Outcome: | The proposed framework aligns models with human preferences across content quality and structural paradigms without supervised fine-tuning and ground-truth references. |