Papers by Bosi Wen
AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024.acl-long)
Copied to clipboard
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. |
| Approach: | They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. |
| Outcome: | The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references. |
Training Language Model to Critique for Better Refinement (2025.findings-acl)
Copied to clipboard
Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang
| Challenge: | Large language models (LLMs) have remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. |
| Approach: | They propose a framework to train critic models using refinement signals to generate feedback loops where critiques guide the model in refining its responses. |
| Outcome: | The proposed framework outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. |
IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation (2026.acl-long)
Copied to clipboard
Bosi Wen, Yilin Niu, Cunxiang Wang, Pei Ke, Xiaoying Ling, Ying Zhang, Aohan Zeng, Hongning Wang, Minlie Huang
| Challenge: | Existing evaluation models for instruction-following have many shortcomings, such as substantial costs and unreliable assessments. |
| Approach: | They propose an LLM critic for fine-grained instruction-following evaluation using a checklist generator and a constraint-level preference optimization method. |
| Outcome: | The proposed model beats strong LLM-as-a-Judge baselines in evaluations under lower computational overhead compared to baselines. |
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for instruction-following lack data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios. |
| Approach: | They propose a meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types and a preference graph for each instruction. |
| Outcome: | Extensive experiments on IF-RewardBench show that the proposed benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks. |
HPSS: Heuristic Prompting Strategy Search for LLM Evaluators (2025.findings-acl)
Copied to clipboard
Bosi Wen, Pei Ke, Yufei Sun, Cunxiang Wang, Xiaotao Gu, Jinfeng Zhou, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Existing efforts to optimize text evaluation prompts neglect the combinatorial impact of multiple factors, leading to insufficient optimization of the evaluation pipeline. |
| Approach: | They propose to integrate 8 key factors for evaluation prompts and integrate them into an algorithm that searches for well-behaved prompting strategies for LLM evaluators. |
| Outcome: | The proposed method outperforms existing methods and human-designed evaluation prompts on four evaluation tasks. |
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)
Copied to clipboard
Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang
| Challenge: | Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references. |
| Approach: | They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings. |
| Outcome: | The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading. |
CharacterGLM: Customizing Social Characters with Large Language Models (2024.emnlp-industry)
Copied to clipboard
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Pei Ke, Guanqun Bi, Libiao Peng, JiaMing Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang
| Challenge: | Character-based dialogue systems (CharacterDial) allow users to customize social characters for social interactions. |
| Approach: | They will collect a large-scale Chinese corpus of characters with diverse categories and behaviors and develop CharacterGLM models to address these challenges. |
| Outcome: | Experiments show that CharacterGLM outperforms most popular open- and closed-source LLMs and performs comparable to GPT-4. |