FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs’ Responsiveness to Human Feedback (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research focuses on benchmarking LLMs in single-turn dialogues, neglecting the nuanced nature of human feedback within real-world usage scenarios. |
| Approach: | They propose a fine-grained, multi-task benchmark designed to evaluate LLMs’ responsiveness to human feedback under real-world usage scenarios in Chinese. |
| Outcome: | The proposed benchmarks show that human feedback can significantly impact LLMs’ responsiveness in real-world usage scenarios. |
Similar Papers
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (2024.acl-long)
Copied to clipboard
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang
| Challenge: | Large Language Models (LLMs) have greatly enhanced dialogue systems, but evaluation of their capabilities remains a challenge. |
| Approach: | They propose a model to evaluate the fine-grained abilities of Large Language Models in multi-turn dialogues. |
| Outcome: | The proposed model evaluates 21 popular chatbots based on MT-Bench-101 . it includes 3 overarching abilities and 13 distinct tasks within multi-turn dialogue scenarios. |
A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on specific predefined model abilities, such as world knowledge, reasoning, etc., making it difficult for users to determine which LLM best suits their particular needs. |
| Approach: | They propose to evaluate large language models from a user-centric perspective and use real-world use cases to identify their effectiveness under distinct intents. |
| Outcome: | The proposed benchmarks achieve a correlation between human preference and the user-reported scenarios and human intents. |
ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models (2023.emnlp-demo)
Copied to clipboard
Baoli Zhang, Haining Xie, Pengfan Du, Junhao Chen, Pengfei Cao, Yubo Chen, Shengping Liu, Kang Liu, Jun Zhao
| Challenge: | Various types of LLMs have recently been rapidly developing, such as Llama2 and ChatGLM2 . |
| Approach: | They propose a benchmark that comprehensively evaluates LLMs across 7 ability dimensions covering 51 tasks. |
| Outcome: | The proposed benchmarks are comprehensive and systematic, with a high level of accuracy and authority. |
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents (2026.findings-acl)
Copied to clipboard
Jiliang Hu, Wenfu Wang, Zuchao Li, Chenxing Li, Yiyang Zhao, Hanzhao Li, Liqiang Zhang, Meng Yu, Dong Yu
| Challenge: | despite advances in multimodal conversational systems, current benchmarks lack comprehensive evaluation across key dimensions. |
| Approach: | They propose a Chinese benchmark built exclusively on real human speech to fill this gap . they assess LALMs across three complementary axes: instruction following, knowledge understanding, robustness . |
| Outcome: | VCB Bench assesses LALMs across three complementary axes: instruction following, knowledge understanding, and robustness . VCBM Bench provides reproducible and fine-grained framework for Chinese voice chat bots . results show significant performance disparities and offer tangible insights for future improvements . |
KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models are striving to grasp commonsense reasoning, but they are lacking in Korean commons- ense benchmarks. |
| Approach: | They present a fine-grained benchmark dataset focused on Korean commonsense reasoning that includes multiple-choice questions across seven error categories. |
| Outcome: | The proposed datasets show that LLMs struggle with Korean commonsense reasoning . human accuracy benchmarked at approximately 85%, while GPT-4’s performance lags at about 74%, and other LLM models demonstrate an average accuracy of around 42%. |
P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs (2025.emnlp-main)
Copied to clipboard
Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou
| Challenge: | Recent advances in large language models showcase varied multilingual capabilities across tasks . previous assessments focused on fundamental natural language processing (NLP) or isolated capability-specific tasks. |
| Approach: | They propose a multilingual multitask benchmark to assess multilingual capabilities . they use a large-scale benchmark covering fundamental and capability-specialized datasets . |
| Outcome: | The proposed benchmark compares models and tasks across languages and tasks and examines knowledge transfer from English to other languages. |
ZhuJiu-Knowledge: A Fairer Platform for Evaluating Multiple Knowledge Types in Large Language Models (2024.naacl-demo)
Copied to clipboard
| Challenge: | evaluating the knowledge of large language models (LLMs) is crucial, and rapid advancement in large language modeling has heightened the importance of model evaluations. |
| Approach: | They propose a fairer benchmark for evaluating multiple knowledge types of LLMs by focusing on commonsense knowledge, world knowledge, and language knowledge. |
| Outcome: | The proposed framework evaluates 14 current mainstream LLMs and provides a detailed discussion and analysis of their results. |
RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing (2025.findings-emnlp)
Copied to clipboard
Hao Xiang, Tianyi Tang, Yang Su, Bowen Yu, An Yang, Fei Huang, Yichang Zhang, Yaojie Lu, Hongyu Lin, Xianpei Han, Jingren Zhou, Junyang Lin, Le Sun
| Challenge: | Existing benchmarks focus on character-centric approach and fail to reflect real-world applications. |
| Approach: | RMTBench is a user-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. |
| Outcome: | RMTBench features 80 diverse characters and over 8,000 dialogue rounds. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can adapt outputs to align with community-specific norms, perspectives and communication styles. |
| Approach: | They propose a benchmark to assess community-specific steering using contrasting reddit communities. |
| Outcome: | STEER-BENCH assesses how well large language models understand community-specific instructions, their resilience to adversarial steering attempts, and their ability to accurately represent cultural and ideological perspectives. |