How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities (2026.acl-long)
Copied to clipboard
Ziwen Xu, Kewei Xu, Haoming Xu, Haiwen Hong, Longtao Huang, Hui Xue, Ningyu Zhang, Yongliang Shen, Guozhou Zheng, Huajun Chen, Shumin Deng
| Challenge: | Large language models (LLMs) are increasingly deployed in socially sensitive domains, yet their unpredictable behaviors pose significant risks. |
| Approach: | They propose a hierarchical benchmark for evaluating LLM controllability across three domains: language features, sentiment, and personality. |
| Outcome: | The proposed framework offers a principled and interpretable framework for safe and controllable LLM behavior serving as a foundation for future research. |
Similar Papers
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can adapt outputs to align with community-specific norms, perspectives and communication styles. |
| Approach: | They propose a benchmark to assess community-specific steering using contrasting reddit communities. |
| Outcome: | STEER-BENCH assesses how well large language models understand community-specific instructions, their resilience to adversarial steering attempts, and their ability to accurately represent cultural and ideological perspectives. |
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
| Challenge: | Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied. |
| Approach: | They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance . |
| Outcome: | The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs. |
Is Your Language Model Ready for Monetization Decisions? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on shopping-centric scenarios and user-facing data, overlooking intermediate decision stages and robustness considerations. |
| Approach: | They propose a multi-task benchmark to evaluate large language models in real-world monetization contexts. |
| Outcome: | The proposed benchmark covers intent understanding, commercial matching, and user behavior modeling. |
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann
| Challenge: | Existing evaluation frameworks focus on English and a handful of high-resource languages, thereby overlooking the realistic performance of large language models in multilingual and lower-resourced scenarios. |
| Approach: | They propose a unified and lightweight framework that integrates 27 benchmarks under a standard ISO 639-3 language identifier system to enable seamless incorporation of new benchmarks. |
| Outcome: | The proposed framework integrates 27 benchmarks under a standard ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. |
SafeLawBench: Towards Safe Alignment of Large Language Models (2025.findings-acl)
Copied to clipboard
Chuxue Cao, Han Zhu, Jiaming Ji, Qichao Sun, Zhenghao Zhu, Wu Yinyu, Josef Dai, Yaodong Yang, Sirui Han, Yike Guo
| Challenge: | Recent studies indicate that large language models (LLMs) may exhibit risks, including threats to the protection of private data and the generation of hallucinations. |
| Approach: | They propose to evaluate LLMs from a legal perspective using the SafeLawBench benchmark. |
| Outcome: | The proposed framework categorizes safety risks into three levels based on legal standards and includes 24,860 multi-choice questions and 1,106 open-domain question-answering tasks. |
How does Misinformation Affect Large Language Model Behaviors and Preferences? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have explored the role of Large Language Models in combating misinformation, but there is still a lack of detailed analysis on the specific aspects and extent to which LLMs are influenced by misinformation. |
| Approach: | They propose to use a benchmark to evaluate LLMs' behavior and knowledge preference toward misinformation to identify their models. |
| Outcome: | The proposed approach is based on 10,346,712 pieces of misinformation and examines knowledge conflicts and stylistic variations. |
SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) lack large-scale, systematically constructed benchmarks for evaluating their alignment with real-world social attitudes. |
| Approach: | They propose a benchmark to assess LLMs' alignment with real-world social attitudes . they find LLM models achieve only 30–40% accuracy when simulating individuals . |
| Outcome: | The proposed benchmark shows that LLMs achieve only 30% accuracy when simulating individuals in complex survey scenarios. |
GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely deployed as autonomous agents capable of following user instructions and making decisions in real-world applications. |
| Approach: | They propose a benchmark to evaluate LLMs' ability to follow domain-oriented guidelines . they evaluate Lms on three critical aspects: adherence to diverse rules, robustness to rule updates . |
| Outcome: | The proposed benchmark evaluates LLMs on three critical aspects: adherence to diverse rules, robustness to rule updates, and alignment with human preferences. |