Value Compass Benchmarks: A Comprehensive, Generative and Self-Evolving Platform for LLMs’ Value Evaluation (2025.acl-demo)
Copied to clipboard
Jing Yao, Xiaoyuan Yi, Shitong Duan, Jindong Wang, Yuzhuo Bai, Muhua Huang, Yang Ou, Scarlett Li, Peng Zhang, Tun Lu, Zhicheng Dou, Maosong Sun, James Evans, Xing Xie
| Challenge: | Current evaluation methods for large language models face two key challenges: 1. evaluation validity and 2. Result interpretation reduce the pluralistic and incommensurable values to one-dimensional scores. |
| Approach: | They propose a platform for comprehensive value diagnosis of large language models (LLMs) that provides a generative evaluation paradigm that automatically creates real-world test items co-evolving with ever-advancing LLMs. |
| Outcome: | The proposed platform provides a framework for comprehensive value diagnosis of large language models (LLMs) with fine-grained scores and case studies across 27 value dimensions for 33 leading LLMs, customized comparisons, and visualized analysis of LLM’s alignment with cultural values. |
Similar Papers
Value Portrait: Assessing Language Models’ Values through Psychometrically and Ecologically Valid Items (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks rely on human annotations that are vulnerable to value-related biases. |
| Approach: | They propose a value portrait benchmark that uses items that capture real-life user-LLM interactions and a rated item based on its similarity to their own thoughts to determine reliability. |
| Outcome: | The proposed framework improves the relevance of assessment results to real-world LLM usage by allowing human subjects to rate items with similarity to their own thoughts and derived correlations between these ratings and the subjects’ actual value scores. |
ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are transforming diverse fields and gaining increasing influence as human proxies. |
| Approach: | They propose a psychometric evaluation pipeline grounded in realistic human-AI interactions to probe value orientations and novel tasks for evaluating value understanding in an open-ended value space. |
| Outcome: | The proposed evaluation pipeline is grounded in realistic human-AI interactions and performs tasks that approximate expert conclusions in value-related extraction and generation tasks. |
Towards Better Value Principles for Large Language Model Alignment: A Systematic Evaluation and Enhancement (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show remarkable performance across tasks . alignment with human values is critical for their responsible development. |
| Approach: | They propose a framework that evaluates value principles along three desirable properties . they propose supervised fine-tuning, reinforcement learning-based approaches . |
| Outcome: | The proposed framework improves value principles along the three desirable properties of LLMs. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on item-level behavioral metrics without capturing how models prioritize competing values as a whole. |
| Approach: | They propose a symmetric human-LLM evaluation framework to measure value-structure alignment . they evaluate 12 LLMs across four model families via 240 replicated Q-sorts . |
| Outcome: | The proposed framework measures value-structure alignment across four model families. |
Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have raised concerns regarding their intrinsic values. |
| Approach: | They propose a psychologically grounded five-factor value system for Large Language Models that integrates psychological principles with cutting-edge AI priorities. |
| Outcome: | The proposed value system meets standard psychological criteria, improves LLM safety prediction, and enhances Llm alignment, when compared to the canonical Schwartz’s values. |
League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
Qianhong Guo, Wei Xie, Xiaofang Cai, Enze Wang, Shuoyoucheng Ma, Xiaobing Sun, Tian Xia, Kai Chen, Xiaofeng Wang, Baosheng Wang
| Challenge: | Large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, but reliable evaluation remains a challenge due to data contamination, opaque operation, and subjective preferences. |
| Approach: | They propose a benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. |
| Outcome: | Experiments on eight mainstream LLMs in mathematics and programming show that the proposed model can distinguish capabilities while maintaining high internal ranking stability. |
The Greatest Good Benchmark: Measuring LLMs’ Alignment with Utilitarian Moral Dilemmas (2024.emnlp-main)
Copied to clipboard
| Challenge: | Our analysis across 15 diverse LLMs reveals consistently encoded moral preferences that diverge from established moral theories and lay population moral standards. |
| Approach: | They propose to evaluate the moral judgments of large language models using utilitarian dilemmas to determine their moral alignment. |
| Outcome: | The findings highlight the ‘artificial moral compass’ of Large Language Models, offering insights into their moral alignment. |
Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) serve diverse global audiences, making it critical for responsible AI deployment across cultures. |
| Approach: | They propose a framework that conceptualizes alignment along three dimensions: Cultural Group, Cultural Elements and Awareness Scope. |
| Outcome: | The proposed framework reveals critical gaps between benchmarks and real-world cultural biases . region dominates cultural group representation, social and political relations dominates coverage . majority of datasets adopt majority-focused Awareness Scope approaches . |
Flames: Benchmarking Value Alignment of LLMs in Chinese (2024.naacl-long)
Copied to clipboard
Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, Yingchun Wang, Dahua Lin
| Challenge: | Existing benchmarks for large language models (LLMs) do not accurately uncover safety vulnerabilities in LLMs. |
| Approach: | They propose a value alignment benchmark called Flames that encompasses both harmlessness principles and a unique morality dimension that integrates specific Chinese values such as harmony. |
| Outcome: | The proposed model performs poorly on Flames, particularly in safety and fairness dimensions. |