CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | a dataset of Chinese large language models is used to measure societal biases . many studies have shown that LLMs exhibit harmful societal biased outputs despite human data . |
| Approach: | They present a Chinese Bias Benchmark dataset that includes over 100K questions constructed by human experts and generative language models. |
| Outcome: | The proposed dataset covers stereotypes and societal biases in 14 social dimensions related to Chinese culture and values. |
Similar Papers
McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets on bias evaluation for large language models focus on English and North American culture and are limited to one task. |
| Approach: | They propose to evaluate Chinese language models' biases from multiple perspectives using a multi-task Chinese Bias Evaluation Benchmark. |
| Outcome: | The proposed model covers 12, 82 subcategories and 5 evaluation tasks covering a wide range of categories and content diversity. |
PakBBQ: A Culturally Adapted Bias Benchmark for QA (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely adopted in language processing applications, but they often perpetuate harmful societal biases. |
| Approach: | They propose a culturally and regionally adapted extension of the original Bias Benchmark for Question Answering dataset to address this gap. |
| Outcome: | The proposed model gains 12% accuracy with disambiguation and stronger counter bias behaviors in Urdu than in English. |
Quite Good, but Not Enough: Nationality Bias in Large Language Models - a Case Study of ChatGPT (2024.lrec-main)
Copied to clipboard
| Challenge: | Nationality is a key demographic element that enhances the performance of large language models, but it has received less scrutiny regarding inherent biases. |
| Approach: | They investigated nationality bias in ChatGPT, a large language model for text generation. |
| Outcome: | The proposed model generates 4,680 discourses about nationality in Chinese and English, with 195 countries, 4 temperature settings, and 3 prompt types. |
TWBias: A Benchmark for Assessing Social Bias in Traditional Chinese Large Language Models through a Taiwan Cultural Lens (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities in natural language processing, but concerns about social bias amplification remain. |
| Approach: | They propose a social bias evaluation benchmark for Traditional Chinese LLMs that integrates chat templates and diverse prompts for comprehensive bias assessment. |
| Outcome: | The proposed model incorporates chat templates and diverse prompts for comprehensive bias assessment focusing on Taiwan's cultural context and prioritizing gender and ethnicity bias evaluation. |
Detecting Subtle Biases: An Ethical Lens on Underexplored Areas in AI Language Models Biases (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly embedded in the daily lives of individuals across diverse social classes. |
| Approach: | They propose to analyze LLMs' responses to 1,016 scenarios categorized into ethical, unethical, and neutral types. |
| Outcome: | The proposed model analyzed 1,016 scenarios categorized into ethical, unethical, and neutral types. |
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022.findings-emnlp)
Copied to clipboard
Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, Helen Meng
| Challenge: | a number of safety concerns hinder the deployment of open-domain dialog systems, such as offensive languages and toxic behaviors, such social bias is difficult to detect. |
| Approach: | They propose a Dial-Bias Framework for analyzing social bias in conversations . they introduce a Chinese social bias dialog dataset and conduct in-depth ablation studies . |
| Outcome: | The proposed framework is the first annotated Chinese social bias dialog dataset . the proposed framework also provides a fine-grained dialog bias measurement benchmark . |
Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have unlocked unprecedented applications of AI. |
| Approach: | They propose to use a social benchmark to evaluate the bias protection provided by Large Language Models (LLMs) with a variety of tasks with varying complexities to assess their effectiveness. |
| Outcome: | The proposed benchmark shows that both ChatGPT and GPT-4 have strong biases with respect to nationality, gender, race, and religion. |
CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models (2024.findings-acl)
Copied to clipboard
Linhao Yu, Yongqi Leng, Yufei Huang, Shang Wu, Haixin Liu, Xinmeng Ji, Jiahui Zhao, Jinwang Song, Tingting Cui, Xiaoqing Cheng, Liutao Liutao, Deyi Xiong
| Challenge: | Recent years have witnessed remarkable progress achieved by large language models in both natural language understanding and generation. |
| Approach: | They propose a large benchmark CMoralEval for moral evaluation of Chinese LLMs . they use a Chinese TV program discussing Chinese moral norms and Chinese moral anomies based on various sources . |
| Outcome: | The proposed dataset is characterized by diversity and authenticity. |
Societal Biases in Language Generation: Progress and Challenges (2021.acl-long)
Copied to clipboard
| Challenge: | Language generation techniques can produce undesirable societal biases that can negatively impact marginalized populations. |
| Approach: | They propose to examine how decoding techniques contribute to biases in language generation . they also conduct experiments to quantify the effects of these techniques . |
| Outcome: | The proposed methods can reduce biases and improve user experience, the authors argue . they also show that the proposed techniques can reduce societal biase . |
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on social biases in language models have focused on only English. |
| Approach: | They propose to use a Chinese dataset for bias evaluation and mitigation of Chinese conversational language models. |
| Outcome: | The proposed dataset includes under-explored bias categories, such as ageism and appearance biases, which received less attention in previous studies. |