Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models exhibit systematic biases across demographic groups. |
| Approach: | They propose to use auditing as uncertainty estimation over a fairness metric . they propose to introduce the Bounded Active Fairness Auditor for query-efficient auditing . |
| Outcome: | The proposed auditing tool reduces query access costs and improves performance over time. |
Similar Papers
A Scalable Entity-Based Framework for Auditing Bias in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to bias evaluation in large language models trade ecological validity for statistical control, or use artificial prompts that lack scale and rigor. |
| Approach: | They propose a framework that uses named entities as probes to measure bias in large language models. |
| Outcome: | The proposed framework reproduces bias patterns observed in natural text, enabling large-scale analysis. |
Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing large language models lack a functional internal sampler to faithfully sample from specified probability distributions . lack of robust sampling mechanisms across diverse application scenarios is a critical functional requirement . |
| Approach: | They propose to use a dual-protocol design to disentangle failure modes . batch generation achieves only modest statistical validity, while independent requests collapse almost entirely . |
| Outcome: | The proposed model fails to enforce uniform answer-position constraints and violates demographic targets in attribute-constrained text-to-image prompt synthesis. |
SAFO: Stable Adaptive Fairness Optimization for LLM-Based Social Survey Simulation (2026.acl-long)
Copied to clipboard
| Challenge: | Social survey simulations are increasingly used to improve minority performance and social-welfare metrics. |
| Approach: | They propose a dynamic utility–fairness optimization framework for LLM-based survey simulation that explicitly targets fairness and training stability. |
| Outcome: | The proposed framework improves minority performance and social-welfare metrics on three large-scale survey datasets from China, the U.S. and Europe. |
The Impossibility of Fair LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for evaluating large language models do not extend to general-purpose AI contexts or are infeasible in practice. |
| Approach: | They analyze a variety of technical fairness frameworks to find inherent challenges . they find that each framework does not logically extend to the general-purpose AI context . |
| Outcome: | The proposed frameworks do not logically extend to the general-purpose AI context or are infeasible in practice due to large amounts of unstructured training data and potential combinations of human populations, use cases, and sensitive attributes. |
A Group Fairness Lens for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focusing on a few groups lack a comprehensive categorical perspective to evaluate LLMs’ potential biases and unfairness. |
| Approach: | They propose to evaluate LLM biases from a group fairness lens using a hierarchical schema characterizing diverse social groups. |
| Outcome: | The proposed method mitigates biases in LLMs from a group fairness lens and encapsulates target-attribute combinations across multiple dimensions. |
FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing safety evaluations may overlook the inherent weaknesses of Large Language Models, despite their benefits. |
| Approach: | They propose a benchmark to evaluate the robustness of Large Language Models under extreme conditions. |
| Outcome: | The proposed approach evaluates the fairness of large language models under extreme conditions. |
TFDP: Token-Efficient Disparity Audits for Autoregressive LLMs via Single-Token Masked Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for auditing autoregressive Large Language Models for disparities are limited and expensive. |
| Approach: | They propose a method to detect disparities in autoregressive Large Language Models by token querying . they propose 'token-focused disparity probing' to measure disparities between sentence pairs . |
| Outcome: | The proposed method detects disparities with 42 times fewer output tokens than previous methods. |
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing evaluation of Large Language Models on static benchmarks is vulnerable to data contamination and leaderboard overfitting. |
| Approach: | LLMEval-Fair framework provides a framework for dynamic evaluation of Large Language Models . evaluators use a proprietary bank of 220k graduate-level questions to analyze model data . |
| Outcome: | LLMEval-Fair provides robust and credible evaluation framework for Large Language Models . it provides a strong empirical validation for the dynamic evaluation paradigm . |
Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in contact-center Quality Assurance (QA) relying on web-scale training data raises concerns regarding demographic and behavioral biases that may distort workforce assessment. |
| Approach: | They propose to evaluate the fairness of large language models on 3,000 real-world transcripts . they find systematic disparities in the CFR and MASD across different dimensions . |
| Outcome: | Evaluating 18 LLMs on 3,000 real-world contact center transcripts, they find systematic disparities . larger, more strongly aligned models show lower unfairness, though fairness does not track accuracy. |
Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest using large language models to make tabular classifications . however, LLMs have been shown to exhibit harmful social biases based on stereotypes and inequalities present in society. |
| Approach: | They propose to use large language models to make tabular classifications . they show that LLMs inherit biases from their training data . |
| Outcome: | The proposed models exhibit harmful biases that reflect stereotypes and inequalities in society. |