Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to eliminate implicit biases in LLMs do not eradicate underlying behavioral bias. |
| Approach: | They propose a framework that uses logic grid puzzles to probe the influence of social stereotypes on logical reasoning and decision making in LLMs. |
| Outcome: | The proposed framework systematically probes the influence of social stereotypes on logical reasoning and decision making in LLMs. |
Similar Papers
Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled automatic generation of chain-of-thought reasoning . however, when reasoning steps reflect social stereotypes, they can reinforce harmful associations and lead to misleading conclusions. |
| Approach: | They propose a method that detects how model predictions change across incremental reasoning steps. |
| Outcome: | The proposed method outperforms a stereotype-free baseline and improves accuracy. |
Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to quantify and quantify social biases in Large Language Models (LLMs) focus on explicit bias, with little attention to implicit bias. |
| Approach: | They propose a self-reflection-based evaluation framework that measures implicit bias and evaluates explicit bias by prompting LLMs to analyze their own generated content. |
| Outcome: | The proposed framework compares explicit and implicit biases in large language models . it demonstrates that explicit bias manifests as mild stereotypes, while implicit bias exhibits strong stereotypes. |
Veracity Bias and Beyond: Uncovering LLMs’ Hidden Beliefs in Problem-Solving Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been aligned to avoid harmful biases and stereotypes, but recent studies have revealed the superficial nature of this alignment. |
| Approach: | They propose to use large language models to avoid harmful biases and stereotypes by assigning personas to LLMs to observe decision discrepancies in social scenarios or asking them to associate specific attributes with social targets. |
| Outcome: | The proposed models attribute fewer correct solutions and more incorrect ones to African-American groups in math and coding, while Asian authorships are least preferred in writing evaluation. |
Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in simulating human behaviour and social intelligence, but they risk perpetuating societal biases, especially when demographic information is involved. |
| Approach: | They propose a framework that measures semantic shifts in responses and an LLM-judged Preference Win Rate to assess how demographic prompts affect response quality across power-disparate social scenarios. |
| Outcome: | The proposed framework measures semantic shifts in responses and an LLM-judged Preference Win Rate (WR) to assess how demographic prompts affect response quality across power-disparate social scenarios. |
Neutral Is Not Unbiased: Evaluating Implicit and Intersectional Identity Bias in LLMs Through Structured Narrative Scenarios (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models often reproduce societal biases, yet most evaluations overlook how such biase evolve across nuanced contexts or intersecting identities. |
| Approach: | They propose a scenario-based evaluation framework built on 100 narrative tasks . they use critical discourse analysis and quantitative linguistic metrics to analyze LLMs . |
| Outcome: | The proposed evaluation framework provides ethically coherent and socially plausible settings for probing model behavior. |
Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that large language models are susceptible to societal biases due to their exposure to human-generated data. |
| Approach: | They propose two strategies to mitigate implicit gender biases in large language models . they create scenarios where implicit gender is present and develop a metric to assess the presence of biase . |
| Outcome: | The proposed methods mitigate implicit biases with self-reflection and fine-tuning. |
A Comparative Study of Explicit and Implicit Gender Biases in Large Language Models via Self-evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on the explicit and implicit biases in large language models (LLMs) focus on either explicit or implicit bias. |
| Approach: | They propose a self-evaluation-based two-stage measurement of explicit and implicit biases within large language models grounded in social psychology. |
| Outcome: | The proposed model is based on two stages of self-evaluation on state-of-the-art LLMs to measure explicit bias toward social targets, where bias is less likely to be self-recognized by the LLM. |
Social Bias Evaluation for Large Language Models Requires Prompt Variations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have tried to evaluate and mitigate social biases accurately using limited prompts. |
| Approach: | They investigate the sensitivity of Large Language Models when changing prompt variations . they found that LLM rankings fluctuate across prompts for both task performance and social bias . |
| Outcome: | The results show that LLM rankings fluctuate when changing prompt variations . |
Cognitive Bias in Decision-Making with LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models inherit societal biases against protected groups and can be subject to functionally resembling cognitive bias. |
| Approach: | They propose a framework to uncover, evaluate, and mitigate cognitive bias in large language models by using a dataset containing 13,465 prompts to evaluate LLM decisions on different cognitive biases. |
| Outcome: | The proposed framework uncovers, evaluates, and mitigates cognitive bias in large language models, particularly in high-stakes decision-making tasks. |
Social Bias Probing: Fairness Benchmarking for Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating social biases in language models have been limited to binary association tests on small datasets. |
| Approach: | They propose a framework for probing language models for social biases by assessing disparate treatment . they use a large-scale benchmark to examine the diversity of identities and stereotypes . |
| Outcome: | The proposed framework expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. |