| Challenge: | Recent research has revealed undesirable biases in NLP data and models . however, these efforts focus of social disparities in the West and are not directly portable to other geo-cultural contexts. |
| Approach: | They propose a framework to re-contextualize NLP fairness research for the Indian context . they build resources for fairness evaluation in the Indian and delve deeper into social stereotypes for Region and Religion . |
| Outcome: | The proposed framework can be generalized to other geo-cultural contexts. |
Similar Papers
FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes (2025.acl-long)
Copied to clipboard
Janki Atul Nawale, Mohammed Safi Ur Rahman Khan, Janani D, Mansi Gupta, Danish Pruthi, Mitesh M Khapra
| Challenge: | Existing studies on fairness of LLMs are largely Western-focused, making them inadequate for culturally diverse countries such as India. |
| Approach: | They propose a benchmark to evaluate fairness of LLMs across 85 identity groups . they consult domain experts to curate over 1,800 socio-cultural topics . |
| Outcome: | The benchmark evaluates LLMs across 85 identities across 85 castes, religions, regions, and tribes. |
Evaluating the Diversity, Equity, and Inclusion of NLP Technology: A Case Study for Indian Languages (2023.findings-eacl)
Copied to clipboard
| Challenge: | In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world’s languages, be equitable, not unduly biased towards any particular language, and be inclusive of all users. |
| Approach: | They propose to use Gini coefficient to assess NLP across all three dimensions to assess diversity, equity, and inclusion across all languages. |
| Outcome: | The proposed evaluation paradigm assesses NLP technologies across all three dimensions and identifies the need for regional-specific choices in model building and dataset creation. |
Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP (2023.eacl-main)
Copied to clipboard
| Challenge: | Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. |
| Approach: | They propose to clarify the current situation and plot a course for meaningful progress in fair learning by making clear inter-relations among the current gamut of methods and their relation to fairness theory. |
| Outcome: | The proposed approach addresses the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research. |
Bias and Fairness in Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | a tutorial will review the history of bias and fairness studies in machine learning and language processing . |
| Approach: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it presents recent community effort to quantify and mitigat bias in natural language processing models . |
| Outcome: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it aims to quantify and mitigate bias in natural language processing models for a wide spectrum of tasks . |
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing fairness metrics quantify the differences in a model’s behaviour across a range of demographic groups. |
| Approach: | They propose to unify existing fairness metrics and compare them to three generalized fairness measures to reveal the connections between them. |
| Outcome: | The proposed measures can be explained by differences in parameter choices, and the results are consistent with previous studies. |
NLP Needs Diversity outside of ‘Diversity’ (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new position paper argues that diversity in NLP is concentrated on a small number of areas surrounding fairness . |
| Approach: | a new position paper argues that diversity in NLP is disproportionately concentrated on fairness areas. |
| Outcome: | a new position paper argues that diversity in NLP is disproportionately concentrated on fairness areas. |
Fairness in Language Models Beyond English: Gaps and Challenges (2023.findings-eacl)
Copied to clipboard
| Challenge: | Language models are inequitable at encoding and re-presentation, but there is much to be studied and criticism for the existing research that remains to be addressed. |
| Approach: | They propose to survey fairness in multilingual and non-English contexts . they argue that it is infeasible to achieve comprehensive coverage in terms of fairness datasets based on English . |
| Outcome: | The proposed methods are infeasible to scale across languages and cultures, the authors argue . they argue that the current methods are too narrowly focused on specific dimensions and types of biases and cannot scale across cultures. |
Socially Aware Bias Measurements for Hindi Language Representations (2022.naacl-main)
Copied to clipboard
| Challenge: | Language representations are an efficient tool used across NLP, but they are strife with encoded societal biases. |
| Approach: | They investigate the encoded biases in Hindi language representations based on cultural and historical contexts . they emphasize the necessity of social-awareness along with linguistic and grammatical artefacts when modeling language representation . |
| Outcome: | The proposed model reflects the cultural and cultural diversity of the region in which it is used . the model is based on the language and culture of the language being used based upon the study . |
Global Voices, Local Biases: Socio-Cultural Prejudices across Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on human biases are heavily skewed towards Western and European languages . despite growing interest in language models, there are several shortcomings in the literature . |
| Approach: | They scale the Word Embedding Association Test to 24 languages and add culturally relevant information for each language. |
| Outcome: | The proposed language models can reflect and often amplify the effects of bias across linguistic, cultural, and societal borders. |
Fairness Beyond Performance: Revealing Reliability Disparities Across Groups in Legal NLP (2025.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that models often make less reliable or overconfident predictions for marginalized groups. |
| Approach: | They evaluate performance and reliability disparities across demographic, regional, and legal attributes across four jurisdictions using the FairLex benchmark. |
| Outcome: | The FairLex benchmark shows that pre-training improves performance and reliability for underrepresented groups. |