FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing (2022.acl-long)
Copied to clipboard
| Challenge: | Using pre-trained language models, we evaluate performance group disparities while none of these techniques guarantee fairness, nor consistently mitigate group disparity. |
| Approach: | They present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. |
| Outcome: | The proposed methods show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparity. |
Similar Papers
Fairness Beyond Performance: Revealing Reliability Disparities Across Groups in Legal NLP (2025.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that models often make less reliable or overconfident predictions for marginalized groups. |
| Approach: | They evaluate performance and reliability disparities across demographic, regional, and legal attributes across four jurisdictions using the FairLex benchmark. |
| Outcome: | The FairLex benchmark shows that pre-training improves performance and reliability for underrepresented groups. |
Are Pretrained Multilingual Models Equally Fair across Languages? (2022.coling-1)
Copied to clipboard
| Challenge: | Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. |
| Approach: | They propose to use a multilingual dataset to examine whether multilingual models are equally fair across languages. |
| Outcome: | The proposed model enables apples-to-apples comparison across languages of group disparities in multilingual language models. |
Fairness in Language Models Beyond English: Gaps and Challenges (2023.findings-eacl)
Copied to clipboard
| Challenge: | Language models are inequitable at encoding and re-presentation, but there is much to be studied and criticism for the existing research that remains to be addressed. |
| Approach: | They propose to survey fairness in multilingual and non-English contexts . they argue that it is infeasible to achieve comprehensive coverage in terms of fairness datasets based on English . |
| Outcome: | The proposed methods are infeasible to scale across languages and cultures, the authors argue . they argue that the current methods are too narrowly focused on specific dimensions and types of biases and cannot scale across cultures. |
Assessing Multilingual Fairness in Pre-trained Multimodal Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent pre-trained multimodal models have shown exceptional capabilities towards connecting images and natural language. |
| Approach: | They propose two new fairness notions for pre-trained multimodal models that consider language as the fairness recipient. |
| Outcome: | The proposed models can be generalized to multilingualism by cross-lingual alignment . the results show that the models are individually fair across languages . |
LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in legal NLP have led to a rapid growth of the field . however, many benchmarks are available only in English and no multilingual benchmark exists . |
| Approach: | They propose to use 11 datasets covering 24 languages to compare NLP models. |
| Outcome: | The proposed benchmarks show that even the best baseline only achieves modest results and ChatGPT struggles with many tasks. |
Bias and Fairness in Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | a tutorial will review the history of bias and fairness studies in machine learning and language processing . |
| Approach: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it presents recent community effort to quantify and mitigat bias in natural language processing models . |
| Outcome: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it aims to quantify and mitigate bias in natural language processing models for a wide spectrum of tasks . |
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2022.acl-long)
Copied to clipboard
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, Nikolaos Aletras
| Challenge: | Laws and their interpretations, legal arguments and agreements are typically expressed in writing. |
| Approach: | They propose a benchmark to evaluate model performance across legal NLU tasks . they also evaluate several generic and legal-oriented models . |
| Outcome: | The proposed model performs better across multiple tasks than previous models. |
Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | An increasing awareness of biased patterns in natural language processing resources such as BERT has motivated many metrics to quantify ‘bias’ and ‘fairness’. |
| Approach: | They combine literature survey, correlation analysis and empirical evaluations to evaluate compatibility of fairness metrics for pre-trained language models and their downstream tasks. |
| Outcome: | The proposed measures are not compatible with each other and highly depend on (i) templates, (ii) attribute and target seeds and (iv) the choice of embeddings. |
Benchmarking Intersectional Biases in NLP (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on fairness of machine learning models has focused on how to debias, but research on the fairness and performance of biased/debiased models on downstream prediction tasks has been limited. |
| Approach: | They assess intersectional bias - fairness across multiple demographic dimensions . they highlight possible causes and make recommendations for future NLP debiasing research. |
| Outcome: | The proposed approaches fare well in terms of fairness-accuracy trade-off, but are unable to effectively alleviate bias in downstream tasks. |
Your fairness may vary: Pretrained language model fairness in toxic text classification (2022.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained, bidirectional language models have revolutionized natural language processing research . authors show that focusing on accuracy measures alone can lead to models with wide variation in fairness characteristics . |
| Approach: | They propose to use two post-processing methods to improve model fairness without retraining . they use pretrained language models of varying sizes on two toxic text classification tasks . |
| Outcome: | The proposed methods improve model fairness without retraining . the results show that the fairness variation is more than just accuracy . |