| Challenge: | Language models exhibit remarkable natural language understanding and generation capabilities, but they have serious flaws, such as societal biases and spurious correlations. |
| Approach: | They argue that interdisciplinary perspectives are essential for developing more fine-grained and valid measures of data diversity. |
| Outcome: | The proposed measures are based on interdisciplinary perspectives and include a variety of datasets. |
Similar Papers
NLP Needs Diversity outside of ‘Diversity’ (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new position paper argues that diversity in NLP is concentrated on a small number of areas surrounding fairness . |
| Approach: | a new position paper argues that diversity in NLP is disproportionately concentrated on fairness areas. |
| Outcome: | a new position paper argues that diversity in NLP is disproportionately concentrated on fairness areas. |
Measure and Improve Robustness in NLP Models: A Survey (2022.naacl-main)
Copied to clipboard
| Challenge: | Despite the performance gains, NLP models are still fragile and brittle to out-of-domain data, adversarial attacks, or small perturbation to the input. |
| Approach: | They propose a survey of how to define, measure and improve robustness in NLP by connecting multiple definitions of robustness and identifying failures. |
| Outcome: | The proposed models are robust against unseen or challenging scenarios, but are still fragile and brittle to out-of-domain data and adversarial attacks. |
On Measures of Biases and Harms in NLP (2022.findings-aacl)
Copied to clipboard
Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, Kai-Wei Chang
| Challenge: | Recent studies show that natural language processing (NLP) technologies propagate societal biases about demographic groups associated with attributes such as gender, race, and nationality. |
| Approach: | They propose a framework for harms and questions to help practitioners understand biases . they propose measurable measures to detect and mitigate biased groups . |
| Outcome: | The proposed framework provides a framework for harms and questions for practitioners to answer to guide the development of bias measures. |
A Systematic Review of Reproducibility Research in Natural Language Processing (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite the recent progress in reproducibility, the field is far from reaching a consensus on how reproducibility should be defined, measured and addressed. |
| Approach: | They propose to provide a wide-angle snapshot of current work on reproducibility in NLP. |
| Outcome: | The proposed work will provide a wide-angle snapshot of current work on reproducibility in NLP. |
Is NLP Ready for Standardization? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a number of scientific fields, including telecommunications, networks and multimedia, lack standards in the field of NLP. |
| Approach: | They propose to examine how NLP lacks standards and how that can impact society, industry and regulations. |
| Outcome: | The proposed standards examine the needs of NLP researchers and industry . they argue that the lack of standards can impact the field, society and industry. |
Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have examined the quality of labeled data in non-English languages. |
| Approach: | They annotate how datasets are created, input text and label sources, tools used to build them and what they study. |
| Outcome: | The results show that language-proficient NLP researchers' estimated availability correlates with dataset availability. |
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing fairness metrics quantify the differences in a model’s behaviour across a range of demographic groups. |
| Approach: | They propose to unify existing fairness metrics and compare them to three generalized fairness measures to reveal the connections between them. |
| Outcome: | The proposed measures can be explained by differences in parameter choices, and the results are consistent with previous studies. |
Challenges and Strategies in Cross-Cultural NLP (2022.acl-long)
Copied to clipboard
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, Anders Søgaard
| Challenge: | Various efforts have been made to accommodate linguistic diversity and serve speakers of many different languages. |
| Approach: | They propose a framework to examine cultural differences in NLP to better serve users . they argue that cultural knowledge, preferences and values can affect NLP practices . |
| Outcome: | The proposed framework examines how cultural knowledge, preferences and values can affect NLP practices. |
A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study aims to assess linguistic diversity of multilingual data sets against a reference language sample . linguistic diversity is typically measured as the number of languages included in the data set . but such measures do not consider structural properties of the included languages . |
| Approach: | They propose to measure linguistic diversity against a reference language sample to maximise linguistic diversity. |
| Outcome: | The proposed measure can be used to identify the types of languages that are not represented in a data set. |
Changing the World by Changing the Data (2021.acl-long)
Copied to clipboard
| Challenge: | a new paper argues that data curation is already happening, and it is changing the world . social biases and spurious patterns are attracting more attention in NLP models . |
| Approach: | They argue that data curation is already happening and will be happening . they argue that social biases and spurious patterns are the main problems . |
| Outcome: | a new paper argues that data curation is already and will be happening, and it is changing the world. |