Systematic Inequalities in Language Technology Performance across the World’s Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have revealed that NLP is limited to a subset of the world’s 6,500 languages. |
| Approach: | They propose a framework for estimating the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP. |
| Outcome: | The proposed framework estimates the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP. |
Similar Papers
Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World (2022.aacl-main)
Copied to clipboard
| Challenge: | Linguistic disparity in the NLP world is widely acknowledged, but the reasons behind it are rarely discussed within the field. |
| Approach: | They propose to categorise languages based on speaker population and vitality . they also analyse the distribution of language data resources and amount of NLP/CL research . |
| Outcome: | The proposed model identifies the reasons for the disparity and suggests ways to overcome it. |
GlobalBench: A Benchmark for Global Progress in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Alham Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, Graham Neubig
| Challenge: | despite advances in NLP, significant disparities in performance across languages still exist . prior benchmarks focused on a limited number of tasks and languages, but now GlobalBench tracks progress on all languages. |
| Approach: | They propose to use global benchmarks to track progress on all NLP datasets in all languages. |
| Outcome: | a new tool tracks progress on all NLP datasets in all languages and tracks per-speaker utility and equity . globalbench is designed to identify the most under-served languages and reward research efforts . a globalbech is available at https://github.com/neulab/globalbench. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020.acl-main)
Copied to clipboard
| Challenge: | 146 papers analyzing "bias" in NLP systems lack normative reasoning, we find . authors propose three recommendations for work analyzing “bias” in Nlp systems . |
| Approach: | They propose three recommendations for analyzing "bias" in NLP systems . they propose to focus on what kinds of system behaviors are harmful, in what ways, to whom, and why . |
| Outcome: | The proposed methods for measuring or mitigating “bias” are poorly matched to their motivations and do not engage critically with literature outside of NLP. |
Evaluating the Diversity, Equity, and Inclusion of NLP Technology: A Case Study for Indian Languages (2023.findings-eacl)
Copied to clipboard
| Challenge: | In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world’s languages, be equitable, not unduly biased towards any particular language, and be inclusive of all users. |
| Approach: | They propose to use Gini coefficient to assess NLP across all three dimensions to assess diversity, equity, and inclusion across all languages. |
| Outcome: | The proposed evaluation paradigm assesses NLP technologies across all three dimensions and identifies the need for regional-specific choices in model building and dataset creation. |
Challenges and Strategies in Cross-Cultural NLP (2022.acl-long)
Copied to clipboard
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, Anders Søgaard
| Challenge: | Various efforts have been made to accommodate linguistic diversity and serve speakers of many different languages. |
| Approach: | They propose a framework to examine cultural differences in NLP to better serve users . they argue that cultural knowledge, preferences and values can affect NLP practices . |
| Outcome: | The proposed framework examines how cultural knowledge, preferences and values can affect NLP practices. |
Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead (2025.emnlp-main)
Copied to clipboard
| Challenge: | African languages are often left behind in state-of-the-art natural language processing systems and large language models. |
| Approach: | They analyze 884 research papers on NLP for African languages published over past five years . they identify key trends shaping the field and outline promising directions . |
| Outcome: | The findings identify key trends shaping the field and outline promising directions . the authors analyze 884 research papers on NLP for African languages published over the past five years . |
Should We Ban English NLP for a Year? (2022.emnlp-main)
Copied to clipboard
| Challenge: | aaron carroll: two thirds of NLP research is devoted to developing technology for speakers of English . carroll says this bias feeds into consumer technologies to widen existing inequality gaps . he says we need to consider more concrete measures to mitigate climate change . |
| Approach: | a new paper argues that NLP is contributing to global inequalities through a digital language divide . a carbon tax, cap-and-trade and car-free Sundays are examples of measures to mitigate climate change . |
| Outcome: | a new paper argues that NLP is contributing to global inequalities through a digital language divide . a carbon tax, cap-and-trade and car-free Sundays are examples of measures to mitigate climate change . |
How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent years have seen many breakthroughs in natural language processing (NLP), transitioning it from a mostly theoretical field to one with many real-world applications. |
| Approach: | They propose a moral philosophy definition of social good and a framework to evaluate the direct and indirect real-world impact of NLP tasks. |
| Outcome: | The proposed framework evaluates the direct and indirect real-world impact of NLP tasks and adopts the methodology of global priorities research to identify priority causes for NLP research. |
A Systematic Review of Reproducibility Research in Natural Language Processing (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite the recent progress in reproducibility, the field is far from reaching a consensus on how reproducibility should be defined, measured and addressed. |
| Approach: | They propose to provide a wide-angle snapshot of current work on reproducibility in NLP. |
| Outcome: | The proposed work will provide a wide-angle snapshot of current work on reproducibility in NLP. |