| Challenge: | Researchers have traditionally recruited native speakers to provide annotations for benchmark datasets, but there are languages for which recruiting native speakers is difficult. |
| Approach: | They recruit 36 language learners and provide two types of additional resources and perform mini-tests to measure their language proficiency. |
| Outcome: | The proposed method improves learners' language proficiency in terms of vocabulary and grammar. |
Similar Papers
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
Building Better: Avoiding Pitfalls in Developing Language Resources when Data is Scarce (2025.acl-long)
Copied to clipboard
| Challenge: | Language is a powerful means of communication and should be regarded as more than just a collection of tokens. |
| Approach: | They collect feedback from individuals directly involved in and impacted by NLP artefacts for medium- and low-resource languages and highlight key issues related to data quality, cultural appropriateness and ethics of common annotation practices. |
| Outcome: | The findings highlight key issues related to data quality, cultural appropriateness, and ethics of common annotation practices. |
Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have examined the quality of labeled data in non-English languages. |
| Approach: | They annotate how datasets are created, input text and label sources, tools used to build them and what they study. |
| Outcome: | The results show that language-proficient NLP researchers' estimated availability correlates with dataset availability. |
Platforms for Non-speakers Annotating Names in Any Language (P18-4)
Copied to clipboard
| Challenge: | Traditionally, native speakers of a language have been asked to annotate a corpus in that language. |
| Approach: | They propose two annotation platforms that allow an English speaker to annotate names for any language without knowing the language. |
| Outcome: | The proposed annotations achieved state-of-the-art performance on two surprise languages and ten languages at TAC-KBP EDL2017. |
What data should I include in my POS tagging training set? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | POS tagging is a crucial task for descriptive linguistics and language documentation . POS tags are not available in all languages, but are used for training sets for understudied languages . |
| Approach: | They compare POS tagging with in-context learning, active learning, and random sampling . they find that POS can deliver reasonable results for communities with limited resources . |
| Outcome: | The proposed training set for Indigenous and endangered languages performs better than random sampling. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
Charting the Linguistic Landscape of Developing Writers: An Annotation Scheme for Enhancing Native Language Proficiency (2024.lrec-main)
Copied to clipboard
| Challenge: | An annotation task was designed to capture orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education (DevEd) courses. |
| Approach: | They propose an annotation task to capture orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education courses. |
| Outcome: | The proposed annotation task captures orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education courses. |
Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey (2026.eacl-long)
Copied to clipboard
| Challenge: | a longstanding strategy to reduce annotation costs is active learning . data annotation is expected to remain important and active learning to stay relevant . |
| Approach: | They conduct an online survey to assess the perceived relevance of data annotation and active learning . they propose a strategy to reduce annotation costs using active learning, an iterative process . |
| Outcome: | The proposed strategies reduce setup complexity and uncertainty cost while maintaining model performance. |
A Short Survey on Sense-Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Understanding. |
| Approach: | They propose to use sense-annotated corpora for supervised Word Sense Disambiguation. |
| Outcome: | The proposed methods have been compared with knowledge-based approaches and have shown to be more efficient when they are available. |
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |