| Challenge: | Existing systems for vocabulary evaluation are based on simple rules and do not account for real-life user learning data. |
| Approach: | They propose to use real-life user vocabulary learning data to evaluate vocabulary . they use language learning data from a phase6 vocabulary trainer to generate a multilingual data set for vocabulary evaluation. |
| Outcome: | The proposed data set provides outstanding results with 95.5 accuracy and F2-score. |
Similar Papers
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)
Copied to clipboard
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou
| Challenge: | XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios . |
| Approach: | They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora. |
| Outcome: | The proposed dataset is labeled in English and includes only natural language understanding tasks. |
Building an English Vocabulary Knowledge Dataset of Japanese English-as-a-Second-Language Learners Using Crowdsourcing (L18-1)
Copied to clipboard
| Challenge: | a dataset for analyzing the English vocabulary of English-as-a-second language learners is available . a vocabulary size test was performed by 100 test takers hired via crowdsourcing . |
| Approach: | They propose a dataset for analyzing the English vocabulary of English-as-a-second language learners. |
| Outcome: | a dataset for analyzing the English vocabulary of English-as-a-second language learners is available online . the results show that the test is reliable and can be predicted with high accuracy . |
The RELX Dataset and Matching the Multilingual Blanks for Cross-Lingual Relation Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches for relation classification are focused on the English language and require lots of training data with human annotations. |
| Approach: | They propose a baseline model based on Multilingual BERT and a new multilingual pretraining setup . they propose 'relationship classification' models that use distant supervision . |
| Outcome: | The proposed model significantly improves the baseline model with distant supervision. |
Huge Automatically Extracted Training-Sets for Multilingual Word SenseDisambiguation (L18-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation is a crucial task in Natural Language Processing . supervised systems need to be trained on word-by-word basis, a problem that is beyond reach for resource-rich languages like English. |
| Approach: | They release six large-scale sense-annotated datasets in multiple languages to pave the way for supervised multilingual Word Sense Disambiguation. |
| Outcome: | The results show that large-scale sense annotations can be used as training sets for supervised systems. |
SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects (2024.eacl-long)
Copied to clipboard
David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, En-Shiun Lee
| Challenge: | despite progress in building multilingual language models evaluation is limited to a few languages with available datasets . despite this, we create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Approach: | They create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Outcome: | The proposed dataset addresses the lack of evaluation dataset for Natural Language Understanding (NLU) for many languages, it is the first publicly available evaluation dataset. |
Multi-lingual Functional Evaluation for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Multilingual competence in large language models is often evaluated via static data benchmarks such as Belebele, M-MMLU and M-GSM. |
| Approach: | They extend existing functional benchmark templates from English to five additional languages that span the range of resources available for NLP: French, Spanish, Hindi, Arabic and Yoruba. |
| Outcome: | The proposed models are translated from English to French, Spanish, Hindi, Arabic and Yoruba. |
BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data (2026.eacl-long)
Copied to clipboard
Jaap Jumelet, Abdellah Fourtassi, Akari Haga, Bastian Bunzeck, Bhargav Shandilya, Diana Galvan-Sosa, Faiz Ghifari Haznitrama, Francesca Padovani, Francois Meyer, Hai Hu, Julen Etxaniz, Laurent Prevot, Linyang He, María Grandury, Mila Marcheva, Negar Foroutan, Nikitas Theodoropoulos, Pouya Sadeghi, Siyuan Song, Suchir Salhan, Susana Zhou, Yurii Paniv, Ziyin Zhang, Arianna Bisazza, Alex Warstadt, Leshem Choshen
| Challenge: | prevailing trend in language modeling research is to prioritize scaling, authors say . from infancy to maturity, English learners acquire language through exposure to less than 100M words . |
| Approach: | They propose a multilingual collection of datasets modeling the language a person observes from birth until they acquire a native language. |
| Outcome: | The proposed models outperform models trained on a fixed, developmentally plausible English corpus on various benchmarks. |
An Open Dataset and Model for Language Identification (2023.acl-short)
Copied to clipboard
| Challenge: | Existing LID systems perform poorly on low-resource languages, causing 'representation washing', where the community is given a false view of the actual progress of low-source NLP. |
| Approach: | They propose a model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033% across 201 languages, outperforming previous work. |
| Outcome: | The proposed model outperforms existing models and datasets on 201 languages and a false positive rate of 0.033%. |
CEFR-based Lexical Simplification Dataset (L18-1)
Copied to clipboard
| Challenge: | Existing tools for lexical simplification are not tailored to language education with word levels and lists of candidates subjective. |
| Approach: | They construct a language dataset for lexical simplification based on CEFR levels . target and candidate words are assigned CEFR-J wordlists and English Vocabulary Profile . |
| Outcome: | The proposed method is based on the common European Framework of References for Languages (CEFR) levels and candidates are selected using an online thesaurus. |
Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models require a more expressive vocabulary to represent all languages . however, increasing the vocabulary size significantly slows down the pre-training speed . |
| Approach: | They propose an algorithm VoCap to determine the desired vocabulary capacity of each language. |
| Outcome: | The proposed algorithm improves cross-lingual model pre-training while reducing side effects of increasing vocabulary size. |