| Challenge: | Annotator-centric active learning addresses the high costs of collecting human annotations by strategically annotating the most informative samples. |
| Approach: | They propose annotator-centric active learning which incorporates an annotation strategy following data sampling to approximate the full diversity of human judgments. |
| Outcome: | The proposed approach improves data efficiency and performs well in annotator-centric evaluations. |
Similar Papers
Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey (2026.eacl-long)
Copied to clipboard
| Challenge: | a longstanding strategy to reduce annotation costs is active learning . data annotation is expected to remain important and active learning to stay relevant . |
| Approach: | They conduct an online survey to assess the perceived relevance of data annotation and active learning . they propose a strategy to reduce annotation costs using active learning, an iterative process . |
| Outcome: | The proposed strategies reduce setup complexity and uncertainty cost while maintaining model performance. |
Active Learning for Abstractive Text Summarization (2022.findings-emnlp)
Copied to clipboard
Akim Tsvigun, Ivan Lysenko, Danila Sedashov, Ivan Lazichny, Eldar Damirov, Vladimir Karlov, Artemy Belousov, Leonid Sanochkin, Maxim Panov, Alexander Panchenko, Mikhail Burtsev, Artem Shelmanov
| Challenge: | Abstractive text summarization (ATS) requires a long document and short summaries. |
| Approach: | They propose a query strategy for AL in abstractive text summarization that uses uncertainty estimation to reduce model performance. |
| Outcome: | The proposed query strategy improves ROUGE and consistency scores for annotated datasets . it also increases the performance of the model, compared to passive annotation. |
FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern machine learning models require a huge collection of precisely labeled data, which can be labor-intensive and time-consuming. |
| Approach: | They propose a collaborative learning framework that interactively distills and filters the task-specific knowledge from LLMs. |
| Outcome: | The proposed framework improves zero-shot performance on eight benchmark datasets without human supervision. |
From Selection to Generation: A Survey of LLM-based Active Learning (2025.acl-long)
Copied to clipboard
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, Julian McAuley
| Challenge: | Large Language Models (LLMs) have been used for selection and training of data for active learning. |
| Approach: | They propose an intuitive taxonomy that categorizes LLM-based active learning techniques and discuss the transformative roles they can play in the active learning loop. |
| Outcome: | The proposed model can generate entirely new data instances and provide more cost-effective annotations with fewer labeled data instances. |
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In subjective tasks, the inclusion of diverse annotators is crucial as their unique perspectives significantly influence the annotations. |
| Approach: | They propose a framework that minimizes the annotation budget while maximizing the predictive performance for each annotator. |
| Outcome: | The proposed framework surpasses the previous SOTA in capturing the annotators’ individual perspectives with as little as 25% of the original annotation budget on two datasets. |
PALS: Personalized Active Learning for Subjective Tasks in NLP (2023.emnlp-main)
Copied to clipboard
Kamil Kanclerz, Konrad Karanowski, Julita Bielaniewicz, Marcin Gruza, Piotr Miłkowski, Jan Kocon, Przemyslaw Kazienko
| Challenge: | Personalized active learning techniques can be used to learn subjective NLP problems . to acquire training data, texts are often randomly assigned to users for annotation . |
| Approach: | They propose to apply an active learning paradigm to a personalized context to learn preferences . they validated their techniques on a Wiki discussion text labeled with aggression and toxicity . |
| Outcome: | The proposed methods outperform random selection and random selection by 30% on three datasets. |
ALANNO: An Active Learning Annotation System for Mortals (2023.eacl-demo)
Copied to clipboard
| Challenge: | Active learning (AL) is a special family of machine learning algorithms designed to reduce labeling costs and improve accuracy. |
| Approach: | They developed an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. |
| Outcome: | ALANNO is an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
A Survey of Active Learning for Natural Language Processing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature surveys on active learning for NLP are too specific or too general, covering deep active learning. |
| Approach: | They propose to use active learning to improve model learning and annotation cost for NLP problems. |
| Outcome: | The proposed approach is based on a large dataset of data-driven machine learning models. |
ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation. |
| Approach: | They propose to use a multi-head model to learn from disagreements in an active learning setting to identify annotators with a single head. |
| Outcome: | The proposed model outperforms a single-head model in terms of uncertainty estimation and prediction while saving 70% of the annotation budget. |