Rebalancing Label Distribution While Eliminating Inherent Waiting Time in Multi Label Active Learning Applied to Transformers (2024.lrec-main)
Copied to clipboard
| Challenge: | Data annotation is a resourceintensive endeavor, necessitating human involvement and expertise. |
| Approach: | They propose to annotate instances to rebalance label distribution by judiciously selecting and limiting the data to be annotated. |
| Outcome: | The proposed method mitigates biases, improves model performance and reduces strategy-dependent disparities. |
Similar Papers
Investigating Active Learning Sampling Strategies for Extreme Multi Label Text Classification (2022.lrec-1)
Copied to clipboard
| Challenge: | Large scale, multi-label text datasets with high numbers of different classes are expensive to annotate due to domain experts taking a lot of time working through all the classes. |
| Approach: | They propose to build classifiers on multi-label text datasets using Active Learning to reduce labeling effort. |
| Outcome: | The proposed classifiers can be used to reduce labeling effort on multi-label datasets. |
Random Label Forests: An Ensemble Method with Label Subsampling For Extreme Multi-Label Problems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multi-label learning require large memory space for text classification . recent studies show that multiple labels are needed for e-commerce applications . |
| Approach: | They propose a distributed ensemble method with label subsampling to share large memory space for handling large-scale labels. |
| Outcome: | The proposed method can reduce memory usage while keeping competitive performance over real-world data sets. |
Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to learning with noisy labels are prone to selection bias and training bias . obtaining large-scale high-quality datasets is expensive and time-consuming in practical scenarios . |
| Approach: | They propose an imbalanced learning with noisy labels task to let model learn from noisy labels . they first conduct debiased sample selection to better separate clean samples from noisy samples . then they feed selected clean samples to active annotator large language models for re-annotating noisy samples. |
| Outcome: | The proposed method is superior to existing methods on synthetic and real-world datasets. |
Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multi-label text classification is a challenging task because it requires capturing label dependencies. |
| Approach: | They propose to use distribution-balanced loss functions to solve label dependency problems in multi-label text classification by capturing label dependencies from a fixed-set of labels. |
| Outcome: | The proposed loss function addresses both the class imbalance and label linkage problems and outperforms other loss functions. |
Active Learning for BERT: An Empirical Study (2020.emnlp-main)
Copied to clipboard
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim
| Challenge: | Existing approaches to deal with data scarcity are active learning (AL) and pre-trained models are not being considered. |
| Approach: | They propose to use active learning techniques to cope with data scarcity in binary text classification scenarios where the annotation budget is very small and the data is often skewed. |
| Outcome: | The proposed methods improve BERT performance in binary text classification scenarios where the annotation budget is very small and the data is often skewed. |
Reinforced Active Learning for Low-Resource, Domain-Specific, Multi-Label Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Modern text classification systems achieve excellent accuracy across tasks and corpora. |
| Approach: | They propose a Reinforcement Learning policy that uses many different aspects of the data and task to select the most informative unlabeled subset dynamically over the course of the AL procedure. |
| Outcome: | The proposed framework outperforms baselines on four complex multi-class, multi-label text classification datasets. |
Learning with Different Amounts of Annotation: From Zero to Many Labels (2021.emnlp-main)
Copied to clipboard
| Challenge: | a lack of annotator agreement can hinder training of NLP systems . we propose a learning algorithm that can learn from training examples with zero, one, or multiple labels. |
| Approach: | They propose an annotation distribution scheme that assigns multiple labels to training examples . they propose a learning algorithm that can learn from training examples with different amount of annotation . |
| Outcome: | The proposed method achieves consistent gains in two tasks, suggesting distributing labels unevenly among training examples can be beneficial for many NLP tasks. |
Don’t waste a single annotation: improving single-label classifiers through soft labels (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for annotating data are limited by ambiguity and lack of context in data samples. |
| Approach: | They challenge the traditional approach of annotating data by only providing a single label for each sample and annotator disagreement is discarded . instead, they use additional annotation information such as confidence, secondary label and disagreement to generate soft labels. |
| Outcome: | The proposed method improves model performance and calibration on the hard label test set. |
D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Bias (2023.findings-acl)
Copied to clipboard
| Challenge: | Infusing clustering with active learning with AL can overcome the bias issue of both AL and traditional annotation methods while exploiting AL’s annotation efficiency. |
| Approach: | They propose an algorithm that dynamically adjusts clustering and annotation efforts in response to an estimated classifier error-rate. |
| Outcome: | The proposed algorithm outperforms baseline AL approaches with pretrained transformers and traditional Support Vector Machines on eight datasets for emotion, hatespeech, dialog act, and book type detection tasks. |
Multi-task Active Learning for Pre-trained Transformer-based Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Multi-task learning requires annotating the same text with multiple annotation schemes, which can be costly and laborious. |
| Approach: | They propose to use multi-task active learning paradigm to optimize annotation processes by iteratively selecting unlabeled examples whose annotation is most valuable for the NLP model. |
| Outcome: | The proposed model minimizes annotation efforts for multi-task NLP models by iterating on the most valuable examples. |