| Challenge: | Existing methods for self-training are interpreted as teacher-student frameworks, where the teacher generates pseudo-labels and the student makes predictions. |
| Approach: | They propose a differentiable self-training method that treats teacher-student as a Stackelberg game where a leader is always in a more advantageous position than a follower. |
| Outcome: | The proposed model outperforms existing methods on semi- and weakly-supervised learning tasks on semi and weak supervised tasks. |
Similar Papers
Self-Training with Weak Supervision (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art deep neural networks require large amounts of labeled training data that is expensive to obtain or not available for many tasks. |
| Approach: | They propose a weak supervision framework that leverages all available data for a given task . they leverage task-specific unlabeled data through self-training with a model that predicts pseudo-labels for instances that may not be covered by weak rules . |
| Outcome: | The proposed framework improves on state-of-the-art datasets on six benchmark tasks. |
Leveraging Training Dynamics and Self-Training for Text Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Semi-supervised learning (SSL) is a promising technique for improving deep learning models when training data is scarce. |
| Approach: | They propose a semi-supervised learning approach that leverages training dynamics of unlabeled data. |
| Outcome: | The proposed method achieves an average increase in F1 score of 3.5% over baselines in low resource settings. |
Rationalizing Transformer Predictions via End-To-End Differentiable Self-Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks are increasingly prevalent across a wide range of applications, driving significant advancements in fields such as natural language processing, computer vision, and beyond. |
| Approach: | They propose an end-to-end differentiable training paradigm for stable training of a rationalized transformer classifier. |
| Outcome: | The proposed model is capable of classifying a sample and scoring input tokens without any explicit supervision and produces class-wise rationales without instabilities. |
Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) methods require a substantial quantity of high-quality annotation for training models. |
| Approach: | They propose a method to reduce the number of incorrect pseudo labels in self-training . they propose 'uncertainty-aware teacher learning' and 'student-student collaboration' |
| Outcome: | The proposed method is superior to state-of-the-art DS-NER denoising methods. |
Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Sets (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for text classification suffer from the semantic drift problem, which is a problem for self-training. |
| Approach: | They propose a semi-supervised text classifier based on self-training using one positive and one negative property of neural networks. |
| Outcome: | The proposed model outperforms ten baseline models in five benchmarks and is additive to language model pretraining. |
Friend-training: Learning from Models of Different but Related Tasks (2023.eacl-main)
Copied to clipboard
| Challenge: | Current self-training methods focus on improving model performance on a single task. |
| Approach: | They propose a cross-task self-training framework where models trained to do different tasks are used in iterative training, pseudo-labeling, and retraining processes to help each other for better selection of pseudo-labeled labels. |
| Outcome: | The proposed framework achieves the best performance compared to baselines on two dialogue understanding tasks. |
A Class-Rebalancing Self-Training Framework for Distantly-Supervised Named Entity Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Distant supervision reduces the reliance on human annotation in named entity recognition tasks. |
| Approach: | They propose a class-rebalancing self-training framework for improving distantly-supervised named entity recognition by using a flexible threshold and a hybrid pseudo label. |
| Outcome: | The proposed model achieves state-of-the-art on five flat and two nested datasets and compares with other methods on the same dataset. |
SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for semi-supervised text classification have shown great performance in few-shot scenarios, where both labeled and unlabeled data are utilized. |
| Approach: | They propose a simple instance-adaptive self-training method for semi-supervised text classification that generates two augmented views for each unlabeled data and trains a meta learner to identify relative strength of augmentations based on the similarity between the original view and the augmented view. |
| Outcome: | The proposed method consistently shows competitive performance with varying sizes of labeled training data compared to existing semi-supervised learning methods. |
Keep Learning: Self-supervised Meta-learning for Learning from Inference (2021.eacl-main)
Copied to clipboard
| Challenge: | A common approach to improve performance of machine learning algorithms involves self-supervised learning on large unlabeled data before fine-tuning on downstream tasks. |
| Approach: | They propose to use model's own class-balanced predictions to back-propagate the loss from the model''s class-balancing predictions (pseudo-labels) this method improves performance of standard backbones such as BERT, Electra, and ResNet-50 on a wide variety of tasks, including question answering on SQuAD and NewsQA . |
| Outcome: | The proposed method outperforms previous approaches on a wide variety of tasks including question answering on SQuAD and NewsQA, benchmark task SuperGLUE, conversation response selection on Ubuntu Dialog corpus v2.0, and image classification on MNIST and ImageNet. |
Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to bridge the linguistic gap between self-training and monolingual named entity recognition (NER) however, due to sub-optimal performance on target languages, the pseudo labels are noisy and limit the overall performance. |
| Approach: | They propose to combine representation learning and pseudo label refinement in one coherent framework to improve self-training for cross-lingual named entity recognition (NER) |
| Outcome: | The proposed method improves cross-lingual named entity recognition (NER) on multiple transfer pairs. |