CIAug: Equipping Interpolative Augmentation with Curriculum Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Current methods for interpolative data augmentation select samples at random, which might make it difficult for the model to generalize better and converge faster. |
| Approach: | They propose a curriculum-based learning method that leverages the relative position of samples in hyperbolic embedding space as a complexity measure to gradually mix up increasingly difficult and diverse samples along training. |
| Outcome: | The proposed method achieves state-of-the-art results over existing methods on 10 benchmark datasets across 4 languages in text classification and named-entity recognition tasks. |
Similar Papers
Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work on data augmentation techniques that interpolate inputs and labels shows strong effectiveness in image classification. |
| Approach: | They propose to integrate mixup to transformer-based pre-trained architecture for NLP tasks while keeping the whole end-to-end training system. |
| Outcome: | The proposed framework improves on GLUEbenchmark and transformer-based learning models while keeping the whole end-to-end training system. |
DMix: Adaptive Distance-aware Interpolative Mixup (2022.acl-short)
Copied to clipboard
| Challenge: | Interpolation-based regularisation methods such as Mixup have shown to be effective for various tasks and modalities. |
| Approach: | They propose an adaptive distance-aware interpolative Mixup that selects samples based on their diversity in the embedding space. |
| Outcome: | The proposed method achieves state-of-the-art on sentence classification over existing methods on 8 benchmark datasets across English, Arabic, Turkish, and Hindi languages while achieving benchmark F1 scores in 3 times less number of iterations. |
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Curriculum learning has improved efficiency across machine learning domains, but remains underexplored for language model pretraining. |
| Approach: | They present a systematic investigation of curriculum learning in LLM pretraining . they use vanilla curriculum learning, pacing-based sampling, and interleaved curricula . |
| Outcome: | The proposed framework accelerates convergence in early and mid-training phases, reducing training steps by 18-45% to reach baseline performance. |
HuCurl: Human-induced Curriculum Discovery (2023.acl-long)
Copied to clipboard
| Challenge: | Existing curriculum learning frameworks can be used to discover effective curricula for NLP tasks based on prior knowledge about sample difficulty. |
| Approach: | They propose a framework for curriculum learning based on prior knowledge about sample difficulty. |
| Outcome: | The proposed framework outperforms existing curriculum learning approaches on several NLP tasks and can prune and weight samples for better learning. |
Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies describe how to apply contrastive learning to the language domain but it is difficult to apply data augmentation methods directly to language modeling. |
| Approach: | They propose a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. |
| Outcome: | The proposed method outperforms baseline models on sentence-level tasks with only 70% of memory compared to the baseline model. |
EPiDA: An Easy Plug-in Data Augmentation Framework for High Performance Text Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for data augmentation do not fully exploit the potential of DA in NLP. |
| Approach: | They propose an easy and plug-in framework for data augmentation to support effective text classification. |
| Outcome: | The proposed framework outperforms existing methods in most cases, but not using agent networks or pre-trained generation networks. |
Dynamic Augmentation Data Selection for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Data augmentation is a popular method for fine-tuning pre-trained language models to increase model robustness and performance. |
| Approach: | They propose a dynamic data selection method to select effective augmentation data from different augmentation sources according to the model’s learning stage by identifying a set of augmentation samples that optimally facilitates the learning process of the most current model. |
| Outcome: | The proposed method outperforms strong baselines on a variety of sentence classification tasks. |
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | a heterogeneous training dataset can vary in characteristics such as domain, translation quality, and degree of difficulty. |
| Approach: | They propose to use reinforcement learning to learn an optimal curriculum for NMT training . they find it can beat uniform baselines and hand-designed, state-of-the-art curricula . |
| Outcome: | The proposed approach beats baselines and hand-designed curricula on English-to-French datasets. |
Dynamic Data Selection and Weighting for Iterative Back-Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate that our methods achieve improvements of up to 1.8 BLEU points over competitive baselines. |
| Approach: | They propose a data selection and weighting strategy to iterate back-translation models and apply it to it . they use a target language to back-transcribe monolingual data, which is of high quality and reflect the target domain. |
| Outcome: | The proposed approach achieves 1.8 BLEU points over baselines on domain adaptation, low-resource, and high-resourced MT settings and on two language pairs. |
A Data Cartography based MixUp for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Using MixUp, additional samples are generated during training by combining random pairs of training samples and their labels. |
| Approach: | They propose a new MixUp strategy that leverages Training Dynamics and allows more informative samples to be combined for generating new data samples. |
| Outcome: | The proposed method achieves competitive performance using a smaller subset of training data compared with strong baselines and yields lower expected calibration error on the pre-trained language model, BERT, on both in-domain and out-of-domain settings. |