| Challenge: | Interpolation-based regularisation methods such as Mixup have shown to be effective for various tasks and modalities. |
| Approach: | They propose an adaptive distance-aware interpolative Mixup that selects samples based on their diversity in the embedding space. |
| Outcome: | The proposed method achieves state-of-the-art on sentence classification over existing methods on 8 benchmark datasets across English, Arabic, Turkish, and Hindi languages while achieving benchmark F1 scores in 3 times less number of iterations. |
Similar Papers
Dynamic Nonlinear Mixup with Distance-based Sample Selection (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to augment data with mixup are limited by the space of synthetic data and its regularization effect. |
| Approach: | They propose a dynamic nonlinear mixup with distance-based sample selection which generates multiple sample pairs based on the distance between each sample. |
| Outcome: | The proposed method outperforms state-of-the-art methods on multiple public datasets. |
HypMix: Hyperbolic Interpolative Data Augmentation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for data augmentation involve performing mathematical operations over the raw input samples or their latent states representations, but these operations are performed in the Euclidean space, simplifying these representations and resulting in noisy interpolations. |
| Approach: | They propose a model-, data-, and modality-agnostic interpolative data augmentation technique operating in the hyperbolic space that captures the complex geometry of input and hidden state hierarchies better than its contemporaries. |
| Outcome: | The proposed technique outperforms state-of-the-art methods on benchmark and low resource datasets across speech, text, and vision modalities. |
A Data Cartography based MixUp for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Using MixUp, additional samples are generated during training by combining random pairs of training samples and their labels. |
| Approach: | They propose a new MixUp strategy that leverages Training Dynamics and allows more informative samples to be combined for generating new data samples. |
| Outcome: | The proposed method achieves competitive performance using a smaller subset of training data compared with strong baselines and yields lower expected calibration error on the pre-trained language model, BERT, on both in-domain and out-of-domain settings. |
DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to improve the robustness of text classification models are token-, sentence-, and hiddenlevel augmentation. |
| Approach: | They propose an interpolation-based data augmentation approach called DoubleMix to improve the robustness of text classification models by learning the “shifted” features in hidden space. |
| Outcome: | The proposed approach outperforms several popular methods on six text classification benchmark datasets and visual analysis shows that the model features are highly interpretable. |
SSMix: Saliency-Based Span Mixup for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | SSMix synthesizes a sentence while preserving the locality of two original texts by span-based mixing and keeping more tokens related to the prediction relying on saliency information. |
| Approach: | They propose a new method where the operation is performed on input text rather than on hidden vectors like previous approaches. |
| Outcome: | The proposed method outperforms hidden-level mixup methods on a wide range of text classification benchmarks including textual entailment, sentiment classification, and questiontype classification. |
CIAug: Equipping Interpolative Augmentation with Curriculum Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Current methods for interpolative data augmentation select samples at random, which might make it difficult for the model to generalize better and converge faster. |
| Approach: | They propose a curriculum-based learning method that leverages the relative position of samples in hyperbolic embedding space as a complexity measure to gradually mix up increasingly difficult and diverse samples along training. |
| Outcome: | The proposed method achieves state-of-the-art results over existing methods on 10 benchmark datasets across 4 languages in text classification and named-entity recognition tasks. |
Mixup Decoding for Diverse Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating multiple translations for source and target languages neglect the one-to-many mapping between the source and the target languages. |
| Approach: | They propose a method to generate different translations for the input sentence by linearly interpolating it with different sentence pairs sampled from the training corpus during decoding. |
| Outcome: | Experiments on WMT’16 en-ro, WMT'14 en de, and WMT ‘17 zh-en show that the proposed method outperforms all previous diverse machine translation methods. |
Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work on data augmentation techniques that interpolate inputs and labels shows strong effectiveness in image classification. |
| Approach: | They propose to integrate mixup to transformer-based pre-trained architecture for NLP tasks while keeping the whole end-to-end training system. |
| Outcome: | The proposed framework improves on GLUEbenchmark and transformer-based learning models while keeping the whole end-to-end training system. |
SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is an NLP task aimed at determining the correct sense of a word in a sentence from discrete sense choices. |
| Approach: | They propose a word-level mixup method that maintains the sense of a target word by increasing the frequency of least frequent senses during training to reduce the distributional bias of senses. |
| Outcome: | The proposed method shows high accuracy on the most frequent sense (MFS) and low accuracy on least frequent senses (LFS). |
SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity (2025.findings-emnlp)
Copied to clipboard
Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye
| Challenge: | Existing methods for pretraining data mixing for large language models neglect significant inter-domain overlaps and commonalities, failing to control the global diversity of the constructed training dataset. |
| Approach: | They propose a sample-wise data mixture approach that performs global cross-domain sampling by systematically evaluating the quality and diversity of each sample. |
| Outcome: | The proposed method exceeds existing domain-based methods in multiple downstream tasks and perplexity assessments. |