| Challenge: | Existing approaches to transfer learning from pretrained language models are frozen and a task-specific head is trained on top of them. |
| Approach: | They propose an alternative approach that trains one or more task-specific layers on top of the language model. |
| Outcome: | The proposed approach outperforms existing methods on the GLUE leaderboard with just 32 training samples. |
Similar Papers
Adversarial Reprogramming of Text Classification Neural Networks (D19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that adversarial examples can cause a machine learning model to misclassify a sample from the classifier's input domain. |
| Approach: | They propose a context-based vocabulary remapping method that performs a computationally inexpensive input transformation to reprogram a victim classification model for a new set of sequences. |
| Outcome: | The proposed method performs a cost-effective input transformation to reprogram a model for a new set of sequences without altering the network architecture or parameters. |
Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown superior performance for non-adversarial methods in more challenging language pairs. |
| Approach: | They propose to use adversarial autoencoder to map monolingual embeddings to a shared space and to put the target encoders as an adversary against the corresponding discriminator. |
| Outcome: | The proposed method is more robust and achieves better performance than previously proposed adversarial and non-adversarial methods. |
Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial. |
| Approach: | They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks. |
| Outcome: | The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks. |
Cross-lingual Multi-Level Adversarial Transfer to Enhance Low-Resource Name Tagging (N19-1)
Copied to clipboard
| Challenge: | Low-resource language name tagging is an important but challenging task. |
| Approach: | They propose a neural architecture that leverages multi-level adversarial transfer to improve name tagging for low-resource languages. |
| Outcome: | The proposed approach outperforms previous approaches on CoNLL data sets. |
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)
Copied to clipboard
| Challenge: | Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures. |
| Approach: | They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. |
| Outcome: | The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty. |
Data Augmentation with Adversarial Training for Cross-Lingual NLI (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to train cross-lingual models with labeled data are subpar, resulting in subpar results. |
| Approach: | They propose a data augmentation strategy that enriches data to reflect more diversity in a semantically faithful way and leverages adversarial training regimens to achieve greater robustness. |
| Outcome: | The proposed approach improves cross-lingual inference by leveraging the data to reflect more diversity in a semantically faithful way. |
Fine-grained Contrastive Learning for Definition Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Recent pre-trained transformer-based definition generation models lack effective representation learning to contain full semantic components of the given word, leading to under-specific definitions. |
| Approach: | They propose a novel contrastive learning method that encourages the model to capture more detailed semantic representations from the definition sequence encoding. |
| Outcome: | The proposed method could generate more specific definitions compared with state-of-the-art models. |
A Reinforced Generation of Adversarial Examples for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation systems fail on less decent inputs, which may harm the credibility of these systems. |
| Approach: | They propose a paradigm that generates adversarial examples using reinforcement learning to expose pitfalls for a given performance metric. |
| Outcome: | The proposed paradigm produces stable attacks with meaning-preserving adversarial examples. |
Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that models can be easily fooled by intentionally designed adversarial examples. |
| Approach: | They propose two efficient approaches for generating adversarial perturbations on embeddings and propose two new approaches to help model learn adversarials more efficiently. |
| Outcome: | The proposed approaches outperform strong baselines on various text classification datasets and the model's performance drops less under adversarial attack. |
Better Synthetic Data by Retrieving and Transforming Existing Datasets (2024.findings-acl)
Copied to clipboard
| Challenge: | despite advances in large language models, task-specific data is not available for many use cases . a new method to improve automated dataset generation uses publicly available datasets . |
| Approach: | They propose a method to make better use of existing datasets to improve automatic dataset generation. |
| Outcome: | The proposed method outperforms existing methods on language-based tasks . it significantly increases diversity and difficulty of generated data on many tasks compared to other methods . |