Flooding-X: Improving BERT’s Resistance to Adversarial Attacks via Loss-Restricted Fine-Tuning (2022.acl-long)
Copied to clipboard
Qin Liu, Rui Zheng, Bao Rong, Jingyi Liu, ZhiHua Liu, Zhanzhan Cheng, Liang Qiao, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing approaches to generating adversarial perturbations scale up the cost of training computational complexity by the number of gradient steps it takes to obtain the adversarials. |
| Approach: | They propose a flood method which aims at better generalization and a criterion to bring hyper-parameter-dependent flooding into effect with a narrowed-down search space by measuring how the gradient steps taken within one epoch affect the loss of each batch. |
| Outcome: | The proposed method improves BERT’s resistance to textual adversarial attacks by a large margin and achieves state-of-the-art robust accuracy on various text classification and GLUE tasks. |
Similar Papers
BERT-ATTACK: Adversarial Attack Against BERT Using BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to generate adversarial samples for discrete data are heuristic replacement strategies that are difficult to implement in continuous data. |
| Approach: | They propose a method to generate adversarial samples using pre-trained masked language models using BERT. |
| Outcome: | The proposed method outperforms state-of-the-art methods in success rate and perturb percentage while remaining fluent and semantically preserved. |
Model Extraction and Adversarial Transferability, Your BERT is Vulnerable! (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models are used for natural language processing (NLP) but when they are deployed as a service, they can suffer from different attacks . |
| Approach: | They propose two defence strategies to protect the target model from adversarial attacks . they show that model extraction and adversarially transferable attacks can be effective . |
| Outcome: | The extracted model can lead to highly transferable adversarial attacks against the target model. |
Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to improve model robustness against word substitution-based adversarial attacks are too slow to generate adversarials on the fly. |
| Approach: | They propose an approach to improve the robustness of BERT models against word substitution-based adversarial attacks by leveraging adversarials for self-supervised contrastive learning. |
| Outcome: | The proposed method improves robustness of BERT models against word substitution-based adversarial attacks without using any labeled data. |
Robust Encodings: A Framework for Combating Adversarial Typos (2020.acl-main)
Copied to clipboard
| Challenge: | Existing procedures to defend against adversarial perturbations of inputs are heuristic in nature and susceptible to stronger attacks. |
| Approach: | They propose a framework that provides robustness without compromises on model architecture. |
| Outcome: | The proposed framework achieves robustness on six tasks while incompatible with state-of-the-art models like BERT. |
BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks (2021.findings-acl)
Copied to clipboard
| Challenge: | adversarial attacks expose important blind spots of deep learning systems, authors show . word and sentence-level attacks tend to be more difficult to defend via spelling correction modules . character-level attack scenarios often involve finding semantic paraphrases of input . |
| Approach: | They propose a model that probabilistically combines context-independent word level information with context-dependent information from BERT's masked language modeling to combat low-level orthographic attacks. |
| Outcome: | The proposed model outperforms a spellchecker and Pruthi's model on a character-level benchmark. |
Towards Improving Adversarial Training of NLP Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for generating NLP adversarial examples involve combinatorial search and expensive sentence encoders for constraining the generated instances. |
| Approach: | They propose to use vanilla adversarial training to train NLP models using a word substitution attack optimized for vanilla adversary training. |
| Outcome: | The proposed approach improves model performance and standard accuracy and can defend against other types of word substitution attacks. |
BAE: BERT-based Adversarial Examples for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have exposed the vulnerability of text classification models to adversarial examples . perturbed versions of the original text are indiscernible by humans and misclassified by the model . |
| Approach: | They propose a black box attack for generating adversarial examples using contextual perturbations from a BERT-masked language model. |
| Outcome: | The proposed attack produces examples with improved grammaticality and semantic coherence compared to previous work. |
GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer-based architectures provide impressive results in many NLP tasks, but obtaining high-quality annotated data is expensive and time consuming. |
| Approach: | They propose a semisupervised learning method that ex- tends the fine-tuning of BERT-like architectures with unlabeled data in a generative adversarial setting. |
| Outcome: | The proposed method reduces the requirement for annotated examples while achieving good performance in sentence classification tasks. |
PlugAT: A Plug and Play Module to Defend against Textual Adversarial Attack (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies have shown that adversarial examples can be easily fooled by adversarially perturbed examples. |
| Approach: | They propose a pluggable defense module PlugAT to provide robust predictions by adding a few trainable parameters to the model inputs while keeping the original model frozen. |
| Outcome: | The proposed model improves robustness over several strong baselines whilst training only 9.1% parameters. |
IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) have shown remarkable performance on diverse NLP tasks through pre-training and fine-tuning. |
| Approach: | They propose a numerically robust IM-connection incorporating a layer of BERT as a solution of Ordinary Differential Equations (ODEs) . Experimental results validate the robustness of IM BERT under various conditions. |
| Outcome: | The proposed model outperforms the existing model on the adversarial GLUE dataset by 5.9%p on low-resource scenarios. |