Robustness to Modification with Shared Words in Paraphrase Identification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Paraphrase identification models have been shown to be vulnerable and lack robustness in tasks such as text classification and natural language inference. |
| Approach: | They propose to modify an example such that a target model makes a wrong prediction by using beam search constrained by heuristic rules and a BERT-masked language model to generate substitution words compatible with the context. |
| Outcome: | The proposed model performance drops dramatically on modified examples, revealing the robustness issue. |
Similar Papers
Evaluating Paraphrastic Robustness in Textual Entailment Models (2023.acl-short)
Copied to clipboard
| Challenge: | Recognizing Textual Entailment models understand language and should be robust to paraphrases. |
| Approach: | They propose to evaluate whether RTE models are robust to paraphrase . they use 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate their models . |
| Outcome: | The evaluation set shows that models change predictions on 8-16% of paraphrased examples, suggesting that there is room for improvement. |
Keys to Robust Edits: From Theoretical Insights to Practical Advances (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for modifying parametric memory are prone to inaccuracies due to conflicting or outdated information. |
| Approach: | They propose a plug-and-play module that disentangles editing keys from native model representations and dynamically adjusts keys via contrastive learning to achieve robustness-specificity balance. |
| Outcome: | The proposed method improves over robustness tests by up to 66.4% while maintaining the success rate unaffected. |
Improving Paraphrase Detection with the Adversarial Paraphrasing Task (2021.acl-long)
Copied to clipboard
| Challenge: | a new adversarial method of paraphrase identification is being used to identify paraphrases based on word overlap and syntax . authors propose a dataset that generates semantically equivalent but lexically and syntactically disparate paraphrase pairs . |
| Approach: | They propose an adversarial method for paraphrase identification that uses word overlap and syntax to identify paraphrases. |
| Outcome: | The proposed method improves paraphrase detection accuracy and speed of generation of datasets. |
Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce disparities in model outcomes have focused on data augmentation, debiasing model embeddings, or adding fairness-based optimization objectives during training. |
| Approach: | They propose to use certified word substitution robustness methods to improve equality of odds and equality of opportunity on multiple text classification tasks. |
| Outcome: | The proposed methods improve equality of odds and equality of opportunity on multiple text classification tasks. |
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we evaluated their robustness on multiple datasets. |
| Approach: | They propose a new metric for assessing model robustness by empirical evaluation of several models on multiple datasets. |
| Outcome: | The proposed metric is based on a set of datasets that are constructed by introducing naturally-occurring, non-malicious perturbations or by generating semantically equivalent paraphrases of input questions or statements. |
Pointwise Paraphrase Appraisal is Potentially Problematic (2020.acl-srw)
Copied to clipboard
| Challenge: | prevailing methods for paraphrase identification models are binary classification problems . current methods do not provide consistent and robust performance on unseen samples and real world problems. |
| Approach: | They propose to use binary classification to evaluate paraphrase identification models . they propose to improve methods for fine-tuning BERT models by pairing two sentences as one sequence . |
| Outcome: | The proposed methods may fail on simple tasks like identifying pairs with two identical sentences. |
Robustness and Adversarial Examples in Natural Language Processing (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to raise awareness of practical concerns about NLP robustness . it aims at addressing the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift . |
| Approach: | This tutorial aims to bring awareness of practical concerns about NLP robustness . it reviews recent studies on analyzing the weakness of NLP systems when facing adversarial inputs . |
| Outcome: | This tutorial aims to bring awareness of practical concerns about NLP robustness . it will examine the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift . |
Revisiting Query Variation Robustness of Transformer Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite their proficiency with natural language, transformer-based large language models are not robust to query variations such as typos and paraphrases. |
| Approach: | They extend their findings to include more recent large language models . they find that instruct-LLMs are more robust to query variations . |
| Outcome: | The proposed model can be prompted for robustness by a set of instruction-tuned LLMs. |
SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions (2020.acl-main)
Copied to clipboard
| Challenge: | State-of-the-art NLP models can be fooled by human-unaware transformations such as synonymous word substitution. |
| Approach: | They propose a method that constructs a stochastic ensemble by applying random word substitutions on the input sentences and leverages the statistical properties to provably certify the robustness. |
| Outcome: | The proposed method outperforms state-of-the-art methods on IMDB and Amazon text classification tasks with practically meaningful certified accuracy. |
Mitigating Paraphrase Attacks on Machine-Text Detection via Paraphrase Inversion (2025.findings-acl)
Copied to clipboard
| Challenge: | Paraphrases applied to machine-generated texts can degrade performance of machine-text detectors. |
| Approach: | They propose an approach which frames the problem as translation from paraphrased text back to the original text. |
| Outcome: | The proposed approach yields an average improvement of +22% AUROC across seven detectors and three different domains. |