Papers with substitution
Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
TNT: Text Normalization based Pre-training of Transformers for Content Moderation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language model pre-training (self-supervised or unsupervised learning) has been widely used in a multitude of language processing tasks such as named entity recognition, sentiment analysis, question answering and content moderation. |
| Approach: | They propose a new language pre-training model TNT for content moderation that uses a combination of masking strategy and text normalization to learn from text. |
| Outcome: | The proposed model outperforms baselines on hate speech classification task and is a potential approach to misspelling correction. |
Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results demonstrate the effectiveness of Tail-to-Tail (TtT) non-autoregressive sequence prediction for Chinese Grammatical Error Correction (CGEC) |
| Approach: | They propose a framework for Chinese Grammatical Error Correction (CGEC) that uses a BERT-initialized Transformer Encoder to model the error positions. |
| Outcome: | The proposed framework solves the problem of Chinese Grammatical Error Correction (CGEC) by modeling the token dependencies. |
KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train large language models on private data are not effective because they rely on a local model for generation, resulting in a performance decline, or expose private data to API servers. |
| Approach: | They propose a client-server framework which enhances synthetic data quality and improves model performance while ensuring privacy. |
| Outcome: | The proposed framework improves model performance and privacy while learning local knowledge from the private data with differential privacy (DP) and distilling professional knowledge from server. |
Semantically Consistent Data Augmentation for Neural Machine Translation via Conditional Masked Language Model (2022.coling-1)
Copied to clipboard
| Challenge: | Neural network models have a large number of parameters to train, but data augmentation is relatively under-explored in natural language processing. |
| Approach: | They propose a bi-directional conditional Masked Language Model (CMLM) that can be conditional on both left and right contexts and the label. |
| Outcome: | The proposed method achieves the best performance on four translation datasets and yields up to 1.90 BLEU points over the baseline. |
Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have shown strong potential in various downstream tasks. |
| Approach: | They propose to model adversarial attack task as a sequential decision-making problem where the whole attack process is sequential with two decision- making problems, i.e., word finder and word substitution. |
| Outcome: | The proposed approach achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT. |
Neural semi-Markov CRF for Monolingual Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our proposed model outperforms all previous approaches for monolingual word alignment. |
| Approach: | They propose a neural semi-Markov CRF alignment model which unifies word and phrase alignments through variable-length spans. |
| Outcome: | The proposed model outperforms existing models on in-domain and out-of-domain evaluations and a QA-based benchmark with human annotations. |
Learning to Substitute Words with Model-based Score Ranking (2025.naacl-long)
Copied to clipboard
| Challenge: | Experimental results show that the proposed approach outperforms both masked language models and large language models. |
| Approach: | They propose a model-based scoring approach to quantify sentence quality . they propose 'loss function' that optimizes alignment between model predictions and sentence scores . |
| Outcome: | The proposed approach outperforms masked language models and large language models in the quantitative analysis of word substitutions. |
MVTamperBench: Evaluating Robustness of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Amit Agarwal, Srikant Panda, Angeline Charles, Hitesh Laxmichand Patel, Bhargava Kumar, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Hansa Meghwani, Karan Gupta, Dong-Kyu Chae
| Challenge: | Multimodal Large Language Models (MLLMs) have been a key advance in video understanding but their vulnerability to adversarial tampering remains underexplored. |
| Approach: | They evaluate MLLMs against five prevalent tampering techniques to assess their robustness . they use a tampered video format to examine the vulnerability of ML models . |
| Outcome: | The benchmark evaluates MLLMs against five prevalent tampering techniques based on 19 video manipulation tasks. |