CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are vulnerable to simple perturbations, causing poor robustness . recent studies show that adversarial training is useless or harmful for the model to detect these semantic changes. |
| Approach: | They propose to use adversarial training to improve the robustness of pre-trained models . they propose to construct negative examples with similar and opposite semantics . |
| Outcome: | Empirical results show that the proposed approach improves on sentiment analysis, reasoning, and reading comprehension tasks. |
Similar Papers
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)
Copied to clipboard
Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu
| Challenge: | Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. |
| Approach: | They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning . |
| Outcome: | The proposed model can encode words into fine-grained representations without modification of production pipelines. |
CLEVE: Contrastive Pre-training for Event Extraction (2021.acl-long)
Copied to clipboard
| Challenge: | Existing EE methods do not model event characteristics from large unsupervised data. |
| Approach: | They propose a contrastive pre-training framework for event extraction to better learn event knowledge from large unsupervised data and their semantic structures. |
| Outcome: | The proposed framework improves on ACE 2005 and MAVEN datasets on event extraction tasks. |
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to improve model robustness against word substitution-based adversarial attacks are too slow to generate adversarials on the fly. |
| Approach: | They propose an approach to improve the robustness of BERT models against word substitution-based adversarial attacks by leveraging adversarials for self-supervised contrastive learning. |
| Outcome: | The proposed method improves robustness of BERT models against word substitution-based adversarial attacks without using any labeled data. |
Learning with Contrastive Examples for Data-to-Text Generation (2020.coling-main)
Copied to clipboard
Yui Uehara, Tatsuya Ishigaki, Kasumi Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, Yusuke Miyao
| Challenge: | Existing models for data-to-text generation generate fluent but sometimes incorrect sentences . Existing studies show that using contrastive examples improves the ability of generating sentences with better lexical choice without degrading the fluency. |
| Approach: | They propose to use models trained on incorrect sentences and learning methods that exploit contrastive examples to reduce such errors. |
| Outcome: | The proposed models generate fluent sentences but often have problematic ones in terms of correctness. |
Reevaluating Adversarial Examples in Natural Language (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art adversarial examples lack a common definition of what constitutes success . human surveys show that to preserve semantics, we need to increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences. |
| Approach: | They propose a unified definition of what constitutes a successful adversarial example . they propose four categories of constraints that are used to define adversarials . |
| Outcome: | The proposed framework is based on the outputs of two state-of-the-art synonym substitution attacks. |
ConRAS: Contrastive In-context Learning Framework for Retrieval-Augmented Summarization (2026.findings-eacl)
Copied to clipboard
| Challenge: | Despite progress in natural language processing, the potential of contrastive learning remains unexplored. |
| Approach: | They propose a framework that injects contrastive objectives into in-context learning-based retrieval-augmented summarization. |
| Outcome: | The proposed framework outperforms state-of-the-art retrieval-augmented methods on three summarization benchmarks showing that it can distinguish between positive and negative samples without parameter updates. |
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2021.acl-long)
Copied to clipboard
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou
| Challenge: | Existing pre-training objectives do not explicitly model relational facts in text . Experimental results show that ERICA can improve typical PLMs on several language understanding tasks, including relation extraction, entity typing and question answering. |
| Approach: | They propose a contrastive learning framework ERICA to obtain a deep understanding of entities and relations in text. |
| Outcome: | The proposed framework can improve PLMs on several language understanding tasks, especially under low-resource settings. |
Semantically Equivalent Adversarial Rules for Debugging NLP models (P18-1)
Copied to clipboard
| Challenge: | Complex machine learning models are often brittle, making different predictions for input instances that are extremely similar semantically. |
| Approach: | They propose to generalize semantically equivalent adversarial rules that induce adversaries on many instances to detect brittle models. |
| Outcome: | The proposed rules generate high-quality local adversaries for more instances than humans and induce four times as many mistakes as human experts. |
Debiased Contrastive Learning of Unsupervised Sentence Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that contrastive learning improves pre-trained language models to derive high-quality sentence representations. |
| Approach: | They propose a framework to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. |
| Outcome: | The proposed framework improves pre-trained language models while pushing apart irrelevant negatives to guarantee the uniformity of the representation space. |