| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
Similar Papers
Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach (2021.naacl-main)
Copied to clipboard
| Challenge: | Fine-tuned pre-trained language models (LMs) have enormous success in many natural language processing tasks, but they still require excessive labeled data in the fine-tuning stage. |
| Approach: | They propose a framework to enable fine-tuning pre-trained language models with weak supervision without any labeled data. |
| Outcome: | The proposed framework outperforms the strongest baseline and achieves competitive performance with fully-supervised fine-tuning methods. |
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained MLMs produce an anisotropic distribution of token representations . this is not ideal for tasks that require discriminative semantic meanings of distinct tokens - a problem that exists in pre-training models . |
| Approach: | They propose a continual pre-training approach that encourages BERT to learn an isotropic distribution of token representations. |
| Outcome: | The proposed approach improves on a wide range of English and Chinese benchmarks. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |
Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train cross-lingual pre-trained language models have shown great success in cross-linguistic sequence labeling tasks. |
| Approach: | They propose a cross-lingual language informative span masking task to eliminate the objective gap between pre-training and fine-tuning stages. |
| Outcome: | The proposed method surpasses the state-of-the-art methods on multiple benchmarks even with limited pre-training data. |
Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods analyze training data with member and non-member contexts, overlooking potential insights from both member and not-member. |
| Approach: | They propose a method that leverages asymmetric distributional shifts induced by member and non-member contexts through contrastive decoding to enhance membership inference. |
| Outcome: | The proposed approach outperforms the current state-of-the-art on the WikiMIA benchmark and is robust against various text manipulation techniques. |
Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Adapting pre-trained language models (PrLMs) to new domains has gained much attention . Adaptation of PrLMs to newdomains is important, but requires fine-tuning . |
| Approach: | They propose to use PrLMs to adapt to new domains without fine-tuning . they use class-aware feature self-distillation to learn discriminative features . |
| Outcome: | The proposed model can learn discriminative features from pre-trained language models without fine-tuning. |
Instance Regularization for Discriminative Language Model Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized independent strategies of ennoising or denosing . Existing methods treat training instances equally throughout the training process . |
| Approach: | They propose to use ennoising and denoising to train discriminative pre-trained language models . they propose to model the complexity of restoring the original sentences from corrupted ones . |
| Outcome: | Experimental results show that the proposed method improves pre-training efficiency, effectiveness, and robustness. |
PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing techniques for natural language understanding and generation use autoencoding and/or autoregressive objectives to train models. |
| Approach: | They propose a self-supervised pre-training scheme that pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus for generating new text conditioned on context. |
| Outcome: | The proposed scheme achieves state-of-the-art results on a variety of language generation benchmarks covering generative question answering, abstractive summarization and conversational response generation. |
NAT: Noise-Aware Training for Robust Neural Sequence Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Sequence labeling systems should perform reliably under ideal conditions and with corrupted inputs. |
| Approach: | They propose two noise-aware training objectives that improve robustness of sequence labeling performed on perturbed inputs. |
| Outcome: | The proposed methods improve robustness on English and German named entity recognition benchmarks. |
Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune pre-trained models to downstream tasks are limited by labeled examples. |
| Approach: | They propose to apply post-training on unlabeled task data before fine-tuning by contrastive learning that considers either token-level or sequence-level similarity. |
| Outcome: | Empirical results show that contrastive masked language modeling surpasses other methods in few-shot settings without the need for data augmentation. |