| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |
Similar Papers
Guiding Attention for Self-Supervised Learning with Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that self-attention patterns in trained models contain a majority of non-linguistic regularities. |
| Approach: | They propose a technique to allow efficient self-supervised learning with bi-directional Transformers by using an auxiliary loss function to guide attention heads to conform to such patterns. |
| Outcome: | The proposed method achieves state-of-the-art in low-resource settings and is agnostic to pre-training objectives. |
Probing for Bridging Inference in Transformer Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained transformer language models are capable of bridging inference, but they lack the commonsense knowledge to capture syntactic information. |
| Approach: | They investigate whether pre-trained transformer language models capture bridging inference . they use a masked token prediction task to investigate attention heads in BERT . |
| Outcome: | The proposed model significantly captures bridging inference, the authors show . the distance between anaphor-antecedent and context plays an important role in the inference . |
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to learning from examples are limited due to the vast number of languages, domains and tasks. |
| Approach: | They propose a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. |
| Outcome: | The proposed approach outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin. |
Pre-Training Transformers as Energy-Based Cloze Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | elucidates close connection between cloze modeling and representation learning over text. |
| Approach: | They propose an energy-based cloze model for representation learning over text . they assign a scalar energy score to each input token indicating how likely it is given context . |
| Outcome: | The proposed model performs better than masked language models and faster than cloze models. |
HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms (2024.naacl-srw)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks. |
| Approach: | They propose two hybrid architectures that combine self-attention and additive attention mechanisms with sub-layer normalization to achieve double the pretraining accuracy of a vanilla-BERT baseline. |
| Outcome: | The proposed architectures outperform BERT-base on two downstream tasks while accelerating inference. |
Investigating Transferability in Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on deep NLP models has centered on probing, a method that involves training classifiers for different tasks on model representations. |
| Approach: | They propose a method for determining the impact of each pretrained layer on transfer task performance by ablation. |
| Outcome: | The proposed method shows that pretraining models improve performance on downstream tasks . the results highlight the limitations of methods that operate on frozen models or single data samples. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for entity representations do not capture information in a knowledge base, and cannot represent entities that do not exist in the KB. |
| Approach: | They propose a pretrained contextualized representation of words and entities based on the bidirectional transformer. |
| Outcome: | The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. |
Syntax-Enhanced Pre-trained Model (2021.acl-long)
Copied to clipboard
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, Nan Duan
| Challenge: | Existing methods that use syntax of text in pre-training and fine-tuning suffer from discrepancy between the two stages. |
| Approach: | They propose a model that utilizes the syntactic structure of text in pre-training and fine-tuning stages. |
| Outcome: | The proposed model achieves state-of-the-art on six public benchmark datasets. |