Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham
| Challenge: | Existing approaches for self-supervision operate at word form level, which serves as a surrogate for the underlying semantic content. |
| Approach: | They propose a method to employ weak-supervision directly at the word sense level, without the use of human annotation. |
| Outcome: | The proposed model achieves significantly improved lexical understanding without human annotation on the ‘Word in Context’ task. |
Similar Papers
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)
Copied to clipboard
| Challenge: | Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives. |
| Approach: | They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining. |
| Outcome: | The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks. |
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for training pre-trained language models have limited practicality due to latency requirements. |
| Approach: | They propose a method that uses a Mixture-of-Experts structure to increase model capacity and inference speed. |
| Outcome: | The proposed method outperforms existing distillation methods on natural language understanding and question answering tasks. |
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent pre-trained language models such as BERT have led to noticeable improvements in semantic similarity detection. |
| Approach: | They propose to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. |
| Outcome: | The proposed method improves on multiple semantic similarity datasets and shows that it is beneficial and currently missing from the original model. |
TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices. |
| Approach: | They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages. |
| Outcome: | The proposed method accelerates inference and reduces model size while maintaining accuracy. |
SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for sign language processing have relied on task-specific models, limiting the potential for transfer learning across tasks. |
| Approach: | They propose a self-supervised contextual representation model that adapts masked token prediction objectives to multi-stream visual sign language input. |
| Outcome: | The proposed model adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams. |
GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge (D19-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) aims to find the exact sense of an ambiguous word in a particular context. |
| Approach: | They propose to integrate gloss knowledge into supervised neural networks for Word Sense Disambiguation (WSD) this paper proposes to fine-tune a pre-trained BERT model and achieve new state-of-the-art results on WSD task. |
| Outcome: | The proposed model achieves state-of-the-art on the word Sense Disambiguation (WSD) task. |
Distilling Knowledge Learned in BERT for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT have revolutionized the state of the art in many language understanding tasks. |
| Approach: | They propose a conditional masked language modeling approach to fine tune BERT on target generation tasks by imposing global sequence-level supervision on conventional Seq2Seq models. |
| Outcome: | The proposed model outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. |
bert2BERT: Towards Reusable Pretrained Language Models (2022.acl-long)
Copied to clipboard
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu
| Challenge: | Pre-training large language models can be expensive and wasteful. |
| Approach: | They propose a method which can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and a two-stage learning method to further accelerate the pre-training. |
| Outcome: | The proposed method can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms (2024.naacl-srw)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks. |
| Approach: | They propose two hybrid architectures that combine self-attention and additive attention mechanisms with sub-layer normalization to achieve double the pretraining accuracy of a vanilla-BERT baseline. |
| Outcome: | The proposed architectures outperform BERT-base on two downstream tasks while accelerating inference. |