Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained masked language models inherit a considerable amount of relational knowledge from the source corpora. |
| Approach: | They propose to specialize pretrained masked language models into relational models from the perspective of network pruning. |
| Outcome: | The proposed model can represent grounded commonsense relations at non-trivial sparsity while being generalizable . the proposed model improves on a wealth of NLP tasks, but we know little about how much knowledge it imparts . |
Similar Papers
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |
Discovering Knowledge-Critical Subnetworks in Pretrained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models encode implicit representations of knowledge in their parameters, but localizing these representations and disentangling them from each other remains an open problem. |
| Approach: | They propose a masking scheme that can be applied to weights and neurons to discover such subnetworks. |
| Outcome: | The proposed method can remove specific knowledge from models while minimizing adverse effects on the original model. |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |
Pre-training Language Models with Deterministic Factual Knowledge (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that Pre-trained Language Models fail to capture factual knowledge robustly. |
| Approach: | They propose to let PLMs learn the deterministic relationship between context and masked content to improve their ability to capture factual knowledge. |
| Outcome: | The proposed methods improve accuracy and consistency of factual knowledge capturing and boost performance of other knowledge-intensive tasks. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance . |
| Approach: | They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible . |
| Outcome: | The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes . |
Commonsense Knowledge Mining from Pretrained Models (D19-1)
Copied to clipboard
| Challenge: | Several approaches have been proposed for training models for commonsense knowledge base completion (CKBC) due to the sparsity of training data. |
| Approach: | They propose a method for generating commonsense knowledge using a large, pre-trained bidirectional language model by transforming relational triples into masked sentences. |
| Outcome: | The proposed method outperforms models trained on held-out test sets on a held-up set, suggesting that it generalizes better than current supervised methods. |
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2021.acl-long)
Copied to clipboard
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou
| Challenge: | Existing pre-training objectives do not explicitly model relational facts in text . Experimental results show that ERICA can improve typical PLMs on several language understanding tasks, including relation extraction, entity typing and question answering. |
| Approach: | They propose a contrastive learning framework ERICA to obtain a deep understanding of entities and relations in text. |
| Outcome: | The proposed framework can improve PLMs on several language understanding tasks, especially under low-resource settings. |
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)
Copied to clipboard
| Challenge: | Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives. |
| Approach: | They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining. |
| Outcome: | The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks. |
Probing Simile Knowledge from Pre-trained Language Models (2022.acl-long)
Copied to clipboard
Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, Chang Su
| Challenge: | Existing approaches to learn generic knowledge from a large corpus are time-consuming and labor-intensive. |
| Approach: | They propose a framework to probe simile knowledge from pre-trained language models to solve SI and SG tasks. |
| Outcome: | The proposed framework solves the SI and SG tasks in a simile triple completion task. |