| Challenge: | Earlier studies have shown that domain-specific LMs are crucial for domain-specific applications. |
| Approach: | They propose a new BERT model for the cybersecurity domain, CTI-BERT . they find it significantly outperforms general-domain and security-domain models . |
| Outcome: | The proposed model outperforms general-domain and security-domain models for cybersecurity tasks. |
Similar Papers
BERT-ATTACK: Adversarial Attack Against BERT Using BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to generate adversarial samples for discrete data are heuristic replacement strategies that are difficult to implement in continuous data. |
| Approach: | They propose a method to generate adversarial samples using pre-trained masked language models using BERT. |
| Outcome: | The proposed method outperforms state-of-the-art methods in success rate and perturb percentage while remaining fluent and semantically preserved. |
Extracted BERT Model Leaks More Information than You Think! (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vulnerable to model extraction attacks . model extraction can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies. |
| Approach: | They propose to launch an attribute-inference attack against an extracted BERT model to prevent privacy leakage. |
| Outcome: | The proposed attack can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies. |
Model Extraction and Adversarial Transferability, Your BERT is Vulnerable! (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models are used for natural language processing (NLP) but when they are deployed as a service, they can suffer from different attacks . |
| Approach: | They propose two defence strategies to protect the target model from adversarial attacks . they show that model extraction and adversarially transferable attacks can be effective . |
| Outcome: | The extracted model can lead to highly transferable adversarial attacks against the target model. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
On the use of BERT for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Existing studies on using pretrained language models for supervised NMT have not been successful. |
| Approach: | They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data. |
| Outcome: | The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets. |
The birth of Romanian BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are available in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. |
| Approach: | They propose to use a Romanian transformer-based language model to pretrained a large text corpus to evaluate the model. |
| Outcome: | The proposed model is open-source and can be used in production. |
Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis (2020.acl-main)
Copied to clipboard
| Challenge: | Cross-domain sentiment classification requires large amounts of labeled data. |
| Approach: | They propose to apply a pre-training language model BERT on unsupervised domain adaptation . they propose to distill domain-specific features in a self-supervised way . |
| Outcome: | The proposed model outperforms state-of-the-art methods on Amazon dataset . it can be applied to the unsupervised domain adaptation task without domain awareness . |
Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)
Copied to clipboard
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang
| Challenge: | Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized. |
| Approach: | They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions. |
| Outcome: | The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks. |
RoChBert: Towards Robust BERT Fine-tuning for Chinese (2022.findings-emnlp)
Copied to clipboard
Zihan Zhang, Jinfeng Li, Ning Shi, Bo Yuan, Xiangyu Liu, Rong Zhang, Hui Xue, Donghong Sun, Chao Zhang
| Challenge: | Pre-trained language models (e.g., BERT) have been proved vulnerable to adversarial texts. |
| Approach: | They propose to fuse Chinese phonetic and glyph features into pre-trained models by using a more comprehensive adversarial graph. |
| Outcome: | The proposed framework outperforms existing methods in significant ways on a wide range of tasks while remaining accurate on benign texts. |
BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks (2021.findings-acl)
Copied to clipboard
| Challenge: | adversarial attacks expose important blind spots of deep learning systems, authors show . word and sentence-level attacks tend to be more difficult to defend via spelling correction modules . character-level attack scenarios often involve finding semantic paraphrases of input . |
| Approach: | They propose a model that probabilistically combines context-independent word level information with context-dependent information from BERT's masked language modeling to combat low-level orthographic attacks. |
| Outcome: | The proposed model outperforms a spellchecker and Pruthi's model on a character-level benchmark. |