fBERT: A Neural Transformer for Identifying Offensive Content (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models such as BERT, XLNET, and XLM-R have outperformed other neural architectures and statistical learning methods in the identification of offensive language and hate speech. |
| Approach: | They present a BERT model retrained on SOLID, the largest English offensive language identification corpus available with over 1.4 million offensive instances. |
| Outcome: | The proposed model outperforms models trained on SOLID, the largest English offensive language identification corpus available with over 1.4 million offensive instances. |
Similar Papers
Teacher and Student Models of Offensive Language in Social Media (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to identify offensive language online use large pre-trained transformer models. however, the inference time, disk, and memory requirements of these models are prohibitively large. |
| Approach: | They propose to transfer knowledge from large transformer models to much smaller neural models to make predictions at the token- and post-level. |
| Outcome: | The proposed model performs 100 times better than transformer models but with 100 times less parameters and much less memory usage. |
SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification (2021.findings-acl)
Copied to clipboard
| Challenge: | toxicity, hate speech, cyberbullying, and cyber-aggression are common themes in social media . authors present a dataset that is limited in size and biased towards offensive language . |
| Approach: | They present an expanded dataset that uses a taxonomy for offensive language identification . they show that using SOLID and OLID yields sizable performance gains . |
| Outcome: | The proposed dataset shows that it performs better than the OLID dataset for two different models. |
Divisive Language and Propaganda Detection using Multi-head Attention Transformers with Deep Learning BERT-based Language Models for Binary Classification (D19-50)
Copied to clipboard
| Challenge: | a team of researchers used a pre-trained BERT language model to train propaganda . the model was based on a cloze comprehension test to answer a question about influence operations . |
| Approach: | team used a BERT language model that was pre-trained on Wikipedia and BookCorpus . they used cloze comprehension tests to train the model to answer a propaganda question . |
| Outcome: | The proposed model was trained on Wikipedia and BookCorpus to answer propaganda questions . the team used a neural network that was pre-trained on the Wikipedia and bookCorpus corpus . |
BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection (2021.eacl-main)
Copied to clipboard
| Challenge: | Pretrained transformer-based models are a common choice for identifying drug events from social media texts. |
| Approach: | They propose to compare transformer-based models with in-domain language pretraining to find out which one is better at ADE detection. |
| Outcome: | The proposed models outperform SpanBERT and PubMedBERT on two benchmarks. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models (2020.acl-demos)
Copied to clipboard
| Challenge: | Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. |
| Approach: | They propose a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. |
| Outcome: | Using exBERT, we can analyze the representations and attentions of large language models and extend them to previously not analyzed models. |
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on societal bias in NLP focuses on race and gender . linguistic background is a unique attribute that has been largely ignored in the field . |
| Approach: | They examine linguistic background to craft plausible adversarial examples that expose biases in popular NLP models. |
| Outcome: | The proposed model improves robustness without sacrificing performance on clean data. |
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models of NLP are fading away, but new ones are needed to maintain their dominance. |
| Approach: | They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs. |
| Outcome: | The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task (2020.lrec-1)
Copied to clipboard
| Challenge: | In the last two years, significant improvements have occurred in NLP with the development of large language models using contextualized word embeddings based on the Google Transformer architecture. |
| Approach: | They performed experiments on data from the Fake News Challenge stage 1 (FNC-1) they used BERT sentence embeddings as a model feature and BERT, XLNet, and RoBERTa transformers to fine-tune them. |
| Outcome: | The proposed model outperforms the winner's system on class-wise F1 scores and achieves state-of-the-art on the stance detection task. |