Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset (2024.naacl-long)
Copied to clipboard
| Challenge: | Hate speech detection models are only as good as the data they are trained on, but adversarial datasets are slow and costly . data sourced from social media suffer from systematic gaps and biases, leading to unreliable models with simplistic decision boundaries. |
| Approach: | They propose a German Adversarial Hate speech Dataset comprising 11k examples . they explore new strategies for supporting annotators and provide manual analysis of disagreements for each strategy . |
| Outcome: | The proposed dataset is challenging even for state-of-the-art hate speech detection models and it significantly improves model robustness. |
Similar Papers
HateGAN: Adversarial Generative-Based Data Augmentation for Hate Speech Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to detect online hate speech depend heavily on labeled datasets for training, which results in poor detection performance of the hate speech class. |
| Approach: | They propose a deep generative reinforcement learning model which augments two commonly-used hate speech detection datasets with the HateGAN generated tweets. |
| Outcome: | The proposed model improves the detection performance of hate speech class regardless of the classifiers and datasets used in the detection task. |
Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets for hate speech detection neglect the cultural diversity within a single language. |
| Approach: | They propose a CR**oss-cultural **E**nglish **Hate* speech dataset that uses culturally hateful keywords to identify posts from four countries plus the United States. |
| Outcome: | The proposed dataset shows that only 56.2% of the posts in CREHate achieve consensus among all countries, with the highest pairwise label difference rate of 26%. |
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021.acl-long)
Copied to clipboard
| Challenge: | Detecting online hate speech has proven difficult and concerns raised about performance, robustness, generalisability and fairness of stateof-the-art models. |
| Approach: | They propose a human-and-model-in-the-loop process for dynamically generating datasets and training better performing hate detection models. |
| Outcome: | The proposed model improves on a dataset of 40,000 hateful entries . the model is harder for annotators to trick and better on HateCheck . |
Generating Counter Narratives against Online Hate Speech: Data and Strategies (2020.acl-main)
Copied to clipboard
| Challenge: | Hate Speech (HS) is a pervasive issue that spreads quickly and widely . research has focused on avoiding undesired effects that come with content moderation . |
| Approach: | They propose to use large scale unsupervised language models to generate responses to hate effectively using large scale models. |
| Outcome: | The proposed methods lack quality data and produce generic/repetitive responses. |
Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages (2022.emnlp-main)
Copied to clipboard
| Challenge: | Hate speech datasets focus on English-language content, hindering effective models . annotating hateful content is expensive, time-consuming and potentially harmful to annotators. |
| Approach: | They propose to use ISO 639-1 codes to fine-tune models on one source language and apply them to another language. |
| Outcome: | The proposed approach performs well on some tasks, but fails on many others. |
Improving Hate Speech Detection with Deep Learning Ensembles (L18-1)
Copied to clipboard
| Challenge: | censorship is a potential risk when addressing these issues with automated text classification methods. |
| Approach: | They propose to use a neural network-based ensemble method to better classify hate speech using a publicly available embedding model and a popular sentiment dataset. |
| Outcome: | The proposed method improves by 5 points on a hate speech corpus from Twitter and a popular sentiment dataset. |
A Benchmark Dataset for Learning to Intervene in Online Hate Speech (D19-1)
Copied to clipboard
| Challenge: | Existing methods to detect online hate speech ignore conversational context . generative hate speech intervention is a novel approach to counter online hate . |
| Approach: | They propose a task where generative hate speech intervention generates responses to intervene during online conversations that contain hate speech. |
| Outcome: | The proposed method can detect and block hate speech and discourage it . it can also generate responses written by Mechanical Turk workers . |
Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study shows that many definitions are being used for equivalent concepts, making most datasets incompatible. |
| Approach: | They analyze six publicly available datasets to determine their similarity and compatibility . they propose to use Fast Text word vectors to analyze similarity between different datasets . |
| Outcome: | The proposed model performs better on similar datasets and worse on more non-offensive samples. |
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | supervised learning is a key component of offensive language detection, but there is little attention given to the quality of annotated data. |
| Approach: | They propose to examine the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. |
| Outcome: | The proposed datasets show that annotators' agreement has a strong effect on classifiers performance and robustness. |
Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments (2022.findings-acl)
Copied to clipboard
| Challenge: | XTREMESPEECH dataset contains 20,297 social media passages from Brazil, Germany, India and Kenya . |
| Approach: | They propose a hate speech dataset containing 20,297 social media passages from Brazil, Germany, India and Kenya. |
| Outcome: | The proposed dataset contains 20,297 social media passages from Brazil, Germany, India and Kenya. |