Evaluating ChatGPT against Functionality Tests for Hate Speech Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models like ChatGPT have shown a great promise in detecting hate speech, but they lack the capability to perform in a holistic fashion. |
| Approach: | They evaluate the ChatGPT model's strengths and weaknesses by performing functional tests across 11 languages to uncover their weaknesses. |
| Outcome: | The proposed model performs poorly across 11 languages and is based on functional tests. |
Similar Papers
HateCheck: Functional Tests for Hate Speech Detection Models (2021.acl-long)
Copied to clipboard
| Challenge: | Hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. |
| Approach: | They propose a suite of functional tests for hate speech detection models that measure model performance on held-out test data and then craft test cases to validate their quality. |
| Outcome: | The proposed tests show that the proposed models perform poorly on a small set of widely-used hate speech datasets. |
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter (2025.acl-long)
Copied to clipboard
Manuel Tonneau, Diyi Liu, Niyati Malhotra, Scott A. Hale, Samuel Fraiberger, Victor Orozco-Olvera, Paul Röttger
| Challenge: | Prior work on automated hate speech detection models has been limited due to systematic biases in evaluation datasets and poor performance across geographies. |
| Approach: | They propose to construct a global hate speech dataset representative of social media settings from tweets posted on September 21, 2022. |
| Outcome: | The proposed dataset covers eight languages and four English-speaking countries and covers eight countries where English is the main language on Twitter. |
Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanish (2025.findings-naacl)
Copied to clipboard
| Challenge: | Hate speech detection deals with many language variants, slang, nuances, and cultural nuances. |
| Approach: | They propose to use large language models to detect hate speech in Rioplatense Spanish . they compare their results to those of a state-of-the-art BERT classifier . |
| Outcome: | The proposed models show lower precision than the state-of-the-art classifier, but are sensitive to highly nuanced cases. |
GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | HateCheck test cases are generic and have simplistic sentence structures that do not match the real-world data. |
| Approach: | They propose a framework to generate more diverse and realistic functional tests from scratch by instructing large language models. |
| Outcome: | The proposed framework generates more diverse and realistic functional tests from scratch by instructing large language models (LLMs). |
HateCheckHIn: Evaluating Hindi Hate Speech Detection Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Hate speech detection models are evaluated on a held-out test data, but they are incapable of identifying weaknesses. |
| Approach: | They propose to use multilingual hate speech detection models to evaluate their performance on social media conversation. |
| Outcome: | The proposed model can detect hate speech in multiple languages using a real-world conversation on social media. |
Probing LLMs for hate speech detection: strengths and vulnerabilities (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent efforts to detect hateful or toxic language using large language models have not used explanation, additional context and victim community information in the detection process. |
| Approach: | They use different prompt variations, input information and victim community information to evaluate large language models in zero shot setting without adding any in-context examples. |
| Outcome: | The proposed models perform significantly better when included in the pipeline than baseline models. |
Delving into Qualitative Implications of Synthetic Data for Hate Speech Detection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on synthetic data for training models for NLP tasks reports mixed results on subjective tasks such as hate speech detection. |
| Approach: | They propose to use synthetic data to train models for highly subjective tasks such as hate speech detection to investigate the potential and specific pitfalls of using it. |
| Outcome: | The proposed model outperforms models trained with real data on hate speech detection tasks, but it fails to accurately reflect real-world data on linguistic dimensions and results in different class distributions. |
Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-Based Hate (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing models for detecting hate expressed with emojis have weaknesses when used for sensitive applications such as content moderation. |
| Approach: | They propose a test suite of 3,930 short-form statements that evaluates hateful language expressed with emoji. |
| Outcome: | The proposed model performs better on emoji-based hate while maintaining strong performance on text-only hate. |
GPTEval: A Survey on Assessments of ChatGPT and GPT-4 (2024.lrec-main)
Copied to clipboard
| Challenge: | emergence of ChatGPT has generated speculation about its potential to disrupt social and economic systems. |
| Approach: | They analyze prior assessments of ChatGPT and GPT-4 to analyze their language and reasoning abilities, scientific knowledge, ethical considerations and existing evaluation methods. |
| Outcome: | The proposed model performs satisfactorily in science knowledge and can answer open questions. |