From the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | a dataset of English posts with annotations of toxic spans is released . sequence labeling models perform best, but rationale extraction methods are promising . |
| Approach: | They propose a dataset for toxic spans detection that includes an annotation of toxic posts . they propose to add generic rationale extraction mechanisms to the model to obtain toxic span information . |
| Outcome: | The proposed framework is based on a dataset of English posts with toxic span annotations . it shows that sequence labeling models perform best, but that rationale extraction methods are promising . |
Similar Papers
Transfer Learning and Prediction Consistency for Detecting Offensive Spans of Text (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing models for toxic span detection only classify text snippets as offensive or not . a novel model seeks to simultaneously predict offensive words and opinion phrases . |
| Approach: | They propose a novel model that seeks to predict offensive words and opinion phrases simultaneously . they also introduce a regularization mechanism to encourage consistency of the model predictions . |
| Outcome: | The proposed model performs well compared to baselines on toxic span detection tasks . it predicts offensive words and opinion phrases to leverage inter-dependencies . |
ToVo: Toxicity Taxonomy via Voting (2025.findings-naacl)
Copied to clipboard
Tinh Son Luong, Thanh-Thien Le, Thang Viet Doan, Linh Ngo Van, Thien Huu Nguyen, Nguyen Thi Ngoc Diep
| Challenge: | Existing toxic content detection models face limitations due to the closed-source nature of training data and the paucity of explanations for their evaluation mechanism. |
| Approach: | They propose a mechanism that integrates voting and chain-of-thought processes to produce a high-quality open-source dataset for toxic content detection. |
| Outcome: | The proposed model improves transparency and customizability while facilitating better fine-tuning for specific use cases. |
ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | toxicity detection has been largely based on social media content, leaving the unique challenges inherent to real-world user-AI interactions insufficiently explored. |
| Approach: | They propose a benchmark to detect toxicity in real-world user-AI conversations . they compare existing models with social media content to find toxicity . |
| Outcome: | The proposed benchmark reveals that existing models fail to recognize toxicity in real-world user-AI conversations. |
Performance and Risk Trade-offs for Multi-word Text Prediction at Scale (2023.findings-eacl)
Copied to clipboard
Aniket Vashishtha, S Sai Prasad, Payal Bajaj, Vishrav Chaudhary, Kate Cook, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury
| Challenge: | Large Language Models (LLMs) generate ethically inappropriate texts even for seemingly innocuous contexts. |
| Approach: | They propose to use large language models to detect and filter toxic content in text prediction tasks by evaluating their toxicity detection approaches against a manually crafted CheckList of harms. |
| Outcome: | The proposed methods are compared against a checklist of harms targeted at different groups and different levels of severity in English. |
Enhancing LLM-based Hatred and Toxicity Detection with Meta-Toxic Knowledge Graph (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to address toxicity issues with large language models are inadequate . lack of domain-specific knowledge leads to false negatives and excessive sensitivity to toxic speech limits freedom of speech. |
| Approach: | They propose a method that leverages graph search on a meta-toxic knowledge graph to enhance hatred and toxicity detection. |
| Outcome: | The proposed method lowers false positive rate and improves toxicity detection performance in out-of-domain scenarios. |
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021.eacl-main)
Copied to clipboard
| Challenge: | et al., 2018a): a poor phrasing may make the conversation go awry. |
| Approach: | They propose a model that can help suggest rephrasings of toxic comments in a more civil manner. |
| Outcome: | The proposed model generates sentences that are more fluent and better at preserving the initial content compared to earlier systems and human evaluation. |
MUDES: Multilingual Detection of Offensive Spans (2021.naacl-demos)
Copied to clipboard
| Challenge: | Identifying offensive spans in texts is the goal of the SemEval-2021 Task 5: Toxic Spans Detection . previous work focused on post level annotations, but identifying offensive span is useful in many ways. |
| Approach: | They propose a Python-based system to detect offensive spans in texts with pre-trained models and a user-friendly web-based interface. |
| Outcome: | The proposed system is based on a Python-based framework and a user-friendly web-based interface. |
♪ Something Just Like TRuST ♪ *: Toxicity Recognition of Span and Target (2026.findings-acl)
Copied to clipboard
| Challenge: | Toxic language is pervasive online, and because LLMs are trained on web data, it generates such content. |
| Approach: | They propose a large-scale dataset that synthesizes toxicity definitions and an annotation scheme . they use a rigorous human annotation process to evaluate the diversity of the annotations . |
| Outcome: | The proposed model outperforms existing models on three tasks and is not reliable. |
Detecting Community Sensitive Norm Violations in Online Conversations (2021.findings-emnlp)
Copied to clipboard
Chan Young Park, Julia Mendelsohn, Karthik Radhakrishnan, Kinjal Jain, Tushar Kanakagiri, David Jurgens, Yulia Tsvetkov
| Challenge: | Existing efforts to identify unacceptable behavior have focused on toxicity as the sole form of community norm violation. |
| Approach: | They propose a dataset that focuses on a more complete spectrum of community norms and their violations in local conversational and global contexts. |
| Outcome: | The proposed model improves the detection of community norm violations in local conversational and global contexts. |
ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for detecting harmful content lack diversity and quality of datasets. |
| Approach: | They propose a framework for synthesizing toxic information from social media datasets . their framework generates a wide variety of synthetic, yet remarkably realistic, examples of toxic information . |
| Outcome: | The proposed framework can generate a wide variety of synthetic, yet remarkably realistic, examples of toxic information. |