Papers by William Yeh
Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are ubiquitous in user-facing applications, yet they still generate undesirable toxic outputs, including profanity, vulgarity, and derogatory remarks. |
| Approach: | They leverage sparse autoencoders to identify toxicity-related directions in residual stream of large language models and perform targeted activation steering using the corresponding decoder vectors. |
| Outcome: | The proposed models surpass baselines in reducing toxicity by up to 20%, though fluency can degrade noticeably on GPT-2 Small and Gemma-2-2B. |