Papers with GPT-2-XL
On Localizing and Deleting Toxic Memories in Large Language Models (2025.findings-naacl)
Copied to clipboard
Anubrata Das, Manoj Kumar, Ninareh Mehrabi, Anil Ramakrishna, Anna Rumshisky, Kai-Wei Chang, Aram Galstyan, Morteza Ziyadi, Rahul Gupta
| Challenge: | Existing methods to reduce toxic generation in large language models are not fully understood. |
| Approach: | They propose to understand the mechanisms that drive toxic generation in large language models by using memory localization to reduce toxic generation. |
| Outcome: | The proposed method reduces toxic generation from 62.86% to 28.61%, but it also improves generation quality. |
How Much Syntactic Supervision is “Good Enough”? (2023.findings-eacl)
Copied to clipboard
| Challenge: | RNNGs with syntactic supervision underperformed RNNs with some syntaktic supervision, whereas RNNS with mild supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. |
| Approach: | They propose a method where syntactic LMs are gradually ablated from full syntatic supervision to zero syntastic supervision by preserving NP, VP, PP, SBAR nonterminal symbols. |
| Outcome: | The proposed method underperforms the RNNGs with zero syntactic supervision, and the LMs with mild syntaktic supervision perform better than the state-of-the-art GPT-2-XL. |
Normalized Narrow Jump To Conclusions: Normalized Narrow Shortcuts for Parameter Efficient Early Exit Transformer Prediction (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that shortcutting pre-trained transformers to final representations can be cheaper and improve model performance but can also increase computational costs. |
| Approach: | They propose Narrow Jump to Conclusions and Normalized Narrow jump to conclusions that reduce shortcut parameter count by over 97%. |
| Outcome: | The proposed approaches outperform Identity shortcuts at early stages and offer stable precision from all transformer block levels for GPT-2-XL, Phi3-Mini and Llama2-7B models. |
Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability (2025.coling-main)
Copied to clipboard
| Challenge: | a new study explores how large language models capture aspects of human linguis-tic ability . large language model performance is limited by the mechanisms behind their performance . |
| Approach: | They employ psycholinguistic paradigms to explore neuron-level representations in language models . they found that large language models exhibit human-like abilities in three tasks . |
| Outcome: | The proposed model shows human-like abilities in sound-shape association, gender association and implicit causality. |