Papers with GPT2-XL
SentSpace: Large-Scale Benchmarking and Evaluation of Text using Cognitively Motivated Lexical, Syntactic, and Semantic Features (2022.naacl-demo)
Copied to clipboard
| Challenge: | SentSpace provides a framework for streamlined evaluation of textual input. |
| Approach: | They describe the design of SentSpace and demonstrate an example use case . they use a web interface for interactive visualization and comparison with large corpora . |
| Outcome: | The framework provides a common framework for evaluation and visualization. |
JAMDEC: Unsupervised Authorship Obfuscation using Constrained Decoding over Small Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to protect the identity and privacy of online authorship are lacking supervision data for diverse authorship and domains. |
| Approach: | They propose an unsupervised inference-time approach to authorship obfuscation that uses a user-controlled, inference time algorithm to oblige the authorship. |
| Outcome: | The proposed method outperforms state-of-the-art methods while performing competitively against a propriety model two orders of magnitudes larger. |
On Localizing and Deleting Toxic Memories in Large Language Models (2025.findings-naacl)
Copied to clipboard
Anubrata Das, Manoj Kumar, Ninareh Mehrabi, Anil Ramakrishna, Anna Rumshisky, Kai-Wei Chang, Aram Galstyan, Morteza Ziyadi, Rahul Gupta
| Challenge: | Existing methods to reduce toxic generation in large language models are not fully understood. |
| Approach: | They propose to understand the mechanisms that drive toxic generation in large language models by using memory localization to reduce toxic generation. |
| Outcome: | The proposed method reduces toxic generation from 62.86% to 28.61%, but it also improves generation quality. |
Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompting has gained tremendous attention as an efficient method for the adaptation of large-scale language models. |
| Approach: | They propose a regularization method that guides a prompt to produce a task context properly. |
| Outcome: | The proposed method improves prediction performance in a zero-shot in-context learning setting without demonstration examples for in-constitu learning. |
WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods focus on single editing, failing to meet the requirements for lifelong editing. |
| Approach: | They propose an approach that selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models. |
| Outcome: | The proposed method improves on GPT2-XL and GPT-J in lifelong editing compared to state-of-the-art methods . |
Subtle Signatures, Strong Shields: Advancing Robust and Imperceptible Watermarking in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have led to an increase in AI-generated text on the Internet, presenting a crucial challenge to differentiate AI-created content from human-written text. |
| Approach: | They propose a novel approach to embed watermarks into LLMs that leverages token prior probabilities to improve detectability and maintain watermark imperceptibility. |
| Outcome: | The proposed method improves detectability and imperceptibility of watermarks by partitioning tokens into two distinct groups based on prior probabilities and employing tailored strategies for each group. |
Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a promising capability for large language models (LLMs) but its underlying mechanism remains unexplored. |
| Approach: | They propose a demonstration compression technique to expedite inference and an analysis framework for diagnosing ICL errors in GPT2-XL. |
| Outcome: | The proposed method improves ICL performance and expedites inference. |
Lifelong Model Editing with Graph-Based External Memory (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for post-training model editing suffer from overfitting and catastrophic forgetting. |
| Approach: | They propose a framework that leverages hyperbolic geometry and graph neural networks for precise and stable model edits. |
| Outcome: | Experiments on CounterFact, CounterFACT+, and MQuAKE with GPT2-XL and GPT-J show that HYPE significantly enhances edit stability, factual accuracy, and multi-hop reasoning. |
Detoxification for LLM: From Dataset Itself (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for large language models focus on post-training stage or inference time, while few tackle the source of toxicity, namely, the dataset itself. |
| Approach: | They propose to localize and rewrite toxic spans in raw corpora with SoCD, which guides an LLM to localized and preserving semantics while preserving toxicity. |
| Outcome: | The proposed method reduces TP from 0.42 to 0.18 and Expected Maximum Toxicity (EMT) from 0.43 to 0.20 on three LLMs. |