Papers with PDR
Improved Language Modeling by Decoding the Past (P19-1)
Copied to clipboard
| Challenge: | Existing methods to improve language modeling performance are based on regularized LSTMs with a large number of parameters and training time. |
| Approach: | They propose a method that decodes the last token in context using the predicted distribution of the next token. |
| Outcome: | The proposed method improves perplexity on the Penn Treebank dataset by 1.8 points and 2.3 points on the WikiText-2 datasets. |
PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection (2026.acl-long)
Copied to clipboard
| Challenge: | Existing likelihood-based methods for detecting pretraining data are limited in black-box, zero-shot settings. |
| Approach: | They propose a training-free and plug-and-play framework that reweights token-level scores to amplify distinct signals from early positions while suppressing noise from later ones. |
| Outcome: | The proposed framework amplifys signals from early positions while suppressing noise from later positions. |