Papers by Houjun Liu
Drop Dropout on Single Epoch Language Model Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Initial dropout was seen as a breakthrough regularization technique that reduced overfitting, yet single-epoch pretraining tasks common to modern LLMs yield minimal overfit. |
| Approach: | They propose to use dropout during single-epoch pretraining to reduce overfitting in language modeling, morpho-syntax, question answering, and MNLI to improve performance. |
| Outcome: | The results show that dropout is not used in large LLMs and improves performance in language modeling, morpho-syntax, question answering, and MNLI. |
ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming approaches prioritize high attack success rate, resulting in high-perplexity prompts. |
| Approach: | a new method uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate. |
| Outcome: | ASTPrompter achieves 5.1 times higher attack success rate on Llama-8.1B . low-perplexity attacks are more difficult to filter and more likely to arise during benign usage . |