Papers by Alexander Wettig
Adapting Language Models to Compress Contexts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based language models have a finite context window and expensive computational cost of processing long text documents. |
| Approach: | They propose to adapt pre-trained LMs into AutoCompressors to compress text into summary vectors . authors propose to use summary vector to speed up inference over long contexts based on a finite context window . |
| Outcome: | The proposed model can compress long contexts into summary vectors, which are accessible as soft prompts. |
Should You Mask 15% in Masked Language Modeling? (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language models (MLMs) traditionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations. |
| Approach: | They revisit the 15% masking rate of MLMs to examine the role of masking in linguistic training. |
| Outcome: | The proposed masking rate outperforms BERT-large size models on GLUE and SQUAD while maintaining 95% accuracy. |
Poisoning Retrieval Corpora by Injecting Adversarial Passages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dense retrievers have outperformed traditional lexical methods in a range of information retrieval tasks, but to what extent can they be safely deployed in real-world applications? |
| Approach: | They propose a method where a malicious user injects a small number of adversarial passages into a retrieval corpus to maximize similarity with a set of training queries. |
| Outcome: | The proposed attack fools retrieval systems into returning top results for queries not seen by the attacker. |
How to Train Long-Context Language Models (Effectively) (2025.acl-long)
Copied to clipboard
| Challenge: | a new study shows that language models can process extremely long contexts with minimal training. |
| Approach: | They use supervised fine-tuning and continued training to evaluate a language model's long-context capabilities. |
| Outcome: | The proposed model outperforms Llama-3.1-8B-Instruct on most long-context tasks . the model can process 512K tokens, one of the longest context windows of LMs . |
Phrase Retrieval Learns Passage Retrieval, Too (2021.emnlp-main)
Copied to clipboard
| Challenge: | Dense retrieval methods have shown great promise over sparse methods in a range of NLP problems. |
| Approach: | They propose to use dense phrase retrieval to learn coarse-level retrieval including passages . they show phrase retrievals can be fine-tuned for more coarse-grained retrieval units . |
| Outcome: | The proposed method improves passage retrieval accuracy and QA performance with fewer passages. |
Finding Dataset Shortcuts with Grammar Induction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on shortcut detection focused on enumerating features like unigrams or bigrams . prior work relied on post-hoc models that reveal qualitative patterns without a clear statistical interpretation . |
| Approach: | They propose to use probabilistic grammars to characterize and discover shortcuts in NLP datasets using context-free grammars and synchronous context- free grammars. |
| Outcome: | The proposed grammars reveal interesting shortcut features in a number of datasets, including simple and high-level features, and automatically identify groups of test examples on which conventional classifiers fail. |