Papers by Orion Weller
You Don’t Have Time to Read This: An Exploration of Document Reading Time Prediction (2020.acl-main)
Copied to clipboard
Orion Weller, Jordan Hildebrandt, Ilya Reznik, Christopher Challis, E. Shannon Tass, Quinn Snell, Kevin Seppi
| Challenge: | Existing work on reading time prediction has focused on word level only predictions . however, previous work has focused only on word levels . |
| Approach: | They perform an experiment to examine how different features of text contribute to the time it takes to read, distributing and collecting data from over a thousand participants. |
| Outcome: | The proposed method combines a large number of machine learning methods with textual and stylistic factors to predict the time it takes to read. |
CLERC: A Dataset for U. S. Legal Case Retrieval and Retrieval-Augmented Analysis Generation (2025.findings-naacl)
Copied to clipboard
Abe Bohan Hou, Orion Weller, Guanghui Qin, Eugene Yang, Dawn Lawrie, Nils Holzenberger, Andrew Blair-Stanek, Benjamin Van Durme
| Challenge: | a dataset of case law is used to train and evaluate models for writing legal analyses . current approaches struggle to find relevant cases and generate legal analyses, authors say . |
| Approach: | They build a dataset of case law to support information retrieval and retrieval-augmented generation. |
| Outcome: | The proposed dataset supports two important backbone tasks: retrieval (IR) and retrieval-augmented generation (RAG). |
When Do Decompositions Help for Machine Reading? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on decompositions of complex questions has focused on multi-step reasoning . but, in machine reading, it is unclear when decomposing is helpful . |
| Approach: | They conduct experiments on decompositions in machine reading to unify recent work . they find that decomposing complex questions can be helpful in zero or limited-data settings . |
| Outcome: | The proposed model can learn decompositions implicitly even with limited data, the study shows . the results are consistent with previous work on decomposing complex questions . |
When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Transfer learning (TL) in natural language processing has seen a surge of interest in recent years . pre-trained models have shown impressive ability to transfer to novel tasks . |
| Approach: | They compare two different methods of transfer learning in natural language processing to find out which is better. |
| Outcome: | The proposed methods perform better when the target task has fewer instances than the supporting task and vice versa. |
Defending Against Disinformation Attacks in Open-Domain Question Answering (2024.eacl-short)
Copied to clipboard
| Challenge: | Existing methods to defend against data poisoning attacks in open-domain question answering are lacking . |
| Approach: | They propose a method that uses query augmentation to find diverse passages that could answer the original question but are less likely to have been poisoned. |
| Outcome: | The proposed method provides gains of nearly 20% exact match across varying levels of data poisoning/knowledge conflicts. |
Streaming Models for Joint Speech Recognition and Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Using end-to-end models for speech translation has become a focus of the ST community . cascaded models have the advantage of including automatic speech recognition output . |
| Approach: | They propose a model that condenses sound waves into translated text and integrates automatic speech recognition outputs into the models. |
| Outcome: | The proposed model is statistically similar to cascading models, but has half the number of parameters. |
CacheNotes: Task-Aware Key-Value Cache Compression for Reasoning-Intensive Knowledge Tasks (2026.eacl-long)
Copied to clipboard
| Challenge: | Current methods for integrating external knowledge into Large Language Models (LLMs) face limitations with broad, multi-source queries, while long-context models are computationally prohibitive. |
| Approach: | They propose a task-aware key-value cache compression method that generates a sequence of CPTs from a corpus and guides a one-time compression of the corpus into a compact, reusable KV cache. |
| Outcome: | The proposed method outperforms Retrieval-Augmented Generation (RAG) on Question-Answering tasks and reduces latency by over 4. |
Pretrained Models for Multilingual Federated Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Federated Learning (FL) is a machine learning technique that trains a model across multiple distributed clients holding local data samples, without ever storing client data in a central location. |
| Approach: | They propose to use pretrained models to study three multilingual language tasks . they also examine impact of non-IID text on FL in naturally occurring data . |
| Outcome: | The proposed methods perform better than centralized learning even when using non-IID partitioning. |
Humor Detection: A Transformer Gets the Last Laugh (D19-1)
Copied to clipboard
| Challenge: | Existing methods to identify humor in text have been limited to identifying humor in the text. |
| Approach: | They propose a model that learns to identify humorous jokes based on Reddit ratings, and employ a Transformer architecture to learn from sentence context. |
| Outcome: | The proposed model outperforms previous work on humor identification tasks with an F-measure of 93.1% for the Puns dataset and 98.6% on the Short Jokes dataset. |
Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic (2024.emnlp-main)
Copied to clipboard
Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme
| Challenge: | Recent language models allow structured reasoning with text, but lack of a clear protocol for discerning entailment causes noisy datasets and limited performance gains. |
| Approach: | They propose a consistent approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference. |
| Outcome: | The proposed approach has higher internal consistency than prior decompositional entailment datasets and significantly improves proof quality and accuracy. |
NevIR: Negation in Neural Information Retrieval (2024.eacl-long)
Copied to clipboard
| Challenge: | Negation is a common everyday phenomenon and has been a consistent area of weakness for language models. |
| Approach: | They ask IR models to rank two documents that differ only by negation . they find that most current information retrieval models do not consider negation. |
| Outcome: | The proposed benchmarks show that most current models do not consider negation . the results are similar to those found in the literature, but are poorer than random ranking . |
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) may hallucinate and generate false information despite pre-training on factual data. |
| Approach: | They propose a new evaluation metric that measures the extent to which model-produced answers are directly found in underlying text corpora. |
| Outcome: | The proposed evaluation metric measures the extent to which model-produced answers are directly found in underlying text corpora. |
The rJokes Dataset: a Large Scale Humor Collection (2020.lrec-1)
Copied to clipboard
| Challenge: | Humor is a complex language phenomenon that depends upon many factors, including topic, date, and recipient. |
| Approach: | They compile a large scale humor dataset from the Reddit r/Jokes subreddit. |
| Outcome: | The proposed dataset provides quantitative metrics for the level of humor in each joke, as determined by subreddit user feedback. |
Exploring the Relationship Between Algorithm Performance, Vocabulary, and Run-Time in Text Classification (2021.naacl-main)
Copied to clipboard
| Challenge: | Many text classification algorithms depend on the size of the corpus’ vocabulary due to their bag-of-words representation. |
| Approach: | They propose to evaluate how preprocessing techniques affect the run-time of models by evaluating ten techniques over four models and two datasets. |
| Outcome: | The proposed methods can reduce run-time with no loss of accuracy while sacrificing up to 65%. |
FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions (2025.naacl-long)
Copied to clipboard
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini
| Challenge: | Modern language models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests. |
| Approach: | They propose a dataset that contains an instruction evaluation benchmark and a training set to help IR models learn to follow instructions. |
| Outcome: | The proposed model improves after fine-tuning on a training set and rigorous instruction evaluation benchmark. |
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2025.acl-long)
Copied to clipboard
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, Iacopo Poli
| Challenge: | Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks compared to larger decoder models. |
| Approach: | They introduce a new transformer model, ModernBERT, which brings modern model optimizations to encoder-only transformer models. |
| Outcome: | The proposed model improves on the BERT transformer model and is faster and more memory efficient than the older models. |
When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets (2024.findings-eacl)
Copied to clipboard
| Challenge: | Using large language models (LMs) for query or document expansion can improve generalization in information retrieval. |
| Approach: | They conduct the first comprehensive analysis of large language models (LMs) for query or document expansion. |
| Outcome: | The proposed expansions improve retrieval performance for weaker models but harm stronger models. |
Learning from Task Descriptions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in supervised NLP has shown significant progress in learning tasks from examples. |
| Approach: | They propose a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. |
| Outcome: | The proposed model achieves 12% on the new dataset, leaving a significant challenge for NLP researchers. |
End-to-End Speech Translation for Code Switched Speech (2022.findings-acl)
Copied to clipboard
Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, Matthias Paulik
| Challenge: | Code switching (CS) is the phenomenon of interchangeably using words and phrases from different languages. |
| Approach: | They propose a new ST corpus that extends the joint transcription and translation setup. |
| Outcome: | The proposed model performs well even when no training data is used. |