Papers by Doug Downey
Intent-aware Schema Generation and Refinement for Literature Review Tables (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ambiguity in reference-based evaluations and lack of editing/refinement methods have slow progress on schema generation. |
| Approach: | They propose a method for augmenting unannotated table corpora with synthesized intents . they propose prompted workflows and fine-tuned models to improve schema generation . |
| Outcome: | The proposed approach significantly improves baseline performance in reconstructing reference schemas. |
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating scientific document representations fail to capture the diversity of relevant tasks. |
| Approach: | They propose a benchmark for training and evaluating scientific document representations that includes 24 challenging and realistic tasks across four formats: classification, regression, ranking and search. |
| Outcome: | The proposed model outperforms existing models by over 2 points absolute. |
Sampling Informative Training Data for RNN Language Models (P18-3)
Copied to clipboard
| Challenge: | Existing methods for selecting training data for recurrent neural network language models are limited by computational costs. |
| Approach: | They propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. |
| Outcome: | The proposed approach outperforms models trained on the Billion Word and Wikitext-103 benchmark corpora. |
Extracting Commonsense Properties from Embeddings with Limited Human Guidance (P18-2)
Copied to clipboard
| Challenge: | Existing methods for learning common sense from text require dozens of hand-annotated frames to connect the property to how it is indirectly reflected in text. |
| Approach: | They propose a method for extracting object-property comparisons from pre-trained embeddings. |
| Outcome: | The proposed approach exceeds previous work but requires less hand-annotated knowledge. |
ABNIRML: Analyzing the Behavior of Neural IR Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc ranking. |
| Approach: | They propose a framework for Analyzing the Behavior of Neural IR ModeLs that includes new types of diagnostic probes that allow us to test several characteristics that are not addressed by previous techniques. |
| Outcome: | The proposed framework tests writing styles, factuality, sensitivity to paraphrasing and word order, and can be used to identify unintended biases. |
Generative Data Augmentation for Commonsense Reasoning (2020.findings-emnlp)
Copied to clipboard
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey
| Challenge: | Recent advances in commonsense reasoning depend on large-scale human-authored training data. |
| Approach: | They propose a generative data augmentation technique that augments human-authored training data by using pretrained language models. |
| Outcome: | The proposed technique outperforms existing methods on commonsense reasoning benchmarks and enhances out-of-distribution generalization. |
PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents (2023.emnlp-demo)
Copied to clipboard
Kyle Lo, Zejiang Shen, Benjamin Newman, Joseph Chang, Russell Authur, Erin Bransom, Stefan Candra, Yoganand Chandrasekhar, Regan Huff, Bailey Kuehl, Amanpreet Singh, Chris Wilhelm, Angele Zamarron, Marti A. Hearst, Daniel Weld, Doug Downey, Luca Soldaini
| Challenge: | Existing tools for working with scientific documents are limited and documents are often in difficult-to-use PDF formats. |
| Approach: | They propose an open-source Python toolkit for analyzing and processing visually-rich scientific documents. |
| Outcome: | PaperMage provides turn-key recipes for common scientific document processing use-cases. |
Few-Shot Self-Rationalization with Natural Language Prompts (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that generate free-text explanations for tasks are limited by human-written explanations. |
| Approach: | They propose to use a standardized collection of natural language prompts to create a model that generates free-text explanations for tasks. |
| Outcome: | The proposed model can predict task labels and generate free-text explanations for predictions . plausibility of human explanations is 76%, while human explanation is 51% . |
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews (2024.acl-long)
Copied to clipboard
| Challenge: | Existing systems that can interpret complex writing feedback and edit documents in response are limited on the most demanding writing tasks. |
| Approach: | They propose to use peer feedback to revise scientific papers based on peer feedback . they provide labels linking each reviewer comment to the specific paper edits made by the author . |
| Outcome: | The proposed model fails to identify which edits correspond to a comment and the original paper. |
SciMON: Scientific Inspiration Machines Optimized for Novelty (2024.acl-long)
Copied to clipboard
| Challenge: | Existing literature-based hypothesis generation models focus on binary link prediction, limiting expressivity of hypotheses. |
| Approach: | They propose a framework that uses literature-based hypothesis generation as input . they use literature-derived literature as background and output natural language ideas . |
| Outcome: | The proposed model improves the ability of language models to generate new scientific directions grounded in literature. |
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on how to encode compositional task structure has been limited by semantic parsing and multihop reasoning for the purpose of Q&A. |
| Approach: | They propose an approach to decomposing a target task into component tasks and fine-tuning smaller LMs on a curriculum of such component tasks. |
| Outcome: | The proposed approach outperforms end-to-end learning even with equal data, and gets better as more component tasks are modeled. |
CARE: Extracting Experimental Findings From Clinical Literature (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing annotation schemas and datasets fail to capture real-world complexity and nuance of experimental findings. |
| Approach: | They propose a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes. |
| Outcome: | The proposed schema captures fine-grained findings as n-ary relations between entities and attributes. |
Estimating Marginal Probabilities of n-grams for Recurrent Neural Language Models (D18-1)
Copied to clipboard
| Challenge: | Recurrent neural network language models (RNNs) only estimate probabilities for complete sequences of text, whereas some applications require context-independent phrase probabilities instead. |
| Approach: | They propose a method to alter the RNNLM training to make it more accurate at marginal estimation. |
| Outcome: | The proposed method is effective compared to baselines including the traditional RNNLM probability and importance sampling approach. |
Construction of the Literature Graph in Semantic Scholar (N18-3)
Copied to clipboard
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, Oren Etzioni
| Challenge: | Fig. 1 summarizes a scalable system for organizing published scientific literature into a heterogeneous graph . authors describe methods used to enable semantic features in www.semanticscholar.org . |
| Approach: | They describe a scalable system for organizing published scientific literature into a heterogeneous graph to facilitate algorithmic manipulation and discovery. |
| Outcome: | The proposed system can be deployed on a scalable platform and report empirical results for each task. |
Ai2 Scholar QA: Organized Literature Synthesis with Attribution (2025.acl-demo)
Copied to clipboard
Amanpreet Singh, Joseph Chee Chang, Dany Haddad, Aakanksha Naik, Jena D. Hwang, Rodney Kinney, Daniel S Weld, Doug Downey, Sergey Feldman
| Challenge: | Ai2 Scholar QA is a free online scientific question answering application . it uses retrieval-augmented generation to answer complex scientific questions . many of these systems are expensive to use and closed-source . |
| Approach: | They propose a retrieval-augmented generation-based scientific question answering application . it uses a Python package and an interactive web app to make the entire pipeline public . they compare it with other similar question-answering applications . |
| Outcome: | The proposed system outperforms other systems on a recent scientific QA benchmark. |
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. |
| Approach: | They propose a method that explicitly models VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve extraction accuracy. |
| Outcome: | The proposed methods show that inserting special tokens denoting layout group boundaries can lead to a 1.9% Macro F1 improvement in token classification. |
Penguins Don’t Fly: Reasoning about Generics through Instantiations and Exceptions (2023.eacl-main)
Copied to clipboard
| Challenge: | Generics express generalizations about the world that are not universally true . commonsense knowledge bases encode some generic knowledge but rarely enumerate exceptions . |
| Approach: | They propose a framework informed by linguistic theory to generate exemplars for generics . they generate 19k exemplar cases for 650 generics and show they outperform a strong baseline . |
| Outcome: | The proposed framework outperforms a baseline framework by 12.8 precision points. |
Generating Literature-Driven Scientific Theories at Scale (2026.acl-long)
Copied to clipboard
| Challenge: | Contemporary automated scientific discovery systems focus on generating experiments, but higher-level activities such as theory building remain underexplored. |
| Approach: | They propose to synthesize theories from scientific literature using literature-grounding versus parametric knowledge. |
| Outcome: | The proposed method matches existing evidence better than parametric LLM memory generation. |
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation (2023.acl-long)
Copied to clipboard
Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi
| Challenge: | Empirical results suggest that scale is not the only way to build commonsense capabilities. |
| Approach: | They propose a commonsense distillation framework that can achieve a competitive level of commonsensing without relying on the benefits of scale. |
| Outcome: | The proposed framework breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model’s own enhanced commonsense acquisition capabilities. |
ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts (2022.emnlp-demos)
Copied to clipboard
Sonia Murthy, Kyle Lo, Daniel King, Chandra Bhagavatula, Bailey Kuehl, Sophie Johnson, Jonathan Borchardt, Daniel Weld, Tom Hope, Doug Downey
| Challenge: | Current systems that automatically define unfamiliar terms only surface a single "best" description for all users, which may not be accessible for all readers, given varying background knowledge. |
| Approach: | They propose an end-to-end system that generates sets of descriptions of scientific concepts . ACCoRD corpus includes 1,275 labeled contexts and 1,787 expert-authored concept descriptions . |
| Outcome: | The proposed system produces diverse descriptions of concepts in terms of reference concepts. |
Stolen Probability: A Structural Weakness of Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Neural Network Language Models (NNLMs) generate probability distributions by using a softmax function. |
| Approach: | They propose to use a dot-product distance metric to generate probability distributions . they show that the dot product distance meter introduces a limitation bounding expressiveness . |
| Outcome: | The proposed model optimizes well with the inductive bias of NNLMs, but it results in suboptimal ordering of the embedding space. |
Using Large Corpus N-gram Statistics to Improve Recurrent Neural Language Models (N19-1)
Copied to clipboard
| Challenge: | a technique that uses large corpus n-gram statistics as a regularizer for training a neural network LM on a smaller corpus is effective, and more time-efficient than training on ngrams. |
| Approach: | They propose a technique that uses large corpus n-gram statistics as a regularizer for training on a smaller corpus. |
| Outcome: | The proposed technique is effective and more time-efficient than training on a larger corpus. |
S2abEL: A Dataset for Entity Linking from Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity linking (EL) is a longstanding problem in natural language processing and information extraction. |
| Approach: | They propose a neural baseline method for EL on scientific tables containing many out-of-knowledge-base mentions and a method that significantly outperforms a generic table EL method. |
| Outcome: | The proposed method significantly outperforms state-of-the-art generic table EL method on scientific tables with many out-of knowledge-base mentions. |
“It doesn’t look good for a date”: Transforming Critiques into Preferences for Conversational Recommendation Systems (2021.emnlp-main)
Copied to clipboard
| Challenge: | Conversational recommendation systems (CRSs) aim to refine options over multiple turns of a conversation, but they are not as flexible as real conversations. |
| Approach: | They propose a method for transforming a user critique into a positive preference . they use a large neural language model to perform critique-to-preference transformation . |
| Outcome: | The proposed method improves recommendations in restaurant domain using a new dataset of restaurant critiques. |
Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that infusing layout features into language models improves processing of visually-rich documents such as scientific papers. |
| Approach: | They propose a method to evaluate layout-infused language models that incorporate layout features into their models to emulate layout distribution shifts. |
| Outcome: | The proposed model performs better under layout distribution shifts than in-distribution conditions. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
Embedding Recycling for Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on embedding recycling have not adequately account for overhead costs. |
| Approach: | They propose to reuse contextualized embeddings from previous runs to speed training and inference of future ones. |
| Outcome: | The proposed technique speeds training and inference with no impact on accuracy. |
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer language models do not leverage information on inter-document relatedness, which limits their document-level representation power. |
| Approach: | They propose a method to generate document-level embeddings using citation graphs. |
| Outcome: | The proposed method outperforms baselines on document-level tasks. |
TOPICAL: TOPIC Pages AutomagicaLly (2024.naacl-demo)
Copied to clipboard
| Challenge: | Topic pages aggregate useful information about an entity or concept into a single concise article. |
| Approach: | They propose a web app that generates topic pages for biomedical entities on demand . they use large language models and retrieval-augmented generation to generate high-quality topics . |
| Outcome: | The proposed method is based on a human evaluation of 150 biomedical topics . it uses large language models and retrieval-augmented generation (RAG) |
CHAMP: Efficient Annotation and Consolidation of Cluster Hierarchies (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Various annotation tasks require a complex hierarchical structure over nodes, where each node is a cluster of items. |
| Approach: | They propose an open source tool that incrementally constructs clusters and hierarchy simultaneously over any type of text. |
| Outcome: | The proposed approach significantly reduces annotation time and guarantees transitivity at the cluster and hierarchy levels. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |