Papers by Nicholas Lourie
Show Your Work with Confidence: Confidence Bands for Tuning Curves (2024.naacl-long)
Copied to clipboard
| Challenge: | a rush to scale up has left us with large, costly language models and little understanding of how different designs compare. |
| Approach: | They propose a method to construct valid confidence bands for tuning curves . they validated their method with ablations and analyze the effect of sample size . |
| Outcome: | The proposed method shows that bootstrap confidence bands do not approximate their target confidence. |
Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Downstream scaling laws aim to predict task performance at larger scales from the model’s performance at smaller scales. |
| Approach: | They conduct a meta-analysis of existing data on downstream scaling laws and find that predictable scaling only occurs in a minority of cases: 39% of the time. |
| Outcome: | The proposed scaling laws only occur in a minority of cases, and seemingly benign changes to the experimental setting can completely change the scaling behavior. |
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020.emnlp-main)
Copied to clipboard
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, Yejin Choi
| Challenge: | Large datasets have become commonplace in NLP research, but the emphasis on quantity has made it challenging to assess the quality of data. |
| Approach: | They propose a model-based tool to characterize and diagnose large datasets . they leverage the behavior of the model on individual instances during training . |
| Outcome: | Experiments on four datasets show that the tool can characterize and diagnose datasets with a model-based tool. |
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation (2022.emnlp-main)
Copied to clipboard
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel Weld
| Challenge: | Effective human evaluation of text generation tasks remains an important, open area for research. |
| Approach: | They propose a system for running standardized human evaluations across different generation tasks. |
| Outcome: | The proposed system produces standardized human evaluations across tasks . it crowdsources predictions and ranks systems on leaderboards . the proposed system is not reproducible over time and different annotator populations . |
Learning from Task Descriptions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in supervised NLP has shown significant progress in learning tasks from examples. |
| Approach: | They propose a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. |
| Outcome: | The proposed model achieves 12% on the new dataset, leaving a significant challenge for NLP researchers. |
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (N19-1)
Copied to clipboard
| Challenge: | Recent work on question answering relies on factoid questions with little general knowledge. |
| Approach: | They propose a dataset to capture commonsense question answering with prior knowledge . they extract multiple-choice questions that discriminate between the source and target concepts . |
| Outcome: | The proposed dataset captures commonsense reasoning beyond associations . it obtains 56% accuracy, well below human performance, which is 89% . |
Writing Code for NLP Research (D18-3)
Copied to clipboard
| Challenge: | upcoming workshop on open source software for NLP aims to share best practices for writing code for Nl research . participants will learn how to write research code that facilitates good science and easy experimentation . |
| Approach: | this tutorial aims to share best practices for writing code for NLP research . participants will learn how to write research code that facilitates good science and easy debugging . |
| Outcome: | the workshop on open source software for NLP aims to share best practices for writing code for Nl research . participants will learn how to write research code that facilitates good science and easy experimentation . |