Papers by David Smith
Detecting Syntactic Change with Pre-trained Transformer Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a fine-tuned BERT model can distinguish between text from the early 1800s and late 1900s . we use it to identify specific instances of syntactic change and specific words for which a new part of speech was introduced. |
| Approach: | They propose to use a BERT-based model to find syntactic differences between English of the early 1800s and that of the late 1900s. |
| Outcome: | The proposed model can distinguish between English of the early 1800s and that of the late 1900s using only syntactic information. |
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens (2025.acl-demo)
Copied to clipboard
Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
| Challenge: | tracing language models' outputs back to training data is a problem because they are trained on text corpora with trillions of tokens . existing methods for tracers have not been scaled to work within this multi-trillion-token setting . |
| Approach: | They propose a system that traces language models' outputs verbatim back to training data . OLMOTRACE retrieves documents from the model's training data that contain exact matches . |
| Outcome: | The proposed system can find verbatim matches between LM output and training data . it can be used to explore fact checking, hallucination, and creativity of language models . |
Constructing a Psychometric Testbed for Fair Natural Language Processing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Psychometric dimensions are important for understanding user behavior in various contexts including health, security, e-commerce, and finance. |
| Approach: | They propose to construct a corpus for psychometric natural language processing related to important dimensions such as trust, anxiety, numeracy, and literacy, in the health domain. |
| Outcome: | The proposed corpus includes 8,502 user-generated responses from 8,502-person survey datasets and includes self-reported demographic information, including race, sex, age, income, and education. |
GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing projection-based methods that project gradients into a lower-dimensional subspace can introduce computational and memory overheads. |
| Approach: | They propose a novel approach that leverages sparse projections to transform gradients into structured sparser updates. |
| Outcome: | The proposed approach significantly reduces memory usage for optimizer states and minimizes memory footprint, computation, and communication costs, leading to substantial throughput improvements. |
Content-based Models of Quotation (2021.eacl-main)
Copied to clipboard
| Challenge: | Prior work has focused on manual feature engineering and development of frameworks to test factors that influence quotability. |
| Approach: | They propose to use quotability identification as a passage ranking problem to evaluate models' performance . they use five datasets that span multiple languages and genres of literature . |
| Outcome: | The proposed model outperforms the existing model on five datasets that span multiple languages and genres of literature. |
Detecting de minimis Code-Switching in Historical German Books (2020.coling-main)
Copied to clipboard
| Challenge: | Code-switching has drawn scholarly attention in computational linguistics and natural language processing from many different perspectives. |
| Approach: | They propose to compare informal code-switching to its appearance in more formal registers by annotating and inspecting the German textarchives. |
| Outcome: | The proposed classifiers can help reduce errors when speech recognition is applied to a large corpus with rare embedded languages. |
Structural Encoding and Pre-training Matter: Adapting BERT for Table-Based Fact Verification (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing research on fact verification focuses on unstructured textual evidence, but it is still underexplored. |
| Approach: | They propose to use a table-based language model to verify textual statements . they use cell embeddings and numerical information to improve accuracy . |
| Outcome: | The proposed method outperforms the state-of-the-art model on question answering tasks even without modeling numerical information. |
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (2023.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov
| Challenge: | Language models have evolved from being research prototypes to commercialized products offered as web APIs. |
| Approach: | They conduct a systematic analysis of the cost and utility of OpenAI’s language model API on multilingual benchmarks in 22 typologically diverse languages. |
| Outcome: | The proposed language model API performs poorly on multiple languages and speakers of a large number of languages are overcharged while obtaining poorer results. |
Multi-Input Attention for Unsupervised OCR Correction (P18-1)
Copied to clipboard
| Challenge: | Existing methods for OCR correction are mostly supervised methods that correct recognition errors in a single output. |
| Approach: | They propose a sequence-to-sequence model with attention and a decoder with attention averaging to search for consensus among multiple sequences. |
| Outcome: | The proposed methods cut the character and word error rates nearly in half on single inputs and can rival supervised methods. |
Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects (2024.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Adelani, Daud Abolade, Noah Smith, Yulia Tsvetkov
| Challenge: | Recent efforts to develop NLP tools for low-resource languages focus on their standard dialects. |
| Approach: | They propose a high-quality parallel text and speech corpus for Yoruba . they use native speakers to collect data from four regional yoruba dialects . |
| Outcome: | The proposed dataset shows that dialect-adaptive finetuning can narrow performance disparities . the dataset will be released publicly under an open license . |
Finite State Machine Pattern-Root Arabic Morphological Generator, Analyzer and Diacritizer (2020.lrec-1)
Copied to clipboard
| Challenge: | Using a finite-state morphologizer, we generate and analyze undiacritized Modern Standard Arabic (MSA) words. |
| Approach: | They propose to use a finite-state Arabic Morphologizer to generate and analyze undiacritized Arabic words and diacritize them. |
| Outcome: | The proposed model generates and analyzes undiacritized Modern Standard Arabic (MSA) words and diacritizes them. |
ROBBIE: Robust Bias Evaluation of Large Generative Language Models (2023.emnlp-main)
Copied to clipboard
David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, Eric Smith
| Challenge: | generative large language models (LLMs) are becoming more performant and prevalent . we need tools to measure and improve their fairness, authors say . |
| Approach: | They propose to compare 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
| Outcome: | The proposed model can be tested on more datasets to better characterize and mitigate biases . the study compared 6 prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |