Papers by David Mimno
Bad Seeds: Evaluating Lexical Methods for Bias Measurement (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for measuring bias use crowd-sourced seed lexicons, but there is little guidance for their selection. |
| Approach: | They use lexicons of different types of social biases and linguistic features to enumerate biased seeds from three English-language corpora. |
| Outcome: | The results show that seed lexicons can be used to measure bias in English-language corpora . the results show the seeds can be re-used in other contexts . |
Show or Tell? Modeling the evolution of request-making in Human-LLM conversations (2026.findings-eacl)
Copied to clipboard
| Challenge: | a new framework to describe request-making segments user input into request content, roles assigned, query-specific context, and task-independent expressions. |
| Approach: | They propose a framework to describe request-making that segments user input into request content, roles assigned, query-specific context, and the remaining task-independent expressions. |
| Outcome: | The proposed framework reveals fundamental and habitual user-LLM interaction patterns beyond individual task completion. |
Practical Correlated Topic Modeling and Analysis via the Rectified Anchor Word Algorithm (D19-1)
Copied to clipboard
| Challenge: | spectral topic models lack reliability in real data and lack of practical implementations. |
| Approach: | They propose to use a spectral topic inference method to infer correlations between topics in real data and a matrix-based approach to inference. |
| Outcome: | The proposed method outperforms tensor-based methods and probabilistic methods in real data and provides a complete guide to correlated topic modeling. |
Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | XLMs can support cross-lingual transfer learning with little to no additional training data. |
| Approach: | They describe a mechanism for cross-lingual transfer learning by measuring the properties of the initial token embedding layer. |
| Outcome: | The proposed model can be used to support cross-lingual transfer learning . the initial token embedding layer is expressive and interpretable . |
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (N18-1)
Copied to clipboard
| Challenge: | Existing work suggests that concepts with concrete visual manifestations are easier to learn than abstract ones. |
| Approach: | They propose an algorithm for automatically computing the visual concreteness of words and topics within multimodal datasets. |
| Outcome: | The proposed algorithm predicts the capacity of machine learning algorithms to learn textual/visual relationships. |
Data Similarity is Not Enough to Explain Language Model Performance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models perform well on many but not all downstream tasks. |
| Approach: | They compare large language models with downstream benchmarks to determine whether similarity measures correlate with model performance. |
| Outcome: | The results show that similarity measures are not correlated with accuracy or each other in other benchmarks. |
Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents (D19-1)
Copied to clipboard
| Challenge: | a structured training objective based on identifying whether collections of images and sentences co-occur in documents can suffice to predict links between specific images and specific sentences. |
| Approach: | They propose algorithms that discover image-sentence relationships without explicit annotation . they experiment on seven datasets of varying difficulty . |
| Outcome: | The proposed algorithms can predict links between images and sentences without explicit multimodal annotation. |
Comparing Text Representations: A Theory-Driven Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP have been made by learning representations that transform complex tasks into simple classification tasks. |
| Approach: | They propose a method to evaluate the compatibility between representations and tasks by fitting text features to specific characteristics of text datasets. |
| Outcome: | The proposed model provides a calibrated, quantitative measure of the difficulty of a classification-based NLP task. |
Authorless Topic Models: Biasing Models Away from Known Structure (C18-1)
Copied to clipboard
| Challenge: | a recent study shows that topic models that highlight differences in authors are often not accurate . authors show that subsampling words that are highly correlated with metadata can reduce topic-metadata correlation . |
| Approach: | They propose three metrics for identifying topics that are highly correlated with metadata . they find that subsampling words causes topic-metadata correlation, improve topic stability . authors propose to use topic models to infer word distributions that correspond to recognizable themes . |
| Outcome: | The proposed model can predict which words cause the phenomenon and improve topic stability and quality. |
A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity (2024.naacl-long)
Copied to clipboard
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, Daphne Ippolito
| Challenge: | a large number of pretraining data design practices are under-documented, authors say . authors: strong performance of modern language models depends on selfsupervised pretraining . |
| Approach: | They propose to pretrain models on data curated at different collection times . they find temporal shift between evaluation data and pretraining data leads to performance degradation . |
| Outcome: | The results validate, quantify, and expose many undocumented intuitions about text pretraining . authors say this practice has outperformed other models in the field . |
Contextualized Topic Coherence Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing topic models that estimate the interpretability of topics are difficult to compare due to their nature as unsupervised models. |
| Approach: | They propose to use contextualized topic coherence metrics to simulate human-centered coherency evaluation while maintaining the efficiency of other automated methods. |
| Outcome: | The proposed metrics better reflect human judgment on topics extracted from short text collections by avoiding highly scored topics that are meaningless to humans. |
Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research examines simple classification tasks, but ability of LMs to classify on complex tasks is less well understood. |
| Approach: | They analyze a Supreme Court opinion annotated by a team of domain experts . they find generative models perform poorly when given instructions equal to human annotators . |
| Outcome: | The proposed model performs poorly when given instructions equal to instructions given to human annotations . strongest results derive from fine-tuning models on the annotated dataset . |
Domain-Specific Lexical Grounding in Noisy Visual-Textual Documents (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing image-text grounding approaches require detailed annotations, authors say . existing methods are difficult to adapt to unlabeled multi-image, multi-sentence documents, they say . |
| Approach: | They propose a method that can learn contextual meanings from unlabeled documents . they demonstrate that a simple unsupervised clustering-based method can be useful . |
| Outcome: | The proposed method is particularly effective for local contextual meanings of a word . existing image-text grounding methods are difficult to adapt to unlabeled multi-image, multi-sentence documents . |