Papers by Katherine Thai
ChapterBreak: A Challenge Dataset for Long-Range Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing long-range language models lack a meaningful evaluation of their discourse-level language understanding capabilities. |
| Approach: | They propose a dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary and asks it to distinguish the beginning of the ground-truth next chapter from n-token segments. |
| Outcome: | The proposed dataset shows that existing models fail to leverage long-range context . |
AI use in American newspapers is widespread, uneven, and rarely disclosed (2026.acl-long)
Copied to clipboard
Jenna Russell, Marzena Karpinska, Destiny Akinode, James Zhou, Katherine Thai, Bradley Emi, Max Spero, Mohit Iyyer
| Challenge: | a large-scale dataset of 186K articles from 1.5K newspapers published in the summer of 2025 is audited. |
| Approach: | They audit 186K articles from 1.5K newspapers published in summer of 2025 . they use Pangram, a state-of-the-art AI detector, to detect whether articles are partially or fully AI-generated . |
| Outcome: | The findings highlight the need for greater transparency and updated editorial standards regarding the use of AI in journalism to maintain public trust. |
Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature (2022.emnlp-main)
Copied to clipboard
Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, Mohit Iyyer
| Challenge: | Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators . a dataset of non-English language novels is used to study literary MT . |
| Approach: | They use a dataset of non-English language novels aligned to human and automatic English translations to study literary MT. |
| Outcome: | The proposed model prefers human translations over machine translations at a rate of 84% . state-of-the-art MT metrics do not correlate with preferences, the study finds . |
One Thousand and One Pairs: A “novel” challenge for long-context language models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context evaluation methods measure surface-level retrieval capabilities, but do not assess performance on the more challenging task of synthesizing distant and underlying information. |
| Approach: | They propose a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books. |
| Outcome: | The proposed model performs better on pairs that require only sentence-level retrieval vs. global reasoning . the proposed model also performs worse on speculative fiction books with extensive world-building . |
RELiC: Retrieving Evidence for Literary Claims (2022.acl-long)
Copied to clipboard
| Challenge: | Literature scholars often provide evidence for claims they make in the form of quotations from a work . a novel evidence retrieval task requires a deep understanding of complex literary phenomena . |
| Approach: | They use a large-scale dataset to formulate a task of literary evidence retrieval . they implement a RoBERTa-based dense passage retriever that outperforms existing methods . |
| Outcome: | The proposed model outperforms pretrained information retrieval baselines, but there is room for improvement. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
Literary Evidence Retrieval via Long-Context Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a recent study shows that long-context language models can exceed human expert performance in literary analysis . despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration. |
| Approach: | They propose a task where a model is given an entire text of a book and a literary criticism with a missing quotation from that work and asked to generate the missing quote. |
| Outcome: | The proposed model outperforms open-weight models in literary evidence retrieval tasks. |