Papers by Maria Antoniak
Bad Seeds: Evaluating Lexical Methods for Bias Measurement (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for measuring bias use crowd-sourced seed lexicons, but there is little guidance for their selection. |
| Approach: | They use lexicons of different types of social biases and linguistic features to enumerate biased seeds from three English-language corpora. |
| Outcome: | The results show that seed lexicons can be used to measure bias in English-language corpora . the results show the seeds can be re-used in other contexts . |
The Empirical Variability of Narrative Perceptions of Social Media Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Identifying stories in social media texts provides a lens through which we can study how individuals and communities process and communicate experiences. |
| Approach: | They construct a taxonomy of crowd workers’ varied and nuanced perceptions of storytelling by open-coding their free-text rationales. |
| Outcome: | The proposed model shows that crowd workers disagree on categorical labels, free-text storytelling rationales, authorial intent, and more. |
Riveter: Measuring Power and Social Dynamics Between Entities (2023.acl-demo)
Copied to clipboard
| Challenge: | Riveter provides a complete pipeline for analyzing verb connotations associated with entities in text corpora. |
| Approach: | et al., 2005, provide a verb-centric analysis pipeline for verb connotations in text corpora . they prepopulate the pipeline with connotation frames of sentiment, power, and agency . lexical frameworks have been foundational tools in social science, digital humanities, and natural language processing . |
| Outcome: | Riveter provides a complete pipeline for analyzing verb connotations associated with entities in text corpora. |
CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming (2025.acl-long)
Copied to clipboard
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
| Challenge: | CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs’ cultural knowledge covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Approach: | They construct a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Outcome: | The proposed model outperforms other models across cultures, while underperforming on questions related to North Africa, South America and Middle East. |
Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a task evaluates how well LLMs recognize poetry, but performance varies by poetic form . performance varying by poetic forms; models struggle to identify unfixed poetic forms . |
| Approach: | They use a benchmark dataset to evaluate how well LLMs recognize poetry . they find that the models can identify fixed poetic forms with high accuracy . |
| Outcome: | The proposed task evaluates how well LLMs recognize poetry features . performance varies significantly by poetic form; models struggle to identify unfixed forms . authors urge more work that builds nuance and ambiguity into humanistic benchmarks . |
Personalized Jargon Identification for Enhanced Interdisciplinary Communication (2024.naacl-long)
Copied to clipboard
| Challenge: | Identifying and translating scientific jargon for individual researchers could speed up research, but current methods of jaron identification rely on corpus-level familiarity indicators rather than modeling researcher-specific needs. |
| Approach: | They collect over 10K term familiarity annotations from 11 computer science researchers and investigate supervised and prompt-based methods to predict individual jargon familiarity. |
| Outcome: | The proposed method improves jargon familiarity prediction by using domain, subdomain, and individual knowledge. |
Research Borderlands: Analysing Writing Across Research Cultures (2025.acl-long)
Copied to clipboard
| Challenge: | a recent study has focused on improving cultural competence of language technologies, but most studies rely on synthetic setups and imperfect proxies of culture. |
| Approach: | They use a human-centered approach to discover and measure language-based cultural norms and cultural competence of large language models (LLMs). |
| Outcome: | The proposed framework identifies cultural norms that vary across research cultures and identifie a lack of cultural competence in LLMs. |
Multi-Modal Framing Analysis of News (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated frame analysis of political communication has been limited by the use of predefined frames and the visual contexts in which they appear. |
| Approach: | They propose a method for doing multi-modal, multi-label framing analysis at scale using large (vision-) language models. |
| Outcome: | The proposed method provides a more complete picture for understanding media bias. |
Social Story Frames: Contextual Reasoning about Narrative Intent and Reception (2026.acl-long)
Copied to clipboard
Joel Mire, Maria Antoniak, Steven R Wilson, Zexin Ma, Achyutarama R Ganti, Andrew Piper, Maarten Sap
| Challenge: | SocialStoryFrames is a formalism for distilling plausible inferences about reader response . authors characterize frequency and interdependence of storytelling intents across communities . |
| Approach: | They propose a formalism for distilling plausible inferences about reader response using conversational context and a taxonomy grounded in narrative theory, linguistic pragmatics, and psychology. |
| Outcome: | The proposed model can be used to analyze reader responses in online communities. |
Where Do People Tell Stories Online? Story Detection Across Online Communities (2024.acl-long)
Copied to clipboard
| Challenge: | Story detection in online communities is a challenging task as stories are scattered across communities and interwoven with non-storytelling spans within a single text. |
| Approach: | They propose a toolkit to detect stories in online communities using an annotated reddit dataset and a codebook adapted to social media context. |
| Outcome: | The proposed toolkit includes an annotation-rich dataset of 502 Reddit posts and comments . it also includes a codebook adapted to the social media context and models to predict storytelling at document and span levels. |
so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite popularity of poetry as both an art form and a generation task for large language models, whitespace has not received sufficient attention from the NLP community. |
| Approach: | They examine how 4k poets have used whitespace in their works . they compare it to 51k LLM-generated poems and 12k unpublished poems posted online . |
| Outcome: | The proposed dataset compares 4k poetry poems with 51k LLM-generated poems and 12k unpublished poems posted in an online community. |
Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models (2025.naacl-long)
Copied to clipboard
Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Maria Antoniak, Bill Yuchen Lin, Niloofar Mireshghallah, Chandra Bhagavatula, Yejin Choi
| Challenge: | Lack of transparency in training data is limiting external oversight and inspection of LLMs for issues such as copyright infringement and data contamination. |
| Approach: | They propose a method to identify training data known to proprietary LLMs without requiring access to model weights or token probabilities by using information-guided probes. |
| Outcome: | The proposed method can identify training data known to proprietary LLMs without access to model weights or token probabilities. |