Papers by Kristina Lerman
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models’ Posteriors (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is the primary method for performing natural language tasks with Large Language Models. |
| Approach: | They examine whether aggregation is a confounding factor in the modeling of subjective tasks . they find it is possible for minority annotators to better align with LLMs . |
| Outcome: | The proposed method is based on aggregation of annotations in a dataset with appropriate priors. |
ALCAP: Alignment-Augmented Music Captioner (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditional approaches to music captioning ignore the intricate interplay between the two . however, a comprehensive understanding of music necessitates the integration of both these elements. |
| Approach: | They propose a method to learn multimodal alignment between audio and lyrics through contrastive learning. |
| Outcome: | The proposed method achieves new state-of-the-art on two music captioning datasets. |
Speaker Turn Modeling for Dialogue Act Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to DA classification model utterances without incorporating the turn changes among speakers throughout the dialogue, thus treating it no different than non-interactive written text. |
| Approach: | They propose to integrate the turn changes in conversations among speakers when modeling DAs by learning conversation-invariant speaker turn embeddings to represent speaker turns in a conversation. |
| Outcome: | The proposed model captures semantics from the dialogue content while accounting for different speaker turns in a conversation. |
Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts (2025.emnlp-main)
Copied to clipboard
Georgios Chochlakis, Peter Wu, Tikka Arjun Singh Bedi, Marcus Ma, Kristina Lerman, Shrikanth Narayanan
| Challenge: | Existing approaches to model complex subjective tasks in natural language are limited by significant variation in annotations. |
| Approach: | They propose a simple in-context learning binary filtering baseline that estimates the reasonableness of a document-label pair. |
| Outcome: | The proposed approach can be integrated into annotation pipelines to enhance signal-to-noise ratios. |
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can adapt outputs to align with community-specific norms, perspectives and communication styles. |
| Approach: | They propose a benchmark to assess community-specific steering using contrasting reddit communities. |
| Outcome: | STEER-BENCH assesses how well large language models understand community-specific instructions, their resilience to adversarial steering attempts, and their ability to accurately represent cultural and ideological perspectives. |
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to label aggregation fail to capture subjective annotations and can lead to biases. |
| Approach: | They propose annotator-aware representations for text for subjective classification tasks that involve learning representations of annotators. |
| Outcome: | The proposed model improves on metrics that assess the performance on capturing individual annotators’ perspectives. |
BigTokDetect: A Clinically-Informed Vision–Language Modeling Framework for Detecting Pro-Bigorexia Videos on TikTok (2026.eacl-long)
Copied to clipboard
Minh Duc Chu, Kshitij Pawar, Zihao He, Roxanna Sharifi, Ross M. Sonnenblick, Magdalayna Curry, Laura DAdamo, Lindsay Young, Stuart Murray, Kristina Lerman
| Challenge: | Social media platforms face escalating challenges in detecting harmful content that promotes muscle dysmorphic behaviors and cognitions (bigorexia). |
| Approach: | They propose a framework for detecting pro-bigorexia content on TikTok using an expert-annotated multimodal benchmark dataset of over 2,200 Tiktok videos labeled by clinical psychiatrists. |
| Outcome: | The proposed framework improves on fine-grained subcategories while commercial models achieve the highest accuracy on primary categories. |
How Susceptible are Large Language Models to Ideological Manipulation? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have the potential to exert substantial influence on public perceptions and interactions with information. |
| Approach: | They examine how LLMs can learn and generalize ideological biases from their instruction-tuning data. |
| Outcome: | The LLMs show a startling ability to absorb ideology from one topic and generalize it to even unrelated ones. |
Reading Between the Tweets: Deciphering Ideological Stances of Interconnected Mixed-Ideology Communities (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies treat ideology as a liberal/conservative binary and fail to capture the spectrum of ideologies that may organically emerge in interconnected online communities. |
| Approach: | They propose a method that uses finetuning language models to probe nuanced ideologies of online communities by analyzing discussions of the 2020 election on Twitter. |
| Outcome: | The proposed approach shows higher alignment than baselines for the proposed approach. |
Community-Cross-Instruct: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Social scientists use surveys to learn opinions and beliefs of populations, but these methods are slow, costly, and prone to biases. |
| Approach: | They propose a framework for aligning large language models to online communities by finetuning instruction-output pairs by an advanced LLM to elicit their beliefs. |
| Outcome: | The proposed framework enables cost-effective and automated surveying of diverse online communities. |
Whose Emotions and Moral Sentiments do Language Models Reflect? (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research has focused on positional alignment, which measures how closely the models mimic the opinions and stances of different social groups. |
| Approach: | They define the problem of affective alignment, which measures how LMs’ emotional and moral tone represents those of different groups. |
| Outcome: | The results show that the models represent the perspectives of some social groups better than others, suggesting a systemic bias within LMs. |
Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in representing individuals and communities, but evaluating their fidelity remains a challenge. |
| Approach: | They propose a framework for aligning large language models with online communities via instruction-tuning and comprehensively evaluating alignment across various aspects of language. |
| Outcome: | The proposed framework shows that it can be used to create high-fidelity representations of people and communities. |
Detecting Polarized Topics Using Partisanship-aware Contextualized Topic Embeddings (2021.findings-emnlp)
Copied to clipboard
| Challenge: | polarization of the news media has been blamed for fanning disagreement, controversy and even violence. |
| Approach: | They propose a method to automatically detect polarized topics from partisan news sources by corpus-contextualized topic embedding a news corpus on a topic and using cosine distance to capture topical polarization. |
| Outcome: | The proposed method captures topical polarization and shows it can retrieve the most polarized topics. |