Papers by Yulia Tsvetkov

TalkUp: Paving the Way for Understanding Empowering Language (2023.findings-emnlp)

Challenge:	Pretrained generative models provide novel ways for users to interact with computers.
Approach:	This tutorial provides an overview of key guardrail mechanisms developed for LLMs along with evaluation methodologies and a detailed security assessment protocol.
Outcome:	This tutorial provides an overview of key guardrail mechanisms developed for LLMs, along with evaluation methodologies and a detailed security assessment protocol.

Copied to clipboard

Lucille Njoo, Chan Park, Octavia Stappart, Marvin Thielk, Yi Chu, Yulia Tsvetkov

Challenge:	Empowerment has rarely been studied in NLP because of its implicit nature . linguistics and psychology research shows how empowerment can impact people by increasing their sense of self-efficacy and self-esteem.
Approach:	They crowdsource Reddit posts labeled for empowerment and use it to train language models that capture empowering and disempowering language.
Outcome:	The proposed dataset can be used to train language models that capture empowering and disempowering language.

Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions (2020.acl-main)

Copied to clipboard

Xiaochuang Han, Byron C. Wallace, Yulia Tsvetkov

Explainability & Interpretability in NLP

Challenge:	Modern deep learning models for NLP are notoriously opaque, and this has motivated efforts to design example-specific approaches to interpret such models.
Approach:	They propose to use influence functions to explain models by highlighting important words in input text to provide models with an explanation.
Outcome:	The proposed approach is particularly useful for natural language inference, a task in which ‘saliency maps’ may not have clear interpretation.

Understanding In-Context Learning via Supportive Pretraining Data (2023.acl-long)

Copied to clipboard

Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, Tianlu Wang

Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation (2021.findings-acl)

Challenge:	In-context learning (ICL) is a form of learning that provides a handful of examples at inference time, but it is not well understood why it emerges as the model has never been specifically trained on such demonstrations.
Approach:	They adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL and compare it with random subsets of pretrain data.
Outcome:	The proposed method improves the model's ICL ability by 18% if it is continued on a small subset of pretraining data.

Copied to clipboard

Prakhar Gupta, Yulia Tsvetkov, Jeffrey Bigham

Dialogue Systems & Conversational Agents

Challenge:	Open-domain neural dialogue models have achieved high performance in response ranking and evaluation tasks.
Approach:	They propose methods for automatically creating adversarial negative training data . they use mask-and-fill and keyword-guided approaches to generate negative examples .
Outcome:	The proposed approaches outperform baseline models in providing informative negative examples for training dialogue systems.

LEXPLAIN: Improving Model Explanations via Lexicon Supervision (2023.starsem-1)

Copied to clipboard

Orevaoghene Ahia, Hila Gonen, Vidhisha Balachandran, Yulia Tsvetkov, Noah A. Smith

Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks (2021.eacl-main)

Challenge:	Existing methods that extract features from input text to explain a classifier's prediction are limiting to models that are faithful to their predictions.
Approach:	They propose a framework for guiding model explanations by supervising them explicitly using task-related lexicons to direct supervise model explanation.
Outcome:	The proposed method improves model explanations without sacrificing performance on sentiment analysis and toxicity detection tasks while demoting spurious correlations with African American English dialects.

Copied to clipboard

Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, David R. Mortensen

RtGender: A Corpus for Studying Differential Responses to Gender (L18-1)

Challenge:	a large amount of work on cross-lingual transfer learning focused on typological and genealogical similarities between languages.
Approach:	They propose three features that capture cross-cultural similarities that manifest in linguistic patterns and quantify distinct aspects of language pragmatics.
Outcome:	The proposed features capture cross-cultural similarities manifest in linguistic patterns and quantify aspects of language pragmatics.

Copied to clipboard

Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, Yulia Tsvetkov

Challenge:	Prior work on linguistic gender difference and communications about gender has focused on language about or portraying persons of a particular gender.
Approach:	They present a multi-genre corpus of 25M comments from five socially and topically diverse sources tagged for the gender of the addressee and 30k annotations for sentiment and relevance of these responses.
Outcome:	The proposed dataset shows that responses to women are more emotive and about the speaker as an individual (rather than about the content being responded to).

Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching (2022.acl-long)

Copied to clipboard

Alissa Ostapenko, Shuly Wintner, Melinda Fricke, Yulia Tsvetkov

Balancing Training for Multilingual Neural Machine Translation (2020.acl-main)

Challenge:	Prior approaches for predicting code-switching only consider shallow linguistic context.
Approach:	They hypothesize that enriching models with speaker information can guide them to pick up on relevant inductive biases.
Outcome:	The proposed model improves on a speaker-driven task in English–Spanish bilingual dialogues by adding sociolinguistically-grounded speaker features as prepended prompts.

Copied to clipboard

Xinyi Wang, Yulia Tsvetkov, Graham Neubig

Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies (D18-1)

Challenge:	Existing methods to train multilingual machine translation models are imbalanced and heterogeneous data is wildly varying.
Approach:	They propose a method that automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages.
Outcome:	The proposed method outperforms baselines on two sets of languages under one-to-many and many-to-1 MT settings and offers flexible control over which languages are optimized.

Copied to clipboard

Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, Yulia Tsvetkov

Challenge:	Amidst growing concern over media manipulation, NLP studies focus on overt strategies like censorship and “fake news”.
Approach:	They propose to use two concepts from political science literature to identify subtler media manipulation strategies . they propose to apply embedding-based methods to cross-lingually project English frames to Russian .
Outcome:	The proposed techniques can be applied to 13 years of the Russian newspaper Izvestia and show that they highlight U.S. moral failings and threats to the U.s.

Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks (2024.acl-long)

Copied to clipboard

Yichen Wang, Shangbin Feng, Abe Hou, Xiao Pu, Chao Shen, Xiaoming Liu, Yulia Tsvetkov, Tianxing He

Yulia Tsvetkov, Vinodkumar Prabhakaran, Rob Voigt

Challenge:	Existing studies on this topic focus on the robustness of specific detectors or particular attack methods.
Approach:	They stress test the detectors’ robustness to malicious attacks under realistic scenarios using LLMs and metric-based detectors.
Outcome:	The proposed methods are based on a set of LLM-based models and their performance is compared under different budget levels.

Socially Responsible NLP (N18-6)

Copied to clipboard

Unsupervised Discovery of Implicit Gender Bias (2020.emnlp-main)

Challenge:	This tutorial will provide an overview of ethical research tools and ethical implications of language technologies.
Approach:	This tutorial will provide an overview of ethical research and practical examples . it will discuss ethical tools to ensure data, algorithms, and models are socially responsible .
Outcome:	This tutorial will provide an overview of ethical research tools and methods . it will discuss philosophical foundations of ethical work along with state of the art techniques .

Copied to clipboard

Anjalie Field, Yulia Tsvetkov

BotPercent: Estimating Bot Populations in Twitter Communities (2023.findings-emnlp)

Challenge:	Social biases are difficult to identify because human judgements in this domain can be unreliable.
Approach:	They propose an unsupervised approach to detecting implicit gender bias in text . their main challenge is forcing the model to focus on signs of implicit bias .
Outcome:	The proposed model reduces the influence of confounds by focusing on signs of implicit bias rather than other artifacts in the data.

Copied to clipboard

Zhaoxuan Tan, Shangbin Feng, Melanie Sclar, Herun Wan, Minnan Luo, Yejin Choi, Yulia Tsvetkov

Challenge:	Existing approaches to bot detection are agnostic to social environments the bots operate in . however, standard approaches are not a good fit for the social environments they operate in.
Approach:	They propose a method that estimates the percentage of Twitter bots given a community . they use Twitter bot detection datasets and feature-, text-, and graph-based models adjusted to a particular community based on Twitter .
Outcome:	The proposed method achieves state-of-the-art in community-level Twitter bot detection across balanced and imbalanced class distribution settings.

Biased LLMs can Influence Political Decision-Making (2025.acl-long)

Copied to clipboard

Jillian Fisher, Shangbin Feng, Robert Aron, Thomas Richardson, Yejin Choi, Daniel W Fisher, Jennifer Pan, Yulia Tsvetkov, Katharina Reinecke

Challenge:	Recent studies have found that biased LLMs can influence decisions in areas such as medical classifications and educational hiring.
Approach:	They conducted two interactive experiments on partisan bias in large language models while completing tasks with either a biased liberal, biased conservative, or unbiased control model.
Outcome:	The results show that prior knowledge of AI is weakly correlated with a reduction of the bias, suggesting that AI education can be crucial for mitigating bias effects.

Machine Translation into Low-resource Language Varieties (2021.acl-short)

Copied to clipboard

Sachin Kumar, Antonios Anastasopoulos, Shuly Wintner, Yulia Tsvetkov

A Survey of Race, Racism, and Anti-Racism in NLP (2021.acl-long)

Challenge:	Current machine translation systems generate a "standard" target language, but many languages have multiple varieties that are different from the standard language.
Approach:	They propose a framework to rapidly adapt machine translation systems to generate different target varieties . they propose to use no parallel data to generate languages close to, but different from, the standard target language .
Outcome:	The proposed model improves on a system that generates Ukrainian and Belarusian in two languages with no parallel data.

Copied to clipboard

Anjalie Field, Su Lin Blodgett, Zeerak Waseem, Yulia Tsvetkov

SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021.emnlp-main)

Challenge:	despite inextricable ties between race and language, little work has considered race in NLP research and development.
Approach:	They survey 79 papers from the ACL anthology that mention race . they find race has been siloed as a niche topic and ignored in many NLP tasks . authors call for inclusion and racial justice in NLP research practices .
Outcome:	The findings highlight the need for inclusion and racial justice in NLP research practices.

Copied to clipboard

Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H Hovy, Yulia Tsvetkov

Explainability & Interpretability in NLP

Challenge:	Existing models that explain text classification predictions are opaque and overfit to spurious artifacts.
Approach:	They propose a novel self-explaining model that explains a text classifier’s predictions using phrase-based concepts.
Outcome:	The proposed model shows that it is adequate, trustworthy and understandable by human judges compared to existing baselines.

Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts (D19-1)

Copied to clipboard

Luke Breitfeller, Emily Ahn, David Jurgens, Yulia Tsvetkov

Efficient Test Time Adapter Ensembling for Low-resource Language Varieties (2021.findings-emnlp)

Challenge:	Existing tools for hate speech detection and sentiment analysis cannot detect veiled offensiveness of microaggressions . linguistic subtlety of micro-aggressives has made it difficult to analyze their exact nature .
Approach:	They propose a typology of microaggressions based on a subset of data . they propose an objective criterion for annotation and an active-learning procedure .
Outcome:	The proposed typology of microaggressions is based on a subset of social media data.

Copied to clipboard

Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, Graham Neubig

Topics to Avoid: Demoting Latent Confounds in Text Classification (D19-1)

Challenge:	Specialized language and task adapters have been proposed to facilitate cross-lingual transfer of multilingual pretrained models.
Approach:	They propose a method that optimizes the ensemble weights of pretrained adapters for each test sentence by minimizing the entropy of its predictions.
Outcome:	The proposed method improves robustness to uncovered languages without training new adapters.

Copied to clipboard

Sachin Kumar, Shuly Wintner, Noah A. Smith, Yulia Tsvetkov

Text Classification

Challenge:	Despite impressive performance on many text classification tasks, deep neural networks tend to learn frequent superficial patterns that are specific to the training data and do not always generalize well.
Approach:	They propose a method that represents latent topical confounds and a model which “unlearns” confounding features by predicting both the label of the input text and the confound.
Outcome:	The proposed model generalizes better and learns features indicative of the writing style rather than the content.

CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming (2025.acl-long)

Copied to clipboard

Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi

ALPACA AGAINST VICUNA: Using LLMs to Uncover Memorization of LLMs (2025.naacl-long)

Challenge:	CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs’ cultural knowledge covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru.
Approach:	They construct a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru.
Outcome:	The proposed model outperforms other models across cultures, while underperforming on questions related to North Africa, South America and Middle East.

Copied to clipboard

Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, Santu Rana

Automatic Extraction of Rules Governing Morphological Agreement (2020.emnlp-main)

Challenge:	Existing studies have shown that pre-trained LLMs emit training data up to 150 more often than in regular operation.
Approach:	They propose a black-box prompt optimization method where an attacker LLM agent uncovers higher levels of memorization in a victim agent .
Outcome:	The proposed method shows 23.7% more overlap with training data compared to state-of-the-art baselines.

Copied to clipboard

Aditi Chaudhary, Antonios Anastasopoulos, Adithya Pratapa, David R. Mortensen, Zaid Sheikh, Yulia Tsvetkov, Graham Neubig

Syntactic Text Processing

Challenge:	Creating a descriptive grammar is an indispensable step for language documentation but it is tedious and time-consuming.
Approach:	They propose a framework for extracting a first-pass grammatical specification from raw text in a concise, human- and machine-readable format.
Outcome:	The proposed framework extracts a grammatical specification that is nearly equivalent to those created with large amounts of gold-standard annotated data.

Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers (2024.naacl-short)

Copied to clipboard

Roy Xie, Orevaoghene Ahia, Yulia Tsvetkov, Antonios Anastasopoulos

Information Extraction & Text Mining

Challenge:	Identifying linguistic differences between dialects of a language often requires expert knowledge and meticulous human analysis.
Approach:	They propose a method to extract distinguishing lexical features of dialects by utilizing interpretable dialect classifiers in the absence of human experts.
Outcome:	The proposed method extracts key language-specific lexical features that contribute to dialectal variations.

BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer (2024.naacl-long)

Copied to clipboard

Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi

LatticeGen: Hiding Generated Text in a Lattice for Privacy-Aware Large Language Model Generation on Cloud (2024.findings-naacl)

Challenge:	Recent advances in few-shot generalization in natural language processing focus on English.
Approach:	They propose a benchmark that unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions.
Outcome:	The proposed framework unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions.

Copied to clipboard

Mengke Zhang, Tianxing He, Tianle Wang, Lu Mi, Niloofar Mireshghallah, Binyi Chen, Hao Wang, Yulia Tsvetkov

Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)

Challenge:	Currently, the server controls the generated text, but users can't keep it private . prompted generation is a common interaction paradigm for large language models on cloud .
Approach:	They propose a protocol where the server handles most of the computation while the client controls the sampling operation.
Outcome:	The proposed protocol protects both prompt and generation under strong attacks.

Copied to clipboard

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, Wen-tau Yih

Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers (2025.tacl-1)

Challenge:	Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations.
Approach:	They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context.
Outcome:	The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks.

Copied to clipboard

Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov

Mitigating Societal Harms in Large Language Models (2023.emnlp-tutorial)

Challenge:	Inductive biases in transformers can cause hierarchical generalization without explicitly encoding structural bias.
Approach:	They investigate sources of inductive bias in transformer models and their training that could cause such preference for hierarchical generalization.
Outcome:	The proposed model can generalize to novel syntactic forms without explicit bias . the proposed model is able to generalize on a dataset with a hierarchical grammar .

Copied to clipboard

Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, Yulia Tsvetkov

Data Swarms: Optimizable Generation of Synthetic Evaluation Data (2026.findings-acl)

Challenge:	Recent studies have highlighted societal harms that can be caused by language generation models deployed in the wild.
Approach:	They propose to use a typology of technical approaches to mitigating harms of language generation models to provide an overview of potential social issues in language generation including toxicity, social biases, misinformation, factual inconsistency, and privacy violations.
Outcome:	The proposed typology addresses toxicity, biases, misinformation, factual inconsistency, and privacy violations in language generation models.

Copied to clipboard

Shangbin Feng, Yike Wang, Weijia Shi, Yulia Tsvetkov

GlobalBench: A Benchmark for Global Progress in Natural Language Processing (2023.emnlp-main)

Challenge:	Extensive experiments demonstrate that Data Swarms outperforms eight data generation baselines across five evaluation objectives.
Approach:	They propose an algorithm to optimize the generation of synthetic evaluation data and advance quantitative desiderata of LLM evaluation.
Outcome:	The proposed algorithm outperforms baseline evaluations and Adversarial Swarms generates harder data while learning from such data.

Copied to clipboard

Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Alham Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, Graham Neubig

Learning to Generate Word- and Phrase-Embeddings for Efficient Phrase-Based Neural Machine Translation (D19-56)

Challenge:	despite advances in NLP, significant disparities in performance across languages still exist . prior benchmarks focused on a limited number of tasks and languages, but now GlobalBench tracks progress on all languages.
Approach:	They propose to use global benchmarks to track progress on all NLP datasets in all languages.
Outcome:	a new tool tracks progress on all NLP datasets in all languages and tracks per-speaker utility and equity . globalbench is designed to identify the most under-served languages and reward research efforts . a globalbech is available at https://github.com/neulab/globalbench.

Copied to clipboard

Chan Young Park, Yulia Tsvetkov

Controlling Dialogue Generation with Semantic Exemplars (2021.naacl-main)

Challenge:	Neural machine translation (NMT) often fails in one-to-many translation, e.g., in the translation of multi-word expressions, compounds, and collocations.
Approach:	They propose a phrase-based NMT model that generates embeddings of words or phrases.
Outcome:	The proposed model performs on par with state-of-the-art phrase-based NMT.

Copied to clipboard

Prakhar Gupta, Jeffrey Bigham, Yulia Tsvetkov, Amy Pavel

Dialogue Systems & Conversational Agents

Challenge:	Existing methods to control dialogue generation are manual labelling and manual editing of data.
Approach:	They propose a method to control dialogue generation using exemplar responses . they use semantic frames present in exemplars to guide response generation .
Outcome:	The proposed model improves coherence while preserving semantic meaning and conversation goals . exemplar responses are handwritten or strategically curated to promote highlevel goals without explicit labels .

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration (2024.emnlp-main)

Copied to clipboard

Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, Yulia Tsvetkov

ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)

Challenge:	Existing alignment paradigms for large language models learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities.
Approach:	They propose a modular framework that "plugs" into a base LLM a pool of smaller but specialized community LMs where models collaborate in distinct modes to support three modes of pluralism: Overton, steerable, and distributional.
Outcome:	The proposed framework “plugs into” a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to support three modes of pluralism: Overton, steerable, and distributional.

Copied to clipboard

Sachin Kumar, Chan Young Park, Yulia Tsvetkov, Noah A. Smith, Hannaneh Hajishirzi

Style Transfer Through Back-Translation (P18-1)

Challenge:	Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups.
Approach:	They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider.
Outcome:	The proposed method improves performance by focusing on group-level preferences rather than individual feedback.

Copied to clipboard

Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, Alan W Black

DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection (2024.findings-acl)

Challenge:	a new method for automatic style transfer is proposed to preserve the meaning of the text while reducing stylistic properties.
Approach:	They propose a method for automatic style transfer that uses latent representations of the input sentence to preserve meaning while reducing stylistic properties.
Outcome:	The proposed method improves on sentiment, gender and political slant styles on three different styles.

Copied to clipboard

Herun Wan, Shangbin Feng, Zhaoxuan Tan, Heng Wang, Yulia Tsvetkov, Minnan Luo

Locating Information Gaps and Narrative Inconsistencies Across Languages: A Case Study of LGBT People Portrayals on Wikipedia (2024.emnlp-main)

Challenge:	Large language models are limited by challenges in factuality and hallucinations to be directly employed off-the-shelf for judging the veracity of news articles.
Approach:	They propose to integrate large language models into the news pipeline by generating news reactions and generating proxy tasks.
Outcome:	The proposed model outperforms state-of-the-art baselines by 16.8% in macro f1-score on seven datasets with three LLMs.

Copied to clipboard

Farhan Samir, Chan Young Park, Anjalie Field, Vered Shwartz, Yulia Tsvetkov

Challenge:	a recent study focuses on comparative text analyses to explain social phenomena and identify systematic biases.
Approach:	They evaluate InfoGap method to locate information gaps and inconsistencies in articles at the fact level, across languages.
Outcome:	The method identifies discrepancies in factual coverage across languages and biographical facts carrying negative connotations are more likely to be highlighted in Russian Wikipedia.

P3Sum: Preserving Author’s Perspective in News Summarization with Diffusion Language Models (2024.naacl-long)

Copied to clipboard

Yuhan Liu, Shangbin Feng, Xiaochuang Han, Vidhisha Balachandran, Chan Young Park, Sachin Kumar, Yulia Tsvetkov

Information Extraction & Text Mining

Challenge:	Existing summarization systems alter the political opinions and stances of news articles in more than 50% of summaries, misrepresenting the intent and perspectives of the authors.
Approach:	They propose a model-based summarization approach controlled by political perspective classifiers that preserves the political stance of a generated summary.
Outcome:	The proposed model outperforms state-of-the-art summarization systems and large language models by up to 13.7% in terms of success rate of stance preservation, with competitive performance on standard metrics of summarizing quality.

Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (2023.emnlp-main)

Copied to clipboard

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov

Unsupervised Keyphrase Extraction via Interpretable Neural Networks (2023.findings-eacl)

Challenge:	Language models have evolved from being research prototypes to commercialized products offered as web APIs.
Approach:	They conduct a systematic analysis of the cost and utility of OpenAI’s language model API on multilingual benchmarks in 22 typologically diverse languages.
Outcome:	The proposed language model API performs poorly on multiple languages and speakers of a large number of languages are overcharged while obtaining poorer results.

Copied to clipboard

Rishabh Joshi, Vidhisha Balachandran, Emily Saldanha, Maria Glenski, Svitlana Volkova, Yulia Tsvetkov

Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021.emnlp-main)

Challenge:	Prior approaches for unsupervised keyphrase extraction relied on heuristic notions of phrase importance via embedding clustering or graph centrality.
Approach:	They propose an approach which defines keyphrases as document phrases that are salient for predicting the topic of the document.
Outcome:	The proposed method alleviates the need for ad-hoc heuristics and achieves state-of-the-art results in scientific publications and news articles.

Copied to clipboard

Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R. Mortensen, Graham Neubig, Yulia Tsvetkov

What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection (2024.acl-long)

Challenge:	Text generation systems are ubiquitous in natural language processing applications, but evaluation of these systems remains a challenge, especially in multilingual settings.
Approach:	They propose a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphologically-rich rules of the language.
Outcome:	The proposed metric can evaluate the morphosyntactic well-formedness of text using its dependency parse and morphologically-rich rules of the language.

Copied to clipboard

Shangbin Feng, Herun Wan, Ningnan Wang, Zhaoxuan Tan, Minnan Luo, Yulia Tsvetkov

Gradient-based Constrained Sampling from Language Models (2022.emnlp-main)

Challenge:	Social media bot detection has always been an arms race between advancements in machine learning and adversarial bot strategies to evade detection.
Approach:	They propose a mixture-of-heterogeneous-experts framework to divide and conquer diverse user information modalities and propose LLM-guided manipulation of user textual and structured information to evade detection.
Outcome:	The proposed framework outperforms state-of-the-art baselines on 1,000 annotated examples while bringing down existing detectors by 29.6% and harming calibration and reliability of bot detection systems.

Copied to clipboard

Sachin Kumar, Biswajit Paria, Yulia Tsvetkov

Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian Media (2022.findings-emnlp)

Challenge:	Large pretrained language models are successful at generating fluent text but are notoriously hard to controllably sample from.
Approach:	They propose a sampling procedure that combines the log-likelihood of the language model with arbitrary constraints in a single energy function and then generates samples in . non-autoregressive manner.
Outcome:	The proposed method improves on text generation with soft and hard constraints and keyword-guided generation.

Copied to clipboard

Chan Young Park, Julia Mendelsohn, Anjalie Field, Yulia Tsvetkov

Challenge:	Information manipulation campaigns rely on textbased news and social media content, and NLP can be a valuable tool in combating them.
Approach:	They propose to use a dataset to examine the use of NLP in public opinion manipulation campaigns in the 2022 Russia-Ukraine war.
Outcome:	The proposed dataset contains 38M+ posts from Russian media outlets on Twitter and VKontakte, as well as public activity and responses, immediately preceding and during the 2022 Russia-Ukraine war.

StructSum: Summarization via Structured Representations (2021.eacl-main)

Copied to clipboard

Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, Yulia Tsvetkov

Gendered Mental Health Stigma in Masked Language Models (2022.emnlp-main)

Challenge:	Abstractive summarization models overfit to training corpora, lack of transparency and layout bias . authors propose incorporating latent and explicit dependencies across sentences in source document .
Approach:	They propose a framework based on document-level structure induction to address layout bias and lack of transparency in abstractive summarization models.
Outcome:	The proposed framework improves coverage of content in the source documents and generates more abstractive summaries by generating more novel n-grams.

Copied to clipboard

Inna Lin, Lucille Njoo, Anjalie Field, Ashish Sharma, Katharina Reinecke, Tim Althoff, Yulia Tsvetkov

SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control (2023.acl-long)

Challenge:	Mental health stigma prevents many individuals from receiving appropriate care, and social psychology studies have shown that mental health tends to be overlooked in men.
Approach:	They propose to use clinical psychology literature to curate prompts, then evaluate models’ propensity to generate gendered words.
Outcome:	The proposed framework captures stigma about gender in mental health and is more likely to predict female subjects than male in sentences about mental health conditions (32% vs. 19%), and this disparity is exacerbated for sentences that indicate treatment-seeking behavior.

Copied to clipboard

Xiaochuang Han, Sachin Kumar, Yulia Tsvetkov

Can LLM Graph Reasoning Generalize beyond Pattern Memorization? (2024.findings-emnlp)

Challenge:	Existing diffusion models for continuous-valued domains have not been adopted for text data.
Approach:	They propose a diffusion-based language model with two key design choices . semi-autoregressive model generates blocks of text and allows local context updates . they evaluate it on unconstrained text generation benchmarks .
Outcome:	The proposed model outperforms autoregressive models on unconstrained text generation benchmarks on uncontrolled text generation.

Copied to clipboard

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, Yulia Tsvetkov

Structured Data in NLP

Challenge:	Existing studies seek to enhance the graph reasoning capabilities of Large Language Models (LLMs) by specialized instruction tuning.
Approach:	They propose to evaluate LLM graph reasoning generalization using in-distribution settings . they propose to use three strategies to improve LLM generalization .
Outcome:	The proposed benchmark evaluates LLM graph reasoning generalization with in-distribution settings only . it shows that LLMs struggle to generalize across reasoning and real-world patterns .

Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updates (2021.findings-emnlp)

Copied to clipboard

Xiaochuang Han, Yulia Tsvetkov

Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings (N19-1)

Challenge:	Existing approaches to interpret black-box models to learn spurious correlations are not well understood.
Approach:	They propose a procedure that leverages model interpretations to update parameters towards a plausible interpretation rather than an interpretation that relies on spurious patterns in data.
Outcome:	The proposed procedure outperforms baseline methods that use adversarial training in a controlled setup.

Copied to clipboard

Thomas Manzini, Lim Yao Chong, Alan W Black, Yulia Tsvetkov

Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker (2023.acl-long)

Challenge:	Existing methods to debias word embeddings in binary settings such as gender and religion are limited to binary labels, whereas word2vec embedders can be used to propagate biases.
Approach:	They propose a method to debias word embeddings in multiclass settings such as gender and religion, extending the work of Bolukbasi et al. (2016).
Outcome:	The proposed method maintains the efficacy in standard NLP tasks while maintaining the utility of embeddings.

Copied to clipboard

Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, Yulia Tsvetkov

KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding (2023.acl-long)

Challenge:	Empirical results show plug-and-play approach to reason about belief states of multiple characters in reading comprehension tasks is more precise and interpretable than previous approaches.
Approach:	They propose a plug-and-play approach to reason about the belief states of multiple characters in reading comprehension tasks via explicit symbolic representation.
Outcome:	The proposed algorithm improves theory of mind of off-the-shelf neural language models without supervision.

Copied to clipboard

Shangbin Feng, Zhaoxuan Tan, Wenqian Zhang, Zhenyu Lei, Yulia Tsvetkov

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023.acl-long)

Challenge:	Existing approaches to infuse knowledge graphs with pre-trained LMs are limited by the input sequence length.
Approach:	They propose a language model that leverages knowledge in local, document-level, and global contexts for long document understanding.
Outcome:	The proposed model achieves state-of-the-art on three long document understanding tasks across 6 datasets/settings.

Copied to clipboard

Shangbin Feng, Chan Young Park, Yuhan Liu, Yulia Tsvetkov

Ethical NLP

Challenge:	Hundreds of studies have highlighted ethical issues in NLP models .
Approach:	They propose to measure media biases in LMs trained on diverse data sources . they focus on hate speech and misinformation detection .
Outcome:	The proposed methods quantify the fairness of downstream NLP models trained on politically biased LMs.

A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation (D19-56)

Copied to clipboard

Gayatri Bhat, Sachin Kumar, Yulia Tsvetkov

On the Blind Spots of Model-Based Evaluation Metrics for Text Generation (2023.acl-long)

Challenge:	Existing methods for word embeddings generate faster training with fewer learnable parameters.
Approach:	They propose a novel margin-based loss that uses only predicted and target embeddings . they argue that the loss is more consistent and interpretable than other margin--based losses .
Outcome:	The proposed model is more consistent and interpretable than other margin-based losses.

Copied to clipboard

Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, Yulia Tsvetkov

When One LLM Drools, Multi-LLM Collaboration Rules (2026.acl-long)

Challenge:	Existing methods for text generation evaluation metrics are lacking in robustness analysis.
Approach:	They propose to use stress tests to test for errors in text generation evaluation metrics . they find that BERTScore is confused by truncation errors in summarization .
Outcome:	The proposed stress tests show that they are insensitive to errors in open-ended generation, translation, and summarization.

Copied to clipboard

Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Shannon Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, Tomas Pfister, Yejin Choi, Yulia Tsvetkov

Challenge:	a single general-purpose LLM is not enough to produce a reliable output, argues this paper . a multi-LLM collaboration approach addresses reliability, democratization, and pluralism .
Approach:	They argue that a single general-purpose LLM is not enough to produce a reliable output . they organize existing multi-LLM collaboration methods into a hierarchy based on access and information exchange .
Outcome:	The proposed method addresses reliability, democratization, and pluralism challenges a single LLM fails to produce a reliable output.

On the Zero-Shot Generalization of Machine-Generated Text Detectors (2023.findings-emnlp)

Copied to clipboard

Xiao Pu, Jingyu Zhang, Xiaochuang Han, Yulia Tsvetkov, Tianxing He

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment (2020.emnlp-main)

Challenge:	rampant proliferation of large language models generates text indistinguishable from human-written language.
Approach:	They train neural detectors on outputs of a new generator and test their performance on held-out generators.
Outcome:	The proposed detectors can be built on training data from medium-sized models.

Copied to clipboard

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects (2024.emnlp-main)

Challenge:	Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer) however, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference.
Approach:	They propose a meta-learning algorithm that adds language-specific parameters as meta-parameters and trains them in a manner that explicitly improves shared layers’ generalization on all languages.
Outcome:	The proposed model improves cross-lingual transferability and generalization on all languages, and improves on the language-specific parameters.

Copied to clipboard

Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Adelani, Daud Abolade, Noah Smith, Yulia Tsvetkov

Fortifying Toxic Speech Detectors Against Veiled Toxicity (2020.emnlp-main)

Challenge:	Recent efforts to develop NLP tools for low-resource languages focus on their standard dialects.
Approach:	They propose a high-quality parallel text and speech corpus for Yoruba . they use native speakers to collect data from four regional yoruba dialects .
Outcome:	The proposed dataset shows that dialect-adaptive finetuning can narrow performance disparities . the dataset will be released publicly under an open license .

Copied to clipboard

Xiaochuang Han, Yulia Tsvetkov

FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge (2023.emnlp-main)

Challenge:	Modern toxic speech detectors are incompetent in recognizing disguised offensive language, such as adversarial attacks that deliberately avoid known toxic lexicons.
Approach:	They propose a framework that fortifies existing toxic speech detectors without a large labeled corpus of veiled toxicity.
Outcome:	The proposed framework is aimed at fortifying existing toxic speech detectors without a large labeled corpus of disguised offensive language.

Copied to clipboard

Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, Yulia Tsvetkov

Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022.emnlp-main)

Challenge:	Existing factuality evaluation models are not robust, especially with respect to entity and relation errors in new domains.
Approach:	They propose a new approach to factuality evaluation that is generalizable across domains . they propose entities-specific facts, facts extracted from external knowledge bases and facts constructed compositionally through knowledge base walks.
Outcome:	The proposed model achieves state-of-the-art on two in-domain news summarization benchmarks and on three out-of domain scientific literature datasets.

Copied to clipboard

Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, Yulia Tsvetkov

Summarization

Challenge:	Abstractive summarization models often generate inconsistent summaries containing factual errors or fabricated content.
Approach:	They propose to generate representative examples of non-factual summaries through infilling language models and train a robust fact-correction model to post-edit them to improve factual consistency.
Outcome:	The proposed model outperforms previous methods in correcting factual errors on two popular summarization datasets.

Teaching LLMs to Abstain across Languages via Multilingual Feedback (2024.emnlp-main)

Copied to clipboard

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Orevaoghene Ahia, Shuyue Stella Li, Vidhisha Balachandran, Sunayana Sitaram, Yulia Tsvetkov

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2023.eacl-main)

Challenge:	Existing studies on LLM abstention focus on English, but they show that it can reduce the accuracy of the model by 20.5% .
Approach:	They propose to teach LLMs to abstain in the face of knowledge gaps by generating multiple feedback items in related languages.
Outcome:	Extensive experiments show that the proposed approach outperforms baselines and achieves 9.2% improvement for low-resource languages.

Copied to clipboard

Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, Yulia Tsvetkov

Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment (2026.findings-acl)

Challenge:	Recent advances in the capacity of large language models to generate human-like text have prompted a heated discourse around the risks of societal harms they introduce.
Approach:	They propose a taxonomy of interventions organized around the different phases where they can be adopted to mitigate harms.
Outcome:	The proposed methods are based on several prior works’ taxonomies of language model risks and provide an overview of strategies for detecting and ameliorating different kinds of risks/harms.

Copied to clipboard

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xinyun Liu, Yulia Tsvetkov

Structured Data in NLP

Challenge:	Existing research has focused on enhancing graph reasoning capabilities of LLMs by supervised fine-tuning on synthetic graph data.
Approach:	They propose to unlock generalizable learning of graph with post-training alignment with synthetic graph data by aligning off-the-shelf LLMs and LLM fine-tuned on synthetic graphs.
Outcome:	The proposed algorithm improves on synthetic graph problems and out-of-domain tasks with implicit graph structures.

Detecting Community Sensitive Norm Violations in Online Conversations (2021.findings-emnlp)

Copied to clipboard

Chan Young Park, Julia Mendelsohn, Karthik Radhakrishnan, Kinjal Jain, Tushar Kanakagiri, David Jurgens, Yulia Tsvetkov

Dialogue Systems & Conversational Agents

Challenge:	Existing efforts to identify unacceptable behavior have focused on toxicity as the sole form of community norm violation.
Approach:	They propose a dataset that focuses on a more complete spectrum of community norms and their violations in local conversational and global contexts.
Outcome:	The proposed model improves the detection of community norm violations in local conversational and global contexts.

Entity-Centric Contextual Affective Analysis (P19-1)

Copied to clipboard

Anjalie Field, Yulia Tsvetkov

Sentiment Analysis

Challenge:	Existing methods for analyzing people portrayals take an unsupervised approach, or rely on domain-specific knowledge.
Approach:	They show how contextualized word embeddings can be used to capture affect dimensions in portrayals of people.
Outcome:	The proposed method can capture affect dimensions in portrayals of men and women . it is biased towards training data, which limits its usefulness to in-domain analyses .

Understanding Ethics in NLP Authoring and Reviewing (2023.eacl-tutorials)

Copied to clipboard

Luciana Benotti, Karën Fort, Min-Yen Kan, Yulia Tsvetkov

Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systems (2026.acl-long)

Challenge:	This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues .
Approach:	This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . the methodology is interactive and participatory, including case studies and working in groups .
Outcome:	This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . the methodology is interactive and participatory, including case studies and working in groups.

Copied to clipboard

Ziyuan Yang, Wenxuan Ding, Shangbin Feng, Yulia Tsvetkov

Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? (2023.findings-emnlp)

Challenge:	Existing research is leveraging multiple language models with diverse skills and strengths to collaborate.
Approach:	They propose mitigation strategies to mitigate the impact of malicious models by employing external supervisors to disable/mask them out to reduce their influence.
Outcome:	The proposed mitigation strategies recover 95.31% of initial performance while making model collaboration systems fully resistant to malicious models remains an open question.

Copied to clipboard

Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, Luke Zettlemoyer

Threat Scenarios and Best Practices to Detect Neural Fake News (2022.coling-1)

Challenge:	Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior.
Approach:	They propose a human readable prompt tuning method that incorporates a fluency constraint to find a distribution of effective and fluent prompts.
Outcome:	The proposed method outperforms baselines by 7.0% across three tasks.

Copied to clipboard

Artidoro Pagnoni, Martin Graciarena, Yulia Tsvetkov

Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation (2022.emnlp-main)

Challenge:	During the COVID-19 pandemic, inaccurate information made it hard for people to find reliable guidance when they needed it.
Approach:	They propose to use pretrained language models to generate fluent, original text . they argue that strong detectors should be released along with new generators .
Outcome:	The proposed system is prone to shortcut learning and should be released along with new generators.

Copied to clipboard

Melanie Sclar, Peter West, Sachin Kumar, Yulia Tsvetkov, Yejin Choi

Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021.naacl-main)

Challenge:	a new framework for sentence summarization is available that can be trained reference-free . a high-quality dataset of sentence-summary pairs with varying degrees of compression ratios is obtained .
Approach:	They propose a framework for sentence summarization that can be trained reference-free . they propose 'referee' that iteratively distills latent knowledge into better models .
Outcome:	The proposed framework outperforms existing models in the use of explicit examples from teacher models without compromising the quality of the summarization.

Copied to clipboard

Artidoro Pagnoni, Vidhisha Balachandran, Yulia Tsvetkov

SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation (2024.naacl-long)

Challenge:	Modern summarization models generate fluent but often factually unreliable outputs.
Approach:	They propose to use human annotations to identify different categories of factual errors and benchmark factuality metrics to improve summarization evaluation.
Outcome:	The proposed method identifies the proportion of different categories of factual errors and benchmarks their human judgements as well as their specific strengths and weaknesses.

Copied to clipboard

Abe Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, Yulia Tsvetkov

David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs (2024.naacl-long)

Challenge:	Existing watermarked generation algorithms employ token-level designs and are vulnerable to paraphrase attacks.
Approach:	They propose a sentence-level watermarking algorithm that uses locality-sensitive hashing to partition the semantic space of sentences.
Outcome:	The proposed algorithm is more robust than the existing state-of-the-art method on paraphrasers and domains, while posing only minor degradations to SemStamp.

Copied to clipboard

Xiaochuang Han, Sachin Kumar, Yulia Tsvetkov, Marjan Ghazvininejad

ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactions (2024.findings-emnlp)

Challenge:	Existing studies of diffusion-based language models have been conducted on a smaller scale.
Approach:	They propose to scale an autoregressive diffusion model from 0.4B to 13B parameters and propose techniques to improve its training and inference efficiency.
Outcome:	The proposed model is able to combine a large general-purpose diffusion model with smaller, but specialized and contextualized diffusion models at inference time.

Copied to clipboard

Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanu Mitra, David Jurgens, Yulia Tsvetkov

Challenge:	VALUESCOPE is a framework that quantifies social norms and values within online communities.
Approach:	They propose a framework that uses language models to quantify social norms and values within online communities.
Outcome:	The proposed framework delineates differences in social norms and tracks evolution of norms in online communities and influence of significant external events like the U.S. presidential elections and the emergence of new sub-communities.

Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models (2024.findings-acl)

Copied to clipboard

Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, Yulia Tsvetkov