Papers by Diyi Yang

SPORTSINTERVIEW: A Large-Scale Sports Interview Benchmark for Entity-centric Dialogues (2022.lrec-1)

Challenge:	Current speech evaluation systems rely on specialized systems for individual audio characteristics and poor correlation between automatic methods and human preferences.
Approach:	They propose a unified evaluation framework for Large Audio Models as a Judge, AudioJudge . they propose specialized judges that can be prompted to perform audio characteristic detection tasks .
Outcome:	The proposed method improves performance across audio characteristic detection and human preference simulation tasks.

Copied to clipboard

Hanfei Sun, Ziyuan Cao, Diyi Yang

Natural Language Interfaces

Challenge:	Existing knowledge grounded dialogue datasets only contain external knowledge from one dimension, which limits the diversity of knowledge sources and may contain unwanted bias.
Approach:	They propose to use two types of external knowledge sources as knowledge grounding in an interview dataset to model human dialogues.
Outcome:	The proposed dataset contains 150K interviews and 34K interviewees . it is larger in size and has more than one dimension of external knowledge linking . however, the performance of the proposed models is far from humans .

CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations (2023.emnlp-main)

Copied to clipboard

Myra Cheng, Tiziano Piccardi, Diyi Yang

Challenge:	Recent work has aimed to capture nuances of human behavior by using LLMs to simulate responses from demographics in social science experiments and public opinion surveys.
Approach:	They propose a framework to characterize LLM simulations using four dimensions: Context, Model, Persona, and Topic.
Outcome:	The proposed framework measures open-ended LLM simulations’ susceptibility to caricature, defined via two criteria: individuation and exaggeration.

The Importance of Modeling Social Factors of Language: Theory and Practice (2021.naacl-main)

Copied to clipboard

Dirk Hovy, Diyi Yang

Linguistic Theories

Challenge:	Current NLP models focus on information content while ignoring language’s social factors.
Approach:	They propose that NLP systems focus on information content while ignoring language’s social factors to improve performance.
Outcome:	The proposed approach improves the performance of existing systems, open up new applications, and increase fairness and usability for all users.

Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)

Copied to clipboard

Personalized Response Generation via Generative Split Memory Network (2021.naacl-main)

Challenge:	Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations.
Approach:	They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers .
Outcome:	The proposed framework could update LLMs without having to retrain the whole model after data removals.

Copied to clipboard

Yuwei Wu, Xuezhe Ma, Diyi Yang

Forgotten Knowledge: Examining the Citational Amnesia in NLP (2023.acl-long)

Challenge:	Despite the success of text generation and dialogue systems, how to endow a text generation system with personality traits remains under-investigated.
Approach:	They propose a model to generate personalized responses on reddit using user profiles and posting histories.
Outcome:	The proposed model improves over the state-of-the-art response generation models.

Copied to clipboard

Janvijay Singh, Mukund Rungta, Diyi Yang, Saif Mohammad

Culture Cartography: Mapping the Landscape of Cultural Knowledge (2025.emnlp-main)

Challenge:	a recent study examines how far back in time we tend to cite papers . citation patterns are correlated with age, age, and other factors .
Approach:	They analyze citation patterns across time and examine temporal changes . they find that 62% of cited papers are from the immediate five years prior to publication .
Outcome:	The authors show that citing papers is the primary method of scientific writing . they show that the trend has reversed and current papers have low temporal diversity .

Copied to clipboard

Caleb Ziems, William Barr Held, Jane Yu, Amir Goldberg, David Grusky, Diyi Yang

Challenge:	Large Language Models (LLMs) can empower users to be more knowledgeable, productive, and creative, but their utility is often diminished for under-represented groups and cultures.
Approach:	They propose a methodology that operationalizes a mixed-initiative approach to finding culture-specific knowledge that is salient to in-group users but unknown to LLMs.
Outcome:	The proposed method improves the accuracy of LLMs on culturally-competent language models by 19.2%.

HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability (2021.acl-long)

Copied to clipboard

Jiaao Chen, Dinghan Shen, Weizhu Chen, Diyi Yang

Learning with Limited Text Data (2022.acl-tutorials)

Challenge:	Using data augmentation to fine-tune pre-trained models with task-specific data has been shown to be ineffective and redundant during fine-timing.
Approach:	They propose a data augmentation technique to regularize pre-trained models and encourage them to learn more generalizable features by dropping contiguous spans during training.
Outcome:	The proposed method outperforms state-of-the-art methods on the GLUE benchmark and consistently exhibits superior generalization performances on out-of distribution and challenging counterexamples.

Copied to clipboard

Diyi Yang, Ankur Parikh, Colin Raffel

To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence (2021.findings-emnlp)

Challenge:	Natural Language Processing (NLP) relies on labeled data to perform state-of-the-art performance . labeles are often required to label large amounts of textual data . this tutorial will provide an overview of labeleing in NLP .
Approach:	This tutorial will provide a systematic overview of methods for learning from limited labeled data.
Outcome:	This tutorial will provide a systematic and up-to-date overview of the proposed methods . it will highlight current challenges and future directions .

Copied to clipboard

Caleb Ziems, Diyi Yang

Ethical NLP

Challenge:	a new study examines the media coverage of police violence in the United States by examining the framing of 82k news articles spanning 7k police killings.
Approach:	They propose an NLP framework to measure entity-centric framing to understand media coverage on police violence in the United States in a new police violence frames corpus of 82k news articles spanning 7k police killings.
Outcome:	The proposed framework reveals significant differences in the way liberal and conservative news sources frame both the issue of police violence and the entities involved.

Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information Maximization (2020.findings-emnlp)

Copied to clipboard

Kunal Chawla, Diyi Yang

Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs (2021.naacl-main)

Challenge:	Formality style transfer is the task of converting informal sentences to grammatically-correct formal sentences.
Approach:	They propose a semi-supervised formality style transfer model that utilizes a language model-based discriminator to maximize the likelihood of the output sentence being formal.
Outcome:	The proposed model outperforms state-of-the-art models in terms of automated metrics and human judgement.

Copied to clipboard

Werewolf Among Us: Multimodal Resources for Modeling Persuasion Behaviors in Social Deduction Games (2023.findings-acl)

Challenge:	Abstractive conversation summarization has received much attention, but it suffers from insufficient, redundant, or incorrect content due to the unstructured and complex characteristics of human-human interactions.
Approach:	They propose to model rich structures in conversations for more precise and accurate conversation summarization by incorporating discourse relations between utterances and action triples in utterrances and designing a multi-granularity decoder to generate summaries by combining all levels of information.
Outcome:	The proposed models outperform state-of-the-art methods and generalize well in other domains in terms of automatic evaluations and human judgments.

Copied to clipboard

Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Anugrah Hayati, James Rehg, Diyi Yang

Modeling Cross-Cultural Pragmatic Inference with Codenames Duet (2023.findings-acl)

Challenge:	Existing studies on persuasive behavior modeling focus on textual dialogues . a multimodal dataset is available for persuasion modeling .
Approach:	They propose a multimodal dataset for modeling persuasive behaviors using visual signals.
Outcome:	The proposed dataset includes 199 dialogue transcriptions and videos captured in a multi-player social deduction game setting and 26,647 utterance level annotations of persuasion strategy and game level annotation of deduction game outcomes.

Copied to clipboard

Omar Shaikh, Caleb Ziems, William Held, Aryan Pariani, Fred Morstatter, Diyi Yang

Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023.emnlp-main)

Challenge:	Existing work on pragmatic reasoning tests using simple word reference games with unidentified speakers and listeners, but speakers' sociocultural background shapes their pragmatic assumptions.
Approach:	They propose a dataset which operationalizes sociocultural pragmatic inference in a word reference game.
Outcome:	The proposed model improves clue-giving and guessing tasks by accounting for background characteristics and the game context.

Copied to clipboard

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, Diyi Yang

Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2022.tacl-1)

Challenge:	Recent advances in scale have enabled large language models to perform NLP tasks zero-shot . however, it is not known whether ChatGPT can serve as a generalist model that can perform many NLP jobs zero- shot.
Approach:	They empirically evaluate ChatGPT's zero-shot learning ability on 20 popular NLP datasets . they find it performs well on many tasks favoring reasoning abilities .
Outcome:	The proposed model can perform many NLP tasks zero-shot without adaptation on downstream data.

Copied to clipboard

Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, Diyi Yang

Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension (2022.acl-long)

Challenge:	causality has not had the same importance in natural language processing, says aaron e. smith . he says research on causality in NLP remains scattered across domains without unified definitions .
Approach:	They propose to consolidate research on causality in NLP across academic areas . they explore potential uses of causal inference to improve robustness, fairness, interpretability .
Outcome:	The proposed method is a unified overview of causal inference for the NLP community.

Copied to clipboard

Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, Mark Warschauer

Question Answering

Challenge:	Existing QA datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements.
Approach:	They propose to use FairytaleQA to generate 10,580 questions based on 278 children-friendly stories to assess model's fine-grained learning skills.
Outcome:	The proposed dataset consists of 10,580 questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations.

NormBank: A Knowledge Bank of Situational Social Norms (2023.acl-long)

Copied to clipboard

Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, Diyi Yang

Challenge:	NormBank is a knowledge bank of 155k situational norms that can be used to ground flexible normative reasoning for interactive, assistive, and collaborative AI systems.
Approach:	They propose a new scheme for hierarchically organizing the seemingly unbounded social norms within a multivalent sociocultural frame.
Outcome:	The proposed framework can be used to ground flexible reasoning for interactive, assistive, and collaborative AI systems.

Summarization of Dialogues and Conversations At Scale (2023.eacl-tutorials)

Copied to clipboard

Diyi Yang, Chenguang Zhu

Natural Language Interfaces

Challenge:	Conversations are the natural communication format for people.
Approach:	This tutorial will survey the cutting-edge methods for summarizing written and spoken conversation.
Outcome:	This tutorial will examine the cutting-edge methods for summarizing written and spoken conversations, covering key sub-areas whose combination is needed for a successful solution.

DynaMiTE: Discovering Explosive Topic Evolutions with User Guidance (2023.findings-acl)

Copied to clipboard

Nishant Balepur, Shivam Agarwal, Karthik Venkat Ramanan, Susik Yoon, Diyi Yang, Jiawei Han

Topic Modeling

Challenge:	Existing Dynamic topic models are either fully supervised, requiring expensive human annotations, or fully unsupervised, producing topic evolutions that often do not cater to a user’s needs.
Approach:	They propose to use a framework that ensembles semantic similarity, category indicative, and time indicative scores to produce informative topic evolutions.
Outcome:	The proposed framework can be used to discover topic evolutions from temporal corpora that align with user-provided category names and uniquely capture topics at each time step.

Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles (2024.emnlp-main)

Copied to clipboard

Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, Diyi Yang

Perceptions of Language Technology Failures from South Asian English Speakers (2024.findings-acl)

Challenge:	Existing methods for improving LLMs in simulations are limited due to privacy concerns and limited domain knowledge.
Approach:	They propose a pipeline that elicits qualitative feedback from a domain-expert and transforms it into a set of principles that govern an LLM-prompted roleplay.
Outcome:	The proposed pipeline shows a 30% improvement in response quality and principle following for the downstream task.

Copied to clipboard

Faye Holt, William Held, Diyi Yang

Challenge:	Existing studies have identified performance disparities between Standard American English and other English dialects, but the degree to which these discrepancies affect user experience is not well understood.
Approach:	They aim to reduce performance gap for South Asian Englishes by surveying their interactions with language technology and comparing their results to a control survey.
Outcome:	The proposed model reduces the performance gap for South Asian Englishes (SAsE) speakers are more likely to recall failures with language technology and to reference specific issues with written language technology than SAmE speakers.

Generative Interfaces for Language Models (2026.findings-acl)

Copied to clipboard

Jiaqi Chen, Yanzhe Zhang, Yutong Zhang, Yijia Shao, Diyi Yang

The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems (2022.acl-long)

Challenge:	Large language models are increasingly seen as assistants, copilots, and consultants . however, their linear request-response format often makes interactions inefficient in multi-turn tasks .
Approach:	They propose a paradigm in which large language models respond to user queries by generating user interfaces that enable more adaptive and interactive engagement.
Outcome:	The proposed paradigm outperforms traditional chat-based interfaces in many tasks and interaction patterns.

Copied to clipboard

Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, Diyi Yang

Continual Sequence Generation with Adaptive Compositional Modules (2022.acl-long)

Challenge:	Moral integrity corpus captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs).
Approach:	They propose a resource that captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs).
Outcome:	The proposed resource captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs).

Copied to clipboard

Yanzhe Zhang, Xuezhi Wang, Diyi Yang

Modeling Gender and Dialect Bias in Automatic Speech Recognition (2024.findings-emnlp)

Challenge:	Existing work on continual sequence generation relies on reuse of existing parameters to learn new tasks or blindly adds new parameters for each new task.
Approach:	They propose to use continual sequence generation to reuse existing parameters for new tasks . current continuous learning models often forget knowledge of dissimilar tasks if data distributions shift .
Outcome:	The proposed framework outperforms state-of-the-art models on various sequences of generation tasks.

Copied to clipboard

Camille Harris, Chijioke Mgbahurike, Neha Kumar, Diyi Yang

Geographic Citation Gaps in NLP Research (2022.emnlp-main)

Challenge:	Dialect and gender-based biases have become an area of concern in language-dependent AI systems.
Approach:	They construct a podcast audio dataset and evaluate its performance . they then refine the models to better understand how finetuning can impact performance.
Outcome:	The proposed model improves on 13 hours of podcast audio transcribed by speakers of four US-based English dialects.

Copied to clipboard

Mukund Rungta, Janvijay Singh, Saif M. Mohammad, Diyi Yang

Leveraging Expert Guided Adversarial Augmentation For Improving Generalization in Named Entity Recognition (2022.findings-acl)

Challenge:	a vast number of papers accepted at top NLP venues come from a handful of western countries and (lately) China.
Approach:	They ask researchers to examine the relationship between geographical location and publication success . they use a dataset of 70,000 papers from the ACL Anthology to examine their citation network .
Outcome:	The proposed dataset of 70,000 papers from the ACL Anthology shows that there are substantial geographical disparities in paper acceptance and citations .

Copied to clipboard

Aaron Reich, Jiaao Chen, Aastha Agrawal, Yanzhe Zhang, Diyi Yang

Multi-VALUE: A Framework for Cross-Dialectal English NLP (2023.acl-long)

Challenge:	Named Entity Recognition (NER) systems perform well on in-distribution data, but perform poorly on examples drawn from a shifted distribution.
Approach:	They propose to use expert-guided heuristics to change entity tokens and their contexts to alter their entity types as adversarial attacks.
Outcome:	The proposed model significantly improves performance on the challenging set and out-of-domain generalization.

Copied to clipboard

Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, Diyi Yang

Machine Translation

Challenge:	Current systems that focus on standard American English are not dialect invariant . current systems focus on a single dialect, which results in performance discrepancies .
Approach:	They propose a resource for evaluating and achieving English dialect invariance . they stress test question answering, machine translation, and semantic parsing .
Outcome:	The proposed system is based on a rule-based translation system spanning 50 English dialects and 189 unique linguistic features.

TADA : Task Agnostic Dialect Adapters for English (2023.findings-acl)

Copied to clipboard

William Held, Caleb Ziems, Diyi Yang

VALUE: Understanding Dialect Disparity in NLU (2022.acl-long)

Challenge:	Existing work on dialectal English NLP is task-specific, using manual annotated dialect data, weak supervision, or data augmentation.
Approach:	They propose a method for task-agnostic dialect adaptation by aligning non-SAE dialects with task-specific adapters from SAE.
Outcome:	The proposed method improves dialectal robustness on 4 dialectal variants of the GLUE benchmark without task-specific supervision.

Copied to clipboard

Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, Diyi Yang

Challenge:	English Natural Language Understanding systems outperform humans on benchmarks like GLUE and SuperGLUE, but they only use textbook Standard American English (SAE) . fewer studies have considered the effects of dialectal differences on performance .
Approach:	They propose a benchmark to evaluate the performance of English natural language understanding systems using a set of lexical and morphosyntactic transformation rules.
Outcome:	The proposed model outperforms humans on GLUE and SuperGLUE, but only on standard American English . the proposed model recruits fluent speakers of African American vernacular english to validate each feature transformation .

Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021.naacl-main)

Copied to clipboard

Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, Diyi Yang

Information Extraction & Text Mining

Challenge:	Existing continual learning methods focus on preserving knowledge from previous tasks . Continual learning is a useful tool for learning over time, but it is not always possible to generalize to new tasks.
Approach:	They propose a disentanglement-based regularization method for continual learning on text classification that disentangles text hidden spaces into generic representations and regularizes them differently to constrain knowledge required to generalize.
Outcome:	The proposed method disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task.

DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification (2022.coling-1)

Copied to clipboard

Hui Chen, Wei Han, Diyi Yang, Soujanya Poria

Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future (2024.findings-acl)

Challenge:	Existing methods to improve the robustness of text classification models are token-, sentence-, and hiddenlevel augmentation.
Approach:	They propose an interpolation-based data augmentation approach called DoubleMix to improve the robustness of text classification models by learning the “shifted” features in hidden space.
Outcome:	The proposed approach outperforms several popular methods on six text classification benchmark datasets and visual analysis shows that the model features are highly interpretable.

Copied to clipboard

Minzhi Li, Weiyan Shi, Caleb Ziems, Diyi Yang

Challenge:	Existing work on social intelligence in NLP does not provide a coherent subfield for researchers to analyze and identify research gaps and future directions.
Approach:	They build a social AI taxonomy and a data library of 480 NLP datasets to analyze existing datasets and evaluate language models’ performance in different social intelligence aspects.
Outcome:	The proposed infrastructure analyzes existing dataset efforts and evaluates language models’ performance in different social intelligence aspects.

Measuring and Addressing Indexical Bias in Information Retrieval (2024.findings-acl)

Copied to clipboard

Caleb Ziems, William Held, Jane Dwivedi-Yu, Diyi Yang

Information Retrieval

Challenge:	Information Retrieval (IR) systems may not optimize rankings for fairness, neutrality, or the balance of ideas.
Approach:	They propose to use a framework to automatically audit IR rankings for indexical biases, or biase in the positional order of documents.
Outcome:	The proposed bias metric can help predict when and how indexical bias will shift a reader’s opinion.

Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles (2024.acl-long)

Copied to clipboard

Julia Kruk, Michela Marchini, Rijul Magu, Caleb Ziems, David Muchlinski, Diyi Yang

“Mistakes Help Us Grow”: Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms (2023.emnlp-main)

Challenge:	a dog whistle is a coded communication that carries a secondary meaning to specific audiences and is often weaponized for racial and socioeconomic discrimination.
Approach:	They propose an approach for word-sense disambiguation of dog whistles from standard speech using Large Language Models.
Outcome:	The proposed method allows disambiguation of dog whistles from standard speech using large language models.

Copied to clipboard

Kunal Handa, Margarett Clapper, Jessica Boyle, Rose Wang, Diyi Yang, David Yeager, Dorottya Demszky

Compositional Data Augmentation for Abstractive Conversation Summarization (2023.acl-long)

Challenge:	GMSL has been shown to significantly reduce disparities in academic achievement and enhance students’ learning outcomes.
Approach:	They develop a coaching tool to reframe unsupportive utterances to GMSL using large language models.
Outcome:	The proposed model outperforms the GMSL-trained teachers in fostering a growth mindset and promoting challenge-seeking behavior.

Copied to clipboard

Siru Ouyang, Jiaao Chen, Jiawei Han, Diyi Yang

Information Extraction & Text Mining

Challenge:	Abstractive conversation summarization systems rely on large-scale annotated summaries, but collecting and annotating these conversations can be time-consuming and labor-intensive.
Approach:	They propose a method for generating diverse and high-quality pairs of conversations and summaries by extracting conversation structures and organizing meaningful conversation snippets.
Outcome:	The proposed method outperforms baseline methods on SAMSum and DialogSum datasets and achieves a 10% increase in ROUGE scores with limited data.

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020.acl-main)

Copied to clipboard

Jiaao Chen, Zichao Yang, Diyi Yang

Benchmarking Machine Translation with Cultural Awareness (2024.findings-emnlp)

Challenge:	Xie et al., 2016) demonstrate that semi-supervised learning models suffer from over-fitting when there is only limited labeled data.
Approach:	They propose a semi-supervised learning method for text classification using a data augmentation method called TMix.
Outcome:	The proposed method outperforms pre-trained and fine-tuned models on several text classification benchmarks.

Copied to clipboard

Binwei Yao, Ming Jiang, Tara Bobinac, Diyi Yang, Junjie Hu

Machine Translation

Challenge:	Existing studies on terminology translation focus on breaking language barriers rather than cultural barriers.
Approach:	They propose a parallel corpus enriched with CSI annotations in 6 language pairs for investigating Cultural-Aware Machine Translation.
Outcome:	The proposed corpus is enriched with CSI annotations in 6 languages and measures translation quality.

When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain (2022.emnlp-main)

Copied to clipboard

Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, Diyi Yang

Impressions: Visual Semiotics and Aesthetic Impact Understanding (2023.emnlp-main)

Challenge:	Pre-trained language models have shown impressive performance on a variety of tasks and domains.
Approach:	They propose a domain specific financial LANGuage model which uses financial keywords and phrases for better masking.
Outcome:	The proposed model outperforms existing models on a variety of tasks and domains.

Copied to clipboard

Julia Kruk, Caleb Ziems, Diyi Yang

Deciphering Stereotypes in Pre-Trained Language Models (2023.emnlp-main)

Challenge:	Existing image captioning and conditional generation models struggle to simulate plausible human responses to images.
Approach:	They propose a dataset to investigate the semiotics of images and how visual features and design choices can elicit specific emotions, thoughts and beliefs.
Outcome:	The proposed dataset improves existing models for image captioning and conditional generation.

Copied to clipboard

Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, Soroush Vosoughi

Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers (2023.eacl-main)

Challenge:	Current approaches for examining stereotypes in PLMs require intricate human knowledge about these stereotypes and entail careful manual curation of examples.
Approach:	They propose a framework for examining stereotype-encoding behavior of PLMs using model probing and textual analyses.
Outcome:	The proposed approach can debiase PLMs without compromising their language modeling capabilities or performance.

Copied to clipboard

William Held, Diyi Yang

ToTTo: A Controlled Table-To-Text Generation Dataset (2020.emnlp-main)

Challenge:	Multilingual transformer-based models demonstrate remarkable zero and few-shot transfer across languages by learning and reusing language-agnostic features.
Approach:	They propose to prune attention heads from a fixed model to reduce interference as a model acquires more languages.
Outcome:	The proposed model can reduce interference by pruning language-specific attention heads.

Copied to clipboard

Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das

Structured Data in NLP

Challenge:	Existing methods for data-to-text generation often hallucinate phrases not supported by the Wikipedia table.
Approach:	They propose a controlled task where annotators directly revise existing Wikipedia sentences to generate a one-sentence description.
Outcome:	The proposed task produces a one-sentence description from a Wikipedia table and highlighted cells.

Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors (2024.acl-long)

Copied to clipboard

Alicja Chaszczewicz, Raj Shah, Ryan Louie, Bruce Arnow, Robert Kraut, Diyi Yang

Focus on the Action: Learning to Highlight and Summarize Jointly for Email To-Do Items Summarization (2022.findings-acl)

Challenge:	Existing mechanisms of providing feedback rely on human supervision . existing mechanisms of delivering feedback largely rely only on human oversight .
Approach:	They propose to leverage large language models to provide contextualized feedback to peer counselors . they construct a publicly available dataset with detailed feedback annotations of 400 conversations .
Outcome:	The proposed method minimizes the risk of potentially harmful and low-quality feedback generation.

Copied to clipboard

Kexun Zhang, Jiaao Chen, Diyi Yang

Summarization

Challenge:	Existing to-do item generation models focus on generating action mentions to provide more structured summaries of email text.
Approach:	They propose a learning to highlight and summarize framework to learn to identify the most salient text and actions and incorporate these structured representations to generate more faithful to-do items.
Outcome:	The proposed model outperforms baseline models and achieves state-of-the-art performance in terms of evaluation and human judgement.

Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment (2026.acl-long)

Copied to clipboard

Woody Haosheng Gan, William Barr Held, Diyi Yang

Planning and Generating Natural and Diverse Disfluent Texts as Augmentation for Disfluency Detection (2020.emnlp-main)

Challenge:	Existing LAM benchmarks with thousands of examples create substantial computational barriers.
Approach:	They examine whether subsets can reliably evaluate large audio models . they find that subset of 50 examples can achieve over 0.93 Pearson correlation with full benchmark .
Outcome:	The proposed method outperforms the full benchmark and subset selection methods.

Copied to clipboard

Jingfeng Yang, Diyi Yang, Zhaoran Ma

Challenge:	Existing approaches to disfluency detection heavily depend on labeled data.
Approach:	They propose a Planner-Generator based disfluency generation model that generates natural disfluent texts as augmented data.
Outcome:	The proposed model outperforms baselines and leads to state-of-the-art performance on Switchboard corpus.

Measure and Improve Robustness in NLP Models: A Survey (2022.naacl-main)

Copied to clipboard

Xuezhi Wang, Haohan Wang, Diyi Yang

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise (2023.emnlp-main)

Challenge:	Despite the performance gains, NLP models are still fragile and brittle to out-of-domain data, adversarial attacks, or small perturbation to the input.
Approach:	They propose a survey of how to define, measure and improve robustness in NLP by connecting multiple definitions of robustness and identifying failures.
Outcome:	The proposed models are robust against unseen or challenging scenarios, but are still fragile and brittle to out-of-domain data and adversarial attacks.

Copied to clipboard

Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, Diyi Yang

Local Additivity Based Data Augmentation for Semi-supervised NER (2020.emnlp-main)

Challenge:	Existing diffusion models have limitations in modeling discrete data, e.g., languages . we present a novel diffusion model for language modeling inspired by linguistic features in languages based on iterative denoising .
Approach:	They propose a method that iteratively denoises and adds corruptions to the textual data through soft-masking to better noise it.
Outcome:	The proposed model achieves better generation quality and lower training cost than current models with better performance.

Copied to clipboard

Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, Diyi Yang

Simple Conversational Data Augmentation for Semi-supervised Abstractive Dialogue Summarization (2021.emnlp-main)

Challenge:	Named Entity Recognition (NER) is one of the first stages in deep language understanding yet current NER models heavily rely on human-annotated data.
Approach:	They propose a Local Additivity based Data Augmentation method for semi-supervised Named Entity Recognition (NER) that creates virtual samples by interpolating sequences close to each other.
Outcome:	The proposed method improves both entity and context learning by adding to training data and extending it to semi-supervised setting.

Copied to clipboard

One Document, Many Revisions: A Dataset for Classification and Description of Edit Intents (2022.lrec-1)

Challenge:	Abstractive conversation summarization models heavily rely on human-annotated summaries.
Approach:	They propose a set of Conversational Data Augmentation methods for semi-supervised abstractive conversation summarization that use random swapping/deletion to perturb the discourse relations inside conversations and dialogue-acts-guided insertion to interrupt the development of conversations.
Outcome:	The proposed methods over several state-of-the-art datasets show that they are more efficient than previous methods.

Copied to clipboard

Dheeraj Rajagopal, Xuchao Zhang, Michael Gamon, Sujay Kumar Jauhar, Diyi Yang, Eduard Hovy

Information Retrieval

Challenge:	Existing methods to understand revisions have failed to provide a deeper understanding of the nature of these edits.
Approach:	They propose to use a Wikipedia revision history dataset to train a classifier that achieves a 90% accuracy in identifying edit intent and a distantly-supervised model that generates .
Outcome:	The proposed model achieves 90% accuracy in identifying edit intent and a best score of 28 ROUGE.

Frustratingly Simple but Surprisingly Strong: Using Language-Independent Features for Zero-shot Cross-lingual Semantic Parsing (2021.emnlp-main)

Copied to clipboard

Jingfeng Yang, Federico Fancellu, Bonnie Webber, Diyi Yang

Explaining Toxic Text via Knowledge Enhanced Text Generation (2022.naacl-main)

Challenge:	Existing training data is limited for languages other than English, so is the performance of the developed parsers.
Approach:	They propose to apply a pre-trained multilingual model to Italian, German and Dutch parsers where only a small number of manually annotated parses are available.
Outcome:	The proposed model improves on six parsers in English and Italian, German and Dutch, with the addition of universal dependency relations and universal POS tags as model-agnostic features.

Copied to clipboard

Rohit Sridhar, Diyi Yang

Inducing Positive Perspectives with Text Reframing (2022.acl-long)

Challenge:	Existing work on toxic speech classification relies on generic and repetitive explanations . elucidating toxic speech can help with downstream tasks such as debiasing .
Approach:	They propose a knowledge-informed encoder-decoder framework to generate toxic text explanations . they use multiple knowledge sources to generate detailed explanations of toxic text .
Outcome:	The proposed model outperforms state-of-the-art models significantly in generating toxic explanations . the proposed model can generate detailed explanations of toxic speech compared to baselines compared with baseline models .

Copied to clipboard

Caleb Ziems, Minzhi Li, Anthony Zhang, Diyi Yang

WIKIBIAS: Detecting Multi-Span Subjective Biases in Language (2021.findings-emnlp)

Challenge:	Sentiment transfer is a text style transfer task that aims to reverse sentiment polarity and reversal in meaning.
Approach:	They propose a task called positive reframing that neutralizes a negative point of view and generates 'positive' perspectives without contradicting original meaning.
Outcome:	The proposed model neutralizes a negative point of view and generates 'positive' perspectives without contradicting the original meaning.

Copied to clipboard

Yang Zhong, Jingfeng Yang, Wei Xu, Diyi Yang

Culturally Aware Natural Language Inference (2023.findings-emnlp)

Challenge:	a particular type of bias is subjective bias, which introduces improper attitudes or presents a statement with the presupposition of truth.
Approach:	They propose to annotate a Wikipedia edits corpus with 4,000 sentence pairs to detect subjective bias.
Outcome:	The proposed dataset can be used as a research benchmark and generalize to multiple domains.

Copied to clipboard

Jing Huang, Diyi Yang

Reasoning

Challenge:	Cultural norms are behavioral rules and conventions shared within specific groups, connecting cultural symbols and values.
Approach:	They propose a task that operationalizes cultural variations in language understanding through a natural language inference task that surfaces cultural variations as label disagreement between annotators from different cultural groups.
Outcome:	The proposed model can be evaluated at which levels it is culturally aware.

Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints (2023.findings-eacl)

Copied to clipboard

Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, Diyi Yang

Controllable Conversation Generation with Conversation Structures via Diffusion Models (2023.findings-acl)

Challenge:	Existing and potential applications of open-ended text generation are farreaching, spanning domains such as QA, story generation, open-end dialogue, and ChatGPT 1 .
Approach:	They propose a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models by a set of structural and stylistic prompts.
Outcome:	The proposed method can be generalized to other large models like BLOOM and OPT.

Copied to clipboard

Mind the Gap: Static and Interactive Evaluations of Large Audio Models (2025.acl-long)

Challenge:	Current generation models fail to effectively utilize rich linguistic and world knowledge to generate coherent long text.
Approach:	They propose a conversation generation framework that incorporates human knowledge and conversation structures with both controllability and interpretability for better conversation generation.
Outcome:	The proposed framework incorporates human knowledge and conversation structures with both controllability and interpretability for better conversation generation.

Copied to clipboard

Minzhi Li, William Barr Held, Michael J Ryan, Kunat Pipatanakul, Potsawee Manakul, Hao Zhu, Diyi Yang

Speech & Audio in NLP

Challenge:	Recent work has focused on evaluating large audio models (LAMs) that directly accept audio inputs.
Approach:	They propose an interactive approach to evaluate large audio models and collect 7,500 LAM interactions from 484 participants.
Outcome:	The proposed model is based on a set of user-generated audio interfaces with 7,500 interactions from 484 participants.

Simulated Misinformation Susceptibility (SMISTS): Enhancing Misinformation Research with Large Language Model Simulations (2024.findings-acl)

Copied to clipboard

Weicheng Ma, Chunyuan Deng, Aram Moossavi, Lili Wang, Soroush Vosoughi, Diyi Yang

Attacking Vision-Language Computer Agents via Pop-ups (2025.acl-long)

Challenge:	Psychological inoculations have shown efficacy in curbing its spread and mitigating its adverse effects at early stages, but their design and optimization typically requires substantial human and financial resources due to the need for repeated experimental trials.
Approach:	They propose to use large language models to simulate participant responses in misinformation studies to mitigate caricatures and stereotypes in the simulations.
Outcome:	The proposed method mitigates caricatures and stereotypes in LLM simulations and enhances response diversity.

Copied to clipboard

Yanzhe Zhang, Tao Yu, Diyi Yang

SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022.findings-naacl)

Challenge:	Existing tools for analyzing and testing VLMs are lacking in understanding what types of attacks are possible and what types are effective.
Approach:	They propose to integrate pop-ups into existing agent testing environments to attack VLM agents by ignoring them.
Outcome:	The proposed attack success rate is 86% and decreases by 47% when integrating pop-ups into existing agent testing environments.

Copied to clipboard

Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, Diyi Yang

TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding (2022.naacl-main)

Challenge:	Recent research shows promising results on combining pretrained language models with canonical utterance for few-shot semantic parsing.
Approach:	They propose a few-shot semantic parsing method that decomposes a problem into a sequence of sub-problems, which correspond to the sub-clauses of the formal language.
Outcome:	The proposed method achieves SOTA performance of BART-based models on GeoQuery and EcommerceQuery, which are two few-shot datasets with compositional data split.

Copied to clipboard

Le Zhang, Zichao Yang, Diyi Yang

SPHERE: An Evaluation Card for Human-AI Systems (2025.findings-acl)

Challenge:	Existing data augmentation methods miss the important characteristic of compositionality, meaning of a complex expression is built from its sub-parts.
Approach:	They propose a compositional data augmentation approach for natural language understanding called TreeMix that leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data enhancing technique to recombine them to generate new sentences.
Outcome:	The proposed approach outperforms current state-of-the-art methods on text classification and SCAN.

Copied to clipboard

Dora Zhao, Qianou Ma, Xinran Zhao, Chenglei Si, Chenyang Yang, Ryan Louie, Ehud Reiter, Diyi Yang, Tongshuang Wu

Task-Agnostic Low-Rank Adapters for Unseen English Dialects (2023.emnlp-main)

Challenge:	Existing evaluation methods and standards for human-AI systems are unclear, especially for large language models.
Approach:	They propose an evaluation card SPHERE which provides a template for evaluation protocols . they outline current evaluation practices and areas for improvement .
Outcome:	The evaluation card provides a template for designing evaluation protocols . it outlines current evaluation practices and areas for improvement .

Copied to clipboard

Zedian Xiao, William Held, Yanchen Liu, Diyi Yang

Green & Sustainable NLP

Challenge:	a recent study found that LLMs are trained on corpora disproportionally weighted in favor of Standard American English . prior work on dialect struggle with generalizing to evolving and emerging dialects in a scalable manner.
Approach:	They propose a method that leverages linguistic knowledge to enable resource-efficient adaptation . their method disentangles dialect-specific and cross-dialectal information .
Outcome:	a new method improves generalization to unseen dialects in a task-agnostic fashion . it achieves the best or most competitive performance across 5 dialects .

Distilling an End-to-End Voice Assistant Without Instruction Training Data (2025.acl-long)

Copied to clipboard

William Barr Held, Yanzhe Zhang, Weiyan Shi, Minzhi Li, Michael J Ryan, Diyi Yang

Social Intelligence in the Age of LLMs (2025.naacl-tutorial)

Challenge:	Recent efforts to train speech-only LLMs have led to models “forging” speech information from text-only models.
Approach:	They propose a paradigm for training Speech Large Language Models without instruction data by using the response of a text-only LLM to transcripts as self-supervision.
Outcome:	The proposed model generalizes to Spoken Question Answering, Classification, and Translation and achieves a 72% win rate compared with state-of-the-art models like Qwen 2 Audio .

Copied to clipboard

Hao Zhu, Bodhisattwa Prasad Majumder, Dirk Hovy, Diyi Yang

Identifying Unlearned Data in LLMs via Membership Inference Attacks (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) are a powerful tool for integrating human-like communication and context-aware interactions into artificial systems.
Approach:	They propose to introduce and overview different aspects of artificial social intelligence and their relationship with LLMs by introducing scientific methods for evaluating social intelligence in LLM.
Outcome:	This tutorial will introduce scientific methods for evaluating social intelligence in LLMs, highlighting the key challenges, and identifying promising research directions.

Copied to clipboard

Advit Deepak, Megan Mou, Jing Huang, Diyi Yang

On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2023.acl-long)

Challenge:	Existing work evaluates approximate unlearning under a retrieval paradigm, where adversaries attempt to extract residual knowledge given partial information of the unlearning target.
Approach:	They propose a framework to evaluate unlearning membership attacks using member inference techniques to exploit the forget set.
Outcome:	The proposed framework assesses whether unlearning leaves behind detectable artifacts that can be exploited to infer membership in the forget set.

Copied to clipboard

Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, Diyi Yang

Demystifying Verbatim Memorization in Large Language Models (2024.emnlp-main)

Challenge:	Prior work has focused on logical reasoning tasks; it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts.
Approach:	They perform a controlled evaluation of zero-shot CoT reasoning in two socially sensitive domains: harmful questions and stereotype benchmarks.
Outcome:	The results show that zero-shot CoT reasoning increases model’s likelihood to produce harmful or undesirable output, but decreases with improved instruction following.

Copied to clipboard

Jing Huang, Diyi Yang, Christopher Potts

Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach (2024.emnlp-main)

Challenge:	Existing studies have shown that Large Language Models (LLMs) memorize long sequences verbatim, with serious copyright and privacy implications.
Approach:	They develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences.
Outcome:	The proposed framework creates a control model M () and a treatment model M with injected sequences.

Copied to clipboard

Yanchen Liu, Mingyu Ma, Wenna Qin, Azure Zhou, Jiaao Chen, Weiyan Shi, Wei Wang, Diyi Yang

Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models (2022.findings-naacl)

Challenge:	Existing studies on susceptibility to misinformation rely on self-reported beliefs, which can be subject to bias, expensive to collect, and challenging to scale for downstream applications.
Approach:	They propose a computational approach to efficiently model users’ latent susceptibility levels by using demographic factors and political ideology as inputs.
Outcome:	The proposed model shows that political leanings and other psychological factors exhibit varying degrees of association with susceptibility to COVID-19 misinformation.

Copied to clipboard

Tianlu Wang, Rohit Sridhar, Diyi Yang, Xuezhi Wang

Human-AI Interaction in the Age of LLMs (2024.naacl-tutorials)

Challenge:	Existing work identifies task-specific shortcuts via human priors or error analyses, which requires extensive expertise and efforts.
Approach:	They propose to automatically identify spurious correlations in NLP models at scale by using existing interpretability methods to extract tokens that significantly affect model’s decision process.
Outcome:	The proposed method can identify spurious correlations in NLP models at scale and mitigate these leads to more robust models in multiple applications.

Copied to clipboard

Diyi Yang, Sherry Tongshuang Wu, Marti A. Hearst

Mitigating Biases in Hate Speech Detection from A Causal Perspective (2023.findings-emnlp)

Challenge:	Large Language Models (LLMs) have revolutionized the capabilities of AI systems.
Approach:	This tutorial will provide an overview of the interaction between humans and Large Language Models (LLMs) it will start with a review of the types of AI models we interact with and walkthrough of the core concepts in Human-AI Interaction.
Outcome:	This tutorial will provide an overview of the interaction between humans and LLMs, exploring the challenges, opportunities, and ethical considerations that arise in this dynamic landscape.

Copied to clipboard

Zhehao Zhang, Jiaao Chen, Diyi Yang

Language Agents: Foundations, Prospects, and Risks (2024.emnlp-tutorials)

Challenge:	Existing methods to detect hate speech are prone to spurious correlations between training data and labels, which could lead to biased treatment of vulnerable and minority groups.
Approach:	They propose to use grammar induction to find grammar patterns for hate speech and analyze this phenomenon from a causal perspective.
Outcome:	The proposed methods can detect hate speech from a causal perspective and are robust to different datasets.

Copied to clipboard

Yu Su, Diyi Yang, Shunyu Yao, Tao Yu

Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering (2021.findings-acl)

Challenge:	Language agents are autonomous agents that can follow language instructions to perform diverse tasks in real-world or simulated environments.
Approach:	They propose to provide a conceptual framework for language agents and a comprehensive discussion on key topics.
Outcome:	The proposed tutorial provides a conceptual framework of language agents and comprehensive discussion on important topic areas.

Copied to clipboard

Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, Manaal Faruqui

Natural Language Interfaces

Challenge:	Disfluencies are an under-studied topic in NLP, even though it is ubiquitous in human conversation.
Approach:	They propose a challenge question answering dataset where humans introduce contextual disfluencies in previously fluent questions.
Outcome:	The proposed dataset shows that existing models degrade significantly when tested on DISFL-QA in a zero-shot setting.

Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency (2023.emnlp-main)

Copied to clipboard

Eric Zelikman, Wanjing Ma, Jasmine Tran, Diyi Yang, Jason Yeatman, Nick Haber

Are Large Language Models Consistent over Value-laden Questions? (2024.findings-emnlp)

Challenge:	Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses.
Approach:	They propose to fine-tune large language models to simulate how previous students would have responded to unseen items to generate high-quality parallel tests.
Outcome:	The proposed test forms are designed to be content-equivalent and produce identical individual scores as the original test form.

Copied to clipboard

Jared Moore, Tanvi Deshpande, Diyi Yang

Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (2021.emnlp-main)

Challenge:	Large language models (LLMs) appear to bias survey answers toward certain values . however, some argue that LLMs are inconsistent to simulate particular values - a recent study .
Approach:	They define value consistency as similarity of answers across paraphrases, related questions and multilingual translations of a question to English, Chinese, German, and Japanese.
Outcome:	The proposed model is consistent across paraphrases, use-cases, translations, and within a topic.

Copied to clipboard

Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, Diyi Yang

An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2023.tacl-1)

Challenge:	Existing studies on explicit or overt hate speech have failed to address a more pervasive form based on coded or indirect language.
Approach:	They propose a theoretically-justified taxonomy of implicit hate speech and a benchmark corpus with fine-grained labels for each message and its implication.
Outcome:	The proposed dataset will serve as a useful benchmark for understanding this multifaceted issue.

Copied to clipboard

Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, Diyi Yang

HypMix: Hyperbolic Interpolative Data Augmentation (2021.emnlp-main)

Challenge:	Existing methods for enhancing data efficiency in limited labeled data are limited.
Approach:	They propose to use data augmentation methods to increase the efficiency of limited data learning in NLP.
Outcome:	The proposed methods perform well on topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks.

Copied to clipboard

Ramit Sawhney, Megh Thakkar, Shivam Agarwal, Di Jin, Diyi Yang, Lucie Flek

DMix: Adaptive Distance-aware Interpolative Mixup (2022.acl-short)

Challenge:	Existing methods for data augmentation involve performing mathematical operations over the raw input samples or their latent states representations, but these operations are performed in the Euclidean space, simplifying these representations and resulting in noisy interpolations.
Approach:	They propose a model-, data-, and modality-agnostic interpolative data augmentation technique operating in the hyperbolic space that captures the complex geometry of input and hidden state hierarchies better than its contemporaries.
Outcome:	The proposed technique outperforms state-of-the-art methods on benchmark and low resource datasets across speech, text, and vision modalities.

Copied to clipboard

Ramit Sawhney, Megh Thakkar, Shrey Pandit, Ritesh Soun, Di Jin, Diyi Yang, Lucie Flek

Challenge:	Interpolation-based regularisation methods such as Mixup have shown to be effective for various tasks and modalities.
Approach:	They propose an adaptive distance-aware interpolative Mixup that selects samples based on their diversity in the embedding space.
Outcome:	The proposed method achieves state-of-the-art on sentence classification over existing methods on 8 benchmark datasets across English, Arabic, Turkish, and Hindi languages while achieving benchmark F1 scores in 3 times less number of iterations.

DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue (2023.acl-long)

Copied to clipboard

William Held, Christopher Hidey, Fei Liu, Eric Zhu, Rahul Goel, Diyi Yang, Rushin Shah

SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs (2025.acl-long)

Challenge:	Existing studies show that multilingual models are less robust for semantic parsing compared to other tasks.
Approach:	They propose a constrained optimization technique to optimize multilingual parsing systems for multilingual use.
Outcome:	The proposed technique outperforms XLM-R and mT5-Large on three benchmarks and significantly outperformed other models.

Copied to clipboard

Michael J Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Barr Held, Diyi Yang

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation (2023.emnlp-main)

Challenge:	Recent calls for pluralistic alignment of Large Language Models encourage adapting models to diverse user preferences.
Approach:	They propose a method to induce synthetic user personas from user interactions for personalized reward modeling.
Outcome:	The proposed approach improves LLM-as-a-judge accuracy by 4.4% on Chatbot Arena.

Copied to clipboard

Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, Diyi Yang

Let’s Make Your Request More Persuasive: Modeling Persuasive Strategies via Semi-Supervised Neural Nets on Crowdfunding Platforms (N19-1)

Challenge:	Annotated data plays a critical role in training models and evaluating their performance.
Approach:	They propose a paradigm for Human-LLM co-annotation of unstructured texts at scale that utilizes uncertainty to estimate LLMs’ annotation capability.
Outcome:	The proposed model outperforms existing models on many text-annotation tasks with up to 21% performance improvement over random baseline.

Copied to clipboard

Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, Eduard Hovy

Designing, Evaluating, and Learning from Humans Interacting with NLP Models (2023.emnlp-tutorial)

Challenge:	Existing models can't quantify persuasiveness of requests or extract successful persuasive strategies.
Approach:	They propose a semi-supervised hierarchical neural network model to quantify persuasiveness and identify persuasive strategies in advocacy requests.
Outcome:	The proposed method outperforms baseline models and offers increased interpretability of persuasive speech.

Copied to clipboard

Tongshuang Wu, Diyi Yang, Sebastin Santy

Human-in-the-loop Abstractive Dialogue Summarization (2023.findings-acl)

Challenge:	This tutorial will cover how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans.
Approach:	They will provide a systematic overview of key considerations and effective approaches for studying human-NLP model interactions.
Outcome:	This tutorial will cover how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans.

Copied to clipboard

Jiaao Chen, Mohan Dodda, Diyi Yang

CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies (2024.findings-emnlp)

Challenge:	Abstractive dialogue summarization systems are trained to maximize the likelihood of human-written summaries, but there is still a huge gap in generating high-quality summary as determined by humans.
Approach:	They propose to incorporate different levels of human feedback into the training process . they ask humans to highlight salient information to be included in summaries .
Outcome:	The proposed model captures human-written summaries and compares them with state-of-the-art models on multiple datasets.

Copied to clipboard

Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Paula, Diyi Yang

Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization (2020.emnlp-main)

Challenge:	CultureBank is a knowledge base built upon users’ self-narratives with 12K cultural descriptors sourced from TikTok and 11K from Reddit.
Approach:	They construct a pipeline to construct cultural knowledge bases from different online communities on a massive scale.
Outcome:	The proposed pipeline improves cultural awareness of language models by evaluating them on two cultural tasks in a zero-shot setting.

Copied to clipboard

Examining the Ordering of Rhetorical Strategies in Persuasive Requests (2020.findings-emnlp)

Challenge:	Existing studies on text summarization focus on single-speaker docs, scientific publications and encyclopedia articles.
Approach:	They propose a multi-view sequence-to-sequence model that extracts conversational structures from unstructured daily chats and incorporates different views to generate dialogue summaries.
Outcome:	The proposed model outperforms state-of-the-art models via automatic evaluation and human judgment on a large-scale dialogue summarization corpus.

Copied to clipboard

Omar Shaikh, Jiaao Chen, Jon Saad-Falcon, Polo Chau, Diyi Yang

Challenge:	Numerous studies have been conducted to understand persuasiveness of text, from explorations of rhetoric in presidential campaigns to the impact of a communicator's likability on persuasiveness.
Approach:	They use a Variational Autoencoder model to disentangle content and rhetorical strategies in textual requests from a large-scale loan request corpus and visualize interplay between content and strategy through an attentional LSTM that predicts the success of textual request.
Outcome:	The proposed model disentangles content and rhetorical strategies in textual requests from a large-scale loan request corpus and visualizes interplay between content and strategy through attentional LSTM that predicts the success rate of textual request.

Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025.naacl-long)

Copied to clipboard

Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, Diyi Yang

EgoNormia: Benchmarking Physical-Social Norm Understanding (2025.findings-acl)

Challenge:	Generative AI has made rapid advances in multimodal understanding and code generation.
Approach:	They construct a first real-world benchmark for multimodal large language models that directly convert visual designs into code implementations by manually curating 484 diverse real-life webpages as test cases.
Outcome:	The proposed model can generate code implementations that directly render into the given reference webpages, given the screenshots as input.

Copied to clipboard

MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin, Caleb Ziems, Yanzhe Zhang, Hao Zhu, Diyi Yang

Future of Work in the Age of LLMs (2026.acl-tutorials)

Challenge:	Existing VLMs lack robust grounded norm understanding, a new study finds . current VLM models lack robust grounding, despite a high score for safety and privacy .
Approach:	They propose a pipeline to generate grounded MCQs from ego-centric videos of human interactions.
Outcome:	The proposed pipeline can generate grounded MCQs from egocentric video . it shows that current VLMs lack robust grounded norm understanding .

Copied to clipboard

Zora Wang, Yijia Shao, David Nguyen, Diyi Yang

EquiBench: Benchmarking Large Language Models’ Reasoning about Program Semantics via Equivalence Checking (2025.emnlp-main)

Challenge:	a tutorial examines the future of work shaped by the interplay of large language models and humans . a series of tutorials examines challenges, opportunities, and ethical considerations in this dynamic landscape .
Approach:	This tutorial examines the future of work shaped by the interplay of LLMs and humans . it examines how LLM-based systems can augment human labor and enhance real-world tasks .
Outcome:	This tutorial examines the future of work shaped by the interplay of LLMs and humans . it examines challenges, opportunities, and ethical considerations in this dynamic landscape .

Copied to clipboard

Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago S. F. X. Teixeira, Diyi Yang, Ke Wang, Alex Aiken

Robustness of Demonstration-based Learning Under Limited Data Scenario (2022.emnlp-main)

Challenge:	EquiBench is a new benchmark to evaluate large language models' ability to reason about program semantics . Unlike natural language, code is executable.
Approach:	They propose a benchmark to evaluate large language models through equivalence checking . EquiBench consists of 2400 program pairs across four languages and six categories .
Outcome:	The proposed benchmark consists of 2400 program pairs across four languages and six categories.

Copied to clipboard

Hongxin Zhang, Yanzhe Zhang, Ruiyi Zhang, Diyi Yang

DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules (2023.emnlp-main)

Challenge:	Current large pretrained language models struggle to learn NLP tasks under limited data scenarios.
Approach:	They propose to augment input with some demonstrations to improve model performance under limited data scenarios.
Outcome:	The proposed demonstrations improve performance on few-shot NER tasks and show that the length of demonstrations and relevance of random tokens are the main factors affecting the model's performance.

Copied to clipboard

Yanchen Liu, William Held, Diyi Yang

Grounding Gaps in Language Model Generations (2024.naacl-long)

Challenge:	Existing large language models (LLMs) that focus on Standard American English (SAE) often suffer from performance degradation when applied to other dialects.
Approach:	They propose a modular approach to imbue SAE-trained models with multi-dialectal robustness . they propose adapters which handle specific linguistic features to imbibe SAe-taught models .
Outcome:	The proposed approach improves performance across multiple dialects and dialects.

Copied to clipboard

Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, Dan Jurafsky

SUBS: Subtree Substitution for Compositional Semantic Parsing (2022.naacl-main)

Challenge:	Effective conversation requires common ground, but it does not emerge spontaneously.
Approach:	They propose a set of grounding acts and metrics that quantify attempted grounding . they find that large language models generate language with less conversational grounding than humans .
Outcome:	The proposed models generate language with less conversational grounding than humans . compared to humans, they generate language that appears to presume common ground .

Copied to clipboard

Jingfeng Yang, Le Zhang, Diyi Yang

Human-AI Collaboration: How AIs Augment Human Teammates (2025.acl-tutorials)

Challenge:	Semantic parsing models fail at compositional generalization due to lack of reasoning ability.
Approach:	They propose to use subtree substitution for compositional data augmentation to increase the number of subtreas with similar semantic functions as exchangeable.
Outcome:	The proposed method improves performance on Scan and GeoQuery, and new SOTA on compositional split of GeoQuery.

Copied to clipboard

Sherry Wu, Diyi Yang, Joseph Chang, Marti A. Hearst, Kyle Lo

Challenge:	Despite the potential of general-purpose models, they are far from perfect, excelling at certain tasks while struggling with others.
Approach:	This tutorial will review recent developments related to human-AI teaming and collaboration.
Outcome:	This tutorial will review recent developments related to human-AI teaming and collaboration.

Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping (2025.naacl-long)

Copied to clipboard

Ryan Li, Yanzhe Zhang, Diyi Yang

Anchor Points: Benchmarking Models with Much Fewer Examples (2024.eacl-long)

Challenge:	Existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration.
Approach:	They propose a benchmark that evaluates state-of-the-art Vision Language Models on converting sketches into webpage prototypes.
Outcome:	The benchmark evaluates state-of-the-art Vision Language Models on automating the conversion of rudimentary sketches into webpage prototypes.

Copied to clipboard

Rajan Vivek, Kawin Ethayarajh, Diyi Yang, Douwe Kiela