Papers by Dirk Hovy

Responsible Evaluation of AI for Mental Health (2026.acl-long)

Challenge:	Existing studies show that incorporating demographic factors in language representations improves performance on downstream NLP tasks.
Approach:	They use continuous language modeling and dynamic multi-task learning to adapt pre-trained Transformers to incorporate demographic information into their representations.
Outcome:	The proposed model shows that the results are consistent with previous studies.

Copied to clipboard

Hiba Arnaout, Anmol Goel, H. Andrew Schwartz, Steffen T. Eberhardt, Dana Atzil-Slonim, Gavin Doherty, Brian Schwartz, Wolfgang Lutz, Tim Althoff, Munmun De Choudhury, Hamidreza Jamalabadi, Raj Sanjay Shah, Flor Miriam Plaza-del-Arco, Dirk Hovy, Maria Liakata, Iryna Gurevych

The Importance of Modeling Social Factors of Language: Theory and Practice (2021.naacl-main)

Challenge:	Existing approaches to evaluating AI tools in this domain remain fragmented and inconsistent.
Approach:	They propose a taxonomy of AI mental health support types that integrates clinical soundness, social context, and equity to provide a structured basis for evaluation.
Outcome:	The proposed framework integrates clinical soundness, social context, and equity, providing a structured basis for evaluation.

Copied to clipboard

Dirk Hovy, Diyi Yang

Linguistic Theories

Challenge:	Current NLP models focus on information content while ignoring language’s social factors.
Approach:	They propose that NLP systems focus on information content while ignoring language’s social factors to improve performance.
Outcome:	The proposed approach improves the performance of existing systems, open up new applications, and increase fairness and usability for all users.

Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance (2025.emnlp-main)

Copied to clipboard

Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth

IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance (2026.tacl-1)

Challenge:	Prior work on persona prompting has shown mixed results on its effectiveness . prior work did not consider when and why personas should affect performance .
Approach:	They analyze literature on persona prompting and distill three desiderata for their effectiveness . they propose mitigation strategies to improve robustness but find they only work for the largest, most capable models .
Outcome:	The authors find that expert personas usually lead to positive or non-significant performance changes . they propose mitigation strategies to improve robustness but only for the largest models .

Copied to clipboard

Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman, Dirk Hovy

PATS: Personality-Aware Teaching Strategies with Large Language Model Tutors (2026.findings-eacl)

Challenge:	Large language models are helping millions of users write texts about diverse issues . issue bias is where an LLM tends to present just one perspective on a given issue .
Approach:	They construct a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance using 3.9k templates and 212 political issues from real user interactions.
Outcome:	The proposed model aligns more with US Democrat than Republican voter opinion on a subset of issues.

Copied to clipboard

Donya Rooein, Sankalan Pal Chowdhury, Mariia Eremeeva, Yuan Qin, Debora Nozza, Mrinmaya Sachan, Dirk Hovy

Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions (2024.lrec-main)

Challenge:	pedagogical theories are not aligned with teaching strategies for educational tasks . quiet students may be disengaged or not thinking critically because they do not speak up .
Approach:	They propose a taxonomy that links pedagogical methods to personality profiles to map teaching strategies to student personality traits.
Outcome:	The proposed model improves the use of less common, high-impact strategies such as role-playing . the model also increases the use less common strategies such role-players .

Copied to clipboard

Flor Miriam Plaza-del-Arco, Alba A. Cercas Curry, Amanda Cercas Curry, Dirk Hovy

Sentiment Analysis

Challenge:	Emotion analysis (EA) is a rapidly growing field in natural language processing . there is no consensus on scope, direction, or methods for EA .
Approach:	They review 154 relevant NLP papers on emotion analysis from the last decade . they ask: how are EA tasks defined in NLP? what are the most prominent emotion frameworks and which emotions are modeled?
Outcome:	The authors examine 154 relevant NLP papers on emotion analysis from the last decade . they find that there is no consensus on scope, direction, or methods .

“My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models (2024.findings-acl)

Copied to clipboard

Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, Barbara Plank

“We will Reduce Taxes” - Identifying Election Pledges with Language Models (2021.findings-acl)

Challenge:	Multiple choice questions are one of the most popular evaluation formats for understanding the capabilities of autoregressive large language models (LLMs).
Approach:	They evaluated how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation.
Outcome:	The proposed evaluation methods are misaligned on all dimensions, reaching mismatch rates over 60%.

Copied to clipboard

Tommaso Fornaciari, Dirk Hovy, Elin Naurin, Julia Runeson, Robert Thomson, Pankaj Adhikari

Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing (P19-1)

Challenge:	a political party's manifestos are published before any election, but do they follow through? a new study uses neural models to distinguish between actual pledges and general statements .
Approach:	They use election manifestos of Swedish and Indian political parties to learn neural models that distinguish actual pledges from generic positions.
Outcome:	The proposed model can predict election year and manifesto's party, while context introduces noise.

Copied to clipboard

Aparna Garimella, Carmen Banea, Dirk Hovy, Rada Mihalcea

Syntactic Text Processing

Challenge:	linguistic studies have shown the prevalence of various lexical and grammatical patterns in texts authored by a person of a particular gender, but models for part-of-speech tagging and dependency parsing have not adapted to account for these differences.
Approach:	They annotate the Wall Street Journal part of the Penn Treebank with the gender information of the articles’ authors and build taggers and parsers trained on this data.
Outcome:	The proposed model can account for gendered differences in syntactic tasks and highlight future venues for developing more accurate taggers and parsers.

Classist Tools: Social Class Correlates with Performance in NLP (2024.acl-long)

Copied to clipboard

Amanda Curry, Giuseppe Attanasio, Zeerak Talat, Dirk Hovy

Text Error Correction

Challenge:	despite growing concerns surrounding fairness and bias in NLP, there is a dearth of studies delving into the effects it may have on NLP systems.
Approach:	They argue that NLP systems’ performance is affected by speakers’ SES, potentially disadvantaging less-privileged socioeconomic groups.
Outcome:	The proposed model shows that NLP systems perform better on tasks with social class, ethnicity and geographical variation than those without social class.

Hey Siri. Ok Google. Alexa: A topic modeling of user reviews for smart speakers (D19-55)

Copied to clipboard

Hanh Nguyen, Dirk Hovy

Information Extraction & Text Mining

Challenge:	Using coherence scores to choose topics, we test whether the results help us to understand user interests and concerns.
Approach:	They analyze user reviews from Best Buy US website for smart speakers to determine whether they provide useful information for product analysis.
Outcome:	The proposed models capture brand performance and differences and differentiate the market into two distinct groups with different properties.

Helpful or Hierarchical? Predicting the Communicative Strategies of Chat Participants, and their Impact on Success (2020.findings-emnlp)

Copied to clipboard

Farzana Rashid, Tommaso Fornaciari, Dirk Hovy, Eduardo Blanco, Fernando Vega-Redondo

Challenge:	a study of 5,500 chat interactions shows that successful communicators are successful in other domains.
Approach:	They annotate chat interactions with four dimensions of interaction styles to predict success . they find successful communicators are also successful in other domains .
Outcome:	The results show that successful communicators are successful in other domains.

Twitter-Demographer: A Flow-based Tool to Enrich Twitter Data (2022.emnlp-demos)

Copied to clipboard

Federico Bianchi, Vincenzo Cutrona, Dirk Hovy

Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Models (2024.findings-emnlp)

Challenge:	199 million people communicate on twitter daily, making it essential to study policy and decision-making.
Approach:	They propose a flow-based tool to augment Twitter data with additional information about tweets and users.
Outcome:	The proposed tool is designed to enhance Twitter data with additional information about tweets and users.

Copied to clipboard

Flor Plaza-del-Arco, Amanda Curry, Susanna Paoli, Alba Cercas Curry, Dirk Hovy

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning (2021.naacl-main)

Challenge:	Previous work has shown that LLMs display biases in emotion attribution along gender lines.
Approach:	They examine how different religions are represented in LLMs by examining emotion attribution and cultural biases.
Outcome:	The findings highlight the need to address and rectify these biases in LLMs.

Copied to clipboard

Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio

Dense Node Representation for Geolocation (D19-55)

Challenge:	Prior work shows that disagreement between annotators can be useful in training models.
Approach:	They propose to use disagreements as an auxiliary task in a multi-task neural network to incorporate disagreements into models.
Outcome:	The proposed method significantly improves performance on NLP tasks beyond the standard approach and prior work.

Copied to clipboard

Tommaso Fornaciari, Dirk Hovy

Representation Learning

Challenge:	Existing methods for geolocation use sparse adjacency matrices of connections, which grow exponentially with the number of users.
Approach:	They propose two methods to learn continuous node representations from social media posts and textual user mentions.
Outcome:	The proposed methods improve performance over previous sparse graph representations.

Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa (2022.acl-demo)

Copied to clipboard

Tommaso Fornaciari, Alexandra Uma, Massimo Poesio, Dirk Hovy

Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? (2026.eacl-long)

Challenge:	Developing better methods for a task is a common feature of the computational linguistics literature.
Approach:	They propose to use bootstrap to compute significance levels with the BOOtSTrap SAmpling procedure to evaluate models that predict hard labels and soft labels as well.
Outcome:	The proposed method can be used to evaluate models that predict hard labels and soft labels on benchmark data sets.

Copied to clipboard

Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Miserlis Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Elliott Ash

“You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases (2020.acl-main)

Challenge:	Variation in human annotation (i.e., disagreements) is common in NLP, but it is unclear whether it is possible to model this variation in LLMs.
Approach:	They evaluate the influence of different reasoning settings on LLM disagreement modeling . RLVR-style reasoning degrades performance in disagreement modeling, they find .
Outcome:	The proposed reasoning settings improve LLM disagreement modeling, while RLVR-style reasoning degrades it.

Copied to clipboard

Dirk Hovy, Federico Bianchi, Tommaso Fornaciari

Machine Translation

Challenge:	a recent study shows that machine translations make older and more male characters sound older and older than the original.
Approach:	They propose to use demographicallyrepresentative data to examine how text is translated . they show that authors sound older and more male than the original .
Outcome:	The results suggest that translation models reflect demographic bias in the training data.

Integrating Ethics into the NLP Curriculum (2020.acl-tutorials)

Copied to clipboard

Emily M. Bender, Dirk Hovy, Alexandra Schofield

BERTective: Language Models and Contextual Information for Deception Detection (2021.eacl-main)

Challenge:	a tutorial aims to teach students how to ethically apply NLP . the tutorial will focus on examples for university classrooms, but ideas can extend to company-internal workshops or tutorials in a variety of organizations.
Approach:	a tutorial aims to empower NLP researchers and practitioners to teach others about ethical NLP . the tutorial will focus on examples for university classrooms, but ideas can extend to company-internal workshops .
Outcome:	a tutorial aims to teach students how to ethically apply NLP techniques . the tutorial will include examples for university classrooms, but ideas can extend to company-internal workshops .

Copied to clipboard

Tommaso Fornaciari, Federico Bianchi, Massimo Poesio, Dirk Hovy

Bridging Fairness and Environmental Sustainability in Natural Language Processing (2022.emnlp-main)

Challenge:	Existing methods to classify texts as truthful or deceptive are limited by the context of the text being analyzed.
Approach:	They propose to use a corpus of Italian dialogues to classify texts as truthful or deceptive.
Outcome:	The proposed models show that not all contexts are equally useful to the task.

Copied to clipboard

Marius Hessenthaler, Emma Strubell, Dirk Hovy, Anne Lauscher

The State of Profanity Obfuscation in Natural Language Processing Scientific Publications (2023.findings-acl)

Challenge:	a lack of research on the interplay between fairness and environmental impact is a problem in natural language processing . fairness is prone to encode and amplify stereotypical social biases, according to several studies .
Approach:	They evaluate a technique to reduce energy consumption of English NLP models by knowledge distillation for its impact on fairness.
Outcome:	The proposed method reduces energy consumption and environmental impact of English NLP models.

Copied to clipboard

Debora Nozza, Dirk Hovy

Capturing Regional Variation with Distributed Place Representations and Geographic Retrofitting (D18-1)

Challenge:	obfuscation is used for English but not other languages, and even then, unevenly.
Approach:	They propose a multilingual community resource called PrOf to standardize profanity obfuscation processes.
Outcome:	The proposed tool can help scientific publications to make hate speech work accessible and comparable, irrespective of language.

Copied to clipboard

Dirk Hovy, Christoph Purschke

Text Clustering

Challenge:	Dialects are one of the main drivers of language variation, a major challenge for natural language processing tools.
Approach:	They use a corpus of 16.8M anonymous online posts to learn continuous document representations of cities.
Outcome:	The proposed method matches dialect areas at different granularities against an existing dialect map.

SocioProbe: What, When, and Where Language Models Learn about Sociodemographics (2022.emnlp-main)

Copied to clipboard

Anne Lauscher, Federico Bianchi, Samuel R. Bowman, Dirk Hovy

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks (2022.naacl-main)

Challenge:	Pre-trained language models have outperformed other models on a wide range of tasks . however, there is still little understanding of their knowledge of higher-level aspects of language .
Approach:	They investigate whether pre-trained language models have knowledge of sociodemographics . they use traditional probing techniques to probe the knowledge of single-GPU PLMs based on multiple English data sets .
Outcome:	The results show that pre-trained language models outperform other models on a wide range of tasks.

Copied to clipboard

Paul Rottger, Bertie Vidgen, Dirk Hovy, Janet Pierrehumbert

Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features (2024.eacl-long)

Challenge:	Labelled data is the foundation of most natural language processing tasks, but there are valid beliefs about what the correct data labels should be.
Approach:	They propose two contrasting paradigms for data annotation that encourage annotator subjectivity . they propose a descriptive paradigm that allows for the surveying and modelling of different beliefs .
Outcome:	The proposed paradigms encourage annotator subjectivity, while the prescriptive paradigm discourages it.

Copied to clipboard

Eliana Pastor, Alkis Koudounas, Giuseppe Attanasio, Dirk Hovy, Elena Baralis

Explainability & Interpretability in NLP

Challenge:	Existing explanations for speech classification models are difficult to interpret and make mistakes.
Approach:	They propose to explain speech classification models by using word-level and paralinguistic attributes to measure the impact of each audio segment aligned with a word on the outcome.
Outcome:	The proposed explanations correctly represent the model’s inner workings and are plausible to humans.

Geolocation with Attention-Based Multitask Learning Models (D19-55)

Copied to clipboard

Tommaso Fornaciari, Dirk Hovy

Challenge:	predicting the location of a social media post requires discretization of the coordinates, but results in poor performance.
Approach:	They propose to combine two approaches to predict location using supervised models . they evaluate a multitask convolutional neural network that predicts both discrete locations and continuous coordinates .
Outcome:	The proposed model outperforms singletask models and prior work on one dataset and shows that correlation between labels and coordinates has a marked impact on the effectiveness of a regression task.

What about “em”? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns (2023.acl-long)

Copied to clipboard

Anne Lauscher, Debora Nozza, Ehm Miltersen, Archie Crowley, Dirk Hovy

Multilinguality

Challenge:	Wrong pronoun translations can discriminate against marginalized groups, e.g., non-binary individuals.
Approach:	They compare 3rd-person pronoun translations to five other languages . they propose to address gender exclusivity in future research .
Outcome:	The proposed method compares translations of gendered vs. gender-neutral pronouns from english to five other languages and vice versa.

Identifying Linguistic Areas for Geolocation (D19-55)

Copied to clipboard

Tommaso Fornaciari, Dirk Hovy

Information Extraction & Text Mining

Challenge:	a recent study shows that social media posts are often given as continuous coordinates . but, the resulting discrete coordinates do not always correspond to existing linguistic areas .
Approach:	They propose an algorithm for clustering coordinates and associating them with towns using point-to-city (P2C) they compare accuracy of a state-of-the-art geolocation model with P2C labels to one with regular k-d tree labels.
Outcome:	The proposed method improves accuracy at 100 miles, but degrades for finer-grained distinctions . iterative k-d tree-based method can cluster coordinates and associate them with towns .

Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender (2022.coling-1)

Copied to clipboard

Anne Lauscher, Archie Crowley, Dirk Hovy

Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps (2024.emnlp-main)

Challenge:	Current modeling of 3rd person pronouns ignores neopronoun phenomena like naive pronounes, which are not (yet) widely established.
Approach:	They propose to validate existing and novel approaches for modeling 3rd person pronouns in language technology and validate them through a survey.
Outcome:	The proposed model excludes non-binary users, while ignoring gender-specific phenomena.

Copied to clipboard

Giuseppe Attanasio, Beatrice Savoldi, Dennis Fucci, Dirk Hovy

Text Generation

Challenge:	a new class of multitasks, multilingual neural networks, has recently pushed the boundaries of speech-related tasks.
Approach:	They evaluate performance of two widely used multilingual automatic speech recognition models . they find clear gender disparities, with the advantaged group varying across languages .
Outcome:	The proposed models are compared on 19 languages from eight language families and two speaking conditions.

Impoverished Language Technology: The Lack of (Social) Class in NLP (2024.lrec-main)

Copied to clipboard

Amanda Cercas Curry, Zeerak Talat, Dirk Hovy

Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification (2025.emnlp-demos)

Challenge:	Existing work on socio-demographic factors has focused on how much a person's socioeconomic status affects their language production and perception.
Approach:	They propose to include socio-economic class in future natural language processing (NLP) research aimed at understanding relationships between socio-demographic factors and language production and perception.
Outcome:	The proposed definition of class can be operationalised by NLP researchers and argue for including socio-economic class in future language technologies.

Copied to clipboard

Chenfei Xiong, Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Zhijing Jin, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Mennatallah El-Assady, Elliott Ash

Information Extraction & Text Mining

Challenge:	Social scientists often need to develop codebooks that can be reliable but require significant human effort.
Approach:	They propose a mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models.
Outcome:	The proposed framework integrates human expertise with automatic annotation guided by large language models.

HONEST: Measuring Hurtful Sentence Completion in Language Models (2021.naacl-main)

Copied to clipboard

Debora Nozza, Federico Bianchi, Dirk Hovy

Compromesso! Italian Many-Shot Jailbreaks undermine the safety of Large Language Models (2024.acl-srw)

Challenge:	4.3% of the time, language models complete a sentence with a hurtful word . authors propose a score to quantify the amount of hurtful sentence completions in a language model.
Approach:	They propose a score to measure hurtful sentence completions in language models . they use a template- and lexicon-based bias evaluation methodology for six languages .
Outcome:	The proposed score measures the amount of hurtful sentences in language models.

Copied to clipboard

Fabio Pernisi, Dirk Hovy, Paul R�ttger

The Pluralistic Moral Gap: Understanding Moral Judgment and Value Differences between Humans and Large Language Models (2026.eacl-long)

Challenge:	Despite efforts to align large language models with ethical guidelines, models can still be induced into unsafe behavior with jailbreaking.
Approach:	They investigate the impact of many-shot jailbreaking on LLMs in italian . they find models exhibit unsafe behaviors even with minimal exposure to harmful prompts .
Outcome:	The proposed model exhibits unsafe behaviors even with minimal exposure to harmful prompts, and this tendency rapidly escalates with more demonstrations.

Copied to clipboard

Giuseppe Russo, Debora Nozza, Paul Röttger, Dirk Hovy

The AI Gap: How Socioeconomic Status Affects Language Technology Interactions (2025.acl-long)

Challenge:	Existing studies have shown that Large Language Models (LLMs) are not fully aligned with human moral judgments.
Approach:	They propose a dataset of 1,618 real-world moral dilemmas paired with a distribution of human moral judgments consisting of a binary evaluation and a free-text rationale to examine how closely LLMs align with human moral judgements.
Outcome:	The proposed model reproduces human judgments only under high consensus; alignment deteriorates sharply when human disagreement increases.

Copied to clipboard

Elisa Bassignana, Amanda Cercas Curry, Dirk Hovy

Challenge:	Socioeconomic status (SES) fundamentally influences how people interact with technology, but it is limited by proxy metrics and synthetic data.
Approach:	They collect 6,482 prompts from previous interactions of 1,000 individuals from ‘diverse socioeconomic backgrounds’ about their use of language technologies and generative AI.
Outcome:	The findings show that higher SES groups have higher levels of abstraction, convey requests more concisely, and topics like ‘inclusivity’ and ‘travel’.

Cross-lingual Contextualized Topic Models with Zero-shot Learning (2021.eacl-main)

Copied to clipboard

Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, Elisabetta Fersini

Low-Resource NLP

Challenge:	Existing topic models are language-specific and cannot be transferred in a transferable manner.
Approach:	They propose a zero-shot cross-lingual topic model that learns topics on one language and predicts them for unseen documents in different languages.
Outcome:	The proposed model learns topics on one language and predicts them for unseen documents in different languages.

The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics (2023.acl-short)

Copied to clipboard

Matthias Orlikowski, Paul Röttger, Philipp Cimiano, Dirk Hovy

Challenge:	Existing work has attempted to model individual annotation behaviour rather than predicting aggregated labels.
Approach:	They propose to model individual annotator behaviour rather than predicting aggregated labels by adding group-specific layers to multi-annotator models to account for sociodemographics.
Outcome:	The proposed model does not significantly improve on toxic content detection tasks.

Narratives at Conflict: Computational Analysis of News Framing in Multilingual Disinformation Campaigns (2024.acl-srw)

Copied to clipboard

Antonina Sinelnik, Dirk Hovy

Reasoning

Challenge:	Existing methods for multilingual framing differ from those used in English-speaking world . framers often use loaded vocabularies to create political images or favor a particular point of view .
Approach:	They use eight years of Russian-backed disinformation campaigns to examine framing . they find that disinformation campaign consistently favors specific framers .
Outcome:	The proposed method underperforms and shows high disagreements in Russian-language articles . the proposed method is based on eight years of Russian-backed disinformation campaigns .

Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)

Copied to clipboard

Federico Bianchi, Silvia Terragni, Dirk Hovy

Predicting News Headline Popularity with Syntactic and Semantic Knowledge Using Multi-Task Learning (D18-1)

Challenge:	Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing .
Approach:	They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models .
Outcome:	The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models.

Copied to clipboard

Sotiris Lamprinidis, Daniel Hardt, Dirk Hovy

Social Intelligence in the Age of LLMs (2025.naacl-tutorial)

Challenge:	Pre-trained word embeddings provide significant improvements over untrained embeddables . Feature analysis reveals structural patterns of headline popularity .
Approach:	They use a multi-task GRU network to model headline popularity . they find that pre-trained word embeddings provide significant improvements over untrained embeddables .
Outcome:	The proposed model improves on pre-trained word embeddings and untrained embeddables . it also improves with the combination of two auxiliary tasks, news-section prediction and part-of-speech tagging .

Copied to clipboard

Hao Zhu, Bodhisattwa Prasad Majumder, Dirk Hovy, Diyi Yang

SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022.acl-long)

Challenge:	Large Language Models (LLMs) are a powerful tool for integrating human-like communication and context-aware interactions into artificial systems.
Approach:	They propose to introduce and overview different aspects of artificial social intelligence and their relationship with LLMs by introducing scientific methods for evaluating social intelligence in LLM.
Outcome:	This tutorial will introduce scientific methods for evaluating social intelligence in LLMs, highlighting the key challenges, and identifying promising research directions.

Copied to clipboard

Emily Dinan, Gavin Abercrombie, A. Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser

Natural Language Interfaces

Challenge:	Several studies discuss the potential harms and benefits of large language models (LLMs) large neural models can replicate and even amplify negative, stereotypical, and derogatory associations in the data.
Approach:	They propose to use a first aid kit to assess the safety of conversational AI in various settings . they propose several future directions and discuss ethical considerations .
Outcome:	The proposed tools can provide estimates of the relative safety of systems in various settings, but they still have several shortcomings.

Biased Tales: Cultural and Topic Bias in Generating Children’s Stories (2025.emnlp-main)

Copied to clipboard

Donya Rooein, Vilém Zouhar, Debora Nozza, Dirk Hovy

Increasing In-Class Similarity by Retrofitting Embeddings with Demographic Information (D18-1)

Challenge:	Personalized stories are often preferred because they reflect a child's interests, experiences, and developmental needs.
Approach:	They analyze a dataset to examine how biases influence protagonists’ attributes and story elements in LLM-generated stories.
Outcome:	The proposed dataset shows that gender stereotypes influence protagonist attributes and story elements in LLM-generated stories.

Copied to clipboard

Dirk Hovy, Tommaso Fornaciari

Information Retrieval

Challenge:	a new method for text classification ignores strong non-linguistic similarities like homophily . authors are typically represented via their linguistic profiles, i.e. information avail-able in the text .
Approach:	They use homophily cues to retrofit text-based author representations with non-linguistic information and introduce a trade-off parameter.
Outcome:	The proposed method improves on two author-attribute prediction tasks with large labels.

“It’s Not Just Hate”: A Multi-Dimensional Perspective on Detecting Harmful Speech Online (2022.emnlp-main)

Copied to clipboard

Federico Bianchi, Stefanie HIlls, Patricia Rossini, Dirk Hovy, Rebekah Tromble, Nava Tintarev

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2020.acl-main)

Challenge:	Detecting offensive content is becoming a critical task in natural language processing . but most datasets use a single binary label for hate or incivility, even though each concept is multi-faceted . a more fine-grained multi-label approach addresses conceptual and performance issues .
Approach:	They propose to use a dataset to annotate offensive online speech with six labels . they propose to apply a more fine-grained approach to predicting incivility and hateful content .
Outcome:	The proposed approach outperforms or matches benchmark datasets on the annotated tweets.

Copied to clipboard

Deven Santosh Shah, H. Andrew Schwartz, Dirk Hovy

Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists (2022.findings-acl)

Challenge:	a growing number of studies address the effect of bias on predictions, but no unifying framework exists . a general phenomenon of biased predictive models in NLP is not recent, authors say .
Approach:	They propose a unifying framework for identifying and reducing bias in natural language processing . they propose to differentiate two consequences of bias and four potential origins of bias .
Outcome:	The proposed framework provides an overview of predictive bias in natural language processing . it differentiates two consequences of bias and four potential origins of bias: label bias, selection bias, model overamplification, and semantic bias.

Copied to clipboard

Giuseppe Attanasio, Debora Nozza, Dirk Hovy, Elena Baralis

On the Gap between Adoption and Understanding in NLP (2021.findings-acl)

Challenge:	E.g., neural hate speech detection models are strongly influenced by identity terms like gay, or women, resulting in false positives, severe unintended bias, and lower performance.
Approach:	They propose a knowledge-free Entropy-based Attention Regularization (EAR) approach to discourage overfitting to training-specific terms.
Outcome:	The proposed model matches or exceeds state-of-the-art performance for hate speech classification and bias metrics on three benchmark corpora in English and Italian.

Copied to clipboard

Federico Bianchi, Dirk Hovy

DADIT: A Dataset for Demographic Classification of Italian Twitter Users and a Comparison of Prediction Methods (2024.lrec-main)

Challenge:	a recent paper argues that current publications foster a gap between adoption and understanding of models . it also makes it easier to meet publication demands with method papers, argues the paper .
Approach:	They argue that current NLP publication models foster a gap between adoption and understanding of models . they argue that everlarger models make it harder to explain how our methods work .
Outcome:	The authors argue that current publications foster a gap between adoption and understanding of models . they argue that the rise of everlarger models makes it harder to explain how our methods work .

Copied to clipboard

Lorenzo Lupo, Paul Bose, Mahyar Habibi, Dirk Hovy, Carlo Schwarz

Information Retrieval

Challenge:	Social scientists increasingly use demographically stratified social media data to study attitudes, beliefs, and behavior of the general public.
Approach:	They validated the DADIT dataset of 30M tweets of 20k Italian Twitter users, along with their bios and profile pictures.
Outcome:	The best XLM-based classifier improves upon the commonly used competitor M3 by up to 53% F1.

Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages (2022.emnlp-main)

Copied to clipboard

Paul Röttger, Debora Nozza, Federico Bianchi, Dirk Hovy

Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution (2024.acl-long)

Challenge:	Hate speech datasets focus on English-language content, hindering effective models . annotating hateful content is expensive, time-consuming and potentially harmful to annotators.
Approach:	They propose to use ISO 639-1 codes to fine-tune models on one source language and apply them to another language.
Outcome:	The proposed approach performs well on some tasks, but fails on many others.

Copied to clipboard

Flor Plaza-del-Arco, Amanda Curry, Alba Cercas Curry, Gavin Abercrombie, Dirk Hovy

XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2024.naacl-long)

Challenge:	Large language models reflect societal norms and biases, especially about gender.
Approach:	They propose to use large language models to examine gendered emotion attribution in five state-of-the-art LLMs to investigate whether emotions are genderes and whether they are influenced by societal stereotypes.
Outcome:	The proposed models exhibit gendered emotions, influenced by gender stereotypes, and the results are consistent with established research in psychology and gender studies.

Copied to clipboard

Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, Dirk Hovy

Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions (2025.acl-long)

Challenge:	Large language models (LLMs) are now being used by millions of people across the world.
Approach:	They propose a test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way.
Outcome:	The proposed test suite identifies eXaggerated Safety behaviours in a systematic way.

Copied to clipboard

Matthias Orlikowski, Jiaxin Pei, Paul Röttger, Philipp Cimiano, David Jurgens, Dirk Hovy