Papers by Benjamin Roth
UniSent: Universal Adaptable Sentiment Lexica for 1000+ Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | Sentiment lexica are vital for sentiment analysis in absence of document-level annotations . linguistic resources are limited for at least a few hundred languages, putting them at risk of extinction . |
| Approach: | They introduce UniSent universal sentiment lexica for 1000+ languages . they use a Bible corpus to project sentiment information from English to other languages based on Twitter data . |
| Outcome: | The proposed method mitigates domain mismatch between Bible and Twitter by using embeddings . it compares to other sentiment seeding methods in a subset of languages with ground truth available . |
Intent Recognition in Doctor-Patient Interviews (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently, up to 20 percent of patients are misdiagnosed in medical training programs. |
| Approach: | They propose to annotate doctor-patient interviews with intent inventory and information retrieval methods that are robust with respect to small amounts of training data. |
| Outcome: | The proposed models provide baseline performance scores on the data set for further research. |
Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on persona prompting has shown mixed results on its effectiveness . prior work did not consider when and why personas should affect performance . |
| Approach: | They analyze literature on persona prompting and distill three desiderata for their effectiveness . they propose mitigation strategies to improve robustness but find they only work for the largest, most capable models . |
| Outcome: | The authors find that expert personas usually lead to positive or non-significant performance changes . they propose mitigation strategies to improve robustness but only for the largest models . |
Interpretable Question Answering on Knowledge Bases and Text (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for ML based question answering models are lacking . a lack of explanation methods has been proposed for QA models . |
| Approach: | They propose an automatic evaluation paradigm for explanation methods in ML based question answering models . they adapt post hoc explanation methods such as LIME and input perturbation to the model . |
| Outcome: | The proposed evaluation paradigm compares explanation methods with human annotations. |
To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable performance in a variety of tasks due to factual knowledge accumulated during pre-training. |
| Approach: | They propose an evaluation protocol that disentangles knowing from applying knowledge and test state-of-the-art LLMs on 49 ambiguous entities. |
| Outcome: | The proposed evaluation protocol disentangles knowing from applying knowledge and tests state-of-the-art LLMs on 49 ambiguous entities. |
Functionality learning through specification instructions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Creating or annotating instances targeting specific functionalities is costly and further training models is expensive. |
| Approach: | They propose to use specification instructions to create specification-augmented prompts for each functionality in a suite and combine them with language models pre-trained on natural instruction data. |
| Outcome: | The proposed test suites can assess models’ performance on specific functionalities on four tasks and models of diverse sizes and families. |
Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies neglect to measure the generalization of their methods to other prompt styles and different sizes of LLMs. |
| Approach: | They propose a framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement. |
| Outcome: | The proposed framework integrates response agreement and focal loss with binary cross-entropy to improve calibration from baselines. |
Joint Bootstrapping Machines for High Confidence Relation Extraction (N18-1)
Copied to clipboard
| Challenge: | Existing semi-supervised bootstrapping methods for relationship extraction lack labeled data. |
| Approach: | They propose a semi-supervised bootstrapping method that protects against semantic drift . they expand entities and templates in parallel and in mutually constraining fashion in each iteration . |
| Outcome: | Experimental results show that BREX improves on state-of-the-art methods for four relationships. |
Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have reported generalization gaps in AI-text detectors, but they lack insights into the causes. |
| Approach: | They propose to analyze generalization behavior of AI-text detectors using linguistic analysis to explain performance variance. |
| Outcome: | The proposed model can generalize across unseen prompts, model families, and domains, but it can't generalize under distribution shifts. |
KnowMAN: Weakly Supervised Multinomial Adversarial Networks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to weakly supervised training lack labeled data . weakly-supervised training can result in heuristic but noisy labels . |
| Approach: | They propose a scheme that allows to control influence of signals associated with specific labeling functions. |
| Outcome: | The proposed scheme improves results compared to weakly supervised learning with a pre-trained transformer language model and a feature-based baseline. |
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing count-based word embeddings are superseded by machine-learning methods like word2vec and GloVe, but in many settings there is not much text data available. |
| Approach: | They propose to use positive pointwise mutual information (PPMI) weighted co-occurrence matrices to compute word embeddings from a corpus using large amounts of text data. |
| Outcome: | The proposed method outperforms word2vec and the state-of-the-art for low-resource settings and obtains competitive results for Maltese and Luxembourgish. |
Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-End Neural Networks (D18-1)
Copied to clipboard
| Challenge: | a new approach for aspect-based sentiment analysis is proposed . we compare the performance of the proposed approach with pipeline approaches . |
| Approach: | They propose a model for aspect-based sentiment analysis that uses a convolutional neural network and fasttext embeddings to combine the two approaches. |
| Outcome: | The proposed model outperforms pipeline approaches in aspects-based sentiment analysis. |
Seeing through the mess: evolutionary dynamics of lexical polysemy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of lexical polysemy suggest multiple senses emerge for words . lexically, multiple sense of a word is the rule rather than the exception, authors say . |
| Approach: | They propose a mathematical model of the evolution of lexical meaning to investigate polysemy . they find conditions under which a sense of a word diversifies itself into multiple senses . |
| Outcome: | The proposed model predicts that diversification is promoted by low frequency and high discriminability . it also shows that the model is robust to a wide range of language variables . |
Domain adaptation for part-of-speech tagging of noisy user-generated text (N19-1)
Copied to clipboard
| Challenge: | Existing POS taggers for canonical German text achieve good results around 97% accuracy, but when applying these trained models to out-of-domain data the performance decreases drastically. |
| Approach: | They propose a neural network that trains an out-of-domain model on a large newswire corpus and transfers those weights by using them as a prior for a model trained on the target domain. |
| Outcome: | The proposed model achieves a tagging accuracy of slightly over 90%, improving on the previous state of the art for this task. |
ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision (2023.emnlp-main)
Copied to clipboard
| Challenge: | A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs). |
| Approach: | They propose an algorithm which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LF. |
| Outcome: | The proposed algorithm denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LF. |
Cross-functional Analysis of Generalization in Behavioral Learning (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for behavioral learning use correlations in training data, but they ignore important model properties such as fairness. |
| Approach: | They propose an analysis method for evaluating behavioral learning considering generalization across dimensions of different granularity levels. |
| Outcome: | The proposed method optimizes behavior-specific loss functions and evaluates models on several partitions of the behavioral test suite controlled to leave out specific phenomena. |
Counterfactual Reasoning with Knowledge Graph Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Knowledge graph embeddings were originally developed to infer true but missing facts in incomplete knowledge repositories. |
| Approach: | They propose a task that requires models to reason on a counterfactual KG. |
| Outcome: | The proposed task connects knowledge graph completion and counterfactual reasoning. |
Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement (P18-1)
Copied to clipboard
| Challenge: | a number of post hoc explanation methods for deep neural networks have been proposed . due to the complexity of the DNNs they explain, these methods are necessarily approximations and come with their own sources of error. |
| Approach: | They propose two evaluation paradigms that cover two important classes of NLP problems . they propose LIMSSE, LRP and DeepLIFT as the most effective explanation methods . |
| Outcome: | The proposed methods are most effective for explaining deep neural networks in NLP . the proposed methods can explain complex models without manual annotation . |
Text-Guided Image Clustering (2024.eacl-long)
Copied to clipboard
Andreas Stephan, Lukas Miklautz, Kevin Sidak, Jan Philip Wahle, Bela Gipp, Claudia Plant, Benjamin Roth
| Challenge: | Current image clustering methods neglect the use of generated textual descriptions. |
| Approach: | They propose to use image captioning and visual question-answering to cluster images . they propose a new approach to inject task- or domain knowledge into image clustering . |
| Outcome: | The proposed method outperforms existing methods on eight image clustering datasets. |
Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations (2026.findings-acl)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often exhibit label variation. |
| Approach: | They extend LiTEx taxonomy to two NLI datasets and jointly analyze label variation and label variation. |
| Outcome: | The proposed model combines explanations as a lens to analyze variation in NLI annotations and examine individual differences in reasoning. |
Do language models accommodate their users? A study of linguistic convergence (2026.eacl-long)
Copied to clipboard
| Challenge: | In this paper, we examine how large language models adapt their language use to the linguistic patterns of their user. |
| Approach: | They examine whether large language models exhibit linguistic convergence, a pragmatic element of human language communication, and compare their results to original human responses. |
| Outcome: | The proposed model language use is significantly different from that of humans. |
ACTC: Active Threshold Calibration for Cold-Start Knowledge Graph Completion (2023.acl-short)
Copied to clipboard
| Challenge: | Knowledge graphs are a graph of information organized as entities, relations, and entities. |
| Approach: | They propose a method to calibrate a scoring model over (entity, relation, entity)-tuples . they use an annotated set of tuple truncated by Logistic Regression or Gaussian Process classifiers . |
| Outcome: | The proposed method finds good per-relation thresholds efficiently based on a limited set of annotated tuples. |
Compact Example-Based Explanations for Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training data influence estimation methods rely on naive selection strategies to provide explanations of a human-interpretable size. |
| Approach: | They propose a retraining-free metric that quantifies how useful a set of examples is for explaining a model's output. |
| Outcome: | The proposed model can predict whether a set of examples supports or undermines the model’s predictions. |
Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions (2026.eacl-long)
Copied to clipboard
| Challenge: | Persona-assigned large language models are used in education, healthcare and sociodemographic simulations. |
| Approach: | They propose a protocol that combines long persona dialogues and evaluation datasets to create dialogue-conditioned benchmarks that can robustly measure long-context effects. |
| Outcome: | The proposed protocol can measure persona fidelity, instruction-following, and safety in long conversations. |
SepLL: Separating Latent Class Labels from Weak Supervision Noise (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for learning from weak labels use heuristics and heurism to create weak labels. |
| Approach: | They propose a weakly supervised learning paradigm that uses human intuitions to create weak (noisy) labels. |
| Outcome: | The proposed model is competitive with the state-of-the-art and yields a new best average performance. |