Papers by Barbara Plank

VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)

Challenge:	In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks.
Approach:	They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training.
Outcome:	The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague.

Copied to clipboard

Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, Barbara Plank

Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective (2026.findings-acl)

Challenge:	Existing work on label variation and annotation errors has focused on them in isolation.
Approach:	They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations.
Outcome:	The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items.

Copied to clipboard

Beiduo Chen, Tiancheng Hu, Caiqi Zhang, Robert Litschko, Anna Korhonen, Barbara Plank

Reasoning

Challenge:	Reasoning-tuned large language models (LLMs) with long Chain-of-Thought excel at single-answer tasks, yet their ability to model Human Label Variation remains underexplored.
Approach:	They conduct systematic disentanglement experiments to isolate the effect of reasoning text from intrinsic model priors on distribution-based tasks.
Outcome:	The proposed model improves distributional alignment, but distributional ranking is governed by model priors.

Stop Measuring Calibration When Humans Disagree (2022.emnlp-main)

Copied to clipboard

Joris Baan, Wilker Aziz, Barbara Plank, Raquel Fernandez

Text Classification

Challenge:	Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., predictive probabilities are a good indication of how likely a prediction is to be correct.
Approach:	They propose to measure calibration to human majority given inherent disagreements on tasks where humans inherently disagree about which class applies.
Outcome:	The proposed measures capture key statistical properties of human judgements including class frequency, ranking and entropy.

If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models (2026.eacl-long)

Copied to clipboard

Jasmin Orth, Philipp Mondorf, Barbara Plank

Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification (2024.acl-long)

Challenge:	Conditional acceptability refers to how plausible a conditional statement is perceived to be.
Approach:	They propose to use conditional acceptability models to assess their models' conditional acceptance . they use probabilistic and semantic cues to assess whether a condition is acceptable or plausible .
Outcome:	The proposed models are sensitive to conditional probability and semantic relevance, but do not align more closely with human judgments.

Copied to clipboard

Shanshan Xu, Santosh T.y.s.s, Oana Ichim, Barbara Plank, Matthias Grabmair

Text Classification

Challenge:	Existing methods for NLP calibration ignore inherent human label variation (HLV) split votes are a problem in high-stakes domains such as legal and medical decisions .
Approach:	They present a case outcome classification dataset with judges' vote distributions and build a taxonomy of disagreement with SV-specific subcategories.
Outcome:	The proposed model is compared against a judge vote distribution and assesses the alignment of perceived difficulty between models and humans.

Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data (2023.findings-acl)

Copied to clipboard

Robert Litschko, Ekaterina Artemova, Barbara Plank

“Seeing the Big through the Small”: Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? (2024.findings-emnlp)

Challenge:	Using zero-shot rankers, cross-lingual IR models are limited by their language coverage.
Approach:	They propose to train ranking models on artificially code-switched data instead of using a dictionary.
Outcome:	The proposed approach is robust towards the ratio of code-switched tokens and extends to unseen languages.

Copied to clipboard

Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, Barbara Plank

Explainability & Interpretability in NLP

Challenge:	Human label variation arises when multiple human annotators provide different labels for valid reasons.
Approach:	They propose to use crowd workers to represent human judgment distributions or expert linguists to provide detailed explanations for their chosen labels.
Outcome:	The proposed model can approximate human judgment distributions using a small number of expert labels and explanations.

Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? (2024.eacl-short)

Copied to clipboard

Joris Baan, Raquel Fernández, Barbara Plank, Wilker Aziz

“My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models (2024.findings-acl)

Challenge:	In modern NLP, neural networks are the de-facto standard to predict complex probability measures from available context.
Approach:	They propose to use a single predictive distribution to evaluate models with disentangled representations of uncertainty about predictions and uncertainty about human labels.
Outcome:	The proposed models are crucial for trustworthy and fair NLP systems, but exploiting a single distribution is limiting.

Copied to clipboard

Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, Barbara Plank

ActiveAED: A Human in the Loop Improves Annotation Error Detection (2023.findings-acl)

Challenge:	Multiple choice questions are one of the most popular evaluation formats for understanding the capabilities of autoregressive large language models (LLMs).
Approach:	They evaluated how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation.
Outcome:	The proposed evaluation methods are misaligned on all dimensions, reaching mismatch rates over 60%.

Copied to clipboard

Leon Weber, Barbara Plank

Challenge:	Annotation error detection (AED) methods flag erroneous annotations for human reannotation . but many of these methods assume a final curation step in which a human annotator decides whether the annotation is erranosome .
Approach:	They propose an annotation error detection method that can detect errors more accurately by repeatedly querying a human for error corrections in its prediction loop.
Outcome:	The proposed method improves on eight datasets spanning five different tasks and can improve accuracy by up to six percentage points.

A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI (2025.findings-acl)

Copied to clipboard

Beiduo Chen, Siyao Peng, Anna Korhonen, Barbara Plank

Explainability & Interpretability in NLP

Challenge:	Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) Large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs, but collecting explanations for every label is still time-consuming.
Approach:	They propose to use Large Language Models (LLMs) as annotators to generate model explanations for a few given human labels.
Outcome:	The proposed models can generate human-provided explanations from human labels, but they are still time-consuming.

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training (2023.findings-emnlp)

Copied to clipboard

Max Müller-Eberstein, Rob van der Goot, Barbara Plank, Ivan Titov

Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models (2026.acl-long)

Challenge:	Contemporary advances in NLP are built on the representational power of latent embedding spaces learned by self-supervised language models (LMs).
Approach:	They use a new information theoretic probing suite to analyze representational subspaces in language models.
Outcome:	The proposed approach compared performance of nine tasks across 2M pre-training steps and five seeds.

Copied to clipboard

Georg Ahnert, Anna-Carolina Haensch, Barbara Plank, Markus Strohmaier

Text Generation

Challenge:	Existing studies focus on generating closed-ended survey responses with large language models, whereas LLMs are typically trained to generate open-ended text.
Approach:	They evaluate the impact of various Survey Response Generation Methods on simulated responses by generating closed-ended responses from large language models.
Outcome:	The proposed methods perform best in individual-level and subpopulation-level alignment.

Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models (2024.emnlp-main)

Copied to clipboard

Philipp Mondorf, Barbara Plank

CrossRE: A Cross-Domain Dataset for Relation Extraction (2022.findings-emnlp)

Challenge:	logical puzzles that involve determining identity of characters require a variety of reasoning skills.
Approach:	They propose a benchmark for suppositional reasoning based on knights and knaves puzzles . they show lower-performing models exhibit a diverse range of reasoning errors .
Outcome:	The proposed benchmark demonstrates that models struggle with suppositional reasoning . lower performing models struggle to grasp the concept of truth and lies, the study finds .

Copied to clipboard

Elisa Bassignana, Barbara Plank

Biomedical Event Extraction as Sequence Labeling (2020.emnlp-main)

Challenge:	Relation Extraction (RE) evaluation is limited to in-domain setups . despite the drought of research on cross-domain RE, its practical importance remains .
Approach:	They propose a cross-domain benchmark for relation extraction which includes multi-label annotations and meta-data to include explanations and flags of difficult instances.
Outcome:	The proposed model includes explanations and flags of difficult instances.

Copied to clipboard

Alan Ramponi, Rob van der Goot, Rosario Lombardo, Barbara Plank

Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set (2025.acl-long)

Challenge:	Empirical results show that BeeSL’s speed and accuracy makes it a viable approach for large-scale real-world scenarios.
Approach:	They propose a joint end-to-end neural information extraction model that recasts the task as sequence labeling and jointly models intermediate tasks via multi-task learning.
Outcome:	Empirical results show that BeeSL outperforms the current best system on the Genia 2011 benchmark by 1.57% absolute F1 score reaching 60.22% F1 .

Copied to clipboard

Florian Eichin, Yang Janet Liu, Barbara Plank, Michael A. Hedderich

Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark (2024.naacl-long)

Challenge:	Existing work on discourse understanding is constrained by framework-dependent discourse representations.
Approach:	They examine whether large language models capture discourse knowledge that generalizes across languages and frameworks.
Outcome:	The proposed model can generalize discourse information across languages and frameworks.

Copied to clipboard

Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, Börje Karlsson, Peiqin Lin, Nikola Ljubešić, Lester James Miranda, Barbara Plank, Arij Riabi, Yuval Pinter

At a Glance: The Impact of Gaze Aggregation Views on Syntactic Tagging (D19-64)

Challenge:	In named entity recognition, the majority of annotation efforts are centered on English, and cross-lingual transfer performance remains brittle.
Approach:	They propose to develop gold-standard named entity recognition benchmarks in many languages using a cross-lingual consistent schema.
Outcome:	The proposed benchmarks will be released to the public in 2022 . they will provide baselines on in-language and cross-lingual learning settings.

Copied to clipboard

Sigrid Klerke, Barbara Plank

Syntactic Text Processing

Challenge:	Recent work uses gaze data at the type level or at the token level and mostly from a single eye-tracking corpus.
Approach:	They propose to use gaze data to capture central tendency or variability of gaze data and to integrate binary phrase chunking and part-of-speech tagging.
Outcome:	The proposed approaches capture the central tendency or variability of gaze data better than proposed local views which retain individual participant information.

Reason to Rote: Rethinking Memorization in Reasoning (2025.emnlp-main)

Copied to clipboard

Yupei Du, Philipp Mondorf, Silvia Casola, Yuekun Yao, Robert Litschko, Barbara Plank

To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity (2024.findings-emnlp)

Challenge:	Large language models readily memorize arbitrary training instances, such as label noise . however, such memorization does not affect generalizable reasoning abilities .
Approach:	They investigate how large language models memorize label noise and why it affects generalizability.
Outcome:	The proposed model performs well on reasoning tasks even when memorized labels are missing . the proposed model is able to generalize to correctly answer "87+19=106"

Copied to clipboard

Anastasiia Sedova, Robert Litschko, Diego Frassinelli, Benjamin Roth, Barbara Plank

Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)

Challenge:	Large language models (LLMs) have remarkable performance in a variety of tasks due to factual knowledge accumulated during pre-training.
Approach:	They propose an evaluation protocol that disentangles knowing from applying knowledge and test state-of-the-art LLMs on 49 ambiguous entities.
Outcome:	The proposed evaluation protocol disentangles knowing from applying knowledge and tests state-of-the-art LLMs on 49 ambiguous entities.

Copied to clipboard

Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank

Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (N19-1)

Challenge:	linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions.
Approach:	They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications.
Outcome:	The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models .

Copied to clipboard

Ravi Shekhar, Aashish Venkatesh, Tim Baumgärtner, Elia Bruni, Barbara Plank, Raffaella Bernardi, Raquel Fernández

Natural Language Interfaces

Challenge:	Existing systems that address the abilities that need to be put to work during conversations are lacking in terms of visual grounding.
Approach:	They propose a visually-grounded dialogue state encoder which integrates visual grounding with dialogue system components.
Outcome:	The proposed system improves the GuessWhat?! game by combining guessing and asking questions with multi-task learning.

Disentangling Subjectivity and Uncertainty for Hate Speech Annotation and Modeling using Gaze (2025.emnlp-main)

Copied to clipboard

Özge Alacam, Sanne Hoeken, Andreas Säuberli, Hannes Gröner, Diego Frassinelli, Sina Zarrieß, Barbara Plank

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning (2021.naacl-main)

Challenge:	Variation is inherent in opinion-based annotation tasks like sentiment or hate speech analysis.
Approach:	They propose to use annotators' confidence ratings to disentangle subjective variation from uncertainty without relying on specific features present in the data.
Outcome:	The proposed approach shows that human gaze patterns offer valuable indicators of subjective evaluation and uncertainty.

Copied to clipboard

Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio

Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models (2025.acl-long)

Challenge:	Prior work shows that disagreement between annotators can be useful in training models.
Approach:	They propose to use disagreements as an auxiliary task in a multi-task neural network to incorporate disagreements into models.
Outcome:	The proposed method significantly improves performance on NLP tasks beyond the standard approach and prior work.

Copied to clipboard

Philipp Mondorf, Sondre Wold, Barbara Plank

Cross-Domain Evaluation of Edge Detection for Biomedical Event Extraction (2020.lrec-1)

Challenge:	Recent advances in mechanistic interpretability have made progress in identifying circuits, the minimal computational subgraphs responsible for a model’s behavior on specific tasks.
Approach:	They propose to analyze circuits for highly compositional subtasks within a transformer-based language model to determine their modularity and how they relate to each other.
Outcome:	The proposed approach shows that the circuits identified exhibit notable node overlap and cross-task faithfulness.

Copied to clipboard

Alan Ramponi, Barbara Plank, Rosario Lombardo

What’s wrong with your model? A Quantitative Analysis of Relation Classification (2024.starsem-1)

Challenge:	Biomedical event extraction systems are evaluated in-domain and on complete event structures only.
Approach:	They present a cross-domain study of edge detection for biomedical event extraction . they analyze differences between five existing gold standard corpora and provide a strong baseline model .
Outcome:	The proposed model shows a drop in performance when the baseline is applied on out-of-domain data.

Copied to clipboard

Elisa Bassignana, Rob van der Goot, Barbara Plank

Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration (2026.findings-acl)

Challenge:	A major trend in NLP research aims at designing more sophisticated setups to improve the state-of-the-art (SOTA) on a target task.
Approach:	They propose an in-depth analysis suite for Relation Classification to be used for prediction tasks.
Outcome:	The proposed model improves over the baseline by >3 Micro-F1 . the proposed model is based on a case study and a preliminary error-guided analysis .

Copied to clipboard

Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang, Ming-Hao Hsu, Florian Eichin, Zhizheng Wu, Alane Suhr, Michael A. Hedderich, David Harwath, David R. Mortensen, Barbara Plank

Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation (2026.eacl-long)

Challenge:	We show that script information is linearly encoded in the activation space of multilingual speech models . modifying activations at inference time induces script change even in unconventional pairings .
Approach:	They propose to add script vectors to activations at test time to induce script change . they also show that script information is linearly encoded in the activation space of multilingual speech models .
Outcome:	The proposed approach can induce script change even in unconventional language-script pairings.

Copied to clipboard

Bolei Ma, Yong Cao, Indira Sen, Anna-Carolina Haensch, Frauke Kreuter, Barbara Plank, Daniel Hershcovich

How to Encode Domain Information in Relation Classification (2024.lrec-main)

Challenge:	Large Language Models (LLMs) are increasingly used to simulate public opinion and other social phenomena.
Approach:	They argue that open-endedness is essential for realistic social simulations . they argue that it captures expressiveness and individuality .
Outcome:	The proposed frameworks can improve measurement and design, support exploration of unanticipated views, and reduce researcher-imposed directive bias.

Copied to clipboard

Elisa Bassignana, Viggo Unmack Gascou, Frida Nøhr Laustsen, Gustav Kristensen, Marie Haahr Petersen, Rob van der Goot, Barbara Plank

Information Retrieval

Challenge:	Existing deep learning models require a lot of training data to obtain high performance.
Approach:	They propose a multi-domain training setup for Relation Classification (RC) they compare different ways to enrich input instances with domain information .
Outcome:	The proposed model improves > 2 Macro-F1 against the baseline setup.

Silver Syntax Pre-training for Cross-Domain Relation Extraction (2023.findings-acl)

Copied to clipboard

Elisa Bassignana, Filip Ginter, Sampo Pyysalo, Rob van der Goot, Barbara Plank

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives (2023.acl-short)

Challenge:	Relation Extraction (RE) is the task of extracting structured knowledge from unstructured text.
Approach:	They exploit the affinity between syntactic structure and semantic RE to obtain low-cost pre-training data.
Outcome:	The proposed model outperforms baseline models in five out of six cross-domain setups without additional annotated data.

Copied to clipboard

Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, Barbara Plank

ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain (2023.acl-long)

Challenge:	Recent studies show that intermediate layer distillation (ILD) objectives improve model compression, but a comprehensive evaluation of distillation objectives in both task-specific and task-agnostic settings is lacking.
Approach:	They propose to use knowledge distillation to improve model compression by transferring knowledge from one model to another.
Outcome:	The proposed framework improves on the task of QNLI with lower teacher layers and higher teacher layers.

Copied to clipboard

Mike Zhang, Rob van der Goot, Barbara Plank

M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis (2025.emnlp-main)

Challenge:	Increasing number of NLP benchmarks highlight need for multilingual models for job-related tasks.
Approach:	They introduce a language model called ESCOXLM-R that uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations taxonomy.
Outcome:	The proposed model outperforms XLM-R-large on short spans and entity-level and surface-level span-F1 tasks on entity- and surface level.

Copied to clipboard

ChengYan Wu, Bolei Ma, Yihong Liu, Zheyu Zhang, Ningyuan Deng, Yanshu Li, Baolan Chen, Yi Zhang, Yun Xue, Barbara Plank

Bleaching Text: Abstract Features for Cross-lingual Gender Prediction (P18-2)

Challenge:	Existing studies focus on English-centric aspects of sentiment analysis, limiting scope for multilingual evaluation and research.
Approach:	They propose to use a multilingual dataset to analyze aspects with associated sentiment elements in text.
Outcome:	The proposed dataset is the most extensive multilingual parallel dataset for ABSA to date.

Copied to clipboard

Rob van der Goot, Nikola Ljubešić, Ian Matroos, Malvina Nissim, Barbara Plank

Position: From Noise to Signal to Selbstzweck - Reframing Human Label Variation in the Era of Post-training in NLP (2026.findings-acl)

Challenge:	Existing gender prediction models rely on lexical and social network features to capture style beyond topic.
Approach:	They propose an alternative to lexical bleaching, i.e., transforming lexicals into more abstract features.
Outcome:	The proposed model performs similar to lexical models, but is less language-, topic-, and platform dependent.

Copied to clipboard

Shanshan Xu, Santosh T.y.s.s, Barbara Plank

Challenge:	Human Label Variation (HLV) refers to legitimate disagreement in annotation . current preference-learning datasets routinely collapse multiple annotations into a single label .
Approach:	They propose to preserve human label variation as an embodiment of pluralism . they argue that disagreement in annotations should be treated as a selfzweck .
Outcome:	The proposed approach preserves pluralism and human pluralismos, the authors argue . they argue that disagreements in annotations should be treated as a selfzweck .

SkillSpan: Hard and Soft Skill Extraction from English Job Postings (2022.naacl-main)

Copied to clipboard

Mike Zhang, Kristian Jensen, Sif Sonniks, Barbara Plank

Low-Resource NLP

Challenge:	Existing studies on Skill Extraction (SE) use crowd-sourced labels or annotations from a predefined skill inventory.
Approach:	They propose a dataset that contains 14.5K sentences and over 12.5K annotated spans.
Outcome:	The proposed model outperforms non-adapted models and single-task outperformed multi-task learning.

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation (2023.emnlp-main)

Copied to clipboard

Robert Litschko, Max Müller-Eberstein, Rob van der Goot, Leon Weber-Genzel, Barbara Plank

Sebastian, Basti, Wastl?! Recognizing Named Entities in Bavarian Dialectal Data (2024.lrec-main)

Challenge:	Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing community has striven to model computationally for decades.
Approach:	They propose to rethink what constitutes tasks and model evaluation in NLP and pursue a more holistic view on language, placing trustworthiness at the center.
Outcome:	The proposed models are based on generative models and are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups.

Copied to clipboard

Siyao Peng, Zihang Sun, Huangyan Shan, Marie Kolm, Verena Blaschke, Ekaterina Artemova, Barbara Plank

Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP (2021.eacl-demos)

Challenge:	Named Entity Recognition (NER) is a fundamental task to extract key information from texts, but annotated resources are scarce for dialects.
Approach:	They present the first dialectal NER dataset for German, BarNER, with 161K tokens annotated on Bavarian Wikipedia articles and tweets.
Outcome:	The proposed dataset improves on bar-wiki and moderately on bartweet with training first on Bavarian .

Copied to clipboard

Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, Barbara Plank

The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models (2024.findings-emnlp)

Challenge:	Multi-task learning (MTL) has become a standard repertoire in natural language processing (NLP) it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters.
Approach:	They propose a toolkit for fine-tuning contextualized embeddings in multi-task settings.
Outcome:	The proposed toolkit supports a variety of natural language processing tasks . it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters.

Copied to clipboard

Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael Hedderich, Barbara Plank, Frauke Kreuter

Kompetencer: Fine-grained Skill Classification in Danish Job Postings via Distant Supervision and Transfer Learning (2022.lrec-1)

Challenge:	Recent advances in Large Language Models have sparked interest in validating human-like cognitive-behavioral traits.
Approach:	They examine whether LLM outputs reflect human-like cognitive-behavioral traits . they find that measuring AOVs embedded within LLMs remains opaque .
Outcome:	The proposed model can be used to evaluate human-like cognitive-behavioral traits . the proposed model could be used in writing assistants and other applications .

Copied to clipboard

Mike Zhang, Kristian Nørgaard Jensen, Barbara Plank

Sort by Structure: Language Model Ranking as Dependency Probing (2022.naacl-main)

Challenge:	Several studies focus on Skill Identification, but there is little work in further categorizing the identified skills.
Approach:	They propose a Danish job posting dataset annotated for nested spans of competences . they use the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy API to obtain fine-grained labels via distant supervision.
Outcome:	The proposed dataset outperforms existing models in the Danish job postings.

Copied to clipboard

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Fine-tuning vs From Scratch: Do Vision & Language Models Have Similar Capabilities on Out-of-Distribution Visual Question Answering? (2022.lrec-1)

Challenge:	Existing algorithms for pre-trained language models lack performance indicators for linguistic tasks such as structured prediction.
Approach:	They propose to measure the degree to which labeled trees are recoverable from an LM’s contextualized embeddings by probing to rank LMs for parsing dependencies in a given language.
Outcome:	The proposed approach predicts the best LM choice 79% of the time using less compute than training a full parser.

Copied to clipboard

Kristian Nørgaard Jensen, Barbara Plank

Neural Unsupervised Domain Adaptation in NLP—A Survey (2020.coling-main)

Challenge:	Large general-purpose pre-trained models have become ubiquitous, but only limited research exists into their robustness.
Approach:	They perform a fine-grained evaluation of two models on Visual Question Answering using out-of-distribution data and a rephrasing analysis.
Outcome:	The proposed model is better than a previous model trained from scratch on the training data alone.

Copied to clipboard

Alan Ramponi, Barbara Plank

Low-Resource NLP

Challenge:	Deep neural networks excel at learning from labeled data, but learning from unlabeled data remains a challenge.
Approach:	They review neural unsupervised domain adaptation techniques which do not require labeled target domain data.
Outcome:	The proposed techniques are more challenging yet widely applicable.

RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs (2025.emnlp-main)

Copied to clipboard

Alberto Testoni, Barbara Plank, Raquel Fernández

Visual Data in NLP

Challenge:	Existing language models that address ambiguity are limited in their ability to address it . ambiguities are an inherent feature of human language, according to research .
Approach:	They propose a dataset targeting referential ambiguity in image-based question answering . they find that current language models lack robust strategies to deal with ambiguities .
Outcome:	The proposed dataset shows that state-of-the-art models fail to address ambiguity . ambiguities are an inherent feature of human language, according to research .

The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (2025.emnlp-main)

Copied to clipboard

Leonardo Bertolazzi, Philipp Mondorf, Barbara Plank, Raffaella Bernardi

ltzGLUE: Luxembourgish General Language Understanding Evaluation (2026.findings-acl)

Challenge:	Existing studies have explored methods to enhance self-correction in large language models, but little attention has been given to understanding the models’ internal mechanisms underlying error detection.
Approach:	They propose to use a large language model to analyze arithmetic errors in four smaller-sized LLMs and identify their internal mechanisms.
Outcome:	The proposed models heavily rely on consistency headstextemdashattention heads that assess surface-level alignment of numerical values in arithmetic solutions.

Copied to clipboard

Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cedric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke

Information Retrieval

Challenge:	ltzGLUE is the first official NLU benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English.
Approach:	They propose a new natural language understanding (NLU) benchmark for Luxembourgish based on the popular GLUE benchmark for English.
Outcome:	The proposed model performs well across many languages and is based on the GLUE benchmark for English.

Slot and Intent Detection Resources for Bavarian and Lithuanian: Assessing Translations vs Natural Queries to Digital Assistants (2024.lrec-main)

Copied to clipboard

Miriam Winkler, Virginija Juozapaityte, Rob van der Goot, Barbara Plank

Machine Translation

Challenge:	xSID datasets for low-resource languages like English are lacking in translations of high-ressource languages . however, many native speakers of such languages may want to use virtual assistants in their mother tongue .
Approach:	They extend a dataset to include two underrepresented languages: Bavarian and Lithuanian . they provide "natural" queries to digital assistants generated by native speakers .
Outcome:	The proposed dataset includes two underrepresented languages: Bavarian and Lithuanian . the results show that translated data can produce overly optimistic scores .

What Media Frames Reveal About Stance: A Dataset and Study about Memes in Climate Change Discourse (2025.findings-emnlp)

Copied to clipboard

Shijia Zhou, Siyao Peng, Simon M. Luebke, Jörg Haßler, Mario Haim, Saif M. Mohammad, Barbara Plank

Multimodality

Challenge:	Media framing is a method of shaping public perceptions of issues, but the interaction between stance and media frame remains unexplored.
Approach:	They propose to use a dataset of climate-change memes annotated with stance and media frames to conceptualize and computationally explore this interaction.
Outcome:	The proposed dataset includes 1,184 climate-change memes sourced from 47 subreddits and enables analysis of frame prominence over time and communities.

Cartography Active Learning (2021.findings-emnlp)

Copied to clipboard

Mike Zhang, Barbara Plank

Low-Resource NLP

Challenge:	Existing methods to label data are limited in their notion of informativeness, due to post-training model uncertainty and batch diversity.
Approach:	They propose a new Active Learning algorithm that exploits the behavior of the model on individual instances during training as a proxy to find the most informative instances for labeling.
Outcome:	The proposed method is competitive to other common AL methods, showing that training dynamics derived from small seed data can be successfully used for AL.

Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study (2025.acl-long)

Copied to clipboard

Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xinpeng Wang, Markus Herklotz, Frauke Kreuter, Barbara Plank, Matthias Aßenmacher

From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification (2023.emnlp-main)

Challenge:	Recent advances in large language models have generated significant interest in their potential for synthetic data generation across various domains.
Approach:	They use open-ended survey data from the German Longitudinal Election Studies to prompt different LLMs to generate synthetic public opinions reflective of German subpopulations by incorporating demographic features into the persona prompts.
Outcome:	The LLM performs better for supporters of left-leaning parties like The Greens and The Left compared to other parties, and matches the least with the right-party AfD.

Copied to clipboard

Shanshan Xu, Santosh T.y.s.s, Oana Ichim, Isabella Risini, Barbara Plank, Matthias Grabmair

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Challenge:	Existing work in explainable COC has been limited to annotations by a single expert.
Approach:	They construct a two-level task-independent taxonomy from a dataset obtained from two experts in the domain of international human rights law . they find disagreements stem from underspecification of the legal context .
Outcome:	The proposed dataset is the first in legal NLP that focuses on human label variation.

Spectral Probing (2022.emnlp-main)

Copied to clipboard

Challenge:	Contextualized embeddings capture linguistic phenomena at distinct layers and frequencies . current studies rely on handcrafted filters and are limited to English .
Approach:	They develop a fully learnable frequency filter to identify spectral profiles for any given task.
Outcome:	The proposed filter enables more granular analyses than handcrafted filters and improves efficiency.

Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties (2024.eacl-long)

Copied to clipboard

Ekaterina Artemova, Verena Blaschke, Barbara Plank

Robustness in NLP

Challenge:	Mainstream cross-lingual task-oriented dialogue systems often overlook the transfer to lower-resource colloquial varieties due to limited test data.
Approach:	They propose to train a model for intent recognition and slot-filling in English and apply it to other languages.
Outcome:	The proposed model performs better than existing models on English and other languages.

What Do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification (2022.acl-srw)

Copied to clipboard

Elisa Bassignana, Barbara Plank

DaN+: Danish Nested Named Entities and Lexical Normalization (2020.coling-main)

Challenge:	Existing RE surveys focus on modeling techniques, but there are few that are based on real-world scenarios.
Approach:	They propose to survey RE datasets and revisit the task definition and its adoption by the community.
Outcome:	The proposed approach improves the reliability of RE evaluations across multiple datasets and reveals significant discrepancies in annotations.

Copied to clipboard

Barbara Plank, Kristian Nørgaard Jensen, Rob van der Goot

Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering (P19-1)

Challenge:	Named Entity Recognition (NER) is a task of finding entities in text, such as locations, organizations, and persons.
Approach:	They propose a multi-domain corpus and annotation guidelines for Danish nested named entities and lexical normalization to support research on cross-lingual cross-domain learning for a less-resourced language.
Outcome:	The proposed model outperforms existing models on the Danish Named Entity Recognition task and shows that it is robust to domain shifts and is highly effective on the least canonical data.

Copied to clipboard

Claudio Greco, Barbara Plank, Raquel Fernández, Raffaella Bernardi

Visual Data in NLP

Challenge:	Existing methods to overcome catastrophic forgetting in visual question answering models are inadequate, but have received little attention within natural language processing.
Approach:	They devise a set of linguistically-informed visual question answering tasks motivated by psycholinguistics and investigate impact of task difficulty on continual learning.
Outcome:	The proposed models differ in the types of questions they ask and show that task difficulty and order matter.

Experimental Standards for Deep Learning in Natural Language Processing Research (2022.findings-emnlp)

Copied to clipboard

Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, Barbara Plank

Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora (2025.findings-emnlp)

Challenge:	a lack of common experimental standards remains an open challenge to the field at large .
Approach:	They propose to distill discussions on experimental standards into a single, widely-applicable methodology.
Outcome:	Using best practices, we can strengthen experimental evidence, improve reproducibility and enable scientific progress.

Copied to clipboard

Robert Litschko, Verena Blaschke, Diana Burkhardt, Barbara Plank, Diego Frassinelli

Genre as Weak Supervision for Cross-lingual Dependency Parsing (2021.emnlp-main)

Challenge:	Dialects exhibit a substantial degree of variation due to the lack of a standard orthography . however, the ability of Large Language Models (LLMs) to process dialects remains understudied .
Approach:	They propose a framework for creating dialect variation dictionaries from monolingual data . they use a dataset to examine how well LLMs can judge Bavarian terms as dialect translations .
Outcome:	The proposed framework can judge dialects as dialect translations, inflected variants or unrelated forms of a given German lemma.

Copied to clipboard

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training (2026.eacl-long)

Challenge:	Recent work shows that monolingual masked language models learn to represent data-driven notions of language variation.
Approach:	They harness genre metadata as a weak supervision signal for targeted data selection in zero-shot dependency parsing.
Outcome:	The proposed method outperforms baseline and embedding-based methods for 12 low-resource language treebanks and three of these target languages.

Copied to clipboard

Felicia Körner, Max Müller-Eberstein, Anna Korhonen, Barbara Plank

Probing for Labeled Dependency Trees (2022.acl-long)

Challenge:	Recent studies have found that Large Language Models process multilingual inputs in shared concept spaces, thought to support generalization and cross-lingual transfer.
Approach:	They investigate the development of language-agnostic concept spaces during pretraining of EuroLLM using the causal interpretability method of activation patching.
Outcome:	The proposed model is language-agnostic and enables cross-lingual transfer . the model is able to process multilingual inputs, but lacks cross-linguistic alignment .

Copied to clipboard

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

Representation Learning

Challenge:	Existing linear probes are limited to extracting undirected or unlabeled dependency parse trees, thereby capturing only a subset of the full task.
Approach:	They propose a linear probe which can extract labeled and directed dependency parse trees from embeddings while using fewer parameters and compute than prior methods.
Outcome:	The proposed method outperforms baselines and prior work in 13 languages and outperformed competitive baselines.

KARRIEREWEGE: A large scale Career Path Prediction Dataset (2025.coling-industry)

Copied to clipboard

Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank

Challenge:	Career path prediction is a growing field, but available data and tools are limited.
Approach:	They propose Karrierewege, a publicly available career path prediction dataset . it combines career path data with ESCO taxonomy to provide valuable resource .
Outcome:	The career path prediction dataset Karrierewege surpasses existing datasets . the dataset is based on the ESCO taxonomy, which provides a common language for occupations and skills .

Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings (2022.lrec-1)

Copied to clipboard

Rob van der Goot, Max Müller-Eberstein, Barbara Plank

Entity Linking in the Job Market Domain (2024.findings-eacl)

Challenge:	Understanding why contextualized embeddings work is still an active area of research.
Approach:	They propose to use a BERT architecture to encode a sub-word, position and a segment embedding as input representations for each sub- word.
Outcome:	The proposed model performs well on single-sentence prediction tasks while swapping segment IDs in paired-sentent tasks.

Copied to clipboard

Mike Zhang, Rob Goot, Barbara Plank

Knowledge Representation

Challenge:	In Natural Language Processing, entity linking (EL) has centered around Wikipedia, but yet remains underexplored for the job market domain.
Approach:	They propose to use a bi-encoder and an autoregressive model to link fine-grained span-level skill mentions to a specific taxonomy entry to quantify labor market demands.
Outcome:	The proposed model outperforms GENRE in strict evaluation, but performs better in loose evaluation.

Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning (2024.acl-long)

Copied to clipboard

Philipp Mondorf, Barbara Plank

Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects (2026.acl-long)

Challenge:	Recent advances in the domain of large language models (LLMs) have showcased their capability in executing deductive reasoning tasks.
Approach:	They examine inferential strategies employed by large language models through a detailed evaluation of their responses to propositional logic problems.
Outcome:	The proposed model shows that it displays reasoning patterns similar to humans, including strategies like supposition following or chain construction.

Copied to clipboard

Verena Blaschke, Miriam Winkler, Barbara Plank

Information Retrieval

Challenge:	Research on cross-dialectal transfer from a standard to a non-standard dialect variety has typically focused on text data.
Approach:	They compare standard-to-dialect transfer in three settings: text models, speech models, and cascaded systems where speech first gets automatically transcribed and then further processed by a text model.
Outcome:	The proposed model performs best on German dialect data while the text-only model perform best on the standard data.

Tracing Multilingual Factual Knowledge Acquisition in Pretraining (2025.findings-emnlp)

Copied to clipboard

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, Hinrich Schuetze

Evaluating Large Language Models for Cross-Lingual Retrieval (2025.findings-emnlp)

Challenge:	Large Language Models are capable of recalling multilingual factual knowledge, but most studies evaluate only the final model, leaving the development of factual recall and crosslingual consistency unexplored.
Approach:	They trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B as a case study.
Outcome:	The results show that fact frequency is the key to a better recall of multilingual facts, regardless of language, and some low-frequency facts in non-English languages can still be correctly recalled.

Copied to clipboard

Longfei Zuo, Pingjun Hong, Oliver Kraus, Barbara Plank, Robert Litschko

Distant Supervision from Disparate Sources for Low-Resource Part-of-Speech Tagging (D18-1)

Challenge:	Large Language Models (LLMs) have been evaluated as second-stage reranking models for monolingual IR, but a systematic comparison is lacking for cross-lingual reranked IR.
Approach:	They propose to use machine translation to evaluate rerankers in cross-lingual IR . they find that LLMs perform better than LLM-based reranked models .
Outcome:	The proposed model improves cross-lingual IR but relies on machine translation for the first stage.

Copied to clipboard

Barbara Plank, Željko Agić

Tagging

Challenge:	Low-resource languages lack manual annotated data to learn basic models such as part-of-speech (POS) taggers.
Approach:	They propose a cross-lingual neural part-of-speech tagger that learns from disparate sources of distant supervision in a uniform framework.
Outcome:	The proposed model scales to hundreds of low-resource languages without access to gold annotated data.

Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languages (2025.coling-main)

Copied to clipboard

Robert Litschko, Oliver Kraus, Verena Blaschke, Barbara Plank

Controlling Reading Ease with Gaze-Guided Text Generation (2026.eacl-long)

Challenge:	lexical gaps between dialects in cross-lingual information retrieval (CLIR) are caused by orthographic variations and different regional expressions.
Approach:	They propose a dataset that consists of seven German dialects extracted from Wikipedia.
Outcome:	The proposed dataset consists of seven German dialects extracted from Wikipedia.

Copied to clipboard

Andreas Säuberli, Darja Jepifanova, Diego Frassinelli, Barbara Plank

Text Generation

Challenge:	Using a gaze-based model, we generate texts with controllable reading ease.
Approach:	They propose a method that predicts gaze patterns to steer language model outputs towards eliciting certain reading behaviors by predicting eye-tracking measures.
Outcome:	The proposed method generates texts with controllable reading ease using eye-tracking with native and non-native speakers of English.

Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuum (2025.findings-naacl)

Copied to clipboard

Ryan Soh-Eun Shim, Barbara Plank

Multimodality

Challenge:	Recent work on dialect variation in NLP treats dialects as discrete categories . dialect variation is a focus of increasing interest in the field .
Approach:	They examine performance differences between Italian dialects by incorporating performance data from different regions of the world.
Outcome:	The results show that performance disparities are due to dialects that are more similar to the standard variety.

EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI (2026.findings-acl)

Copied to clipboard

Longfei Zuo, Barbara Plank, Siyao Peng

Explainability & Interpretability in NLP

Challenge:	Human label variation (HLV) arises when multiple labels are valid for the same instance.
Approach:	They propose a framework for generating and validating explanations to detect errors using large language models (LLMs) EVADE framework provides broader explanation coverage and requires less human intervention .
Outcome:	The proposed framework provides broader explanation coverage, requires less human intervention, and delivers better downstream performance in predicting label distributions.

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations (2026.findings-acl)

Copied to clipboard

Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Benjamin Roth, Barbara Plank

ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation (2023.emnlp-main)

Challenge:	Natural Language Inference (NLI) datasets often exhibit label variation.
Approach:	They extend LiTEx taxonomy to two NLI datasets and jointly analyze label variation and label variation.
Outcome:	The proposed model combines explanations as a lens to analyze variation in NLI annotations and examine individual differences in reasoning.

Copied to clipboard

Xinpeng Wang, Barbara Plank

Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (P18-1)

Challenge:	Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation.
Approach:	They propose to use a multi-head model to learn from disagreements in an active learning setting to identify annotators with a single head.
Outcome:	The proposed model outperforms a single-head model in terms of uncertainty estimation and prediction while saving 70% of the annotation budget.

Copied to clipboard

Sebastian Ruder, Barbara Plank

MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)

Challenge:	Existing neural models for learning under domain shifts only evaluate on a single task, on proprietary datasets, or compare to weak baselines.
Approach:	They propose a multi-task tri-training method that reduces time and space complexity of classic bootstrapping approaches.
Outcome:	The proposed method outperforms the state-of-the-art for sentiment analysis on two benchmarks.

Copied to clipboard

Raoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. Hedderich

What’s the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns (2025.acl-long)

Challenge:	Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness .
Approach:	They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics.
Outcome:	The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language.

Copied to clipboard

Michael A. Hedderich, Anyi Wang, Raoyuan Zhao, Florian Eichin, Jonas Fischer, Barbara Plank

MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank (2024.lrec-main)

Challenge:	Existing evaluation methods for prompting for large language models have limitations such as being labor-intensive or lacking insights.
Approach:	They propose a new approach that automatically distinguishes between random variations and systematic differences in language model outputs by using token patterns.
Outcome:	The proposed method combines both automation and human analysis to provide new insights into established prompt data.

Copied to clipboard

Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, Barbara Plank

Syntactic Text Processing

Challenge:	Despite the success of the Universal Dependencies (UD) project, there is still a lack of diversity within high-resource languages and their closely related non-standard languages and dialects.
Approach:	They propose to annotate Bavarian with part-of-speech and syntactic dependency information manually in UD and to highlight morphosyntactical differences between the closely related languages.
Outcome:	The proposed treebank covers multiple genres including wiki, fiction, grammar examples, social, non-fiction and Bavarian.

NNOSE: Nearest Neighbor Occupational Skill Extraction (2024.eacl-long)

Copied to clipboard

Mike Zhang, Rob Goot, Min-Yen Kan, Barbara Plank

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding (2021.naacl-main)

Challenge:	a new method for extracting occupational skills from text is needed to overcome the scarcity of skills across datasets.
Approach:	They propose a method that leverages multiple datasets to extract occupational skills from text . they propose NNOSE to extract neighboring skills from other datasets .
Outcome:	The proposed method improves infrequent skill extraction without additional fine-tuning.

Copied to clipboard

Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet Üstün, Marija Stepanović, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, Barbara Plank

Crossing Domains without Labels: Distant Supervision for Term Extraction (2025.emnlp-industry)

Challenge:	Lack of publicly available evaluation data for low-resource languages limits progress in SLU . despite advances in neural modeling for slot and intent detection, datasets for SLU remain limited.
Approach:	They propose a joint learning approach with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer.
Outcome:	The proposed model can learn English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer.

Copied to clipboard

Elena Senger, Yuri Campbell, Rob Van Der Goot, Barbara Plank

Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation (2025.emnlp-main)

Challenge:	Current state-of-the-art methods require expensive human annotation and struggle with domain transfer, limiting their practical deployment.
Approach:	They propose a benchmark spanning seven diverse domains to evaluate ATE performance . they propose psuedo-labels and post-hoc heuristics to ensure generalizability .
Outcome:	The proposed model outperforms supervised cross-domain encoder models and few-shot learning baselines on the document- and corpus-levels and its GPT-4o teacher on the benchmark.

Copied to clipboard

Beiduo Chen, Yang Janet Liu, Anna Korhonen, Barbara Plank

Evaluating Pixel Language Models on Non-Standardized Languages (2025.coling-main)

Challenge:	Recent advances in large language models have shown the power of chain-of-thought reasoning in improving complex decision-making tasks.
Approach:	They propose a pipeline that generates chain-of-thought (CoT) explanations from CoTs with improved accuracy.
Outcome:	The proposed pipeline outperforms a direct generation method and baselines on three datasets.

Copied to clipboard

Alberto Muñoz-Ortiz, Verena Blaschke, Barbara Plank

LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference (2025.emnlp-main)

Challenge:	pixel-based models can be used to transfer learning from standard languages to dialects . pretrained language models achieve strong results for languages seen during training, but their performance declines with out-of-domain dialects.
Approach:	They compare pixel-based models to token-based ones to evaluate dialects . standard german is tokenized in a more meaningful way, whereas the Bern dialect is tokenize in pixel form .
Outcome:	The proposed models outperform token-based models in part-of-speech tagging, dependency parsing and intent detection for zero-shot dialect evaluation by up to 26 percentage points in some scenarios, though not in Standard German.

Copied to clipboard

Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Barbara Plank

Reasoning

Challenge:	Existing evidence of human label variation in Natural Language Inference (NLI) however, within-label variation is an additional challenge.
Approach:	They propose a linguistically-informed taxonomy for categorizing free-text explanations in English that captures different reasoning strategies behind NLI explanations with a particular focus on within-label variation.
Outcome:	The proposed taxonomy can be used to classify explanations in English using a linguistically-informed taxonomies.

Cross-Lingual Cross-Domain Nested Named Entity Evaluation on English Web Texts (2021.findings-acl)

Copied to clipboard

Barbara Plank

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)

Challenge:	Named Entity Recognition (NER) is a key task in Natural Language Processing, but most existing work on NER ignores the recognition of nested entities.
Approach:	They propose to annotate five web domains for nested named entities on top of the English Web Treebank (EWT) . they propose to use the English web treebank to perform cross-domain evaluations.
Outcome:	The proposed dataset covers five domains and includes transfer results from German and Danish.

Copied to clipboard

Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, Barbara Plank

Text Generation

Challenge:	In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways.
Approach:	They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty.
Outcome:	The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references.

IndirectQA: Understanding Indirect Answers to Implicit Polar Questions in French and Spanish (2024.lrec-main)

Copied to clipboard

Christin Müller, Barbara Plank

Natural Language Interfaces

Challenge:	polar questions are common in spoken dialogues and expect exactly one of two answers (yes/no) but conversational systems struggle to interpret them.
Approach:	They propose to use subtitle data to interpret indirect answers in french and spanish . they use subtitles to broaden polar questions to include also implicit polar ones .
Outcome:	The proposed corpus of indirect answers shows that the task is challenging and challenging . the baseline accuracy scores drop from 61.43 on english to 44.06 for french and Spanish .

The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (2022.emnlp-main)

Copied to clipboard

Barbara Plank

Evidence > Intuition: Transferability Estimation for Encoder Selection (2022.emnlp-main)

Challenge:	a paper argues that human label variation impacts all stages of the ML pipeline . human label variations are often considered noise due to disagreement, subjectivity in annotation or multiple plausible answers.
Approach:	They propose to reconcile different notions of human label variation and propose a repository of publicly-available datasets with un-aggregated labels.
Outcome:	The proposed approaches are compared with publicly available datasets with un-aggregated labels and identify gaps.

Copied to clipboard

Elisa Bassignana, Max Müller-Eberstein, Mike Zhang, Barbara Plank