Papers by Sina Zarrieß

Modeling Referential Gaze in Task-oriented Settings of Varying Referential Complexity (2022.findings-aacl)

Challenge:	a lot of research aims to mitigate these problems by introducing specific computational solutions.
Approach:	They examine how large language models engage in the interactive process of repair in multi-turn dialogues around solvable and unsolvable math questions.
Outcome:	The models respond to user-initiated repair differently from one another . the models exhibit their own characteristic form of unreliability in the context of repair .

Copied to clipboard

Özge Alacam, Eugen Ruppert, Sina Zarrieß, Ganeshan Malhotra, Chris Biemann, Sina Zarrieß

Psycholinguistics

Challenge:	Referential gaze is a fundamental phenomenon for psycholinguistics and human-human communication.
Approach:	They propose a multimodal NLP task to predict when the gaze is referential . they train a sequential attention-based LSTM model and a transformer encoder architecture to model referential gaze and transfer gaze features to unseen situated settings .
Outcome:	The proposed model can be applied to situations with different referential complexities . the proposed model is based on an attention-based LSTM model and a multivariate transformer encoder architecture .

Prompting Across Time: Evaluating LLMs on Historical and Contemporary Offensive Language (2026.findings-acl)

Copied to clipboard

Sanne Hoeken, Sophie Jasmin Spliethoff, Silke Schwandt, Özge Alacam, Sina Zarrieß

Object Naming in Language and Vision: A Survey and a New Dataset (2020.lrec-1)

Challenge:	Existing research on large language models and hate speech detection has focused on contemporary data.
Approach:	They propose to use a modular prompt design to evaluate early-modern English invectives . they propose to widen the scope of NLP research on hate speech beyond the contemporary domain .
Outcome:	The proposed model outperforms a modern hate-speech benchmark on Early Modern English invectives . the results show that the model is more robust to contextual and contextual factors than the current model .

Copied to clipboard

Carina Silberer, Sina Zarrieß, Gemma Boleda

Visual Data in NLP

Challenge:	Object naming has been studied in Psycholinguistics, but has received little attention in Computational Linguistics.
Approach:	They propose a dataset that provides 36 name annotations for each of 25K objects in images selected from VisualGenome.
Outcome:	The proposed dataset shows that people choose certain names for objects, on average.

Conceptual Pacts for Reference Resolution Using Small, Dynamically Constructed Language Models: A Study in Puzzle Building Dialogues (2024.lrec-main)

Copied to clipboard

Julian Hough, Sina Zarrieß, Casey Kennington, David Schlangen, Massimo Poesio

Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks? (2025.findings-acl)

Challenge:	Existing large language models can be fine-tuned offline but are large and resource-intensive.
Approach:	They propose to use a simple reference resolver to simulate a conceptual pact process over time with different conversation pairs.
Outcome:	The proposed model performs better than a pre-trained model with exhaustive retraining after each prediction, while being more transparent, faster and less resource-intensive.

Copied to clipboard

Simeon Junker, Manar Ali, Larissa Koch, Sina Zarrieß, Hendrik Buschmeier

Exploring Text Recombination for Automatic Narrative Level Detection (2022.lrec-1)

Challenge:	Existing models are unable to resolve references to abstract visual stimuli, such as color patches and color grids, but their pragmatic capabilities are still a challenge for state-of-the-art MLLMs.
Approach:	They investigate whether multimodal large language models are able to resolve references to abstract visual stimuli, such as color patches and color grids, in a well-known reference resolution paradigm.
Outcome:	The proposed model can resolve references to abstract visual stimuli in dyadic reference games.

Copied to clipboard

Nils Reiter, Judith Sieker, Svenja Guhr, Evelyn Gius, Sina Zarrieß

Challenge:	Existing annotation workflows do not scale well to the annotation of complex narrative phenomena.
Approach:	They propose a workflow for narrative level detection that includes operationalization and a model . they propose generating training data synthetically to improve the prediction results .
Outcome:	The proposed workflow improves predictions by using training data synthetically.

Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets (2026.eacl-long)

Copied to clipboard

Omar Momen, Emilie Sitter, Berenike Herrmann, Sina Zarrieß

Knowledge Supports Visual Language Grounding: A Case Study on Colour Terms (2020.acl-main)

Challenge:	Novel metaphor comprehension involves complex semantic processes and linguistic creativity.
Approach:	They propose a cloze-style surprisal method that conditions on full-sentence context.
Outcome:	The proposed method shows that LM surprisal yields moderate correlations with scores/labels of metaphor novelty.

Copied to clipboard

Simeon Schüz, Sina Zarrieß

Visual Data in NLP

Challenge:	In human cognition, world knowledge supports the perception of object colours . a lot of recent work in Language & Vision has looked at grounding language in real-world sensory information.
Approach:	They propose to integrate visual information and object-specific knowledge via hard-coded or learned fusion to improve visual grounding of colour terms in realistic objects.
Outcome:	The proposed models outperform a baseline model that predicts colour terms solely from visual inputs but show interesting differences when predicting atypical colours of so-called colour diagnostic objects.

WikiScenes with Descriptions: Aligning Paragraphs and Sentences with Images in Wikipedia Articles (2024.starsem-1)

Copied to clipboard

Özge Alaçam, Ronja Utescher, Hannes Grönner, Judith Sieker, Sina Zarrieß

Multimodality

Challenge:	Existing work on processing image-text alignment in multimodal documents has been unsupervised, facing the challenge of missing evaluation and training data.
Approach:	They propose to provide one of the first datasets that provides ground-truth annotations of image-text alignments in multi-paragraph multi-image articles.
Outcome:	The proposed dataset can be used to study phenomena of visual language grounding in longer documents and assess retrieval capabilities of language models trained on captioning data.

Disentangling Subjectivity and Uncertainty for Hate Speech Annotation and Modeling using Gaze (2025.emnlp-main)

Copied to clipboard

Özge Alacam, Sanne Hoeken, Andreas Säuberli, Hannes Gröner, Diego Frassinelli, Sina Zarrieß, Barbara Plank

Eyes Don’t Lie: Subjective Hate Annotation and Detection with Gaze (2024.emnlp-main)

Challenge:	Variation is inherent in opinion-based annotation tasks like sentiment or hate speech analysis.
Approach:	They propose to use annotators' confidence ratings to disentangle subjective variation from uncertainty without relying on specific features present in the data.
Outcome:	The proposed approach shows that human gaze patterns offer valuable indicators of subjective evaluation and uncertainty.

Copied to clipboard

Özge Alacam, Sanne Hoeken, Sina Zarrieß

Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions (2023.findings-eacl)

Challenge:	Hate speech is a complex and subjective phenomenon.
Approach:	They propose a dataset that provides gaze data collected in a hate speech annotation experiment and introduce a first gaze-integrated HSD model.
Outcome:	The proposed dataset provides gaze data from hate speech annotation experiments.

Copied to clipboard

Henrik Voigt, Jan Hombeck, Monique Meuschke, Kai Lawonn, Sina Zarrieß

Multimodality

Challenge:	Existing language and vision models can be used for language understanding in 3D environments . however, existing models lack specific properties and biases that limit their performance .
Approach:	They propose a framework that uses a camera to generate images from different viewpoints and evaluate them in terms of their similarity to natural language descriptions.
Outcome:	The proposed model performs poorly on most canonical views and fine-tunes using hard negative sampling and random contrasting yields good results even under conditions with little available training data.

Rationalizing Transformer Predictions via End-To-End Differentiable Self-Training (2024.emnlp-main)

Copied to clipboard

Marc Brinner, Sina Zarrieß

Text Classification

Challenge:	Neural networks are increasingly prevalent across a wide range of applications, driving significant advancements in fields such as natural language processing, computer vision, and beyond.
Approach:	They propose an end-to-end differentiable training paradigm for stable training of a rationalized transformer classifier.
Outcome:	The proposed model is capable of classifying a sample and scoring input tokens without any explicit supervision and produces class-wise rationales without instabilities.

SceneGram: Conceptualizing and Describing Tangrams in Scene Context (2025.findings-acl)

Copied to clipboard

Simeon Junker, Sina Zarrieß

Cognitive Modeling

Challenge:	Current systems show mixed results in reproducing human variation in object naming . figurative descriptions for abstract stimuli remain a major challenge in vision and language research .
Approach:	They propose to analyze human references to tangrams placed in different scene contexts . they analyze the richness and variability of conceptualizations found in human references .
Outcome:	The proposed model does not account for the richness and variability of human references.

Know What You Don’t Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories (P19-1)

Copied to clipboard

Sina Zarrieß, David Schlangen

Can LLMs Ground when they (Don’t) Know: A Study on Direct and Loaded Political Questions (2025.acl-long)

Challenge:	a lot of recent and traditional research on pragmatically informative object descriptions has focused on the task of correctly labelling objects of novel categories.
Approach:	They extend a neural generator to become a pragmatic speaker reasoning about uncertain object categories.
Outcome:	The proposed model improves the accuracy of the listener's communication with unfamiliar objects.

Copied to clipboard

Clara Lachenmaier, Judith Sieker, Sina Zarrieß

Exploring Semantic Spaces for Detecting Clustering and Switching in Verbal Fluency (2022.coling-1)

Challenge:	Using large language models, interlocutors can reach mutual understanding even when they do not possess perfect knowledge.
Approach:	They examine whether loaded questions lead LLMs to engage in active grounding and correct false user beliefs in connection to their level of knowledge and their political bias.
Outcome:	The proposed model can answer direct knowledge questions and loaded questions that presuppose misinformation, while ignoring false user beliefs.

Copied to clipboard

Özge Alacam, Simeon Schüz, Martin Wegrzyn, Johanna Kißler, Sina Zarrieß

Text Clustering

Challenge:	Existing evaluations of word/concept representations on verbal fluency tasks rely on human annotations of clusters and switches between sub-categories.
Approach:	They analyze word/concept representations in an experimental verbal fluency dataset . they find that ConceptNet embeddings outperforms other semantic representations .
Outcome:	The proposed method outperforms other semantic representations by a large margin.

SemCSE-Multi: Multifaceted and Decodable Embeddings for Aspect-Specific and Interpretable Scientific Domain Mapping (2026.acl-long)

Copied to clipboard

Marc Brinner, Sina Zarrieß

Subword models struggle with word learning, but surprisal hides it (2025.acl-short)

Challenge:	Existing embedding models rely on implicit, imprecise and fixed notion of similarity to evaluate scientific abstracts.
Approach:	They propose a framework for generating multifaceted embeddings of scientific abstracts . they propose an unsupervised procedure that produces aspect-specific summarizing sentences .
Outcome:	The proposed framework captures distinct, individually specifiable aspects in isolation . it then trains embedding models to map semantically related summaries to nearby positions . the proposed framework is evaluated in the domains of invasion biology and medicine .

Copied to clipboard

Bastian Bunzeck, Sina Zarrieß

Evaluating Diversity in Automatic Poetry Generation (2024.emnlp-main)

Challenge:	Subword LMs struggle to discern words and non-words with high accuracy, character LM models do this easily and consistently.
Approach:	They propose to model word learning in subword and character language models with the psycholinguistic lexical decision task.
Outcome:	The results suggest that word learning and syntactic learning are separable in character LMs.

Copied to clipboard

Yanran Chen, Hannes Gröner, Sina Zarrieß, Steffen Eger

Text Generation

Challenge:	Existing models for creative text generation are not evaluated regarding how different generated poems are from existing training sets.
Approach:	They evaluate the diversity of automatically generated poetry by comparing distributions of generated poetry to distributions in human poetry along structural, lexical, semantic and stylistic dimensions.
Outcome:	The proposed model types show that style-conditioning and character-level modeling increases diversity across virtually all dimensions.

Plots Made Quickly: An Efficient Approach for Generating Visualizations from Natural Language Queries (2024.lrec-main)

Copied to clipboard

Henrik Voigt, Kai Lawonn, Sina Zarrieß

Model Interpretability and Rationale Extraction by Input Mask Optimization (2023.findings-acl)

Challenge:	Existing methods for generating visualizations from natural language queries have not fully incorporated state-of-the-art techniques such as pre-trained LMs.
Approach:	They propose to generate a valid Vega-Lite specification from a data frame and a query as input and render it as a visualization.
Outcome:	The proposed model scales better with pre-trained LMs than current state-of-the-art models on the NL2VIS benchmark nvBench.

Copied to clipboard

Marc Brinner, Sina Zarrieß

Explainability & Interpretability in NLP

Challenge:	Existing methods for creating explanations for black-box models struggle with deriving easily interpretable explanations.
Approach:	They propose a model-agnostic method to generate extractive explanations for neural network predictions using masking parts of the input that the model does not consider indicative of the respective class.
Outcome:	The proposed method achieves state-of-the-art results in a paragraph-level rationale extraction task, showing that this task can be performed without training a specialized model.

How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models (2026.findings-acl)

Copied to clipboard

Judith Sieker, Sina Zarrieß

SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts (2025.emnlp-main)

Challenge:	Large language models (LLMs) are increasingly studied as repositories of linguistic knowledge.
Approach:	They compare LLMs’ performance as pragmatic listeners and as pragmatic speakers . they find a robust asymmetry between pragmatic evaluation and pragmatic generation .
Outcome:	The proposed models perform better as listeners than speakers, and produce more appropriate language than speakers.

Copied to clipboard

Marc Felix Brinner, Sina Zarrieß

Humans Meet Models on Object Naming: A New Dataset and Analysis (2020.coling-main)

Challenge:	SemCSE is an unsupervised method for learning semantic embeddings of scientific texts .
Approach:	They propose an unsupervised method for learning semantic embeddings of scientific texts that leverages LLM-generated scientific summaries to train a model that positions semantically related summary closer together in the embeddable space.
Outcome:	The proposed method achieves state-of-the-art performance on the SciRepEval benchmark for scientific text embeddings, highlighting the benefits of a semantically focused training approach.

Copied to clipboard

Carina Silberer, Sina Zarrieß, Matthijs Westera, Gemma Boleda

Visual Data in NLP

Challenge:	Existing object naming datasets that use only images with a bounding box are noisy . a human-like model behavior is not stable across domains, a study finds .
Approach:	They use MN v2 to verify object naming datasets with dozens of valid names per object . they find that human-like model behavior is not stable across domains .
Outcome:	The proposed model confuses people and clothing objects more frequently than humans do.

Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas (2025.coling-main)

Copied to clipboard

Bastian Bunzeck, Daniel Duran, Leonie Schade, Sina Zarrieß

Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them (2025.findings-emnlp)

Challenge:	Existing studies on LMs have focused on linguistic generalizations and representations from developmentally plausible data.
Approach:	They propose to use phoneme- and grapheme-based language models to learn linguistic units at and below the word level.
Outcome:	The proposed models can achieve strong performance on syntactic and novel benchmarks and match grapheme-based models in standard tasks and novel evaluations.

Copied to clipboard

Marc Felix Brinner, Tarek Al Mustafa, Sina Zarrieß