Papers by Sandro Pezzelle
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used in verbal creative tasks. |
| Approach: | They propose a divergent association task that focuses on novelty, ignoring appropriateness, a core component of creativity. |
| Outcome: | The proposed model scores are lower than baselines with no creative abilities, undermining its validity for model evaluation. |
A Psycholinguistic Analysis of BERT’s Representations of Compounds (2023.eacl-main)
Copied to clipboard
| Challenge: | 'compound' semantic representations are based on the semantics of constituent words, and are lexical items like any other word. |
| Approach: | They leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD) and semantic transparency (ST). |
| Outcome: | The proposed representations are based on a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD) and semantic transparency (ST). |
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts (D19-1)
Copied to clipboard
| Challenge: | gradable adjectives of size are relative, i.e., determined by the context. |
| Approach: | They propose to model how the meaning of gradable adjectives of size can be learned from visually-grounded contexts by using four tasks to determine whether an object is ‘big’ or ‘small’. |
| Outcome: | The proposed model can learn subtending the meaning of size adjectives, but their performance decreases while moving from simple to more complex tasks. |
EaSe: A Diagnostic Tool for VQA based on Answer Diversity (2021.naacl-main)
Copied to clipboard
| Challenge: | EASE is a diagnostic tool for Visual Question Answering (VQA) it quantifies the difficulty of an image, question sample. |
| Approach: | They propose a diagnostic tool which quantifies the difficulty of an image, question sample. |
| Outcome: | The proposed tool can be used to select the most-informative samples for training/fine-tuning. |
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind (2023.findings-acl)
Copied to clipboard
| Challenge: | Adaptation is a process in human communication by which a speaker tunes its language to that of a listener to achieve communicative success. |
| Approach: | They propose a visual-based referential game between a knowledgeable speaker and a listener with limited visual and linguistic experience to model this adaptation mechanism. |
| Outcome: | The proposed model improves on plug-and-play approaches to controlled language generation without finetuning the speaker’s underlying language model. |
Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for linguistic representations of words are based on information extracted from large text corpora, and the sensory-motor experiences humans have with the world play an important role in determining word meaning. |
| Approach: | They propose to use contextualized word representations to learn semantic representations of words that align with human semantic intuitions. |
| Outcome: | The proposed models are shown to be more efficient on concrete word pairs than on abstract ones. |
How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effects (2026.findings-acl)
Copied to clipboard
| Challenge: | a number of theories have been proposed to account for content effects in large language models, including the dual-process theory of reasoning, but the mechanisms behind content effects remain unclear. |
| Approach: | They propose to encode validity and plausibility concepts in LLMs by aligning them in representational geometry. |
| Outcome: | The proposed model conflates validity and plausibility, and vice versa. |
Who is the richest club in the championship? Detecting and Rewriting Underspecified Questions Improve QA Performance (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) perform well on well-posed factual queries, yet standard question-answering (QA) benchmarks remain far from solved. |
| Approach: | They propose an LLM-based classifier to identify underspecified questions and apply it to several widely used QA datasets. |
| Outcome: | The proposed classifier detects underspecified questions in QA datasets and significantly improves on them. |
Vision-Language Models Align with Human Neural Representations in Concept Processing (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that transformer-based vision-language models capture the multimodality of concept processing in the human brain. |
| Approach: | They analysed multiple VLMs employing different strategies to integrate visual and textual modalities, along with language-only counterparts. |
| Outcome: | The transformer-based vision-language models outperform language-only models in two experimental conditions, while only some outperformed the language-based models. |
Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Visual storytelling is a task of generating a story for a sequence of several temporally-ordered images or video frames. |
| Approach: | They propose a method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. |
| Outcome: | The proposed method improves on the foundation model LLaVA but only slightly compared to TAPM, a 50-times smaller visual storytelling model. |
Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts (2020.emnlp-main)
Copied to clipboard
| Challenge: | Subsequent references exploit the common ground accumulated by the interlocutors and tend to be shorter and reuse expressions that were effective in previous mentions. |
| Approach: | They propose a model that generates first and subsequent references in visually grounded dialogue . they also implement a reference resolution system to assess the referring effectiveness . |
| Outcome: | The proposed model produces better, more effective referring utterances than one not grounded in the dialogue context. |
From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions (2025.acl-long)
Copied to clipboard
Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici
| Challenge: | Large Language Models excel at solving individual problems in isolation, but are they able to effectively collaborate over long-term interactions? |
| Approach: | They propose to use a multi-session dataset to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. |
| Outcome: | The proposed model performs poorly when instructions are spread across sessions, suggesting that they are not able to integrate information over long interactions. |
Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze (2020.emnlp-main)
Copied to clipboard
| Challenge: | a long tradition of cognitive studies shows that the interplay between language and vision is complex. |
| Approach: | They propose an approach to image description generation where visual processing is modelled sequentially. |
| Outcome: | The proposed model exploits gaze-driven attention to produce better descriptions . it sheds light on human cognitive processes by comparing different ways of aligning gaze with language production. |
Dealing with Semantic Underspecification in Multimodal NLP (2023.acl-long)
Copied to clipboard
| Challenge: | a linguistic signal can leave underspecified information, such as gender and number . this problem is a crucial feature that boosts language's storage and processing efficiency . |
| Approach: | They argue that intelligent systems must deal with semantic underspecification . it is a feature that boosts language's storage and processing efficiency, they argue . they argue that the problem is not a bug but a problem that could negatively affect performance . |
| Outcome: | a new paper shows that systems that aim at mastering language must deal with semantic underspecification . it shows that human speakers can integrate semantically-underspecified linguistic signals with non-linguistic information . |
Some of Them Can be Guessed! Exploring the Effect of Linguistic Context in Predicting Quantifiers (P18-2)
Copied to clipboard
| Challenge: | cloze deletion test is a test that requires the learner to understand the context and vocabulary in order to identify the correct word. |
| Approach: | They collect data from human participants and test various models in a local and a global context condition to examine the role of linguistic context in predicting quantifiers. |
| Outcome: | The proposed models outperform humans in a local and global context and are only slightly better in the latter. |
Naming, Describing, and Quantifying Visual Objects in Humans and LLMs (2024.acl-short)
Copied to clipboard
| Challenge: | Recent work has highlighted that speakers display a wide range of variability when asked to utter sentences, resulting in inter-speaker variability but also variability over time for the same speaker. |
| Approach: | They evaluate Vision & Language Large Language Models (VLLMs) on three categories where humans show great subjective variability concerning the distribution over plausible labels. |
| Outcome: | The proposed models can mimic human distributions over plausible labels, but fail to assign quantifiers, a task that requires more accurate, high-level reasoning. |
They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse (2025.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that large language models lack the pragmatic capabilities needed to interpret highly implicit content. |
| Approach: | They propose to use transcribed italian political speeches to test their ability to interpret implicit content. |
| Outcome: | The proposed model provides a fully correct explanation in only one-fourth of cases in the open-ended generation setup. |
Comparatives, Quantifiers, Proportions: a Multi-Task Model for the Learning of Quantities from Vision (N18-1)
Copied to clipboard
| Challenge: | Comparatives, vague quantification, and proportional estimation are not learned at the same time nor governed by the same rules during language acquisition. |
| Approach: | They propose to combine sets comparison, vague quantification, and proportional estimation to learn them together from visual scenes. |
| Outcome: | The proposed model can generalize to unseen combinations of target/non-target objects. |
GROOViST: A Metric for Grounding Objects in Visual Storytelling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Several evaluation metrics for visual storytelling do not consider images at all . authors propose a novel evaluation tool that accounts for cross-modal dependencies and temporal misalignments . |
| Approach: | They propose a visual storytelling evaluation tool that evaluates visual grounding . they use cross-modal dependencies, temporal misalignments and human intuitions . |
| Outcome: | The proposed evaluation tool accounts for cross-modal dependencies, temporal misalignments and human intuitions on visual grounding. |
The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language-and-vision models have impressive performance in downstream tasks, but it remains unclear whether this improves understanding of image-text interaction. |
| Approach: | They propose to use BLA to evaluate multimodal models on basic linguistic constructions that even preschool children can typically master. |
| Outcome: | The proposed model improves basic language skills in a zero-shot learning setting. |
When Language Models Fall in Love: Animacy Processing in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Animacy is not always expressed directly in language, but it manifests indirectly in English . atypically animate entities are easier to remember and prioritized in visual processing . |
| Approach: | They find that LMs behave much like humans when presented with entities whose animacy is typical. |
| Outcome: | The proposed model can learn about animacy even when presented with atypically animate entities. |
Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST! (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models are unable to identify and interpret semantically underspecified sentences when explicitly prompted. |
| Approach: | They propose a Dataset of semantically underspecified Sentences grouped by type to study whether pre-trained language models correctly identify and interpret underspecify sentences. |
| Outcome: | The proposed dataset shows that pre-trained language models are reasonably able to identify and interpret underspecified sentences when explicitly prompted. |
Big Generalizations with Small Data: Exploring the Role of Training Samples in Learning Adjectives of Size (D19-64)
Copied to clipboard
| Challenge: | In previous work, models have been shown to fail in generalizing to unseen adjective-noun combinations. |
| Approach: | They propose a visual reasoning task dealing with quantities that challenges models to learn the meaning of size adjectives from visually-grounded contexts. |
| Outcome: | The proposed task is based on a visual reasoning task dealing with quantities and shows that seeing some of the cases during training helps a model understand the rule subtending the task. |
Describing Images Fast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing models of visuo-linguistic variation are weak to moderately trained to capture such a variation in visual outputs. |
| Approach: | They use a corpus of Dutch image descriptions with eye-tracking data to investigate the nature of the variation in visuo-linguistic signals. |
| Outcome: | The proposed model lacks biases about what makes a stimulus complex for humans and what leads to variations in human outputs. |
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision (2020.findings-emnlp)
Copied to clipboard
| Challenge: | BD2BB is a language and vision benchmark that requires multimodal models combine complementary information from the two modalities. |
| Approach: | They propose a novel language and vision benchmark that requires multimodal models combine complementary information from both modalities. |
| Outcome: | The proposed model is easy for humans, but poor for humans . it compares state-of-the-art models against human speakers to show that it performs well. |