Papers by Raffaella Bernardi
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments (2022.coling-1)
Copied to clipboard
| Challenge: | embodied AI tasks require a strong understanding of verbs and their corresponding actions. |
| Approach: | They propose a controlled benchmark for embodied action understanding using a simulated environment and a visual feature extractor. |
| Outcome: | The proposed benchmark achieves 81.4% accuracy and high inter-annotator agreement . the proposed model falls behind human models in a zero-shot scenario . |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
Ask No More: Deciding when to guess in referential visual dialogue (C18-1)
Copied to clipboard
| Challenge: | Using a task-oriented visual dialogue model, we add a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess. |
| Approach: | They augment a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess. |
| Outcome: | The proposed model can be enhanced with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess. |
Teaching Small Language Models to Learn Logic through Meta-Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly evaluated on reasoning tasks, yet their logical abilities remain contested. |
| Approach: | They propose to apply few-shot meta-learning to large language models' reasoning domain to enable them to acquire abstract inference patterns that generalize to novel structures. |
| Outcome: | The proposed model outperforms GPT-4o and o3-mini on a syllogistic reasoning task. |
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (N19-1)
Copied to clipboard
Ravi Shekhar, Aashish Venkatesh, Tim Baumgärtner, Elia Bruni, Barbara Plank, Raffaella Bernardi, Raquel Fernández
| Challenge: | Existing systems that address the abilities that need to be put to work during conversations are lacking in terms of visual grounding. |
| Approach: | They propose a visually-grounded dialogue state encoder which integrates visual grounding with dialogue system components. |
| Outcome: | The proposed system improves the GuessWhat?! game by combining guessing and asking questions with multi-task learning. |
Grounded Textual Entailment (C18-1)
Copied to clipboard
Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, Albert Gatt
| Challenge: | Existing models for entailment analysis are not performing well in visual information-based models. |
| Approach: | They propose to use a visual representation of the Textual Entailment task to compare visual-grounded models with a multimodal version of the SNLI dataset. |
| Outcome: | The proposed model performs better when there is an image of the “world” or “situation” . |
How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effects (2026.findings-acl)
Copied to clipboard
| Challenge: | a number of theories have been proposed to account for content effects in large language models, including the dual-process theory of reasoning, but the mechanisms behind content effects remain unclear. |
| Approach: | They propose to encode validity and plausibility concepts in LLMs by aligning them in representational geometry. |
| Outcome: | The proposed model conflates validity and plausibility, and vice versa. |
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored methods to enhance self-correction in large language models, but little attention has been given to understanding the models’ internal mechanisms underlying error detection. |
| Approach: | They propose to use a large language model to analyze arithmetic errors in four smaller-sized LLMs and identify their internal mechanisms. |
| Outcome: | The proposed models heavily rely on consistency headstextemdashattention heads that assess surface-level alignment of numerical values in arithmetic solutions. |
A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024.emnlp-main)
Copied to clipboard
| Challenge: | syllogistic reasoning is a deductive reasoning skill that is crucial in everyday problem-solving and decision-making experiences. |
| Approach: | They propose to study the reasoning abilities of Large Language Models (LLMs) they propose to use supervised fine-tuning and chain-of-thought reasoning to investigate their results. |
| Outcome: | The proposed models exhibit reasoning biases, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning. |
The Price of Thought: A Multilingual Analysis of Reasoning, Performance, and Cost of Negotiation in Large Language Models (2026.findings-eacl)
Copied to clipboard
Sherzod Hakimov, Roland Bernard, Tim Leiber, Karl Osswald, Kristina Richert, Ruilin Yang, Raffaella Bernardi, David Schlangen
| Challenge: | Negotiation is a fundamental challenge for AI agents as it requires an ability to reason strategically, model opponents, and balance cooperation with competition. |
| Approach: | They propose to use a self-play setup to compare commercial and open-weight large language models to their vanilla counterparts in three different languages to examine trade-offs between performance and cost. |
| Outcome: | The proposed model improves GPT-5's performance by 31.4 % while increasing its cost by nearly 400 %. |
Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests (2025.findings-emnlp)
Copied to clipboard
Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, Raffaella Bernardi
| Challenge: | MMLU and BBH are three evaluation paradigms for language learning models . interactive games are superior to standard benchmarks in discriminating models based on human cognitive assessments . |
| Approach: | They examine three evaluation paradigms: standard benchmarks, interactive games and cognitive tests . they examine whether interactive games are more effective at discriminating LLMs . |
| Outcome: | The results show that interactive games are superior to standard benchmarks in discriminating models. |
Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) often perform poorly in generating informative questions, as measured by expected information gain (EIG). |
| Approach: | They propose to use a large language model to enhance the informativeness of LLM-generated questions in 20-question game dialogues by applying a Direct Preference Optimization algorithm to generate low-EIG and high-EI questions. |
| Outcome: | The proposed method produces more effective questions even in domains different from those used to train the DPO model. |
Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering (P19-1)
Copied to clipboard
| Challenge: | Existing methods to overcome catastrophic forgetting in visual question answering models are inadequate, but have received little attention within natural language processing. |
| Approach: | They devise a set of linguistically-informed visual question answering tasks motivated by psycholinguistics and investigate impact of task difficulty on continual learning. |
| Outcome: | The proposed models differ in the types of questions they ask and show that task difficulty and order matter. |
EPIC: Multi-Perspective Annotation of a Corpus of Irony (2023.acl-long)
Copied to clipboard
Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, Davide Bernardi
| Challenge: | EPIC is the first annotated corpus for irony analysis based on data perspectivism . a recent trend in natural language processing (NLP) postulates that the disagreement among annotators in a language resource is a valuable source of knowledge, rather than noise that ought to be minimized or discarded. |
| Approach: | They propose to annotate an English perspectivist irony corpus based on data perspectivism . they validate the model by creating perspective-aware models that encode the perspectives of annotators grouped according to their demographic characteristics. |
| Outcome: | The proposed model can capture different perspectives on irony among different groups of annotators, and is more confident than non-perspectivist models. |
Some of Them Can be Guessed! Exploring the Effect of Linguistic Context in Predicting Quantifiers (P18-2)
Copied to clipboard
| Challenge: | cloze deletion test is a test that requires the learner to understand the context and vocabulary in order to identify the correct word. |
| Approach: | They collect data from human participants and test various models in a local and a global context condition to examine the role of linguistic context in predicting quantifiers. |
| Outcome: | The proposed models outperform humans in a local and global context and are only slightly better in the latter. |
All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark (2025.findings-emnlp)
Copied to clipboard
Davide Testa, Giovanni Bonetta, Raffaella Bernardi, Alessandro Bondielli, Alessandro Lenci, Alessio Miaschi, Lucia Passaro, Bernardo Magnini
| Challenge: | MAIA evaluates visual language models on video-related tasks using reasoning categories that aim to disentangle language and vision relations. |
| Approach: | a native-italian benchmark is designed for fine-grained investigation of the reasoning abilities of visual language models on videos. |
| Outcome: | The benchmark evaluates visual language models on two aligned tasks and a visual question-answering task. |
Comparatives, Quantifiers, Proportions: a Multi-Task Model for the Learning of Quantities from Vision (N18-1)
Copied to clipboard
| Challenge: | Comparatives, vague quantification, and proportional estimation are not learned at the same time nor governed by the same rules during language acquisition. |
| Approach: | They propose to combine sets comparison, vague quantification, and proportional estimation to learn them together from visual scenes. |
| Outcome: | The proposed model can generalize to unseen combinations of target/non-target objects. |
Playpen: An Environment for Exploring Learning From Dialogue Game Feedback (2025.emnlp-main)
Copied to clipboard
Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia
| Challenge: | In this paper, we investigate whether Dialogue Games—goal-directed and rule-governed activities driven predominantly by verbal actions—can also serve as a source of feedback signals for learning. |
| Approach: | They introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning, direct alignment and reinforcement learning with Group Relative Policy Optimization. |
| Outcome: | The proposed model improves performance on unseen instances, but negatively impacts other skills, while interactive learning shows balanced improvements without loss of skills. |
Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy (2021.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art systems generate questions that sound unnatural to humans and are grammatically correct. |
| Approach: | They propose to use beam search re-ranking to generate a model that guides an effective goal-oriented strategy by asking questions that confirm the model’s conjecture about the referent. |
| Outcome: | The proposed model is more natural and effective than beam search decoding without re-ranking on the GuessWhat?! game. |
The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues (2021.eacl-main)
Copied to clipboard
| Challenge: | chit-chat and task-oriented dialogue models are evaluated on their task success metric, but the best model is usually chosen based on task success. |
| Approach: | They compare models playing different games to find out which one is best . they find that this discrepancy is model- and task-agnostic . |
| Outcome: | The proposed model can generate utterances that are indistinguishable from human dialogues by learning to ground, encode, and decode words that do not occur frequently in the training set. |
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision (2020.findings-emnlp)
Copied to clipboard
| Challenge: | BD2BB is a language and vision benchmark that requires multimodal models combine complementary information from the two modalities. |
| Approach: | They propose a novel language and vision benchmark that requires multimodal models combine complementary information from both modalities. |
| Outcome: | The proposed model is easy for humans, but poor for humans . it compares state-of-the-art models against human speakers to show that it performs well. |
“I’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in GuessWhat?! (2021.acl-srw)
Copied to clipboard
| Challenge: | a problem with natural language generation systems is the generation of tokens that are unrelated to the source input. |
| Approach: | They propose two new models to play the GuessWhat?! referential game . they propose to adapt the best visual processing models available to mitigate this issue . |
| Outcome: | The proposed models generate few hallucinations compared to other models available in the literature. |