Papers by David Schlangen
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In the Minecraft Collaborative Building Task, two players collaborate to build a building using 3D blocks. |
| Approach: | They propose to use large language models to model the Builder's sequence of actions in the Minecraft Collaborative Building Task. |
| Outcome: | The proposed methods significantly improve performance over baseline methods and provide detailed analysis for future work. |
A Corpus of Controlled Opinionated and Knowledgeable Movie Discussions for Training Neural Conversation Models (2020.lrec-1)
Copied to clipboard
| Challenge: | Fully data driven Chatbots suffer from inconsistent behaviour across their turns due to a general difficulty in controlling parameters like their assumed background personality and knowledge of facts. |
| Approach: | They propose a model that is based on pre-specified facts and opinions and validates the dialogues for adherence to their given fact and opinion profile. |
| Outcome: | The proposed model is able to generate opinionated responses that are judged to be natural and knowledgeable and show attentiveness. |
When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality (2024.acl-long)
Copied to clipboard
| Challenge: | In incremental models, one interpretation is possible, but models that can revise can do so if the ambiguity is resolved. |
| Approach: | They propose an interpretable way to analyse incremental states in a bidirectional way . they propose to use a model that can update internal states to reflect the garden path effect . |
| Outcome: | The proposed model shows that it can perform revisions and recover if the label is incorrect. |
Conceptual Pacts for Reference Resolution Using Small, Dynamically Constructed Language Models: A Study in Puzzle Building Dialogues (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing large language models can be fine-tuned offline but are large and resource-intensive. |
| Approach: | They propose to use a simple reference resolver to simulate a conceptual pact process over time with different conversation pairs. |
| Outcome: | The proposed model performs better than a pre-trained model with exhaustive retraining after each prediction, while being more transparent, faster and less resource-intensive. |
Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to apply incremental processing to NLUs but this is computationally expensive and does not scale efficiently for long sequences. |
| Approach: | They propose to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental quality. |
Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples (2023.eacl-main)
Copied to clipboard
| Challenge: | Using an extensional description of a visual input, we show that a model can produce referring expressions from visual inputs, whereas simpler baselines do not. |
| Approach: | They propose to use a visual dataset to generate referring expressions from visual inputs. |
| Outcome: | The proposed model achieves BLEU@1 score and sentence accuracy, whereas baselines do not. |
Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset (2023.eacl-main)
Copied to clipboard
| Challenge: | In visual instruction-following dialogue games, players can engage in repair mechanisms in the face of an ambiguous or underspecified instruction. |
| Approach: | They annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game. |
| Outcome: | The proposed dataset contains lexically and semantically diverse iCRs produced self-motivatedly by players deciding to clarify in order to solve the task successfully. |
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (2021.acl-short)
Copied to clipboard
| Challenge: | a language benchmark is a task devised that is restricted enough to be managable with current methods, but is deemed challenging enough to serve as a benchmark. |
| Approach: | They propose to use a language task as a benchmark and a baseline model to argue it is challenging enough to be a good one. |
| Outcome: | The proposed language benchmarks are based on a dataset and a language task . the proposed benchmarks can be used to measure progress towards the goal of the research . |
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a common approach to coherence evaluation is shuffling the sentence order of a text, creating incoherent text samples that need to be discriminated from the original. |
| Approach: | They propose an extendable set of test suites addressing different aspects of discourse and dialogue coherence. |
| Outcome: | The proposed evaluation paradigm is suited to evaluate linguistic qualities that contribute to the notion of coherence. |
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
Know What You Don’t Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories (P19-1)
Copied to clipboard
| Challenge: | a lot of recent and traditional research on pragmatically informative object descriptions has focused on the task of correctly labelling objects of novel categories. |
| Approach: | They extend a neural generator to become a pragmatic speaker reasoning about uncertain object categories. |
| Outcome: | The proposed model improves the accuracy of the listener's communication with unfamiliar objects. |
New or Old? Exploring How Pre-Trained Language Models Represent Discourse Entities (2022.coling-1)
Copied to clipboard
| Challenge: | Recent research shows pre-trained language models learn to encode syntactic knowledge to a certain degree. |
| Approach: | They propose to investigate the information-status of entities as discourse-new or discourse-old . they use binary classification and sequence labeling to investigate their ability to encode syntactic knowledge . |
| Outcome: | The proposed models encode information on whether an entity has been introduced before or not in the discourse. |
The Price of Thought: A Multilingual Analysis of Reasoning, Performance, and Cost of Negotiation in Large Language Models (2026.findings-eacl)
Copied to clipboard
Sherzod Hakimov, Roland Bernard, Tim Leiber, Karl Osswald, Kristina Richert, Ruilin Yang, Raffaella Bernardi, David Schlangen
| Challenge: | Negotiation is a fundamental challenge for AI agents as it requires an ability to reason strategically, model opponents, and balance cooperation with competition. |
| Approach: | They propose to use a self-play setup to compare commercial and open-weight large language models to their vanilla counterparts in three different languages to examine trade-offs between performance and cost. |
| Outcome: | The proposed model improves GPT-5's performance by 31.4 % while increasing its cost by nearly 400 %. |
Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests (2025.findings-emnlp)
Copied to clipboard
Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, Raffaella Bernardi
| Challenge: | MMLU and BBH are three evaluation paradigms for language learning models . interactive games are superior to standard benchmarks in discriminating models based on human cognitive assessments . |
| Approach: | They examine three evaluation paradigms: standard benchmarks, interactive games and cognitive tests . they examine whether interactive games are more effective at discriminating LLMs . |
| Outcome: | The results show that interactive games are superior to standard benchmarks in discriminating models. |
A Corpus of Natural Multimodal Spatial Scene Descriptions (L18-1)
Copied to clipboard
| Challenge: | Existing work on multimodal spatial descriptions combines speech and hand gestures to form a corpus of multimodal descriptions. |
| Approach: | They present a corpus of multimodal spatial descriptions as commonly occurring in route giving tasks. |
| Outcome: | The proposed corpus of multimodal spatial descriptions is more amenable to computational analysis and useable for learning natural computer interfaces. |
TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches for incremental processing use RNNs or Transformers, which consume whole sequences and are by nature non-incremental. |
| Approach: | They propose a two-pass model for AdaPtIve Revision to obtain an incremental supervision signal for learning an adaptive revision policy. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to restart-incremental Transformers while showing little degradation on full sequences. |
clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents (2023.emnlp-main)
Copied to clipboard
Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, David Schlangen
| Challenge: | Recent work suggests large language models can be understood as (simulators of) such agents. |
| Approach: | They propose a method for systematic evaluation of "Situated Language Understanding Agents" they propose implementing a framework for implementing rules to be played in "self-play" |
| Outcome: | The proposed model can be evaluated in game-like settings, the authors show . they show that the model can follow game-play instructions and perform better than existing models . |
On General Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent paper suggests that the evidence underspecifies the understanding of large language models. |
| Approach: | They propose to use a "general language understanding" benchmark to examine what it could mean in machines. |
| Outcome: | The proposed model can be used to ground questions of the adequacy of benchmarking methods. |
The slurk Interaction Server Framework: Better Data for Better Dialog Models (2022.lrec-1)
Copied to clipboard
| Challenge: | slurk is a lightweight dialog data collection and testing tool for crowdsourcing platforms. |
| Approach: | They present a lightweight dialog server that allows to set up dialog data collections and run experiments. |
| Outcome: | The slurk software allows to set up dialog data collections and run experiments with no limitations on the number of participants. |
Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have led to language model-based systems that do a good job at creating natural dialogue behaviour but are often verbose and brittle. |
| Approach: | They propose a game that requires two players to coordinate on vision and language observations. |
| Outcome: | The proposed game achieves high success rates when bootstrapped with heuristic partner behaviors that implement insights from the analysis of human-human interactions. |
Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models (2025.coling-main)
Copied to clipboard
Sherzod Hakimov, Yerkezhan Abdullayeva, Kushal Koshti, Antonia Schmidt, Yan Weiser, Anne Beyer, David Schlangen
| Challenge: | Existing evaluation paradigms for text-only models are largely limited to a limited number of tasks and require little or no data and training cost. |
| Approach: | They propose to use a game-based evaluation paradigm to evaluate multimodal models by a goal-oriented game (self) play. |
| Outcome: | The proposed evaluation paradigm is more efficient than current methods for text-only models and is more cost-effective than existing methods. |
Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models have demonstrated robust performance on various language tasks using zero-shot or few-shot learning paradigms. |
| Approach: | They propose to use open-source, open-access language models to make visual input accessible to the model using separate verbalisation models. |
| Outcome: | The proposed model can handle visual input but also require strong reasoning component. |
Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers (2023.findings-acl)
Copied to clipboard
| Challenge: | In collaborative situations, communication is performed as signalling and recognizing, and the ability to act on these signals is crucial for future machine learning models to collaborate and interact with humans naturally. |
| Approach: | They propose to use a referential language game as an example of a collaborative joint activity to evaluate intra-episodic feedback given by a teacher. |
| Outcome: | The proposed model can generalize on aspects of scene complexity and perform better than providing only the initial statement. |
Space Efficient Context Encoding for Non-Task-Oriented Dialogue Generation with Graph Attention Transformer (2021.acl-long)
Copied to clipboard
| Challenge: | Recent Transformer-based models aim to integrate fixed background context into non-task-oriented dialogue systems, but the context length is fixed in these architectures, which restricts how much background or dialogue context can be kept. |
| Approach: | They propose a more concise encoding for background context structured in the form of knowledge graphs by expressing the graph connections through restrictions on the attention weights. |
| Outcome: | The proposed architecture reduces space requirements without negative effects on the precision of reproduction of knowledge and perceived consistency. |
Playpen: An Environment for Exploring Learning From Dialogue Game Feedback (2025.emnlp-main)
Copied to clipboard
Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia
| Challenge: | In this paper, we investigate whether Dialogue Games—goal-directed and rule-governed activities driven predominantly by verbal actions—can also serve as a source of feedback signals for learning. |
| Approach: | They introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning, direct alignment and reinforcement learning with Group Relative Policy Optimization. |
| Outcome: | The proposed model improves performance on unseen instances, but negatively impacts other skills, while interactive learning shows balanced improvements without loss of skills. |
Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge (2022.acl-short)
Copied to clipboard
| Challenge: | Existing evaluation methods for visual dialogue models are not consistent . |
| Approach: | They propose a theory-based evaluation method to examine to what degree visual dialogue models incrementally build up representations that do scorekeeping. |
| Outcome: | The proposed method aims to determine to what degree models build up representations that are appropriate to do scorekeeping of shared commitments throughout a visual dialogue. |
Representations of Fact, Fiction and Forecast in Large Language Models: Epistemics and Attitudes (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models estimate and calibrate confidence of large language models with verbalized uncertainty, but they lack a careful examination of the linguistic knowledge of uncertainty encoded in the latent space of LLMs. |
| Approach: | They draw on typological frameworks of epistemic expressions to evaluate LLMs’ knowledge of epistenetic modality, using controlled stories. |
| Outcome: | The proposed models generate expressions matching the strength of evidence and are not robust in generating epistemic expressions. |