Papers by Alexander Koller
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
Compositional Generalization without Trees using Multiset Tagging and Latent Permutations (2023.acl-long)
Copied to clipboard
| Challenge: | Seq2seq models struggle with compositional generalization in semantic parsing, i.e. generalizing to unseen compositions or deeper recursion of phenomena that the model handles correctly in isolation. |
| Approach: | They propose a new way of parameterizing and predicting permutations by combining input tokens with multisets of output tokens and a method to backpropagate through the solver. |
| Outcome: | The proposed model outperforms pretrained models and prior work on realistic semantic parsing tasks that require generalization to longer examples. |
Normalizing Compositional Structures Across Graphbanks (2020.coling-main)
Copied to clipboard
| Challenge: | Graph-based meaning representations (MRs) exhibit structural differences that reflect different theoretical and design considerations, presenting challenges to uniform linguistic analysis and cross-framework semantic parsing. |
| Approach: | They propose a method to normalize MRs at the compositional level by linguistically-grounded rules. |
| Outcome: | The proposed method increases the match in compositional structure between MRs and improves multi-task learning in a low-resource setting. |
Script Parsing with Hierarchical Sequence Modelling (2021.starsem-1)
Copied to clipboard
| Challenge: | Script knowledge is a category of commonsense knowledge that describes how people conduct everyday activities sequentially. |
| Approach: | They propose a hierarchical sequence model and transfer learning to do script parsing with a sequence model that accurately tags script participants. |
| Outcome: | The proposed model improves state of the art of event parsing by over 16 points F-score and, for the first time, accurately tags script participants. |
Generalized chart constraints for efficient PCFG and TAG parsing (P18-2)
Copied to clipboard
| Challenge: | Existing pruning techniques limit chart constraints to PCFGs and cannot be applied to more expressive grammars. |
| Approach: | They propose to apply chart constraints to more expressive grammars and a neural tagger which predicts chart constraints at very high precision. |
| Outcome: | The proposed technique accelerates both PCFG and TAG parsing by two orders of magnitude while improving accuracy. |
ADaPT: As-Needed Decomposition and Planning with Language Models (2024.findings-naacl)
Copied to clipboard
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot
| Challenge: | Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. |
| Approach: | They propose an approach that explicitly plans and decomposes complex sub-tasks when the LLM is unable to execute them. |
| Outcome: | The proposed approach significantly outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft. |
Zero-shot Script Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Existing resources cover only a small number of tasks, limiting its practical usefulness. |
| Approach: | They propose a zero-shot learning approach to script parsing which enables us to acquire script knowledge without domain-specific annotations. |
| Outcome: | The proposed model outperforms a previous model with scenario-specific supervision and achieves 68.1/74.4 average F1 for event / participant parsing. |
AMR dependency parsing with a typed semantic algebra (P18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representations (AMRs) are graphs which describe the predicate-argument structure of a sentence. |
| Approach: | They propose a semantic parser which parses strings into tree representations of the compositional structure of an AMR graph. |
| Outcome: | The proposed parser outperforms baselines and standard neural techniques for supertagging and dependency tree parsing. |
Semantic Expressive Capacity with Bounded Memory (P19-1)
Copied to clipboard
| Challenge: | Existing methods for semantic parsing use placeholders to represent relations between sentences and semantic representations. |
| Approach: | They show that compositional parsers can remember unbounded number of placeholders . this is the first study of this kind to describe relations between sentences and semantic representations based on projective mechanisms. |
| Outcome: | The proposed method can represent relations between sentences and semantic representations without using nonprojective mechanisms. |
Scope-enhanced Compositional Semantic Parsing for DRT (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional semantic parsers for DRT struggle to produce well-formed representations due to the complexity of the sentence. |
| Approach: | They propose a compositional, neurosymbolic semantic parser for DRT that uses a novel mechanism for predicting quantifier scope. |
| Outcome: | The proposed model produces well-formed outputs and performs well on complex sentences. |
What’s the Meaning of Superhuman Performance in Today’s NLU? (2023.acl-long)
Copied to clipboard
Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli
| Challenge: | Recent research has focused on developing larger pretrained language models and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities. |
| Approach: | They propose to use benchmarks such as SuperGLUE and SQUAD to evaluate PLMs' abilities in language understanding, reasoning, and reading comprehension to assess their performance. |
| Outcome: | The proposed benchmarks have serious limitations affecting comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks. |
Graph-Based Meaning Representations: Design and Processing (P19-4)
Copied to clipboard
| Challenge: | This tutorial focuses on representing and processing sentence meaning in the form of labeled directed graphs. |
| Approach: | This tutorial will briefly review relevant background in formal and linguistic semantics . it will also briefly define a unified abstract view on different flavors of semantic graphs - and associated terminology . |
| Outcome: | The tutorial will briefly review relevant background in formal and linguistic semantics . |
Predicting generalization performance with correctness discriminators (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models estimate accuracy of models on unlabeled test data, but they hide their own uncertainty. |
| Approach: | They propose a model that establishes upper and lower bounds on the accuracy without requiring gold labels for the unseen data. |
| Outcome: | The proposed model establishes upper and lower bounds on accuracy without requiring gold labels for the unseen data. |
SLOG: A Structural Generalization Benchmark for Semantic Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional generalization benchmarks focus on lexical generalisation, the interpretation of novel lexicals in syntactic structures familiar from training. |
| Approach: | They propose a semantic parsing dataset that extends COGS with 17 structural generalization cases to evaluate how well models generalize to new complex linguistic expressions. |
| Outcome: | The proposed model generalization accuracy is far below the near-perfect accuracy of existing models on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models’ lexical and structural generalization capacities. |
Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate instructions using large language models require spatiotemporal awareness of multiple objects and their surroundings. |
| Approach: | They propose a tree-based evaluation metric for LLM-generated step-by-step assembly instructions that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. |
| Outcome: | The proposed metric better correlates with manually-annotated error counts, and is more robust against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity. |
Aligning Actions Across Recipe Graphs (2021.emnlp-main)
Copied to clipboard
| Challenge: | a recipe explains step by step how to cook a dish, but recipes differ in which cooking actions they describe explicitly, how they describe them, and in which order. |
| Approach: | They propose a recipe corpus which annotates cooking steps in recipes at sentence level . they train a neural model to predict recipes on ARA and model it for automatic understanding . |
| Outcome: | The proposed model can predict recipes with fine-grained structural information . it shows that recipes can be explained in different ways, or not at all . |
Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests (2025.findings-emnlp)
Copied to clipboard
Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, Raffaella Bernardi
| Challenge: | MMLU and BBH are three evaluation paradigms for language learning models . interactive games are superior to standard benchmarks in discriminating models based on human cognitive assessments . |
| Approach: | They examine three evaluation paradigms: standard benchmarks, interactive games and cognitive tests . they examine whether interactive games are more effective at discriminating LLMs . |
| Outcome: | The results show that interactive games are superior to standard benchmarks in discriminating models. |
Procedural Environment Generation for Tool-Use Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to curation of tool-use training data are non-interactive and/or non-compositional. |
| Approach: | They propose a pipeline for the procedural generation of interactive tools and compositional tool-use data. |
| Outcome: | The proposed pipeline improves on a range of tool-use benchmarks and sets the new SoTA for two metrics on the NESTFUL dataset. |
Simple and effective data augmentation for compositional generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | Compositional generalization is the ability of a system to correctly predict the meaning of complex sentences when trained on simpler sentences. |
| Approach: | They propose to use data augmentation methods to generate additional training data by sampling from an augmentation distribution to generalize to the out-of-distribution test data. |
| Outcome: | The proposed method outperforms existing methods that sampled from the training distribution and outperformed existing methods. |
Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Inductive biases play a critical role in NLP, especially in learning from limited data and generalizing systematically outside of the training distribution. |
| Approach: | They propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform syntactic transformations of dependency trees given a description of the transformation. |
| Outcome: | The proposed model can perform syntactic transformations and generalize semantic parsing with attention heads that keep track of which syntaktic transformation needs to be applied to which token. |
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020.acl-main)
Copied to clipboard
| Challenge: | a priori, large neural language models are described as understanding or capturing meaning on tasks that are ostensibly meaningsensitive. |
| Approach: | They argue that a system trained only on form has no way to learn meaning . they argue that this is due to a misunderstanding of the relationship between form and meaning - which is a misconception in NLP . |
| Outcome: | The proposed model can't learn meaning because it only uses form as training data, the authors argue . they argue that a clear understanding of the distinction between form and meaning will guide the field towards better science around natural language understanding. |
Compositional generalization with a broad-coverage semantic parser (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent work has shown that compositional generalization on COGS is difficult and complex. |
| Approach: | They propose a compositional semantic parser that solves compositional generalization on COGS dataset. |
| Outcome: | The AM parser solves compositional generalization on the COGS dataset. |
Compositional Semantic Parsing across Graphbanks (P19-1)
Copied to clipboard
| Challenge: | Existing semantic parsers that map sentences to graph-based meaning representations are hand-designed for specific graphbanks. |
| Approach: | They propose a compositional neural semantic parser which achieves competitive accuracies across graphbanks. |
| Outcome: | The proposed system achieves competitive accuracies across a variety of graphbanks. |
A Knapsack by Any Other Name: Presentation impacts LLM performance on NP-hard problems (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EHOP is a collection of NP-hard optimization problems expressed in natural language . state-of-the-art LLMs solve textbook problems more accurately than their real-life counterparts, but they lack a truly robust reasoning mechanism. |
| Approach: | They introduce a dataset of everyday hard optimization problems (EHOP) which includes problem formulations found in computer science textbooks, versions dressed up as problems that could arise in real life, and variants with inverted rules. |
| Outcome: | The proposed dataset includes problem formulations found in computer science textbooks, versions dressed up as problems that could arise in real life, and variants with inverted rules. |
A Corpus of German Abstract Meaning Representation (DeAMR) (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representations (AMRs) are semantic graphs that abstract away from surface syntax and capture the meaning of who does what to whom in a sentence. |
| Approach: | They propose to use German Abstract Meaning Representation (Deutsche AMR) to represent the structure and semantics of German. |
| Outcome: | The proposed framework is based on an annotated corpus of 400 DeAMR in German and is validated through inter-annotator agreement. |
SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation (2024.acl-long)
Copied to clipboard
| Challenge: | Popular neural architectures lack strong structural inductive biases for seq2seq NLP tasks . previous work shows that these models struggle with systematic generalization . |
| Approach: | They propose to inject a structural inductive bias into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. |
| Outcome: | The proposed method improves few-shot learning and generalization of FST-like models. |
Language models can learn implicit multi-hop reasoning, but only if they have lots of training data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies explore the ability of language models to solve multi-hop reasoning tasks without chain of thought. |
| Approach: | They propose to use GPT2-style language models to train k-hop reasoning models . they show that the required training data grows exponentially in k . |
| Outcome: | The proposed models can learn implicit reasoning without chain-of-thoughts, the authors show . their training data grows exponentially in k, and the required number of transformer layers grows linearly in the model. |
LLMs syntactically adapt their language use to their conversational partner (2025.acl-short)
Copied to clipboard
| Challenge: | Adapting to the language of a communication partner is associated with increased success in goal-oriented conversations. |
| Approach: | They construct a corpus of conversations between large language models (LLMs) and measure their syntactic adaptation. |
| Outcome: | The proposed model can adapt to the language of the conversational partner in at least a rudimentary way. |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
Generating Instructions at Different Levels of Abstraction (2020.coling-main)
Copied to clipboard
| Challenge: | Using hierarchical planning, technical instructions can be described at different levels of abstraction. |
| Approach: | They propose a method from AI planning which can capture the structure of complex objects neatly. |
| Outcome: | The proposed method can capture the structure of complex objects neatly. |
Playpen: An Environment for Exploring Learning From Dialogue Game Feedback (2025.emnlp-main)
Copied to clipboard
Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia
| Challenge: | In this paper, we investigate whether Dialogue Games—goal-directed and rule-governed activities driven predominantly by verbal actions—can also serve as a source of feedback signals for learning. |
| Approach: | They introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning, direct alignment and reinforcement learning with Group Relative Policy Optimization. |
| Outcome: | The proposed model improves performance on unseen instances, but negatively impacts other skills, while interactive learning shows balanced improvements without loss of skills. |
Story Generation with Rich Details (2020.coling-main)
Copied to clipboard
| Challenge: | Recent neural story generation systems have been able to produce coherent stories. |
| Approach: | They propose a model that features an outliner, which proceeds the main story line to realize global coherence, and a detailer, which supplies relevant details to the story in a locally coherent manner. |
| Outcome: | The proposed model outperforms baseline models in the informativeness and coherence tests on human participants. |
Structural generalization is hard for sequence-to-sequence models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Sequence-to-sequence models have been successful across many NLP tasks, but they have low generalization accuracy . |
| Approach: | They propose to use linguistic knowledge to overcome generalization limitations of seq2seq models . they show that human beings are able to understand and produce linguistic structures they have never observed before . |
| Outcome: | The proposed models can overcome this limitation by having linguistic knowledge built in. |
Compositional Generalisation with Structured Reordering and Fertility Layers (2023.eacl-main)
Copied to clipboard
| Challenge: | Seq2seq models struggle with compositional generalisation, i.e. generalising to new and potentially more complex structures than seen during training. |
| Approach: | They propose a flexible end-to-end differentiable neural model that composes two structural operations: a fertility step and a reordering step. |
| Outcome: | The proposed model outperforms seq2seq models on compositional splits of realistic semantic parsing tasks. |
Fast semantic parsing with well-typedness guarantees (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing algorithms for AM dependency parsing are slow and do not support linguistic principles. |
| Approach: | They propose an A* parser and a transition-based parsing algorithm which guarantee well-typedness and improve parse speed by up to 3 orders of magnitude. |
| Outcome: | The proposed algorithms guarantee well-typedness and improve parsing speed by up to 3 orders of magnitude while maintaining or improving accuracy. |