Papers by Philipp Mondorf
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Conditional acceptability refers to how plausible a conditional statement is perceived to be. |
| Approach: | They propose to use conditional acceptability models to assess their models' conditional acceptance . they use probabilistic and semantic cues to assess whether a condition is acceptable or plausible . |
| Outcome: | The proposed models are sensitive to conditional probability and semantic relevance, but do not align more closely with human judgments. |
Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | logical puzzles that involve determining identity of characters require a variety of reasoning skills. |
| Approach: | They propose a benchmark for suppositional reasoning based on knights and knaves puzzles . they show lower-performing models exhibit a diverse range of reasoning errors . |
| Outcome: | The proposed benchmark demonstrates that models struggle with suppositional reasoning . lower performing models struggle to grasp the concept of truth and lies, the study finds . |
Reason to Rote: Rethinking Memorization in Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models readily memorize arbitrary training instances, such as label noise . however, such memorization does not affect generalizable reasoning abilities . |
| Approach: | They investigate how large language models memorize label noise and why it affects generalizability. |
| Outcome: | The proposed model performs well on reasoning tasks even when memorized labels are missing . the proposed model is able to generalize to correctly answer "87+19=106" |
Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in mechanistic interpretability have made progress in identifying circuits, the minimal computational subgraphs responsible for a model’s behavior on specific tasks. |
| Approach: | They propose to analyze circuits for highly compositional subtasks within a transformer-based language model to determine their modularity and how they relate to each other. |
| Outcome: | The proposed approach shows that the circuits identified exhibit notable node overlap and cross-task faithfulness. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored methods to enhance self-correction in large language models, but little attention has been given to understanding the models’ internal mechanisms underlying error detection. |
| Approach: | They propose to use a large language model to analyze arithmetic errors in four smaller-sized LLMs and identify their internal mechanisms. |
| Outcome: | The proposed models heavily rely on consistency headstextemdashattention heads that assess surface-level alignment of numerical values in arithmetic solutions. |
Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in the domain of large language models (LLMs) have showcased their capability in executing deductive reasoning tasks. |
| Approach: | They examine inferential strategies employed by large language models through a detailed evaluation of their responses to propositional logic problems. |
| Outcome: | The proposed model shows that it displays reasoning patterns similar to humans, including strategies like supposition following or chain construction. |