Papers by Michael Hanna
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments (2022.coling-1)
Copied to clipboard
| Challenge: | embodied AI tasks require a strong understanding of verbs and their corresponding actions. |
| Approach: | They propose a controlled benchmark for embodied action understanding using a simulated environment and a visual feature extractor. |
| Outcome: | The proposed benchmark achieves 81.4% accuracy and high inter-annotator agreement . the proposed model falls behind human models in a zero-shot scenario . |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
Transformer-specific Interpretability (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Transformers are dominant play-ers in various scientific fields, but their inner workings remain opaque. |
| Approach: | This tutorial presents a trending approach to interpreting Transformers . it uses specific features of the Transformer architecture to quantify context- mixing interactions . |
| Outcome: | This tutorial aims to show how a new trending approach can be applied to Transformer-based models. |
Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a recent study has found LMs focus on processing syntactic relations, but not structural information. |
| Approach: | They examine the mechanisms underlying garden path sentence processing in LMs . they use sparse autoencoders to identify interpretable features that determine which continuation . |
| Outcome: | The proposed model lacks syntactic features and shallow heuristics to perform incremental sentence processing. |
The Functional Relevance of Probed Information: A Case Study (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that transformer models like BERT rely on number information encoded in their representations of sentences’ subjects and head verbs when performing subject-verb agreement. |
| Approach: | They propose to use probing to find out which words contain functionally relevant information encoded in the representations of subject plurality and words that agree with it in number in BERT. |
| Outcome: | The proposed model only uses the subject plurality information encoded in its representations of the subject and words that agree with it in number. |
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers (2024.findings-acl)
Copied to clipboard
| Challenge: | Previous work has examined performance disparities between AAL speakers and White Mainstream English speakers . but, this work has not sought to understand the impacts of these disparities on AAL speaker. |
| Approach: | They examine the experiences of African American Language (AAL) speakers when using language technologies. |
| Outcome: | The authors interview 19 AAL speakers to understand performance disparities . they find that speakers often undertake invisible labor to successfully use language technologies . |
When Language Models Fall in Love: Animacy Processing in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Animacy is not always expressed directly in language, but it manifests indirectly in English . atypically animate entities are easier to remember and prioritized in visual processing . |
| Approach: | They find that LMs behave much like humans when presented with entities whose animacy is typical. |
| Outcome: | The proposed model can learn about animacy even when presented with atypically animate entities. |
Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST! (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models are unable to identify and interpret semantically underspecified sentences when explicitly prompted. |
| Approach: | They propose a Dataset of semantically underspecified Sentences grouped by type to study whether pre-trained language models correctly identify and interpret underspecify sentences. |
| Outcome: | The proposed dataset shows that pre-trained language models are reasonably able to identify and interpret underspecified sentences when explicitly prompted. |