Papers by Jesse Thomason
The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge (2025.findings-naacl)
Copied to clipboard
| Challenge: | Sign language models could make language technologies more accessible to deaf and hard-of-hearing signers, but the supply of accurately labeled data struggles to meet the demand associated with training large, end-to-end architectures. |
| Approach: | They construct an American Sign Language Knowledge Graph from 11 sources of linguistic knowledge and use it to train neuro-symbolic models on ASL video input tasks. |
| Outcome: | The proposed model achieves 91% accuracies for isolated sign recognition, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos. |
ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments (2022.emnlp-main)
Copied to clipboard
Arjun Akula, Spandana Gella, Aishwarya Padmakumar, Mahdi Namazifar, Mohit Bansal, Jesse Thomason, Dilek Hakkani-Tur
| Challenge: | Embodied Vision and Language Task Completion requires an embodied agent to interpret natural language instructions and egocentric visual observations to navigate through and interact with environments. |
| Approach: | They propose to use ALFRED to test whether models can adapt to tasks not seen during training that require the same types of language understanding as ALFred. |
| Outcome: | The proposed model can adapt to tasks that require the same types of language understanding as ALFRED-L. |
Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Widely used multi-modal pretrained models generalize poorly to out-of-distribution data, suggesting shortcomings in the VLE2E pipeline. |
| Approach: | They develop a segment-combine test for multi-image queries and contrast set for cross-benchmark transfer. |
| Outcome: | The proposed method shows that it is possible to train both neural and neuro-symbolic models in the same way. |
Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multistep question answering have shown promise in generating multistep solutions, but they lack robustness. |
| Approach: | They propose a framework that trains a model to robustly answer multistep questions by generating and answering sub-questions. |
| Outcome: | The proposed framework outperforms neuro-symbolic methods on a DROP contrast set and GPT-3.5 on QA adversarial sets. |
Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (2024.findings-acl)
Copied to clipboard
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Chandu
| Challenge: | ReCoVERR reduces the over-abstention of a vision-language system with low tolerance for inaccurate predictions without increasing the error rate of the system’s predictions. |
| Approach: | They propose an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system’s predictions. |
| Outcome: | ReCoVERR reduces the over-abstention of a vision-language system without increasing the error rate of the system’s predictions. |
Words that make SENSE: Sensorimotor Norms in Learned Lexical Token Representations (2026.findings-acl)
Copied to clipboard
| Challenge: | Empirical studies suggest that comprehending action, perceptual and abstract concepts elicits rapid, automatic activity in modality-specific brain areas. |
| Approach: | They propose a model that predicts Lancaster sensorimotor norms from word lexical embeddings. |
| Outcome: | The proposed model predicts Lancaster sensorimotor norms from word lexical embeddings. |
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for identifying object referents of language expressions consider target and distractor objects independently and pool multiple views before grounding. |
| Approach: | They propose a model that selects an object referent based on language that distinguishes between two similar objects and a multi-view approach to grounding in context model which reduces the relative error by 12.9% . |
| Outcome: | The proposed model improves on the SNARE object reference task with a relative error reduction of 12.9% and an absolute improvement of 2.7%. |
When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning is a method for adapting large language models to tasks with instructions or examples. |
| Approach: | They propose a method to decompose the output of large language models into components . they propose component reweighting, which learns to linearly re-scale component activations from a few labeled examples. |
| Outcome: | The proposed method improves by 6.0% accuracy points over 24 examples given 24 examples on Llama-2-7B. |
Experience Grounds Language (2020.emnlp-main)
Copied to clipboard
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian
| Challenge: | aaron carroll: language understanding research is held back by a failure to relate language to the physical world it describes and to social interactions it facilitates. carroll says successful linguistic communication relies on a shared experience of the world. |
| Approach: | They propose to use a broader physical and social context to address communication problems . they argue that the current success of representation learning approaches is limited . |
| Outcome: | a new study suggests that the current success of representation learning requires a parallel tradition of research on the broader physical and social context of language to address the deeper questions of communication. |
Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has found that explanations can easily convince users that inaccurate VLM predictions are correct. |
| Approach: | They propose to evaluate two complementary qualities of VLM-generated explanations via two quality scoring functions to improve their accuracy. |
| Outcome: | The proposed explanations improve accuracy on the A-OKVQA, VizWiz, and MMMU-Pro tasks by 11.1%, including a 15.4% reduction in falsely believing incorrect predictions. |
RMM: A Recursive Mental Model for Dialogue Navigation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on language-guided robots focuses on the latter, but little attention is paid to the guiding agent. |
| Approach: | They propose a two-agent task where one agent navigates and asks questions that a second, guiding agent answers. |
| Outcome: | The proposed model can be generalized to novel environments. |
Why Do Some Inputs Break Low-Bit LLM Quantization? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-bit weight-only quantization reduces memory usage but disproportionately affects certain examples. |
| Approach: | They analyze quantization errors of 50 pairs of methods on large language models and test their hypothesis . |
| Outcome: | The proposed method reduces the memory footprint of large language models while maintaining reasonable performance across benchmarks. |
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (N19-1)
Copied to clipboard
| Challenge: | Existing work on unimodal approaches often lacks dataset biases . we present unimod ablations on three recent datasets in visual navigation and QA . |
| Approach: | They propose unimodal ablations for visual navigation and QA using egocentric vision . they argue that unimodulated models better capture and reflect dataset biases . |
| Outcome: | The proposed models outperform full models on visual navigation and QA tasks with language only on three recent datasets. |
Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on the ability of localization methods to pinpoint LLM components for memorized data are lacking. |
| Approach: | They propose to use a subset of LLM weights to evaluate localization methods . they propose to measure how much dropping out identified neurons deletes a memorized sequence. |
| Outcome: | The proposed methods show promising localization ability, despite differences in their evaluations. |
Large Language Models Do Multi-Label Classification Differently (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. |
| Approach: | They propose to use initial probability distributions to analyze output distributions of LLMs at each label generation step to find out how LLM models perform multi-label classification. |
| Outcome: | The proposed methods improve alignment and predictive performance over existing methods. |
Improving Sign Recognition with Phonology (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing work does not consider sign language phonology, but none leverages it . a recent study has shown that sign language recognition models lack structure . |
| Approach: | They explicitly recognize the role of phonology in sign production to train models for isolated sign language recognition . they train models that take in pose estimations of a signer producing a single sign to predict its phonological characteristics . |
| Outcome: | The proposed model improves sign recognition accuracy by 9% on the WLASL benchmark . the study could accelerate linguistic research in the domain of signed languages . |
Can VLMs Recall Factual Associations From Visual References? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic deficiency in the multimodal grounding of Vision Language Models is identified . VLMs can recall factual associations when provided a textual reference to an entity . |
| Approach: | They identify a systematic deficiency in the multimodal grounding of Vision Language Models . they show that VLMs struggle to link their internal knowledge of an entity with its image representation . |
| Outcome: | The study shows that VLMs struggle to link internal knowledge with image representations . the findings provide recommendations for future research . |
Efficient End-to-End Visual Document Understanding with Rationale Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-processing tools such as optical character recognition (OCR) can map document image inputs to textual tokens, then large language models (LLMs) can reason over text. |
| Approach: | They propose a method that integrates outputs of OCR tools and larger multimodal models as intermediate "rationales" a student model is trained to predict rationales and answers based on visual documents . |
| Outcome: | The proposed model outperforms the base model on three visual document understanding benchmarks with only 1% higher computational cost. |
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions (2022.acl-long)
Copied to clipboard
| Challenge: | Vision-and-Language Navigation (VLN) is a research topic that is gaining attention in the field of artificial intelligence. |
| Approach: | They propose to build an embodied agent that can communicate with humans in natural language and navigate in real 3D environments. |
| Outcome: | This paper reviews current studies in the emerging field of vision-and-language navigation . it highlights limitations and opportunities for future work . |
Iterative Formalization and Planning in Partially Observable Environments (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to formalize an environment into the Planning Domain Definition Language (PDDL) have been shown to improve performance and control. |
| Approach: | They propose a framework to iteratively formalize, plan, grow, and refine PDDL representations by decomposing the environment and the goal into fully observable episodes. |
| Outcome: | The proposed framework improves planning success and exhibits robustness against problem complexity compared to end-to-end approaches. |
Language Models Can Infer Action Semantics for Symbolic Planners from Environment Feedback (2025.naacl-long)
Copied to clipboard
| Challenge: | Symbolic planners can discover sequences of actions from initial to goal states given domain-specific action semantics. |
| Approach: | They propose a domain induction task that requires an expert to define action semantics of domain to enable symbolic planning. |
| Outcome: | The proposed model boosts plan success rate from 36.4% to 100% on 7 environments. |
Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms for learning unimodal vision-only or language-only tasks are limited by the size and computational load of fine-tuning large-scale pre-trained neural networks. |
| Approach: | They propose a transformer-based CL architecture for learning bimodal vision-and-language tasks by increasing the number of the learnable parameters dynamically and using knowledge distillation. |
| Outcome: | The proposed model reaches state-of-the-art on vision-and-language tasks. |