Papers by Joyce Chai
In-Context Analogical Reasoning with Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences. |
| Approach: | They apply large pre-trained language models to visual Raven’s Progressive Matrices (RPM) and use language-based abstractions to support analogy in AI systems. |
| Outcome: | The proposed language-based abstractions outperform human models on Raven’s Progressive Matrices and supervised vision-based methods. |
Commonsense Justification for Action Explanation (D18-1)
Copied to clipboard
| Challenge: | a recent study examines the commonsense reasoning used by humans to justify an AI prediction. |
| Approach: | They propose an approach that models object relations/attributes of the world as latent variables and jointly learns a performer that predicts actions and an explainer that gathers commonsense evidence to justify the action. |
| Outcome: | The proposed model achieves significantly higher performance in both action prediction and justification. |
DANLI: Deliberative Agent for Following Natural Language Instructions (2022.emnlp-main)
Copied to clipboard
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, Joyce Chai
| Challenge: | Recent work on embodied AI agents that can perform tasks by following human language instructions is limited by reactive methods, which are insufficient for long-horizon complex tasks. |
| Approach: | They propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience. |
| Outcome: | The proposed agent achieves greater than 70% improvement over reactive baselines on the challenging TEACh benchmark. |
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021.findings-acl)
Copied to clipboard
| Challenge: | despite recent progress, learning new tasks through language instructions remains a challenging problem. |
| Approach: | They propose a hierarchical task learning approach that decomposes task learning into three sub-problems and a model that addresses each sub-probability in a unified manner. |
| Outcome: | The proposed model achieves the state-of-the-art performance on the AL-FRED benchmark . it decomposes task learning into three sub-problems and addresses them in a unified manner . |
From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive performance in various language tasks, but are prone to spurious correlations and illusory information. |
| Approach: | They propose to use pre-trained language models to justify decisions with formalized, coherent reasoning chains. |
| Outcome: | The proposed strategies improve coherence of rationalizations yielding state-of-the-art results on Tiered Reasoning for Intuitive Physics (TRIP). |
Benchmarking and Improving LLM Robustness for Personalized Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations focus on whether a model’s responses align with a user’s preferences, but factuality is an important yet overlooked dimension. |
| Approach: | They propose a scalable framework for evaluating robustness of large language models in personalization and a new dataset, PERGData. |
| Outcome: | The proposed framework improves robustness by 25% across models. |
Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on coding tutoring, but their capabilities in guiding users to solve complex tasks remain underexplored. |
| Approach: | They propose a novel agent workflow, Trace-and-Verify, which combines knowledge tracing to estimate a student’s knowledge state and turn-by-turn verification to ensure effective guidance toward task completion. |
| Outcome: | The proposed agent workflow achieves significantly higher success rates than existing tutoring agents. |
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent inquiries reveal a lack of robust ToM in large language models . current models focus on different aspects of ToM and are prone to shortcuts and data leakage. |
| Approach: | They propose to taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. |
| Outcome: | The proposed model breaks ToM into individual components and treats LLMs as agents physically and socially situated in interactions with humans. |
Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that word acquisition is an efficient, supervised, and continual process. |
| Approach: | They develop a computational process for word acquisition through comparative learning . they frame the acquisition of words as representation-symbol mapping . |
| Outcome: | The proposed method can be used to learn the meaning of a word efficiently and efficiently. |
Experience Grounds Language (2020.emnlp-main)
Copied to clipboard
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian
| Challenge: | aaron carroll: language understanding research is held back by a failure to relate language to the physical world it describes and to social interactions it facilitates. carroll says successful linguistic communication relies on a shared experience of the world. |
| Approach: | They propose to use a broader physical and social context to address communication problems . they argue that the current success of representation learning approaches is limited . |
| Outcome: | a new study suggests that the current success of representation learning requires a parallel tradition of research on the broader physical and social context of language to address the deeper questions of communication. |
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks. |
| Approach: | They propose a commonsense reasoning dataset with dense annotations that allows multi-tiered evaluation of machines’ reasoning process. |
| Outcome: | The proposed model can achieve high end performance but struggle to support predictions with valid supporting evidence. |
Learning to Mediate Disparities Towards Pragmatic Communication (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work explores pragmatic reasoning based on Rational Speech Act (RSA) and Theory of Mind in communication (Zhu et al., 2021). |
| Approach: | They propose a framework where the speaker attempts to learn the speaker-listener disparity and adjust the speech accordingly by adding a light-weighted disparity adjustment layer into working memory on top of speaker’s long-term memory system. |
| Outcome: | The proposed framework can learn and adapt to different types of listeners by adding a light-weighted disparity adjustment layer into working memory on top of speaker’s long-term memory system. |
What Action Causes This? Towards Naive Physical Action-Effect Prediction (P18-1)
Copied to clipboard
| Challenge: | a new task on naive physical action-effect prediction addresses the relationship between concrete actions and their effects on the state of the physical world as depicted by images. |
| Approach: | They propose a task that harnesses web image data to facilitate action-effect prediction. |
| Outcome: | The proposed approach harnesses web image data through distant supervision to facilitate learning for action-effect prediction. |
Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? (2023.findings-emnlp)
Copied to clipboard
Yuwei Bao, Keunwoo Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yossef, Alex de la Iglesia, Megan Su, Xiao Zheng, Joyce Chai
| Challenge: | despite advances in AI, it remains a challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. |
| Approach: | They propose to use a multimodal benchmark dataset to study whether interactive task guidance systems can be quickly adapted to perceptually enabled tasks. |
| Outcome: | The proposed models demonstrate fair performances in some cases with no training . the results will provide a stepping stone for future work on situated task guidance . |
Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties (2024.emnlp-main)
Copied to clipboard
| Challenge: | Emergent In-context Learning on Videos induces in-contact learning over video and text . eILeV-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions. |
| Approach: | They implement Emergent In-context Learning on Videos (EILeV) that induces in-contact learning over video and text by capturing key properties of pre-training data. |
| Outcome: | The proposed training paradigm outperforms off-the-shelf VLMs in few-shot video narration for novel, rare actions. |
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | GOVA examines grounding and bootstrapping in open-world language learning. |
| Approach: | They propose a visually-grounded language model that uses grounding as an objective . they propose GOVA to investigate grounding and bootstrapping in open-world language learning . |
| Outcome: | The proposed model is faster and faster grounded than previous models, the authors show . they show that grounding helps the model to learn unseen words more rapidly and robustly . |
Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Models (2026.acl-long)
Copied to clipboard
Ruixuan Deng, Xiaoyang Hu, Miles Gilberti, Shane Storks, Aman Taxali, Mike Angstadt, Chandra Sripada, Joyce Chai
| Challenge: | Recent work has focused on layerwise interpretations, lacking fine-grained interpretation of specific features and their interaction. |
| Approach: | They identify semantically coherent, context-consistent network components in large language models . they use sparse autoencoders to coactivate sparsity features from a handful of prompts . |
| Outcome: | The proposed model can capture concepts and relations more comprehensively than individual features while maintaining specificity. |
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos (2025.emnlp-main)
Copied to clipboard
Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon
| Challenge: | Recent advances in conversational AI have been substantial, but developing real-time tasks guidance systems remains a challenge. |
| Approach: | They propose a data curation pipeline that synthesizes dialogues from annotated egocentric videos and a suite of automatic evaluation metrics that validated through extensive human studies. |
| Outcome: | The proposed framework synthesizes dialogues from annotated egocentric videos and validates them through extensive human studies. |
SafetyALFRED: Evaluating Safety-Conscious Planning of Vision Language Models (2026.findings-acl)
Copied to clipboard
Josue Torres-Fonseca, Naihao Deng, Yinpei Dai, Shane Storks, Yichi Zhang, Rada Mihalcea, Casey Kennington, Joyce Chai
| Challenge: | Existing safety evaluations focus on hazard recognition through disembodied question answering (QA) settings, but lack a critical gap in evaluating an agent. |
| Approach: | They evaluate multimodal large language models with six categories of kitchen hazards . they propose a safety-based approach that prioritizes multi-step corrective actions . |
| Outcome: | The proposed model can recognize hazards in QA settings, but average mitigation success rates are low . the proposed model is based on the embodied agent benchmark ALFRED . |
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn compositional concepts from previous experience are based on systematic generalization, productivity and substitutivity. |
| Approach: | They propose a retrieval-enhanced meta-learning model to solve the visually grounded compositional concept learning problem by meta-training retrieved primitive concepts from episodes constructed by the retriever. |
| Outcome: | The proposed model outperforms other baselines and the retrieval module plays an important role in this compositional learning process. |
Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, but statistical bias in benchmark data and probing studies has recently called into question their true capabilities. |
| Approach: | They propose to evaluate systems through a measure of prediction coherence by using two existing language understanding benchmarks with different properties to demonstrate its versatility. |
| Outcome: | The proposed evaluation framework is quick, effective, and versatile to provide insight into the coherence of machines’ predictions. |
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Creating embodied, situated agents able to move in, communicate naturally about, and collaborate on human terms in the physical world has been a persisting goal in artificial intelligence (Winograd, 1972). |
| Approach: | They propose to use a 3D Minecraft dataset to model the beliefs of human partners in situ to enable theory of mind modeling in situated interactions. |
| Outcome: | The proposed model can be used to model human collaborative behaviors in the 3D virtual blocks world of Minecraft. |
NLP Reproducibility For All: Understanding Experiences of Beginners (2023.acl-long)
Copied to clipboard
| Challenge: | a study with 93 students in an introductory NLP class shows that beginners' programming skill and comprehension of research papers have a limited impact on their effort spent completing the exercise. |
| Approach: | a study conducted with 93 students in an introductory NLP course questioned them on their programming background and programming background. |
| Outcome: | The results show that beginners' programming skill and comprehension of research papers have a limited impact on their effort spent on the exercise. |
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for embodied agents to learn and perform tasks use low-level instructions, which may not reflect natural human communication. |
| Approach: | They propose to use different types of language inputs to facilitate reinforcement learning (RL) embodied agents. |
| Outcome: | The proposed methods show that agents trained with diverse and informative language can achieve enhanced generalization and fast adaptation to new tasks in an open world. |
DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents (2022.findings-emnlp)
Copied to clipboard
Ziqiao Ma, Benjamin VanDerPloeg, Cristian-Paul Bara, Yidong Huang, Eui-In Kim, Felix Gervits, Matthew Marge, Joyce Chai
| Challenge: | Empowering autonomous driving agents with the ability to navigate in a continuous and dynamic environment is critical. |
| Approach: | They propose a novel interactive simulation platform that enables the creation of unexpected situations on the fly to support empirical studies on situated communication with autonomous driving agents. |
| Outcome: | The proposed platform enables the creation of unexpected situations on the fly to support empirical studies on situated communication with autonomous driving agents. |
Transparent and Coherent Procedural Mistake Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Procedural mistake detection (PMD) is a problem of classifying whether a human user has successfully executed a task. |
| Approach: | They extend PMD to require generating visual self-dialog rationales to inform decisions . they leverage a natural language inference model to formulate two automated metrics for coherence of generated rationale. |
| Outcome: | The proposed model improves on a reframed task with a natural language inference model and a multi-faceted metrics visualization of common outcomes. |
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual illusions are a phenomenon that is often seen in human perception but are not always faithful to the physical world. |
| Approach: | They build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusion in state-of-the-art VLMs. |
| Outcome: | The proposed dataset reveals that larger models are closer to human perception and more susceptible to visual illusions. |
Learning Language through Grounding (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial provides a historical overview of grounding and discusses its use in computational linguistics and in computational language processing. |
| Approach: | They introduce the concept of grounding and discuss future directions and open challenges . they will delve into recent progress in learning lexical semantics, syntax, and complex meanings through various forms of ground. |
| Outcome: | This course will provide an overview of the field of grounding and discuss future directions and challenges related to large language models and scaling. |
Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have adopted a non-interactive training paradigm, and refined pre-trained models through feedback afterward. |
| Approach: | They propose a trial-and-demonstration learning framework that incorporates student trials, teacher demonstrations, and a reward conditioned on language competence at various developmental stages. |
| Outcome: | The proposed framework accelerates word acquisition for student models of equal and smaller numbers of parameters and a strong correlation between the frequency of words in trials and learning curves. |