Papers by Seungwhan Moon
SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks do not capture the complexity of structured, step-by-step reasoning essential in physics and related domains. |
| Approach: | They propose a large-scale synthetic benchmark of 15K university-level physics problems . they use structured, step-by-step reasoning and executable Python code to produce the ground-truth solution. |
| Outcome: | The proposed model is based on a set of 15K university-level physics problems with three question types. |
Multimodal Named Entity Disambiguation for Noisy Social Media Posts (P18-1)
Copied to clipboard
| Challenge: | Social media posts often contain unstructured text or images, making opinion mining challenging. |
| Approach: | They propose a new task for multimodal social media captions with named entities annotated and linked to external knowledge bases. |
| Outcome: | The proposed model outperforms state-of-the-art text-only NED models . it predicts correct entities in knowledge graph embeddings space, showing its efficacy and potentials . |
Embodied Executable Policy Learning with Language-based Scene Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing Large Language models with text inputs lack the capability to evolve with non-expert interactions with environments. |
| Approach: | They propose a novel learning paradigm that generates robots’ executable actions in the form of text, derived solely from visual observations. |
| Outcome: | The proposed learning paradigm surpasses baselines and can adapt to the target tasks effectively. |
SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing task-oriented dialog datasets do not situate the dialog in the user’s multimodal context. |
| Approach: | They propose to use a dataset to study multimodal task-oriented dialogs in the shopping domain to situate them in the user’s multimodal context. |
| Outcome: | The proposed dataset includes 11K task-oriented user->assistant dialogs (117K utterances) in the shopping domain, grounded in immersive and photo-realistic scenes. |
Adding Chit-Chat to Enhance Task-Oriented Dialogues (2021.naacl-main)
Copied to clipboard
Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, Claire Cardie
| Challenge: | Existing dialogue systems focus on functional goals, open-domain chatbots on socially engaging conversations. |
| Approach: | They propose to add chit-chat to ENhance Task-ORiented dialogues by a human-assisted data collection approach to augment task-oriented dialogues with minimal annotation effort. |
| Outcome: | The proposed models can code-switch between task and chit-chat to be more engaging, interesting, knowledgeable, and humanlike while maintaining competitive task performance. |
Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks (2023.findings-eacl)
Copied to clipboard
Pedro Rodriguez, Mahmoud Azab, Becka Silvert, Renato Sanchez, Linzy Labson, Hardik Shah, Seungwhan Moon
| Challenge: | Existing benchmarks for text-to-video retrieval are incomplete, resulting in false negatives . a recent state-of-the-art model gains 25% recall points, but this is not the case for TVR. |
| Approach: | They propose to retire video captioning datasets as TVR benchmarks . they propose to annotate and release additional caption-video pairs to mitigate this flaw . |
| Outcome: | The proposed method fails to accurately reflect reality, despite lack of purpose-built benchmarks. |
Zero-Shot Dialogue State Tracking via Cross-Task Transfer (2021.emnlp-main)
Copied to clipboard
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, Pascale Fung
| Challenge: | Existing approaches to training a dialogue state tracking model require extensive annotated dialogue data. |
| Approach: | They propose to transfer cross-task knowledge from general question answering corpora to QA model that can handle zero-shot DST. |
| Outcome: | The proposed model improves existing zero-shot and few-shot results on MultiWoz and shows better generalization ability in unseen domains. |
VideoMind: Thinking in Steps for Long Video Understanding (2026.eacl-industry)
Copied to clipboard
Shubhang Bhatnagar, Renxiong Wang, Kapil Krishnakumar, Adel Ahmadyan, Zhaojiang Lin, Lambert Mathias, Xin Luna Dong, Babak Damavandi, Narendra Ahuja, Seungwhan Moon
| Challenge: | Multimodal Large Language Models struggle with Long Video Understanding due to their limited context window and the distributed nature of salient information across many redundant frames. |
| Approach: | They propose a training framework that mimics a human reasoning process to train Long Video Understanding models. |
| Outcome: | The proposed framework achieves 77.6% performance on Video MME, LongVideo, and MLVU benchmarks while yielding 5% improvement on Llama 4 Scout. |
Large Language Models as Zero-shot Dialogue State Tracker through Function Calling (2024.acl-long)
Copied to clipboard
Zekun Li, Zhiyu Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Dong, Adithya Sagar, Xifeng Yan, Paul Crook
| Challenge: | Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. |
| Approach: | They propose a method for solving dialogue state tracking (DST) with large language models through function calling. |
| Outcome: | The proposed approach improves zero-shot DST, allowing adaptation to diverse domains without extensive data collection or model tuning. |
Information Seeking in the Spirit of Learning: A Dataset for Conversational Curiosity (2020.emnlp-main)
Copied to clipboard
| Challenge: | Open-ended human learning and information-seeking systems often ignore the user’s pre-existing knowledge. |
| Approach: | They propose to use pre-existing user knowledge to build a model that reproduces human assistant policies and improves over a bert content model by 13 mean reciprocal rank points. |
| Outcome: | The proposed model reproduces human assistant policies and improves over a bert content model by 13 mean reciprocal rank points. |
Navigating Connected Memories with a Task-oriented Dialog System (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen an increasing trend in the volume of personal media captured by users thanks to smartphones and smart glasses. |
| Approach: | They propose to use dialogs for connected memories to query media collection . they use a multimodal dialog simulator and manual paraphrasing to obtain natural language utterances. |
| Outcome: | The proposed dataset contains 11.5k userassistant dialogs grounded in simulated personal memory graphs. |
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams (2023.acl-long)
Copied to clipboard
Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, Seungwhan Moon
| Challenge: | Existing models lack a large-scale benchmark to capture user–assistant interactions . et al., 2022: 145-160. |
| Approach: | They propose a video-grounded task-oriented dialog dataset that captures real-world AI-assisted user scenarios in VR. |
| Outcome: | The proposed dataset captures real-world AI-assisted user scenarios in VR. |
SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM (2024.findings-emnlp)
Copied to clipboard
Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul Crook, Yifan Xu, Babak Damavandi, Xin Dong, Christos Faloutsos, Lei Li, Seungwhan Moon
| Challenge: | Vision-extended LLMs have made significant strides in VQA, but they still encounter significant difficulties in handling queries involving long-tail entities. |
| Approach: | They propose a benchmark to test models' ability to identify entities and provide detailed, entity-specific knowledge by combining 10 images and 10 knowledge-intensive QA pairs. |
| Outcome: | The proposed model outperforms existing methods on the SnapNTell dataset, achieving a 66.5% improvement in the BELURT score. |
Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue StateTracking (2021.naacl-main)
Copied to clipboard
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, Rajen Subba
| Challenge: | Existing models for zero-shot cross-domain dialogue state tracking require in-domain data to model a new domain. |
| Approach: | They propose a slot descriptions enhanced generative approach for zero-shot cross-domain DST by encoding a dialogue context and a slots with a pre-trained encoder and generating slot value in auto-regressive manner. |
| Outcome: | The proposed model significantly improves state-of-the-art results in zero-shot cross-domain setting. |
TRACE: A Framework for Analyzing and Enhancing Stepwise Reasoning in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Evaluating large vision-language models has focused on final-answer correctness, but this metric is often insufficient and misleading. |
| Approach: | They propose a framework that decomposes complex multimodal tasks into Auxiliary Reasoning Sets (ARS) ARS decomposition reveals how consistently a model reasons across sub-questions with structured dependencies. |
| Outcome: | a new framework improves diagnostic evaluation of large vision-language models . it decomposes complex multimodal tasks into auxiliary reasoning sets with structured dependencies . the framework pinpoints reasoning failures and exposes errors overlooked by standard evaluation . |
Memory Grounded Conversational Reasoning (D19-3)
Copied to clipboard
| Challenge: | Existing methods to retrieve and browse memories are keyword based searches or catalog based browsing systems. |
| Approach: | They propose a conversational system which engages the user through a multi-modal, multi-turn dialog over the user’s memories. |
| Outcome: | The proposed system can perform QA over memories and make suggestions to surface related events or facts from past memories to make conversations more engaging and natural. |
IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to align motion sensors with text and video are limited in their scale and limited in the use of IMU models. |
| Approach: | They propose to project IMU motion sensor recordings into the joint representation space of Contrastive Language-Image Pre-training (CLIP) they introduce several new IMU-based Wearable AI applications such as motion-based media search, or an LM-based multimodal reasoning with motion sensor data. |
| Outcome: | The proposed approach significantly improves downstream performance when fine-tuned for each application, demonstrating its universal usage as a new pre-trained resource. |
Situated and Interactive Multimodal Conversations (2020.coling-main)
Copied to clipboard
Seungwhan Moon, Satwik Kottur, Paul Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, Alborz Geramifard
| Challenge: | Situated Interactive MultiModal Conversations (SIMMC) is a new direction for virtual assistants that handle multimodal inputs and perform multimodal actions. |
| Approach: | They propose to use Situated Interactive MultiModal Conversations (SIMMC) to train agents to take multimodal actions grounded in a co-evolving multimodal context. |
| Outcome: | The proposed model will be made publicly available. |
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks do not have enough annotations to analyze video-grounded dialogue systems and understand their capabilities and limitations in isolation. |
| Approach: | They present a Diagnostic Dataset for Video-grounded dialogue with minimal biases and detailed annotations for the different types of reasoning over the spatio-temporal space of video. |
| Outcome: | The proposed system is based on 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video. |
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos (2025.emnlp-main)
Copied to clipboard
Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon
| Challenge: | Recent advances in conversational AI have been substantial, but developing real-time tasks guidance systems remains a challenge. |
| Approach: | They propose a data curation pipeline that synthesizes dialogues from annotated egocentric videos and a suite of automatic evaluation metrics that validated through extensive human studies. |
| Outcome: | The proposed framework synthesizes dialogues from annotated egocentric videos and validates them through extensive human studies. |
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model (2024.emnlp-industry)
Copied to clipboard
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar
| Challenge: | Prior work on LLMs focused on models that combine text and one other modality, such as image encoders or proprietary models that are not open sourced. |
| Approach: | They propose a unified model that reasons over diverse input modality signals and generates textual responses. |
| Outcome: | The proposed model performs better on multimodal tasks than industry-leading models . |
OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs (P19-1)
Copied to clipboard
| Challenge: | Existing models that use a large-scale knowledge graph to create a conversational reasoning model are domain-agnostic and scalable. |
| Approach: | They propose a conversational reasoning model that strategically traverses through a large-scale common fact knowledge graph to introduce engaging and contextually diverse entities and attributes. |
| Outcome: | The proposed model retrieves more natural responses than state-of-the-art models in both in-domain and cross-domain tasks. |
User Memory Reasoning for Conversational Recommendation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing systems that update user preferences via asking relevant questions are unable to dynamically maintain and reason over their knowledge for current (and possibly future) recommendations. |
| Approach: | They propose a new memory graph (MG) -> Conversational Recommendation parallel corpus with 7K+ human-to-human role-playing dialogs and a graph-based reasoning model that updates MG from unstructured utterances and predicts optimal dialog policies based on updated MG. |
| Outcome: | The proposed model is based on a large-scale user memory bootstrapped from real-world user scenarios and can be easily updated from unstructured utterances. |
Normalized Contrastive Learning for Text-Video Retrieval (2022.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. |
| Approach: | They propose a normalized contrastive learning algorithm that normalizes the sum retrieval probabilities of each instance so that every text and video instance is fairly represented. |
| Outcome: | Empirical results show that NCL brings significant gains in text-video retrieval on different model architectures without any architecture engineering. |
KETOD: Knowledge-Enriched Task-Oriented Dialogue (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies treat task-oriented dialogue and chit-chat as separate domains . a new dataset is created to integrate both types of dialogue into a single system . |
| Approach: | They propose to integrate task-oriented dialogue and knowledge-grounded chit-chat into a single model by using a dataset. |
| Outcome: | The proposed models improve the performance of knowledge-enriched dialogues while maintaining a competitive task-oriented dialog performance. |
Continual Learning in Task-Oriented Dialogue Systems (2021.emnlp-main)
Copied to clipboard
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, Zhiguang Wang
| Challenge: | Existing continuous learning systems are not designed to add new domains and functionalities through time without incurring the high cost of retraining the whole system. |
| Approach: | They propose a first-ever continual learning benchmark for task-oriented dialogue systems . they propose 'architecture' method based on residual adapters to implement continual training . |
| Outcome: | The proposed architectural method performs better than multitask learning while being 20X faster in learning new domains. |
NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing conversational systems are agent-centric, which assumes the user utterances will closely follow the system ontology. |
| Approach: | They build a dataset that maps user preferences to an ontology and model user preferences as estimated distributions over the system ontologies. |
| Outcome: | The proposed system can be used to push existing research from agent-centric to user-centric. |
Multimodal Named Entity Recognition for Short Social Media Posts (N18-1)
Copied to clipboard
| Challenge: | Social media posts often contain inconsistent or incomplete syntax and lexical notations with limited textual contexts. |
| Approach: | They propose a task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data . they use a dataset called SnapCaptions to build upon the state-of-the-art NER models . |
| Outcome: | The proposed model outperforms existing models on noisy user-generated data . it uses a deep image network and generic modality attention module . |