Papers by Spandana Gella
StarFlow: Generating Structured Workflow Outputs From Sketch Images (2026.eacl-long)
Copied to clipboard
Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan A. Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian
| Challenge: | Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. |
| Approach: | They propose a framework for generating structured workflow outputs from sketches using vision-language models to automate the process. |
| Outcome: | The proposed framework outperforms large vision-language models in the task of generating structured workflow outputs from sketches and diagrams. |
A Dataset for Telling the Stories of Social Media Videos (D18-1)
Copied to clipboard
| Challenge: | Existing datasets focused on pre-selected human activities, whereas social media videos contain a great diversity of topics. |
| Approach: | They propose a large-scale dataset for video description as a new challenge for multi-sentence video description. |
| Outcome: | The proposed dataset contains 20k videos with 123k sentences, temporally aligned to the video. |
ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments (2022.emnlp-main)
Copied to clipboard
Arjun Akula, Spandana Gella, Aishwarya Padmakumar, Mahdi Namazifar, Mohit Bansal, Jesse Thomason, Dilek Hakkani-Tur
| Challenge: | Embodied Vision and Language Task Completion requires an embodied agent to interpret natural language instructions and egocentric visual observations to navigate through and interact with environments. |
| Approach: | They propose to use ALFRED to test whether models can adapt to tasks not seen during training that require the same types of language understanding as ALFred. |
| Outcome: | The proposed model can adapt to tasks that require the same types of language understanding as ALFRED-L. |
An Evaluation of Image-Based Verb Prediction Models against Human Eye-Tracking Data (N18-2)
Copied to clipboard
| Challenge: | Recent research in language and vision has developed models for predicting and disambiguating verbs from images. |
| Approach: | They propose a verb prediction model and visual sense disambiguation model for verbs . they ask whether the image regions a model identifies as salient correlate with human intuitions about visual verbs. |
| Outcome: | The proposed model can predict verbs from images, but it is unclear to what extent it captures human intuitions about visual verbs. |
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2023.findings-emnlp)
Copied to clipboard
Prakhar Gupta, Yang Liu, Di Jin, Behnam Hedayatnia, Spandana Gella, Sijia Liu, Patrick Lange, Julia Hirschberg, Dilek Hakkani-Tur
| Challenge: | Dialogue models are able to generate fluent and interesting responses, but they can be difficult to control and may produce non-engaging, unsafe results. |
| Approach: | They propose a framework for controlling dialogue model behavior using natural language rules, or guidelines, which provide information about the context they are applicable to and what should be included in the response. |
| Outcome: | The proposed framework is effective in three open-domain dialogue response generation tasks and is consistent with the developer's expectations and intent. |
Multimodal Abstractive Summarization for How2 Videos (P19-1)
Copied to clipboard
| Challenge: | Abstractive summarization is a task of producing a shorter version of the content in the document while preserving its information. |
| Approach: | They propose a new evaluation metric that measures semantic adequacy rather than fluency of abstractive summarization tasks. |
| Outcome: | The proposed model integrates information from different sources into a coherent output. |
Cross-lingual Visual Verb Sense Disambiguation (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown that visual context improves cross-lingual sense disambiguation for nouns. |
| Approach: | They extend their work to the task of cross-lingual verb sense disambiguation by using a dataset annotated with English, German, and Spanish verbs. |
| Outcome: | The proposed model improves the results of a text-only machine translation system when used for a multimodal translation task. |
Analyzing the Limits of Self-Supervision in Handling Bias in Language (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that natural language models can perform tasks with little to no in-context supervision . a number of tasks are performed using self-supervised pre-training . |
| Approach: | They define and comprehensively evaluate how well natural language taskprompting captures the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. |
| Outcome: | The proposed model performs to wide varying degrees across bias dimensions . the model is largely challenged when prompted to perform these tasks . |
FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on single-hop, single-modality, or short texts, limiting real-world applications . despite advances in visual question answering, this multihop setting remains underexplored due to a lack of quality datasets. |
| Approach: | They propose a framework for creating a high-quality dataset for multimodal multihop question answering . they use a 5-stage pipeline to acquire relevant multimodal documents from Wikipedia . |
| Outcome: | The proposed framework outperforms existing methods on multimodal multihop question answering datasets. |
ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval (2025.emnlp-industry)
Copied to clipboard
Ahmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar
| Challenge: | Existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval. |
| Approach: | They propose a document retrieval model that bridges the gap between multimodal representation learning and document retrievals by providing external knowledge as context. |
| Outcome: | The proposed model achieves 3.61% improvement over existing retrieval models on the ViDoRe V2 benchmark, showing stronger generalization to out-of-domain benchmarks. |
Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Embodied task completion requires an agent to predict environment actions to complete tasks based on natural language instructions and egocentric visual observations. |
| Approach: | They propose a method to generate human-human dialogues and use them as training data for plan prediction. |
| Outcome: | The proposed model outperforms language-only models but falls short of oracle plans. |
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that pre-trained language models perform poorly on challenging datasets where spurious correlations do not hold. |
| Approach: | They propose to use multi-task learning to improve generalization from minority examples . they propose to combine MTL with auxiliary tasks to improve performance . |
| Outcome: | The proposed model generalizes from minority examples without hurting in-distribution performance. |
Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior implementations of NMN use pre-defined and fixed textual inputs in their module instantiation. |
| Approach: | They propose to parameterize the module arguments to reduce the number of modules in NMN by up to 75% without any loss in performance. |
| Outcome: | The proposed model outperforms the state-of-the-art model on CLEVR-Ref+ dataset with +8.1% improvement in accuracy and +4.3% on full test set. |
Using In-Context Learning to Improve Dialogue Safety (2023.findings-emnlp)
Copied to clipboard
Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, Dilek Hakkani-Tur
| Challenge: | Recent work has highlighted safety issues with large neural-based conversational models. |
| Approach: | They propose a retrieval-based approach for reducing bias and toxicity in chatbot responses . they retrieve demonstrations of safe responses to similar dialogue contexts to generate a response . |
| Outcome: | The proposed method reduces bias and toxicity in three chatbot models . it can be used in compliment to existing dialogue safety approaches, such as RLHF. |
Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020.acl-main)
Copied to clipboard
| Challenge: | Visual referring expression recognition is a task that requires natural language understanding in the context of an image. |
| Approach: | They propose to use contrastive learning and multi-task learning to increase the robustness of ViLBERT, the current state-of-the-art model for this task. |
| Outcome: | The proposed methods are 12% to 23% lower in performance than the established progress for this task. |
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation (2025.emnlp-main)
Copied to clipboard
Rabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Spandana Gella, Sai Rajeswar
| Challenge: | Existing benchmarks focus on specific aspects of web tasks but lack comprehensive coverage. |
| Approach: | They propose a multilingual benchmark that evaluates three core web tasks: (1) website visual question answering, (2) code editing involving HTML/CSS/JavaScript, and (3) mockup-to-code generation. |
| Outcome: | The proposed model performs well on basic information extraction, but struggles with reasoning and grounding, editing code to preserve functionality, and generating design-to-code that maintains hierarchy and supports multilingual content. |
Multimodal Large Language Models for Human-AI Interaction: Foundations, Agents, and Inclusive Applications (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Approach: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Outcome: | This tutorial covers foundations, agentic capabilities, and inclusive applications of multimodal large language models. |