Papers by Dinesh Manocha
Do Vision-Language Models Understand Compound Nouns? (2024.naacl-short)
Copied to clipboard
| Challenge: | Open-vocabulary vision-language models (CLIP) are emerging as a promising new paradigm for text-to-image retrieval. |
| Approach: | They propose a benchmark to evaluate the effectiveness of open-vocabulary vision-language models (CLIP) for text-to-image retrieval using contrastive loss. |
| Outcome: | The proposed framework improves CN understanding of CLIP by 8.25% on Compun. |
DOC-RAG: ASR Language Model Personalization with Domain-Distributed Co-occurrence Retrieval Augmentation (2024.lrec-main)
Copied to clipboard
| Challenge: | Extensive experiments on three user-specific speech-to-text tasks show that DOC-RAG significantly outperforms strong baselines with an 8-15% improvement in terms of perplexity and a 4-7% reduction in terms in terms . of Word Error Rates. |
| Approach: | They propose a domain-distributed co-occurrence augmentation approach to improve automatic speech recognition of rare word patterns in unseen domains by using n-gram co-existence distributions. |
| Outcome: | Experiments on three user-specific speech-to-text tasks show that DOC-RAG outperforms baselines with an 8-15% improvement in terms of perplexity and a 4-7% reduction in terms in terms . of Word Error Rates. |
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents (2025.emnlp-main)
Copied to clipboard
Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha
| Challenge: | Flowcharts are a critical tool for visualizing decision-making processes, but their non-linear structure and complex visual-textual relationships make it difficult to interpret them using LLMs. |
| Approach: | They propose a task of Fine-grained Flowchart Attribution to trace components grounding a flowchart referring LLM response. |
| Outcome: | The proposed agent mitigates visual hallucinations in LLM answers over baselines by 10–14% on a FlowExplainBench dataset. |
ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations (2024.findings-acl)
Copied to clipboard
Sreyan Ghosh, Chandra Kiran Evuru, Sonal Kumar, Utkarsh Tyagi, S Sakshi, Sanjoy Chowdhury, Dinesh Manocha
| Challenge: | Neural image classifiers often rely on non-predictive features that are spuriously correlated with the class labels in training data. |
| Approach: | They propose a language-guided data augmented with images without spurious correlations that can be used to augment training datasets for robust learning. |
| Outcome: | The proposed model improves the worst-group classification accuracy of prior methods by 1% - 38%. |
EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding (2025.emnlp-main)
Copied to clipboard
Ashish Seth, Utkarsh Tyagi, Ramaneswaran Selvakumar, Nishit Anand, Sonal Kumar, Sreyan Ghosh, Ramani Duraiswami, Chirag Agarwal, Dinesh Manocha
| Challenge: | Multimodal Large Language Models excel at visual perception and reasoning in third-person and egocentric videos, but are prone to hallucinations, generating coherent yet inaccurate responses. |
| Approach: | They propose to use a benchmark to evaluate MLLM hallucinations in egocentric videos. |
| Outcome: | EGOILLUSION comprises 1,400 videos paired with 8,000 human-annotated open and closed-ended questions designed to trigger hallucinations in both visual and auditory cues in egocentric videos. |
DIAGRAMS : A Review Framework for Reasoning-Level Attribution in Diagram QA (2026.acl-demo)
Copied to clipboard
Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Manan Suri, Raviteja Bommireddy, Dinesh Manocha, Puneet Mathur, Vivek Gupta
| Challenge: | Diagram question answering (Diagram QA) requires reasoning-level attribution that links each question-answer pair to all visual regions needed to derive the answer. |
| Approach: | They propose a diagram question-answer review framework that decouples interface logic from dataset-specific JSON structures through an internal meta-schema and dataset adapters. |
| Outcome: | The proposed framework achieves 85.39% precision and 75.30% recall across six diagram QA datasets. |
AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Boyd-Graber, Tianyi Zhou, Dinesh Manocha
| Challenge: | Large vision-language models are prone to hallucinations, where contextual cues in an image can trigger the language module to produce overconfident and incorrect reasoning about abnormal or hypothetical objects. |
| Approach: | They propose to automate the generation of hallucination-related questions using images . they propose to use three image manipulation strategies to induce hallucinosity . |
| Outcome: | The proposed approach reduces human bias in crafting such examples and improves accuracy. |
DocInfer: Document-level Natural Language Inference using Optimal Evidence Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Documentlevel NLI is an important problem for many tasks including verification of factual correctness of documents. |
| Approach: | They propose a document-level natural language inference model that builds a hierarchical document graph enriched through inter-sentence relations and performs paragraph pruning using the novel SubGraph Pooling layer. |
| Outcome: | The proposed model performs on a legal judicial reasoning task with a dataset enriched with document graphs and a proposed evidence selection algorithm. |
CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | a low-resource dataset is limited in training data, so generating task-specific data is challenging. |
| Approach: | They propose a data augmentation technique that prompts off-the-shelf instruction-following Large Language Models to generate augmentations. |
| Outcome: | The proposed technique outperforms baselines on 11 datasets spanning 3 tasks and 3 low-resource settings. |
ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions (2024.acl-long)
Copied to clipboard
Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Evuru, Ramaneswaran S, S Sakshi, Dinesh Manocha
| Challenge: | ABEX is a novel and effective generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks. |
| Approach: | They propose a novel generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks based on a paradigm for generating diverse forms of an input document . |
| Outcome: | The proposed method outperforms all baselines qualitatively with improvements of 0.04% - 38.8%. |
TIMERS: Document-level Temporal Relation Extraction (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods for temporal relation extraction focus on extracting temporal relations between event pairs present in the same sentence or adjacent sentences, mostly ignoring document-level pairs. |
| Approach: | They propose a TIME, Rhetorical and Syntactic-aware model for document-level temporal relation classification in the English language that leverages rhetorical discourse features and temporal arguments from semantic role labels. |
| Outcome: | The proposed model outperforms previous methods on the TDDiscourse, TimeBank-Dense, and MATRES datasets due to its discourse-level modeling. |
DocScript: Document-level Script Event Prediction (2024.lrec-main)
Copied to clipboard
Puneet Mathur, Vlad I. Morariu, Aparna Garimella, Franck Dernoncourt, Jiuxiang Gu, Ramit Sawhney, Preslav Nakov, Dinesh Manocha, Rajiv Jain
| Challenge: | Existing script event prediction frameworks such as ChatGPT and FlanT5 lack the ability to learn long-range dependencies between events. |
| Approach: | They propose a novel script event prediction task which aims to predict the next event from a candidate list of narrative events in long-form documents. |
| Outcome: | The proposed architecture can learn sequential ordering between events at the document scale. |
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (2024.emnlp-main)
Copied to clipboard
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha
| Challenge: | We propose a novel large-scale audio-language model with advanced audio understanding and reasoning abilities. |
| Approach: | They propose a general-purpose large audio-language model with advanced audio understanding and reasoning abilities that integrates an LLM with multiple types of audio representations. |
| Outcome: | The proposed model outperforms existing models on audio understanding tasks by 1%-84%. |
Structured Uncertainty guided Clarification for LLM Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to clarifying tasks fail when user instructions are ambiguous or incomplete. |
| Approach: | They propose a principled formulation of structured uncertainty that operates directly over tool parameters and their domains. |
| Outcome: | The proposed framework improves when2call accuracy and training-time sample efficiency. |
PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from related Example Banks (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive few-shot learning capabilities through in-context learning. |
| Approach: | They propose a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages. |
| Outcome: | The proposed approach outperforms existing frameworks for retrieving examples on low-resource Indic languages. |
PAT: Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio Classification (2025.naacl-long)
Copied to clipboard
| Challenge: | Audio-Language Models (ALMs) have demonstrated remarkable performance in zero-shot audio classification. |
| Approach: | They propose a training-free method that enhances audio and language representations using mutual feedback. |
| Outcome: | The proposed method outperforms vanilla zero-shot evaluation with significant margins of 0.42%-27.0%. |
Saliency-Aware Interpolative Augmentation for Multimodal Financial Prediction (2024.lrec-main)
Copied to clipboard
Samyak Jain, Parth Chhabra, Atula Tejaswi Neerkaje, Puneet Mathur, Ramit Sawhney, Shivam Agarwal, Preslav Nakov, Sudheer Chava, Dinesh Manocha
| Challenge: | Recent advances in the Financial AI realm have expanded the scope of data and methods they use, such as textual and audio cues from financial earnings calls, but limitations exist. |
| Approach: | They propose a Saliency-guided Hierarchical Mixup augmentation technique for multimodal financial prediction tasks. |
| Outcome: | The proposed technique outperforms state-of-the-art methods by 3-7% on financial earnings and conference call datasets. |
DocTime: A Document-level Temporal Dependency Graph Parser (2022.naacl-main)
Copied to clipboard
Puneet Mathur, Vlad Morariu, Verena Kaynig-Fittkau, Jiuxiang Gu, Franck Dernoncourt, Quan Tran, Ani Nenkova, Dinesh Manocha, Rajiv Jain
| Challenge: | Document dependency graphs (TDGs) are used to understand the temporal relations between events mentioned in a document and to improve downstream tasks such as timeline creation and time-aware summarization. |
| Approach: | They propose a temporal dependency graph parser that takes input from a text document and produces a graph that incorporates longer range dependencies. |
| Outcome: | The proposed framework outperforms existing models on three datasets and improves tasks such as timeline creation, time-aware summarization, and temporal information extraction. |
Can LLM’s Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis (2024.naacl-short)
Copied to clipboard
| Challenge: | 83.3% of users find the synthesized instructions accurately capture the details of the environment and show characteristics similar to those of human-generated instructions. |
| Approach: | They propose an algorithm that uses in-context learning to condition an LLM to generate instructions using just a few references. |
| Outcome: | The proposed algorithm is platform-agnostic and 83.3% of users find it to be accurate and similar to human-generated instructions. |
DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding (2024.emnlp-main)
Copied to clipboard
Manan Suri, Puneet Mathur, Franck Dernoncourt, Rajiv Jain, Vlad Morariu, Ramit Sawhney, Preslav Nakov, Dinesh Manocha
| Challenge: | Document structure editing involves manipulating localized textual, visual, and layout components in document images based on user’s requests. |
| Approach: | They propose a framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs) by localizing edit regions of interest and disambiguating user edit requests into edit commands. |
| Outcome: | The proposed framework outperforms baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12%) tasks. |
CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on detecting explicit hate speech has focused on indirect or coded language. |
| Approach: | They propose a context synergized neural network that integrates user- and conversational-contexts for detecting implicit hate speech in online conversations. |
| Outcome: | The proposed framework outperforms baselines on 6 hate speech datasets and shows that it is highly efficient. |
APoLLo : Unified Adapter and Prompt Learning for Vision Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | APoLLo improves generalization capabilities of vision-language pretrained models . despite being largely successful in terms of generalization, these models are difficult to fine-tune for few-shot learning-based downstream tasks. |
| Approach: | They propose a multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. |
| Outcome: | The proposed approach improves generalization capabilities of vision-language pretrained models . it achieves a relative gain of 6.03% over MaPLe on 10 diverse datasets . |
IntCoOp: Interpretability-Aware Vision-Language Prompt Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt-tuning frameworks lack interpretability, limiting their ability to understand compositional nature of images. |
| Approach: | They propose a prompt-tuning method that integrates compositional attributes into manual prompts to enhance image-text alignment scores. |
| Outcome: | The proposed method improves CoOp performance by 7.35% across 10 diverse datasets. |
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for document QA for visually rich documents outperform unimodal and long-context LLMs by 12-20%. |
| Approach: | They propose a multimodal Retrieval Augmented Generation approach that integrates visual and textual retrieval with linguistic reasoning. |
| Outcome: | The proposed approach outperforms unimodal and long-context LLM benchmarks for document QA by 12-20%. |
PersonaLM: Language Model Personalization via Domain-distributed Span Aggregated K-Nearest N-gram Retrieval Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language modeling tools for automatic speech recognition (ASR) are difficult to personalize. |
| Approach: | They propose a domain-distributed Span-Aggregated K-nearest N-gram retrieval augmentation to improve language modeling for automatic speech recognition (ASR) personalization. |
| Outcome: | The proposed model outperforms baselines on Wikitext-103, UserLibri, and ASAP datasets with a 10-16% improvement in perplexity and a 5-8% reduction in word error rates. |
TAME-RD: Text Assisted Replication of Image Multi-Adjustments for Reverse Designing (2024.findings-acl)
Copied to clipboard
Pooja Guhan, Uttaran Bhattacharya, Somdeb Sarkhel, Vahid Azizi, Xiang Chen, Saayan Mitra, Aniket Bera, Dinesh Manocha
| Challenge: | a new model to reverse design images can be used to replicate image edits on other images based on human instructions in natural language . a study of a dataset of 100K source and edited images shows improvements in accuracy and concordance correlation coefficient . |
| Approach: | They propose a reverse-designing model that automatically learns from image editing operations and natural language instructions to learn fully specified edit operations. |
| Outcome: | The proposed model improves accuracy and concordance correlation scores on multiple datasets. |
MULTIVOX: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions (2025.emnlp-main)
Copied to clipboard
Ramaneswaran Selvakumar, Ashish Seth, Nishit Anand, Utkarsh Tyagi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha
| Challenge: | omni models lack spoken dialogues, which is essential for assessing conversational and auditory capabilities of voice assistants. |
| Approach: | They propose a benchmark to evaluate the ability of voice assistants to integrate paralinguistic speech features into their models. |
| Outcome: | The multivox voice assistant benchmark evaluates the ability of models to integrate spoken and visual cues including paralinguistic speech features for truly multimodal understanding. |
EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | EH-MAM is a self-supervised learning approach for speech representation learning . prior methods used random masking schemes to learn speech representations . |
| Approach: | They propose a self-supervised approach that automatically selects hard regions during SSL training and introduces them to the model for reconstruction. |
| Outcome: | The proposed approach outperforms state-of-the-art models across low-resource speech recognition and SUPERB benchmarks by 5%-10%. |
Imposter: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models (2025.coling-main)
Copied to clipboard
| Challenge: | ImPoster is a novel algorithm for generating a target image of a ‘source’ subject performing a 'driving' action. |
| Approach: | They propose an unsupervised approach that generates a target image of a ‘source’ subject performing a driving action from a single pair of inputs along with the text descriptions of the two images. |
| Outcome: | The proposed algorithm is completely unsupervised and does not require access to additional annotations like keypoints or pose. |
Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation (2025.findings-acl)
Copied to clipboard
Sreyan Ghosh, Mohammad Sadegh Rasooli, Michael Levit, Peidong Wang, Jian Xue, Dinesh Manocha, Jinyu Li
| Challenge: | Generative Error Correction (GEC) is a powerful post-processing method to boost the performance of Automatic Speech Recognition systems. |
| Approach: | They propose a method to augment GEC models with retrieved entities to improve accuracy in out-of-domain and out-od scenarios. |
| Outcome: | The proposed method outperforms baseline models on multiple datasets and settings. |
ProSE: Diffusion Priors for Speech Enhancement (2025.naacl-long)
Copied to clipboard
Sonal Kumar, Sreyan Ghosh, Utkarsh Tyagi, Anton Jeran Ratnarajah, Chandra Kiran Reddy Evuru, Ramani Duraiswami, Dinesh Manocha
| Challenge: | deterministic deep learning models have been used for speech enhancement, but generative models have shown promise. |
| Approach: | They propose a method to apply diffusion probabilistic models to speech enhancement using priors in a latent space. |
| Outcome: | The proposed method achieves state-of-the-art performance on synthetic and real-world datasets while consuming less computational costs. |
FIGMA: Towards FIne-Grained Music retrievAl (2026.acl-long)
Copied to clipboard
| Challenge: | Existing music retrieval models fail to retrieve fine-grained musical attributes when using coarse semantic queries. |
| Approach: | They propose a multi-view contrastive architecture that captures high-level semantic context and fine-grained musical attributes within a unified representation space. |
| Outcome: | The proposed method outperforms existing CLAP-based music retrieval models on multiple benchmarks. |
ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER (2023.acl-long)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a task of detecting linguistically complex named entities in low-context text. |
| Approach: | They propose a keyword-based augmentation approach to address the context-entity mismatch issue in complex name recognition (NER) they use selective masking to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entity. |
| Outcome: | The proposed approach outperforms baseline methods on monolingual, cross-lingual, and multilingual complex NER in various low-resource settings. |
ChartLens: Fine-grained Visual Attribution in Charts (2025.acl-long)
Copied to clipboard
| Challenge: | MLLMs suffer from hallucinations, where generated text fails to align with visual inputs. |
| Approach: | They propose a chart attribution algorithm that uses segmentation-based techniques to identify chart objects and employs set-of-marks prompting with MLLMs for fine-grained visual attribution. |
| Outcome: | The proposed algorithm improves fine-grained attributions by 26-66% . |
DocFin: Multimodal Financial Prediction and Bias Mitigation using Semi-structured Documents (2022.findings-emnlp)
Copied to clipboard
Puneet Mathur, Mihir Goyal, Ramit Sawhney, Ritik Mathur, Jochen Leidner, Franck Dernoncourt, Dinesh Manocha
| Challenge: | Existing research focuses on textual and audio modalities of financial disclosures but ignores the rich tabular data available in financial reports. |
| Approach: | They propose to combine tabular financial data with text transcripts and audio recordings to improve stock volatility and price movement prediction by 5-12% and reduce gender bias by over 30%. |
| Outcome: | The combined data improves stock volatility and price movement prediction by 5-12% and reduces gender bias caused due to audio-based neural networks by over 30%. |
Do Audio-Language Models Understand Linguistic Variations? (2025.naacl-short)
Copied to clipboard
Ramaneswaran Selvakumar, Sonal Kumar, Hemant Kumar Giri, Nishit Anand, Ashish Seth, Sreyan Ghosh, Dinesh Manocha
| Challenge: | Existing open-vocabulary audio language models struggle to generalize to linguistic variations in textual queries. |
| Approach: | They propose a novel technique to learn audio-language representations agnostic to linguistic variations by reformulating contrastive loss used in CLAP architectures. |
| Outcome: | The proposed approach improves the performance of the open-vocabulary audio language models by 0.8%-13% across benchmarks and enhances robustness to linguistic variation. |
Engagement Undermines Safety: How Stereotypes and Toxicity Shape Humor in Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly used for creative writing and engagement content, raising safety concerns about their outputs. |
| Approach: | They evaluate how funniness optimization in large language models couples with harmful content by jointly measuring humor, stereotypicality, and toxicity. |
| Outcome: | The proposed model couples humor, stereotypicality, and toxicity with harmful outputs . the results suggest a bias amplification loop between generators and evaluators . |
RELIC: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples (2025.findings-emnlp)
Copied to clipboard
Soumya Suvra Ghosal, Vaibhav Singh, Akash Ghosh, Soumyabrata Pal, Subhadip Baidya, Sriparna Saha, Dinesh Manocha
| Challenge: | a new reward model for low-resource Indic languages is proposed . a preference-based training approach is prohibitively expensive, authors say . |
| Approach: | a new in-context learning framework is proposed to train a retriever to select in-constext examples from low-resource Indic languages. |
| Outcome: | a new in-context learning framework for reward modeling in low-resource Indic languages is developed . the proposed framework outperforms existing examples on three preference datasets . |
PolyAudio: Advancing Multi-Audio Reasoning in Large Audio Language Models with Interleaved Multi-Audio Contexts (2026.findings-acl)
Copied to clipboard
Sonal Kumar, Sreyan Ghosh, Yueqian Lin, S Sakshi, Ashish Seth, Yiran Chen, Ramani Duraiswami, Dinesh Manocha
| Challenge: | Large Audio Language Models have shown impressive performance on single-clip tasks . however, their ability to reason over interleaved multi-audio contexts remains limited . |
| Approach: | They propose a LALM that targets multi-audio understanding via instruction tuning rather than massive-scale pre-training. |
| Outcome: | The proposed model outperforms baseline models on multi-audio tasks while maintaining robustness. |
DALE: Generative Data Augmentation for Low-Resource Legal NLP (2023.emnlp-main)
Copied to clipboard
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh Tyagi, Dinesh Manocha
| Challenge: | DALE addresses the challenges existing frameworks pose in generating effective data augmentations of legal documents. |
| Approach: | They propose a generative Data Augmentation framework for low-resource legal NLP that exploits domain-specific language characteristics of templated legal documents to mask collocated spans of text. |
| Outcome: | The proposed framework outperforms baseline frameworks on 13 datasets and 4 low-resource settings. |