Papers by Sumit Shekhar
CaRVE: Critiquing and Refining Visual Elaborations for Figurative Language Illustrations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-image frameworks for figurative illustration rely on proprietary models or human supervision to achieve adequate alignment. |
| Approach: | They propose a critique-driven framework that uses VLM feedback to refine visual elaborations for figurative image generation. |
| Outcome: | The proposed framework outperforms existing figurative image-to-text pipelines on human-supervised visual elaborations. |
Looking Beyond the Pixels: Evaluating Visual Metaphor Understanding in VLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Visual metaphors are a complex vision–language phenomenon that requires both perceptual and conceptual reasoning to understand. |
| Approach: | They introduce a visual metaphor dataset featuring 2177 synthetic and 350 human-annotated images and benchmark several SOTA VLMs on two tasks: Visual Metaphor Captioning (VMC) and Visual Metamorphosis VQA (VM-VQA). |
| Outcome: | The proposed model outperforms standard few-shot baselines on visual metaphors and VM-VQA tasks. |
STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Chart Question Answering (CQA) is a task of answering natural language questions about visualisations in the chart image. |
| Approach: | They propose a method for Chart Question Answering which improves the question/answering through sequential elements localization, question encoding and then, a structural transformer-based learning approach. |
| Outcome: | The proposed method outperforms state-of-the-art methods on various chart Q/A datasets while outperforming even human baseline. |
“Let’s not Quote out of Context”: Unified Vision-Language Pretraining for Context Assisted Image Captioning (2023.acl-industry)
Copied to clipboard
| Challenge: | Large enterprises have several teams to create their content for the purpose of marketing, campaigning, or even maintaining a brand presence. |
| Approach: | They propose a new unified Vision-Language (VL) model with a focus on context-assisted image captioning where the caption is generated based on both the image and its context. |
| Outcome: | The proposed model achieves state-of-the-art with an improvement of up to 8.34 CIDEr score on the benchmark news image captioning datasets. |
DynamicTOC: Persona-based Table of Contents for Consumption of Long Documents (2022.naacl-main)
Copied to clipboard
Himanshu Maheshwari, Nethraa Sivakumar, Shelly Jain, Tanvi Karandikar, Vinay Aggarwal, Navita Goyal, Sumit Shekhar
| Challenge: | Long documents are tedious to read through and can be authored by multiple entities . traditional document navigation is through a Table of Contents (ToC) but there is no way to highlight information relevant to different personas. |
| Approach: | They propose a dynamic table of content-based navigator that highlights sections of interest . DYNAMICTOC is augmented with short questions to assist users in understanding underlying content . |
| Outcome: | The proposed navigator highlights sections of interest in documents as per the aspects relevant to different personas. human and automatic evaluations suggest the efficacy of both end-to-end pipeline and different components. |
Unveiling the Invisible: Captioning Videos with Metaphors (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that Vision-Language models cannot understand visual metaphors in memes and adverts. |
| Approach: | They propose a task to describe visual metaphors in videos using a manually created dataset and a new metric called Average Concept Distance to automatically evaluate creativity. |
| Outcome: | The proposed system performs comparable to existing video language models on the proposed task and can be used for future research. |
Open-World Factually Consistent Question Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for question generation suffer from factual inconsistencies and incorrect entities and are not answerable from the input paragraph. |
| Approach: | They propose a data processing technique based on de-lexicalization for consistent question generation across domains and a model that is generic across question-generation models. |
| Outcome: | The proposed method produces entity-level factually consistent questions without significant impact on traditional metrics. |
A Neural CRF-based Hierarchical Approach for Linear Text Segmentation (2023.findings-eacl)
Copied to clipboard
Inderjeet Nair, Aparna Garimella, Balaji Vasan Srinivasan, Natwar Modani, Niyati Chhaya, Srikrishna Karanam, Sumit Shekhar
| Challenge: | Existing methods to segment unformatted text and transcripts explicitly train to predict segment boundaries, but they fail to provide a large annotated dataset. |
| Approach: | They propose a method to generate hierarchical segmentation structures based on Wikipedia annotations by using a neural conditional random field. |
| Outcome: | The proposed method outperforms or achieves competitive performance when compared to previous state-of-the-art algorithms. |
Imposter: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models (2025.coling-main)
Copied to clipboard
| Challenge: | ImPoster is a novel algorithm for generating a target image of a ‘source’ subject performing a 'driving' action. |
| Approach: | They propose an unsupervised approach that generates a target image of a ‘source’ subject performing a driving action from a single pair of inputs along with the text descriptions of the two images. |
| Outcome: | The proposed algorithm is completely unsupervised and does not require access to additional annotations like keypoints or pose. |