Papers by Sriparna Saha
Copied to clipboard
| Challenge: | Existing studies have shown that sarcasm is reflected by the intended meaning of the speaker's utterance. |
| Approach: | They propose to extend the MUStARD dataset to enclose dialogue acts for each dialogue . they propose a dialogue act-aided multi-modal transformer network for sarcasm identification model . |
| Outcome: | The proposed model improves performance in dialogue act-aided sarcasm identification compared to sardasmatic identification alone. |
Copied to clipboard
| Challenge: | DRISHTIKON is a first-of-its-kind multimodal and multilingual benchmark centered exclusively on Indian culture. |
| Approach: | They evaluate a wide range of vision-language models across zero-shot and chain-of-thought settings and use them to evaluate cultural understanding of generative AI systems. |
| Outcome: | The DRISHTIKON dataset covers 15 languages, all states and union territories, and incorporating over 64,000 aligned text-image pairs. |
Copied to clipboard
| Challenge: | Recent laws like “right to explanations” have spurred research in developing interpretable models . a recent study has shown that multimodal explanations improve performance in generating textual justifications . |
| Approach: | They propose to use visual and textual modalities to explain why a given meme is cyberbullying . they use a Contrastive Language-Image Pretraining approach to generate textual justifications . |
| Outcome: | The proposed model improves performance in visual and textual explanations and identifies the visual evidence supporting a decision. |
Copied to clipboard
| Challenge: | This survey provides the first in-depth review of multilingual reasoning in Language Models. |
| Approach: | This survey provides the first in-depth review of multilingual reasoning in LMs. |
| Outcome: | The present study provides the first in-depth review of multilingual reasoning in LMs. |
Copied to clipboard
| Challenge: | Existing efforts to generate simple charts have focused on generating simple infographics from text-heavy documents. |
| Approach: | They propose to generate statistical infographics composed of multiple sub-charts that are contextually accurate, insightful, and visually aligned. |
| Outcome: | The proposed framework outperforms both open-source and closed LLMs in text-to-statistical infographic generation. |
Copied to clipboard
| Challenge: | Using a dataset of 931 videos with 4021 code-mixed Hindi-English utterances, we find that video content with multiple modalities is more accurate and more accurate than textual content. |
| Approach: | They propose to use a dataset to analyze toxic content in video content in non-English languages by leveraging language models. |
| Outcome: | The proposed framework achieves an Accuracy and Weighted F1 score of 94.29% and 94.35% for the first time in its class. |
Copied to clipboard
| Challenge: | Strong retrieval models are increasingly important in knowledge-intensive domains. |
| Approach: | They propose a benchmark to evaluate multimodal retrieval models in medical settings . they examine 1.2 million text documents and 164K multimodal queries . |
| Outcome: | The proposed model spans 5 domains,16 medical fields, and 4 distinct tasks with over 1.2 Million text documents and 164K multimodal queries. |
Copied to clipboard
| Challenge: | In recent past, social media has emerged as an active platform in the context of healthcare and medicine. |
| Approach: | They propose to use a novel adversarial learning approach to capture medical sentiments expressed in a medical blog to analyze the user's opinions on health-related issues. |
| Outcome: | The proposed framework can capture the user's opinions on health-related issues at a medical blog level. |
Copied to clipboard
| Challenge: | Existing work on multimodal summarization does not consider the topic of the content. |
| Approach: | They propose a topic-aware MS system which performs two tasks simultaneously: differentiating images into "on-topic" and "off-topic". |
| Outcome: | The proposed system outperforms the state-of-the-art approach by 1.7 % in ROUGE-L metric. |
Copied to clipboard
| Challenge: | Existing task-oriented conversational agents assume that end-users will always have a pre-determined and servable task goal, which results in dialogue failure in hostile scenarios, such as goal unavailability. |
| Approach: | They propose to build an end-to-end multi-modal persuasive dialogue system incorporating a personalized persuasive module aided goal controller and goal persuader. |
| Outcome: | The proposed system achieves user tasks even in goal unavailability scenarios by persuading them towards a similar and servable goal. |
Copied to clipboard
| Challenge: | Existing models based on textual data do not capture context beyond the sentence. |
| Approach: | They propose a framework that enables the model to learn multi-omnics biological information about entities (proteins) with the help of additional multi-modal cues like molecular structure. |
| Outcome: | The proposed model is generalized and optimized for protein-protein interaction task and benefited from additional domain-specific cues. |
Copied to clipboard
| Challenge: | Speech Act Classification determining the communicative intent of an utterance has been investigated widely over the years as a standalone task. |
| Approach: | They propose a multi-modal, emotion-TA dataset called EmoTA from open-source Twitter dataset and a Dyadic Attention Mechanism framework that integrates intra-modal and inter-modal attention to fuse multiple modalities. |
| Outcome: | The proposed framework boosts the performance of the primary task, i.e., TA classification (TAC), by benefitting from the two secondary tasks, namely, Sentiment and Emotion Analysis compared to its uni-modal and single task TAC variants. |
Copied to clipboard
| Challenge: | a recent WHO report highlights a drastic doctor-to-patient ratio . telehealth is one of the most impactful sectors where AI advances can bring a significant revolution . |
| Approach: | They propose an image-guided encoder-decoder model that uses contextual attention to create detailed visual-guides for multimodal documents. |
| Outcome: | The proposed model outperforms state-of-the-art models on multimodal question and dialogue summarization tasks. |
Copied to clipboard
| Challenge: | Poems are a distinct form of literature, with meanings that transcend beyond the literal words. |
| Approach: | They propose a framework to generate images that visually represent the meanings of poems using prompt tuning and a PoeKey algorithm to extract emotions, visual elements, and themes from poems. |
| Outcome: | The proposed framework generates images that visually represent the meanings of poems and their images. |
Copied to clipboard
| Challenge: | Language models excel in syntactic and semantic analysis, while small language models struggle in region-specific contexts. |
| Approach: | They evaluate SANSKRITI on leading Large Language Models, Indic Language Model, and Small Language Model (SLM) it covers 16 key attributes of Indian culture including rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities. |
| Outcome: | The SANSKRITI dataset covers 16 attributes of Indian culture . it reveals that many models struggle in region-specific contexts . |
Copied to clipboard
| Challenge: | DeFactoX integrates Direct Preference Optimization (DPO) with Curriculum learning to align machine-generated explanations with human reasoning. |
| Approach: | They propose a framework that integrates Direct Preference Optimization (DPO) with Curriculum learning to align machine-generated explanations with human reasoning. |
| Outcome: | The proposed framework combines Direct Preference Optimization (DPO) with Curriculum learning to align machine-generated explanations with human reasoning. |
Copied to clipboard
| Challenge: | Increasing use of virtual tutors has allowed for more efficient, personalized, and interactive AI-based learning experiences. |
| Approach: | They propose a task of Multi-modal Perspective based Dialogue Summarization (MM-PerSumm) that summarizes educational dialogues from three unique perspectives: the Student, the Tutor, and a Generic viewpoint. |
| Outcome: | The proposed model can summarize educational dialogues from three perspectives, while student-oriented summaries should distill learning points, track progress, and suggest scope for improvement. |
Copied to clipboard
| Challenge: | In automatic essay grading, essay traits are important for scoring the essay holistically . a single-task learning system gives the best results for scoring essays holistically and scoring essay traits. |
| Approach: | They propose a way to score essays using a multi-task learning approach . they compare the MTL-based BiLSTM system to a single-task Learning approach based on LSTMs and BiLStms . |
| Outcome: | The proposed system gives better results for scoring essay holistically and scoring essay traits. |
Copied to clipboard
| Challenge: | Considerable work on Dialogue Act Classification (DAC) has been done on textual inputs. |
| Approach: | They propose to use a multimodal Emotion aware Dialogue Act dataset to explore the role of multi-modality and emotion recognition in DAC. |
| Outcome: | The proposed dataset shows that multi-modality and emotion recognition improves DAC performance compared to uni-modal and single task DAC variants. |
Copied to clipboard
| Challenge: | Social media platforms are important resources for investigating mental health of users. |
| Approach: | They propose a new dataset for Causal Analysis of Mental health in Social media posts (CAMS) they crawl and annotate 3155 Reddit data and reannotate a publicly available SDCNL dataset . |
| Outcome: | The proposed model outperforms existing models on 3155 Reddit posts and 1896 instances of the dataset. |
Copied to clipboard
| Challenge: | Recent advances in large language models have significantly improved conversational recommender systems performance. |
| Approach: | They propose a framework that reframes conversational recommendation as a structured decision-making process optimized for multi-dimensional recommendation quality. |
| Outcome: | The proposed framework improves on ReDial, INSPIRED, and MUSE while maintaining competitive response quality. |
Copied to clipboard
| Challenge: | Existing methods of identifying ADRs are reliable but time-consuming and offer a limited amount of ADR relevant information. |
| Approach: | They propose a neural network-inspired multi-task learning framework that can simultaneously extract ADRs from various sources. |
| Outcome: | The proposed framework achieves state-of-the-art performance on three publicly available real-world benchmark pharmacovigilance datasets, a Twitter dataset from PSB 2016 Social Me- dia Shared Task, CADEC corpus and Medline ADR corpus. |
Copied to clipboard
| Challenge: | idioms provide a fascinating gateway to creativity, cultural values, historical context, and diverse perspectives inherent to diverse linguistic traditions. |
| Approach: | They propose a multimodal idiom corpus enriched with seven idiomatic tones . they propose idiomic hybridization framework that embeds multiple idiomatic expert opinions . |
| Outcome: | The proposed framework achieves 5–6% performance gains across advanced vision language models. |
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have produced strong performance in mathematical reasoning and code generation, but medical reasoning remains challenging because it requires domain knowledge. |
| Approach: | They propose a multilingual medical reasoning dataset with open-ended reasoning queries with a single verifiable answer that spans thirteen languages. |
| Outcome: | The proposed framework outperforms baselines and scales effectively across thirteen languages. |
Copied to clipboard
| Challenge: | Existing models of disease diagnosis using AI do not use knowledge infusion. |
| Approach: | They propose a transformer-based, knowledge-infused multi-modal medical dialogue generation framework . they propose 'discourse-aware' image identifier that recognizes signs and their severity . |
| Outcome: | The proposed model outperforms state-of-the-art models by 7.84% in the english language. |
Copied to clipboard
| Challenge: | Existing studies on complaint identification are limited to text. |
| Approach: | They propose a meta-learning-based multi-modal multi-task framework for identifying complaints using emotion recognition and sentiment analysis as auxiliary tasks. |
| Outcome: | The proposed framework outperforms baselines and state-of-the-art approaches in centralized and federated meta-learning settings. |
Copied to clipboard
| Challenge: | Language Models (LMs) are primarily evaluated on globally popular sports, often overlooking regional and indigenous sporting traditions. |
| Approach: | They propose to use multiple-choice questions (MCQs) to assess LMs' understanding of traditional sports across 60 countries and 6 continents. |
| Outcome: | The new benchmark will be publicly available, fostering research in culturally aware AI systems. |
Copied to clipboard
| Challenge: | ADEs are a serious public health concern and cost healthcare systems billions of dollars . despite advancements in healthcare, ADE detection remains a significant challenge . |
| Approach: | They propose a multimodal adverse drug event detection dataset that merges ADE-related textual information with visual aids to enhance patient safety. |
| Outcome: | The proposed dataset integrates ADE-related textual information with visual aids to improve patient safety and healthcare accessibility. |
Copied to clipboard
| Challenge: | Temporal reasoning is a vital component of human communication and understanding, yet remains an underexplored area within the context of Large Language Models (LLMs). |
| Approach: | They propose to use 3 prompting strategies to evaluate 8 different LLMs across 6 datasets and 2 Code Generation LMs to perform the analysis. |
| Outcome: | The proposed models perform better on NLP tasks than the standard models on the same dataset. |
Copied to clipboard
| Challenge: | Complaining is an expression of negative emotions communicated due to a discrepancy between reality and expectations. |
| Approach: | They propose to use an explainable complaint dataset to generate a commonsense-aware generative framework that can predict the complaint cause, severity level, emotion, and polarity of the text. |
| Outcome: | The proposed model can predict the complaint cause, severity level, emotion, and polarity of the text in addition to detecting whether it is a complaint or not. |
Copied to clipboard
| Challenge: | Existing approaches to search result clustering use multiple views and visual and textual views. |
| Approach: | They propose to use multi-view learning to learn search results from web-snippets . they propose to obtain a single consensus partitioning after consulting two views . |
| Outcome: | The proposed approach on a benchmark dataset shows that visual and text-based views can achieve better clustering. |
Copied to clipboard
| Challenge: | Existing studies on content moderation of toxic memes focus on text-based content . current research neglects the widespread influence of multimodal content like memes . |
| Approach: | They propose a framework leveraging Large Language Models and Visual Language Model (VLMs) for meme intervention. |
| Outcome: | The proposed framework enables users to generate relevant and effective responses to toxic memes. |
Copied to clipboard
| Challenge: | Existing reviews focus on a few high-resource languages or embed Indian languages within broad multilingual settings, limiting coverage of low-resourced and culturally diverse varieties. |
| Approach: | They present a unified survey of Indian NLP resources, covering 200+ datasets, 50+ benchmarks, and 100+ models, tools, and systems across text, speech, multimodal, and culturally grounded tasks. |
| Outcome: | The proposed survey covers 200+ datasets, 50+ benchmarks, and 100+ models, tools, and systems across text, speech, multimodal, and culturally grounded tasks. |
Copied to clipboard
| Challenge: | Using vision language models, we examine demographic biases in VLMs across gender, race, age, and skin tone. |
| Approach: | They propose a benchmark for uncovering demographic biases in Vision Language Models . they propose 'Gras Bias Score' to quantify bias in VLMs based on gender, race, age and skin tone . |
| Outcome: | The proposed model achieves 98, far from the unbiased ideal of 0. |
Copied to clipboard
| Challenge: | COSMMIC is a multimodal, multilingual dataset featuring nine major Indian languages. |
| Approach: | They propose a multimodal, multilingual multimodal multimodal dataset that integrates text, images and user feedback to enhance summarization. |
| Outcome: | The proposed dataset is based on 4,959 article-image pairs and 24,484 reader comments with ground-truth summaries available in all included languages. |
Copied to clipboard
| Challenge: | a large dataset of news articles spanning 20 languages is lacking for keyword extraction. |
| Approach: | They propose a large-scale multi-lingual keyword extraction dataset for 11 of 20 languages . authors believe it will help advance the field of automatic keyword extraction . |
| Outcome: | The proposed dataset is the first for 11 of 20 languages and is based on 540K+ news articles from the BBC News network. |
Copied to clipboard
| Challenge: | Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect. Existing transferable adversarials are less effective in the medical domain. |
| Approach: | They propose a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. |
| Outcome: | The proposed method induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. |
Copied to clipboard
| Challenge: | a novel method to enhance imagery in poetic language is proposed . weighted prompt manipulation is a new approach to enhance poetry images . current diffusion models struggle to interpret metaphorical language, symbolism, and nuanced themes. |
| Approach: | They propose a weighted prompt manipulation technique that modifies attention weights and text embeddings within diffusion models to enhance or suppress specific words' influence in the final generated image. |
| Outcome: | The proposed technique enhances or suppresses the influence of specific words in the final generated image, leading to semantically richer and more contextually accurate visualizations. |
Copied to clipboard
| Challenge: | a large dataset of document-image pairs and annotated multi-modal summarization data is needed for multi-lingual modeling . encoder-decoder models represent information comprising multiple modalities. |
| Approach: | They propose to use a multi-lingual summarization dataset to analyze multi-modal summarizing using multi-linguistic annotated data. |
| Outcome: | The proposed dataset is the largest multi-lingual multi-modal summarization dataset for 13 languages and consists of cross-lingual summarizing data for 2 languages. |
Copied to clipboard
| Challenge: | Existing textual methods for protein-protein interaction identification have been used to perform most of the recent PPI tasks in BioNLP domain. |
| Approach: | They propose to incorporate multimodal cues into existing textual data to improve the automatic identification of PPI. |
| Outcome: | The proposed multi-modal datasets outperform baseline methods and unimodal approaches in predicting protein interactions. |
Copied to clipboard
| Challenge: | Existing benchmarks for numerical reasoning in multilingual Indic languages are inadequate . e.g., FinVQA is a framework for evaluating financial numerical reasoning . |
| Approach: | They propose a framework that combines supervised fine-tuning with constraint-aware decoding to promote faithful numerical reasoning. |
| Outcome: | The proposed framework spans English, Hindi, Bengali, Marathi, Gujarati, and Tamil . it combines supervised fine-tuning with constraint-aware decoding to promote faithful numerical reasoning . |
Copied to clipboard
| Challenge: | a study conducted by the pew Internet & American Life Project 1 shows that almost 80 percent of Internet users have explored health-related topic online. |
| Approach: | They propose to crawl medical forums with opinions about medical condition self narrated by users. |
| Outcome: | The proposed system is based on opinions about medical condition self-narrated by users on medical forums. |
Copied to clipboard
| Challenge: | a significant gap exists in understanding code-mixed languages and the need for explainability in this context. |
| Approach: | They propose to annotate posts with four labels to identify bullies in code-mixed languages . they propose to use a generative framework to reimagine the multitask problem as a text-to-text generation task. |
| Outcome: | The proposed model outperforms baseline models and state-of-the-art models on the BullyExplain dataset. |
Copied to clipboard
| Challenge: | a new reward model for low-resource Indic languages is proposed . a preference-based training approach is prohibitively expensive, authors say . |
| Approach: | a new in-context learning framework is proposed to train a retriever to select in-constext examples from low-resource Indic languages. |
| Outcome: | a new in-context learning framework for reward modeling in low-resource Indic languages is developed . the proposed framework outperforms existing examples on three preference datasets . |
Copied to clipboard
| Challenge: | Collaboration between doctors and AI scientists is leading to personalized models to stream-line healthcare tasks and improve productivity. |
| Approach: | They propose to use alignment techniques to combine a doctor-patient dialogue with a visual component of the BART model. |
| Outcome: | The proposed model in-tegrates visual components with the BART ar-chitecture. |
Copied to clipboard
| Challenge: | Mental health disorders are one of the primary causes of disability worldwide . lack of qualified and competent mental health professionals is a major problem . we propose a virtual assistant that can act as the first point of contact and comfort for mental health patients. |
| Approach: | They propose a virtual assistant that can act as the first point of contact and comfort for mental health patients. |
| Outcome: | The proposed system outperforms baselines in the evaluation of 7k dyadic conversations from a peer-to-peer support platform. |
Copied to clipboard
| Challenge: | a growing need to understand and alleviate FMs' propensity to produce hallucinated outputs, especially in high-stakes applications. |
| Approach: | They propose a framework for detecting and mitigating hallucination in FMs . they synthesize recent advancements in detection and mitigation techniques . |
| Outcome: | The proposed framework provides valuable insights for researchers, developers, and practitioners. |
Copied to clipboard
| Challenge: | Empirically, we show that the optimisation of multi-modal DAC, SA and ER tasks produces better results compared to its different counterparts. |
| Approach: | They propose a dual attention mechanism that integrates sentiment tags into a multi-modal conversational framework that integrate modal attentions and multiple loss optimization. |
| Outcome: | The proposed framework integrates sentiment tags for each utterance and learns generalized features across multiple tasks. |