Papers by Pranava Madhyastha
Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering (2023.findings-eacl)
Copied to clipboard
| Challenge: | Current explanation generation models are trained to select the best answers from Multiple-Choice questions or to classify single-word answers to a predetermined vocabulary. |
| Approach: | They propose a multitask learning approach towards a Unified Model for Answer and Explanation generation (UMAE) UMAE models surpass the prior state-of-the-art answer accuracy on A-OKVQA by 10 15%, show competitive results on OK-VQA and VCR, and demonstrate promising out-of domain performance on VQA-X. |
| Outcome: | The proposed model outperforms the state-of-the-art model on A-OKVQA and VCR and shows promising out-of domain performance on VQA-X. |
VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions (P19-1)
Copied to clipboard
| Challenge: | Existing methods for evaluating image description generation systems are subjective and expensive to scale. |
| Approach: | They propose a new image-aware metric for evaluating image description generation systems . it estimates the faithfulness of a generated caption with respect to the content of the actual image . |
| Outcome: | The proposed metric achieves high correlation with human judgments on two well-known datasets and is competitive with metrics that depend on and rely exclusively on human references. |
Chart Question Answering from Real-World Analytical Narratives (2025.acl-srw)
Copied to clipboard
| Challenge: | a dataset for chart question answering is constructed from visualization notebooks . data visualizations are an essential modality for communicating complex information about data. |
| Approach: | They propose a dataset for chart question answering constructed from visualization notebooks . they use real-world, multi-view charts paired with natural language questions . |
| Outcome: | The proposed dataset is constructed from student-authored visualization notebooks . it features real-world, multi-view charts paired with natural language questions . initial evaluations highlight significant performance gaps . |
Belief Revision Based Caption Re-ranker with Visual Semantic Information (2022.coling-1)
Copied to clipboard
| Challenge: | Xu et al., 2015; You e t al, 2016) aimed at generating a natural language description for a given image. |
| Approach: | They propose a visual re-ranking approach that leverages visual-semantic measures to identify the ideal caption that maximally captures the visual information in the image. |
| Outcome: | The proposed approach improves the performance of image caption generation systems without training or fine-tuning. |
Simultaneous Machine Translation with Visual Context (2020.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous machine translation (SiMT) aims to reproduce human interpretation, where an interpreter translates spoken utterances as they are produced. |
| Approach: | They propose to add visual context to siMT to compensate for the missing source context . they show visual-grounded models are much better than commonly used global features . |
| Outcome: | The proposed models reach up to 3 BLEU points improvement under low latency scenarios. |
Deep Copycat Networks for Text-to-Text Generation (D19-1)
Copied to clipboard
| Challenge: | Text-to-text generation tasks require copying words from the input to the output. |
| Approach: | They propose a transformer-based pointer network for text-to-text generation which generates more abstractive summaries and a further extension of this architecture for automatic post-editing. |
| Outcome: | The proposed model outperforms existing models in text-to-text generation tasks and improves translation accuracy. |
Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on multimodality in simultaneous machine translation have highlighted the challenges for the agent to maintain good translation quality while learning an optimal translation path. |
| Approach: | They propose a multimodal approach to simultaneous machine translation using reinforcement learning with strategies to integrate visual and textual information in both the agent and the environment. |
| Outcome: | The proposed multimodal approach improves translation quality while keeping latency low while providing visual cues. |
Probing the Need for Visual Context in Multimodal Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Current work on multimodal machine translation (MMT) suggests that the visual modality is either unnecessary or only marginally beneficial. |
| Approach: | They propose to use the visual modality to combine visual and textual information to generate better translations by partially depriving models from source-side textual context. |
| Outcome: | The proposed model can combine visual and textual information to generate better translations under limited textual context. |
Numerical reasoning in machine reading comprehension tasks: are we there yet? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning based machine reading comprehension models have achieved near-human performance on a variety of benchmarks, but are they capable of learning to reason? |
| Approach: | They propose to use a DROP benchmark to measure machine reading comprehension and investigate models that have achieved near-human performance over standard metrics. |
| Outcome: | The DROP benchmark has inspired the design of specialized BERT and embedding the results into a specialized model. |
Discrete Reasoning Templates for Natural Language Understanding (2021.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to reasoning over multiple parts of a passage provide little evidence of their reasoning process, especially with regards to why specific operands are chosen for a reasoning task. |
| Approach: | They propose a method that decomposes complex questions into subquestions that can take advantage of single-span extraction models and derives the final answer according to instructions in a predefined reasoning template. |
| Outcome: | The proposed approach is interpretable and requires little supervision while competing with the state-of-the-art models. |
Theoretical Conditions and Empirical Failure of Bracket Counting on Long Sequences with Linear Recurrent Networks (2023.eacl-srw)
Copied to clipboard
| Challenge: | Existing studies have shown that linear RNNs with unbounded activation functions are difficult to train effectively and do not learn exact counting behaviour. |
| Approach: | They propose to identify the necessary conditions for a linear single-cell RNN to have the ability to count and to investigate how these conditions relate to the empirical behaviour of trained linear RNN models. |
| Outcome: | The proposed model is a linear single-cell RNN with an unbounded activation function and a Dyck-1-like balanced bracket language. |
BERTGen: Multi-task Generation through BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU). |
| Approach: | They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks. |
| Outcome: | The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts. |
Towards preserving word order importance through Forced Invalidation (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models are insensitive to word order . performance on NLU tasks remains unchanged even after permuting the word . |
| Approach: | They propose a simple approach called Forced Invalidation to force the model to identify permuted sequences as invalid samples. |
| Outcome: | The proposed approach significantly improves the sensitivity of the models to word order on English NLU and QA tasks over BERT-based and attention-based models over word embeddings. |
IYKYK: Using language models to decode extremist cryptolects (2026.eacl-long)
Copied to clipboard
Christine de Kock, Arij Riabi, Zeerak Talat, Michael Sejr Schlichtkrull, Pranava Madhyastha, Eduard Hovy
| Challenge: | Extremist groups develop complex in-group language to exclude or mislead outsiders . general purpose LLMs cannot consistently detect or decode extremist language . |
| Approach: | They evaluate the ability of current language technologies to detect and interpret the cryptolects of two online extremist platforms. |
| Outcome: | The proposed models can detect and interpret extremist language better than current models. |
Learning and Enforcing Context-Sensitive Control for LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been able to achieve syntactic correctness but ensuring semantic validity requires additional mechanisms. |
| Approach: | They propose a framework that automatically learns context-sensitive constraints from LLM interactions through syntactic exploration and constraint exploitation. |
| Outcome: | The proposed framework outperforms larger models and state-of-the-art models in learning and generation of large LLMs. |
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020.coling-main)
Copied to clipboard
| Challenge: | a few popular metrics are still used to evaluate language generation systems despite their known limitations. |
| Approach: | They propose to use automatic metrics to evaluate language generation systems . they show that they prefer system outputs to human-authored texts . |
| Outcome: | The proposed metrics are insensitive to correct translations of rare words and can yield high scores when given a single sentence as system output for the entire test set. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Distilling Translations with Visual Awareness (P19-1)
Copied to clipboard
| Challenge: | Existing work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. |
| Approach: | They propose a translate-and-refine approach to multimodal machine translation where images are only used by a second stage decoder to generate a good first draft translation and to improve over this draft. |
| Outcome: | The proposed approach generates a good translation and improves over the draft by making better use of the target language textual context and making use of visual context. |
Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study has shown that human-like working memory constraints can be integrated into the Transformer architecture . our model incorporates fixed-width windows and temporal decay based attention mechanisms . |
| Approach: | They propose to integrate working memory constraints into the Transformer architecture . they use fixed-width windows and temporal decay-based attention mechanisms . |
| Outcome: | The proposed models show that they can learn better when training data is scarce . the findings suggest that such constraints may serve as a beneficial bias guiding models towards more robust representations . |