Papers by Arjun Akula
ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments (2022.emnlp-main)
Copied to clipboard
Arjun Akula, Spandana Gella, Aishwarya Padmakumar, Mahdi Namazifar, Mohit Bansal, Jesse Thomason, Dilek Hakkani-Tur
| Challenge: | Embodied Vision and Language Task Completion requires an embodied agent to interpret natural language instructions and egocentric visual observations to navigate through and interact with environments. |
| Approach: | They propose to use ALFRED to test whether models can adapt to tasks not seen during training that require the same types of language understanding as ALFred. |
| Outcome: | The proposed model can adapt to tasks that require the same types of language understanding as ALFRED-L. |
CPL: Counterfactual Prompt Learning for Vision and Language Models (2022.emnlp-main)
Copied to clipboard
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Wang
| Challenge: | Existing prompt tuning methods tend to learn spurious or entangled representations, leading to poor generalization to unseen concepts. |
| Approach: | They propose a prompt tuning technique that tunes the learnable prompt for pre-trained vision and language models. |
| Outcome: | The proposed method improves few-shot performance on vision and language tasks over existing prompt tuning methods. |
Seeing Beyond: Enhancing Visual Question Answering with Multi-Modal Retrieval (2025.coling-industry)
Copied to clipboard
| Challenge: | Multi-modal Large language models still suffer from model hallucination and lack of specific knowledge when answering challenging questions. |
| Approach: | They propose to use a multi-modal retrieval augmented generation method to integrate knowledge from all modalities into a model to enable alignment between query and knowledge. |
| Outcome: | The proposed method achieves significant performance improvement on the VQA dataset. |
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models (2023.acl-industry)
Copied to clipboard
| Challenge: | Image ad understanding is a crucial task with wide real-world applications, but is under-explored in the machine learning community due to the lack of foundational vision-language models (VLMs) . |
| Approach: | They propose a simple feature adaptation strategy to fuse multimodal information for image ads and further empower it with knowledge of real-world entities. |
| Outcome: | The proposed strategy fuses multimodal information for image ads and empowers it with knowledge of real-world entities. |
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual question answering models leverage spurious biases and take shortcuts to improve performance. |
| Approach: | They propose a semi-automatic framework for generating disentangled shifts by introducing a controllable visual question-answer generation module that generates highly-relevant question-announcer pairs with the desired dataset style. |
| Outcome: | The proposed framework generates highly-relevant and diverse question-answer pairs with the desired dataset style. |
Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior implementations of NMN use pre-defined and fixed textual inputs in their module instantiation. |
| Approach: | They propose to parameterize the module arguments to reduce the number of modules in NMN by up to 75% without any loss in performance. |
| Outcome: | The proposed model outperforms the state-of-the-art model on CLEVR-Ref+ dataset with +8.1% improvement in accuracy and +4.3% on full test set. |
Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020.acl-main)
Copied to clipboard
| Challenge: | Visual referring expression recognition is a task that requires natural language understanding in the context of an image. |
| Approach: | They propose to use contrastive learning and multi-task learning to increase the robustness of ViLBERT, the current state-of-the-art model for this task. |
| Outcome: | The proposed methods are 12% to 23% lower in performance than the established progress for this task. |
PRISM: A New Lens for Improved Color Understanding (2024.emnlp-industry)
Copied to clipboard
| Challenge: | PRISM is a visual representation learner that can grasp the nuances of precise colors without compromising CLIP’s performance on established benchmarks. |
| Approach: | They propose a method that extends CLIP's ability to grasp the nuances of precise colors by utilizing a curated dataset of 100 image-text pairs that can be effortlessly repurposed for fine-tuning. |
| Outcome: | The proposed method improves CLIP's ability to grasp the nuances of precise colors without compromising CLIP’s performance on established benchmarks. |