Papers by Sugato Basu
CPL: Counterfactual Prompt Learning for Vision and Language Models (2022.emnlp-main)
Copied to clipboard
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Wang
| Challenge: | Existing prompt tuning methods tend to learn spurious or entangled representations, leading to poor generalization to unseen concepts. |
| Approach: | They propose a prompt tuning technique that tunes the learnable prompt for pre-trained vision and language models. |
| Outcome: | The proposed method improves few-shot performance on vision and language tasks over existing prompt tuning methods. |
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models (2023.acl-industry)
Copied to clipboard
| Challenge: | Image ad understanding is a crucial task with wide real-world applications, but is under-explored in the machine learning community due to the lack of foundational vision-language models (VLMs) . |
| Approach: | They propose a simple feature adaptation strategy to fuse multimodal information for image ads and further empower it with knowledge of real-world entities. |
| Outcome: | The proposed strategy fuses multimodal information for image ads and empowers it with knowledge of real-world entities. |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations (2020.emnlp-main)
Copied to clipboard
| Challenge: | A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings. |
| Approach: | They propose to use visual attention to build robust benchmark datasets and models that can generalize well in real-world settings. |
| Outcome: | The proposed models show that human-generated references vary drastically in different datasets/tasks, revealing the nature of each task. |
PRISM: A New Lens for Improved Color Understanding (2024.emnlp-industry)
Copied to clipboard
| Challenge: | PRISM is a visual representation learner that can grasp the nuances of precise colors without compromising CLIP’s performance on established benchmarks. |
| Approach: | They propose a method that extends CLIP's ability to grasp the nuances of precise colors by utilizing a curated dataset of 100 image-text pairs that can be effortlessly repurposed for fine-tuning. |
| Outcome: | The proposed method improves CLIP's ability to grasp the nuances of precise colors without compromising CLIP’s performance on established benchmarks. |
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation (2021.eacl-main)
Copied to clipboard
Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
| Challenge: | Outdoor vision-and-language navigation (VLN) tasks require visual grounding to generate correct actions. |
| Approach: | They propose a multimodal text style transfer learning approach to mitigate data scarcity in outdoor vision-and-language navigation tasks. |
| Outcome: | The proposed approach outperforms baseline models on the outdoor vision-and-language navigation task, improving task completion rate by 8.7% relative to the baseline models. |