Papers by Amita Kamath
Text encoders bottleneck compositionality in contrastive vision-language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal models are often unable to reason about simple spatial relations or attribute attachments. |
| Approach: | They first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture . then train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL model. |
| Outcome: | The proposed model can reconstruct captions from single-vector text representations produced by several models on a broader range of scenes compared to previous models. |
What’s “up” with vision-language models? Investigating their struggle with spatial reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has re-surfaced a concern that has long plagued vision-language models: poor performance on simple tasks like attribute attachment, counting, etc. |
| Approach: | They evaluate 18 vision-language models and find they perform poorly on VQAv2 . they find that popular vision-linguistic pretraining corpora lack reliable data for learning spatial relationships . |
| Outcome: | The new models are compared with existing datasets on what'sup and visual-language models . they achieve 56% accuracy on the new benchmarks compared to 99% for humans . |
Selective Question Answering under Domain Shift (2020.acl-main)
Copied to clipboard
| Challenge: | Deployed question answering (QA) models need to know when to abstain from answering questions that diverge from their training data. |
| Approach: | They propose a selective question answering under domain shift in which a QA model is tested on a mixture of in-domain and out-of-domain data and must answer (i.e., not abstain on) as many questions as possible. |
| Outcome: | The proposed method answers 56% of questions while maintaining 80% accuracy. |