Papers by Jaisidh Singh
(Almost) Free Modality Stitching of Foundation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-modal foundation models often use modality-specific (uni-modal) models as sub-components, which are stitched together via a connector module. |
| Approach: | They propose a framework that allows for optimal uni-modal model selection and connector training by leveraging hypernetworks. |
| Outcome: | The proposed framework reduces the cost of searching for the best performing uni-modal model pair by 10 while matching the ranking and trained connector performance across diverse multi-modal benchmarks. |