Papers with projector
Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that capture multiple modalities with a single input length are unable to handle this computational burden. |
| Approach: | They propose a query-based cross-modal projector that compresses visual tokens based on input through the cross-attention mechanism. |
| Outcome: | The proposed projector reduces the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLMs. |
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (LVLMs) learn visual capacity through visual instruction tuning. |
| Approach: | They propose a method for LVLMs to be trained by selective layers tuning . they propose removing non-critical layers outside the visual region . |
| Outcome: | The proposed approach preserves nearly 99% of visual performance and improves textual task results while reducing training time. |
The Dominance of Text Space: Unveiling the Asymmetric Nature of Cross-Modal Alignment in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for cross-modal alignment assume a symmetric interaction between visual and textual modalities, implying that both spaces adapt to each other. |
| Approach: | They propose a method that regularizes the projector to maintain the geometric structure of the text embedding space via spectral filtering. |
| Outcome: | The proposed method preserves the LLM’s inherent linguistic capabilities and reduces object hallucination significantly better than standard fine-tuning methods. |