Papers with ViT
T-MAD: Target-driven Multimodal Alignment for Stance Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Multimodal Stance Detection struggle with generalizing to unseen targets and handling modality inconsistencies. |
| Approach: | They propose a multimodal stability detection model which captures target-specific relationships and balances modality contributions by iterative reasoning. |
| Outcome: | Experiments on the MMSD and MultiClimate datasets show that the proposed model outperforms state-of-the-art models with optimal results achieved using RoBERTa, ViT, and an iterative depth of 5. |
MDC3: A Novel Multimodal Dataset for Commercial Content Classification in Bengali (2025.naacl-srw)
Copied to clipboard
| Challenge: | Identifying commercial posts in resource-constrained languages remains a challenge for automatic text classification tasks. |
| Approach: | They propose a dataset for Bengali social media posts classified as commercial and noncommercial . they include an annotation guideline to aid future dataset creation in resource-constrained languages . |
| Outcome: | The proposed dataset is based on an annotation guideline for future dataset creation in resource-constrained languages. |
FLIQA-AD: a Fusion Model with Large Language Model for Better Diagnose and MMSE Prediction of Alzheimer’s Disease (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing classification and regression models that only extract finer-grained information from magnetic resonance imaging (MRI) may not be effective for Alzheimer's disease (AD). |
| Approach: | They propose to use a 3D Adapter in a Vision Transformer to extract the patient's EHR information and questions related to the disease as text prompts. |
| Outcome: | The proposed model can discriminate and predict the corresponding MMSE score based on the extracted brain structural information and textual content . |
Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation (2026.acl-long)
Copied to clipboard
Xi Xiao, Chenrui Ma, Yunbei Zhang, Chen Liu, Zhuxuanzi Wang, Yanshu Li, Lin Zhao, Guosheng Hu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a key parameter-efficient fine-tuning method . however, its effectiveness is hampered by semantic drift and structural incoherence . |
| Approach: | They propose a low-rank Adaptation framework that tackles semantic drift and structural incoherence by pruning task-irrelevant directions. |
| Outcome: | Experiments on large language models, vision models, and vision models show that the proposed framework outperforms LoRA and advanced dynamic rank allocation and sparsity-based methods. |
ART: Attention-Regularized Transformers for Multi-Modal Robustness (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to enhancing robustness are domain-specific or lack formal guarantees. |
| Approach: | They propose a framework that enhances robustness across modalities by regularizing attention maps under adversarial perturbations. |
| Outcome: | The proposed framework improves robustness across modalities and training on IMDB, QNLI, CIFAR-10, Cifar-100, and Imagenette. |
Multimodal Language Models See Better When They Look Shallower (2025.emnlp-main)
Copied to clipboard
Haoran Chen, Junyan Lin, Xinghao Chen, Yue Fan, Jianfeng Dong, Xin Jin, Hui Su, Jinlan Fu, Xiaoyu Shen
| Challenge: | Existing studies show that multimodal large language models extract visual features from the final layers of a pretrained Vision Transformer. |
| Approach: | They propose a feature fusion method that strategically incorporates shallower layers . they propose MLLMs that extract visual features from the final layers of a pretrained Vision Transformer . |
| Outcome: | The proposed method outperforms deep layers on fine-grained visual tasks . it is the first comprehensive study of visual layer selection for MLLMs . |
SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics (2024.naacl-long)
Copied to clipboard
Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin Iancu, Koushik Sen
| Challenge: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
| Approach: | They propose a tool that analyzes transformer-based models and freezes less-contributory layers during fine-tuning to reduce the overall on-device memory usage. |
| Outcome: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite the success of Large Vision-Language Models, they suffer from hallucination. |
| Approach: | They propose a training-free strategy that "D**ive into" the attention of LVLMs to "R**educe" object hallucination by using classification tokens of ViT. |
| Outcome: | The proposed method reduces the impact of outlier tokens on LVLMs . the proposed method is based on LLaVA-1.5, LLvaVA-NeXT and InstructBLIP . |
Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Foundation Models (FMs) have demonstrated success in a wide range of applications, but their optimization often requires access to sensitive data. |
| Approach: | They propose a framework that combines FMs and Federated Learning to enable privacy-preserving and collaborative learning across multiple end-users. |
| Outcome: | The proposed framework combines benefits of FMs and Federated Learning (FL) it enables privacy-preserving and collaborative learning across multiple end-users. |
Quantifying Character Similarity with Vision Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Off-the-shelf string matching methods are widely used to link entities across datasets, but they are not available for many settings. |
| Approach: | They propose to use augmented digital fonts to measure character substitution costs for OCR’ed documents by using vision transformers. |
| Outcome: | The proposed method significantly improves record linkage compared to other string matching methods. |
Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines the visual representation of concrete concepts using images from Bing and YFCC. |
| Approach: | They examine the variability in visual representations by using images of concrete and abstract concepts from Bing and YFCC. |
| Outcome: | The proposed model can distinguish between concrete and abstract concepts using basic visual features, the authors show . their model outperforms other models in the nearest neighbor analysis, but it is more complex and requires more visual features . |