Papers by Raja Giryes
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
Mamba Knockout for Unraveling Factual Information Flow (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has introduced Mamba-based SSM architectures that rival Transformer performance in various settings. |
| Approach: | They propose to use attentional interpretability techniques originally developed for Transformers to trace how information is transmitted and localized across tokens and layers. |
| Outcome: | The proposed model disentangles how distinct features enable token-to-token information exchange or enrich individual tokens, thus offering a unified lens to understand Mamba internal operations. |
Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language models excel at extracting and reasoning about information from images, yet their capacity to leverage internal knowledge about specific entities remains underexplored. |
| Approach: | They propose a dataset which allows separating entity recognition and question answering . they hypothesize that this decline arises from limitations in how information flows from image tokens to query tokens. |
| Outcome: | The proposed model performance drops when the entity is presented visually rather than textually. |
CARES: Context-Aware Resolution Selector for VLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large vision–language models process images at native or high resolution to remain effective across tasks. |
| Approach: | They propose a lightweight preprocessing module that predicts the minimum sufficient input resolution for large vision–language models. |
| Outcome: | CARES predicts when a pre-trained VLM's response converges to its peak ability to answer correctly, reducing compute by up to 80%. |
Low Resource Sequence Tagging using Sentence Reconstruction (2020.acl-main)
Copied to clipboard
| Challenge: | This work revisits the task of training sequence tagging models with limited resources using transfer learning. |
| Approach: | They propose a new loss that relies on sentence reconstruction from normalized embeddings. |
| Outcome: | The proposed method achieves 0.6 F1 score on Dutch datasets using only one sample. |
ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to curation text-image data are noisy and lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. |
| Approach: | They propose a metric that evaluates caption text without an image reference to measure its concreteness and relevancy. |
| Outcome: | The proposed method detects the concreteness of captions without an image reference and correlates with human evaluation of concreteness in both single-word and caption-level texts. |
ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline (2026.acl-long)
Copied to clipboard
| Challenge: | Constructed languages (conlangs) have played diverse roles in art, philosophy, and international communication. foundation models have revolutionized creative generation in text, images, and beyond. |
| Approach: | They propose a multi-hop pipeline that decomposes language design into modular stages . they use LLMs' metalinguistic reasoning capabilities to encourage diversity . |
| Outcome: | The proposed pipeline decomposes language design into modular stages . it leverages LLMs’ metalinguistic reasoning capabilities to encourage diversity and self-refinement feedback to encourage consistency and typological diversity. |
Mitigating Open-Vocabulary Caption Hallucinations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for image captioning ignore the long-tailed nature of hallucinations . a new framework is proposed to address hallucines in image captions in the open-vocabulary setting . |
| Approach: | They propose a framework to address hallucinations in image captioning in the open-vocabulary setting. |
| Outcome: | The proposed framework surpasses the CHAIR benchmark in diversity and accuracy in open-vocabulary captioning. |