Papers with vision

82 papers
Social Norms Guide Reference Resolution (2022.naacl-main)

Copied to clipboard

Challenge: Existing tools for natural language resolution fail to handle ambiguous referents . ambiguity arises when the language is underspecified or there are multiple candidate referent.
Approach: They investigate how pragmatic modulators outside of the linguistic content are critical for correct interpretation of referents in underspecified contexts.
Outcome: The proposed method can be used to resolve referents in human environments.
ViLMedic: a framework for research at the intersection of vision and language in medical AI (2022.acl-demo)

Copied to clipboard

Challenge: Multimodal medical AI is a growing field of interest, especially for tasks that involve multimodal data.
Approach: They propose a vision-and-language medical library to improve multimodal medical predictions and enable new applications.
Outcome: The vision-and-language medical library aims to improve reproducibility and speed up progress across medical AI . it contains a dozen implementations replicating state-of-the-art results on medical datasets . the library is extensible by researchers but also simple for practitioners .
EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot (2024.acl-demos)

Copied to clipboard

Challenge: EmpathyEar is an open-source, avatar-based multimodal empathetic chatbot . currently, ERG systems rely on text, sound, and vision .
Approach: They propose an open-source, avatar-based multimodal empathetic chatbot to fill the gap in traditional text-only ERG systems.
Outcome: The proposed system enables users to generate emotional responses to user queries . it can also generate avatars with talking faces and synchronized speeches .
Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checking (2022.findings-emnlp)

Copied to clipboard

Challenge: Chinese Spell Checking (CSC) aims to detect and correct Chinese spelling errors.
Approach: They propose a framework which renders Chinese Spell Checking model to learn heterogeneous knowledge from the dictionary in terms of phonetics, vision, and meaning.
Outcome: The proposed framework renders the CSC model to learn heterogeneous knowledge from the dictionary in terms of phonetics, vision, and meaning.
Long-range Sequence Modeling with Predictable Sparse Attention (2022.acl-long)

Copied to clipboard

Challenge: Existing approaches to capture global context dependencies in sequence modeling suffer from quadratic complexity in time and memory usage.
Approach: They propose an efficient Transformer architecture for fast long-range sequence modeling with a sparse attention matrix and a hidden state cross module.
Outcome: The proposed architecture outperforms the standard multi-head attention and its variants in various long-sequence tasks with low computational costs.
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities (2023.acl-demo)

Copied to clipboard

Challenge: Several pre-training models of different modalities are showing a rising trend of homogeneity in their model structures.
Approach: They propose a toolkit that supports pre-training models of different modalities.
Outcome: The proposed toolkit can match the performance of the original implementations on text, vision, and audio benchmarks.
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)

Copied to clipboard

Challenge: a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study.
Approach: They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains.
Outcome: a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not.
Chronocept: Instilling a Sense of Time in Machines (2026.eacl-srw)

Copied to clipboard

Challenge: Human cognition is deeply intertwined with a sense of time, known as Chronoception, which allows us to judge how long facts remain valid and when knowledge becomes outdated.
Approach: They propose a model that captures nuanced patterns of emergence, decay, and peak relevance using skew-normal curves fitted along semantically decomposed temporal axes.
Outcome: The proposed model captures nuanced patterns of emergence, decay, and peak relevance in two datasets.
Scene-Text Aware Image and Text Retrieval with Dual-Encoder (2022.acl-srw)

Copied to clipboard

Challenge: Existing studies on image and text retrieval using a dual-encoder model have not shown their effectiveness for fast inferences.
Approach: They propose a dual-encoder model that connects vision and language in the same semantic space and integrates scene-text and visual information into a model.
Outcome: The proposed model can interpret scene-text and surrounding visual information better than cross-encoder models.
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing foundation models can only perform the best in one type of understanding tasks.
Approach: They propose a method for training a general foundation model, X-FM, using text, image, and image-text data.
Outcome: The proposed method outperforms existing foundation models on language, vision, and vision-language understanding tasks.
CMTD: Cognitive Modeling with Traits and Distortions for Multimodal Emotion Recognition in Conversations (2026.findings-acl)

Copied to clipboard

Challenge: Experimental results show that traits temper negativity bias from distortions, and cognitive modeling with psychological, visual, and acoustic information can improve the performance of MERC.
Approach: They propose a framework for multimodal emotion recognition in conversations that takes advantage of stable personality traits, dynamic cognitive distortions, visual and acoustic features of interlocutors to enhance the emotional intelligence of LLMs.
Outcome: Experimental results show that traits temper negativity bias from distortions, and cognitive modeling with psychological, visual, and acoustic information can improve the performance of MERC.
VKIE: The Application of Key Information Extraction on Video Text (2023.emnlp-industry)

Copied to clipboard

Challenge: Existing methods for extracting structured information from videos are coarse-grained at segment level and unable to capture finegrained information at the entity level.
Approach: They propose a task for extracting hierarchical key information from visual texts on videos . they decouple the task into four subtasks and propose two implementation solutions .
Outcome: The proposed solutions achieve remarkable performance and efficient inference speed on a well-defined dataset.
Multimodal Logical Inference System for Visual-Textual Entailment (P19-2)

Copied to clipboard

Challenge: Recent studies of multimodal inference provide challenging tasks such as visual question answering and visual reasoning.
Approach: They propose an unsupervised multimodal logical inference system that can prove entailment relations between texts and images by combing semantic parsing and theorem proving.
Outcome: The proposed system can handle semantically complex sentences for visual-textual inference.
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (N19-1)

Copied to clipboard

Challenge: Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image.
Approach: They construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset and use it to benchmark performance of standard visual dialog models.
Outcome: The proposed model is based on a large diagnostic dataset for studying multi-round reasoning in visual dialog.
Read Before Grounding: Scene Knowledge Visual Grounding via Multi-step Parsing (2025.coling-main)

Copied to clipboard

Challenge: Existing VG datasets use simple textual descriptions with limited attribute and spatial information between images and text.
Approach: They propose a method that transforms visual knowledge into concise, information-dense visual descriptions.
Outcome: The proposed method significantly improves performance of multimodal grounding models.
Measuring Social Biases in Grounded Vision and Language Embeddings (2021.naacl-main)

Copied to clipboard

Challenge: Existing methods to measure social biases in word embeddings are limited to visually grounded word embeds . a new study generalizes word embedment associations to visually ground word embeddas .
Approach: They generalize word embeddings' biases to visually grounded word embeds . they propose two generalizations that answer questions about how biase, language, and vision interact .
Outcome: The proposed measures are applied to a new dataset that includes 10,228 images from COCO, Conceptual Captions, and Google Images.
Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidth (2026.eacl-long)

Copied to clipboard

Challenge: despite advances in transformers, their theoretical limitations in discrete reasoning remain a critical open problem.
Approach: They synthesize recent advances from three theoretical perspectives to clarify structural and computational barriers transformers face when performing symbolic computations.
Outcome: The proposed models excel at pattern matching and interpolation, but they face bottlenecks in communication and depth constraints.
MemeCap: A Dataset for Captioning and Interpreting Memes (2023.emnlp-main)

Copied to clipboard

Challenge: a new dataset aims to understand meme captioning tasks using visual metaphors . vision and language models are proving to be effective in image captioning and visual question answering tasks .
Approach: They present a dataset that contains 6.3K memes and 6.3k meme captions . they show that vision and language models still struggle with visual metaphors despite their advanced capabilities .
Outcome: The proposed dataset contains 6.3K memes along with the title of the post containing the meme, meme captions, literal image caption, and visual metaphors.
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis (2020.coling-main)

Copied to clipboard

Challenge: Existing studies focus on learning the joint representation of multiple modalities, ignoring useful knowledge contained in language modal.
Approach: They propose to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features.
Outcome: The proposed method improves the fusion representation of multimodal features on a YouTube and video dataset.
Situated and Interactive Multimodal Conversations (2020.coling-main)

Copied to clipboard

Challenge: Situated Interactive MultiModal Conversations (SIMMC) is a new direction for virtual assistants that handle multimodal inputs and perform multimodal actions.
Approach: They propose to use Situated Interactive MultiModal Conversations (SIMMC) to train agents to take multimodal actions grounded in a co-evolving multimodal context.
Outcome: The proposed model will be made publicly available.
An Empirical Study of Multimodal Model Merging (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing studies have shown that model merging can generate a multi-task solution without synchronous training.
Approach: They propose to merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient architecture.
Outcome: The proposed model merging outperforms naive models on various tasks with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% ADE20k.
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing datasets are often informed by established research directions in the NLP community.
Approach: They propose a benchmark to evaluate the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks.
Outcome: The proposed benchmark evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks.
Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features (2024.eacl-long)

Copied to clipboard

Challenge: Existing explanations for speech classification models are difficult to interpret and make mistakes.
Approach: They propose to explain speech classification models by using word-level and paralinguistic attributes to measure the impact of each audio segment aligned with a word on the outcome.
Outcome: The proposed explanations correctly represent the model’s inner workings and are plausible to humans.
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023.acl-short)

Copied to clipboard

Challenge: Existing models focus on improving the efficiency of self-attention, but in practice they may be slower, especially given modest input lengths that are typical of many tasks.
Approach: They propose a novel local-attention variant of a self-supervised speech model that uses input length thresholds to identify bottlenecks.
Outcome: The proposed model is based on a self-attention-based model with a high input length threshold.
Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis (2022.acl-long)

Copied to clipboard

Challenge: Existing approaches to multimodal Aspect-Based Sentiment Analysis (MABSA) ignore crossmodalalignment and use pre-trained visual and textual models.
Approach: They propose a multimodal multimodal encoder-decoder framework for MABSA that uses a unified multimodal decoder architecture for all the pretrainingand downstream tasks.
Outcome: The proposed framework outperforms state-of-the-art approaches on three MABSA subtasks.
Control Image Captioning Spatially and Temporally (2021.acl-long)

Copied to clipboard

Challenge: Existing methods to generate image captions with user intention are still under exploration.
Approach: They propose a model that connects Contrastive constraints and Attention Guidance in a loop manner and engages explicit spatial and temporal constraints to the generating process.
Outcome: The proposed model improves performance on a trace-controlled image captioning task.
Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization (2023.acl-long)

Copied to clipboard

Challenge: Existing studies on multimodal abstractive summarization focus on how to use extracted visual features to produce a concise summary given the multimodal data.
Approach: They propose to improve the visual quality of the multimodal abstractive summarization model by capturing summary-oriented visual features.
Outcome: The proposed approach achieves state-of-the-art under 44 languages and is highly effective on high-resource English datasets.
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation (2023.findings-acl)

Copied to clipboard

Challenge: Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision.
Approach: They propose a framework for multimodal machine translation that utilizes large-scale non-triple data and a multimodal translation dataset.
Outcome: The proposed method can significantly improve translation performance with more non-triple data.
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025.findings-acl)

Copied to clipboard

Challenge: LLaVA-7B demonstrated a decline in safety alignment ability on multi-modal inputs compared to its LLM backbone.
Approach: They propose a method to recover alignment ability from LLM backbone while preserving functional capabilities of VLMs.
Outcome: The proposed framework recovers alignment ability that is inherent in the LLM backbone with minimal impact on fluency and linguistic capabilities of pre-trained VLMs.
xGQA: Cross-Lingual Visual Question Answering (2022.findings-acl)

Copied to clipboard

Challenge: a lack of multilingual multimodal datasets has hindered multimodal vision and language modeling efforts.
Approach: They propose a multilingual evaluation benchmark for the visual question answering task . they extend the established English GQA dataset to 7 typologically diverse languages .
Outcome: The proposed methods outperform current state-of-the-art models in zero-shot cross-lingual settings, but the accuracy remains low across languages.
Visual Enhanced Entity-Level Interaction Network for Multimodal Summarization (2024.findings-naacl)

Copied to clipboard

Challenge: Existing methods to generate concise summarizations rely on coarse-grained textual and visual information, but they are underutilized.
Approach: They propose a Visual Enhanced Entity-Level Interaction Network to address underutilization of multimodal inputs at a fine-grained level.
Outcome: The proposed model outperforms existing models on two MMS datasets and proposes new metrics to measure factual consistency of entities in the output.
Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)

Copied to clipboard

Challenge: Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP.
Approach: They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning.
Outcome: The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning.
Fusion of Detected Objects in Text for Visual Question Answering (D19-1)

Copied to clipboard

Challenge: Recent neural architectures such as Transformer and BERT allow for multimodal context, which may help model the meaning of words in general and also sharpen its understanding of instances of words.
Approach: They propose a neural architecture that combines vision and natural language to advance models of multimodal context.
Outcome: The proposed architecture achieves the highest performance on the Visual Commonsense Reasoning benchmark and the best performance to date on the public leaderboard.
Transformer-Exclusive Cross-Modal Representation for Vision and Language (2021.findings-acl)

Copied to clipboard

Challenge: a number of approaches to crossmodal representation have been used, but transformer architecture has taken over the recurrent neural networks in natural language processing tasks.
Approach: They propose to use transformer architecture to handle cross-modal representations for vision and language with compatible performance to convolutional neural networks.
Outcome: The proposed model outperforms recurrent neural networks in vision and language representations with transformer architecture.
LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models (2025.findings-naacl)

Copied to clipboard

Challenge: Multi-modal Large Language Models (MLLMs) incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments.
Approach: They propose a language-guided vision token pruning method that can be integrated into existing MLLMs with minimal architectural changes.
Outcome: The proposed method reduces vision tokens by 90% and preserves model performance.
Analyzing (In)Abilities of SAEs via Formal Languages (2025.naacl-long)

Copied to clipboard

Challenge: Autoencoders have been used for finding interpretable and disentangled features underlying neural network representations in both image and text domains, but there is a lack of corresponding results for the text domain.
Approach: They propose to train sparse autoencoders (SAEs) on a synthetic testbed of formal languages to find interpretable latents in models trained on formal languages.
Outcome: The proposed approach promotes learning of causally relevant features in a formal language setting.
On the Representation Geometry of LoRA Model Merging (2026.findings-acl)

Copied to clipboard

Challenge: Existing methods for low-rank Adaptation (LoRA) fine-tuning focus on globally shared structure . combining SVD with CUR improves performance of LoRA model merging .
Approach: They propose a training-free method that combines SVD and CUR decomposition to improve LoRA merging performance.
Outcome: The proposed procedure improves on vision and language benchmarks.
Grounding language acquisition by training semantic parsers using captioned videos (D18-1)

Copied to clipboard

Challenge: a new method for parsing sentences using captioned videos is being developed . we use video clips to ground the semantics of language, but without annotations .
Approach: They develop a semantic parser that is trained in a grounded setting using captioned videos . they use a corpus of sentences paired with videos without other annotations to train it .
Outcome: The proposed parser recovers the meaning of English sentences despite no annotations . learning a grounded semantic parsers can expand the range of data that parseurs can be trained on .
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)

Copied to clipboard

Challenge: Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA).
Approach: They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks.
Outcome: The proposed framework is used to train and evaluate multi-modal retrievers.
GOBench: Stage-Wise Diagnostics and the Visual Paradox in Multimodal Graph Optimization (2026.findings-acl)

Copied to clipboard

Challenge: Existing benchmarks fail to represent multimodal problem specifications, score outcomes only and cannot localize where failures occur along the modeling pipeline.
Approach: They propose a Graph Optimization benchmark that aligns multiple modalities with solver-derived oracles and a diagnostic protocol that evaluates intermediate artifacts as well as end results.
Outcome: Graph Optimization benchmark (GOBench) evaluates intermediate artifacts as well as end results . vision reliably increases inference cost, while reliability impact is regime-dependent . current benchmarks fail to represent multimodal problem specifications, fail to localize failures .
Multi Modal Distance - An Approach to Stemma Generation With Weighting (L18-1)

Copied to clipboard

Challenge: Stemma generation is a task where manuscripts are copied and copied from each other and from M. Existing methods to generate stemma using unweighted token similarity weighting have been used.
Approach: They propose to use a distance model to weight the texts of M1 and M2 to estimate the most likely tree from a series of mapping processes.
Outcome: The proposed method is small in the experimental scenario(s) it is based on psycholinguistically gained distance matrices of letters in three modalities: vision, audition and motorics.
DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding (2026.eacl-long)

Copied to clipboard

Challenge: Recent work shows that hyperscaling of data and parameter count in LLMs is yielding diminishing improvement when weighed against training costs.
Approach: They propose to insert multimodal tokens directly into the middle of the model to bypass the early layers.
Outcome: The proposed method reduces training and inference costs while preserving performance.
Measure and Improve Robustness in NLP Models: A Survey (2022.naacl-main)

Copied to clipboard

Challenge: Despite the performance gains, NLP models are still fragile and brittle to out-of-domain data, adversarial attacks, or small perturbation to the input.
Approach: They propose a survey of how to define, measure and improve robustness in NLP by connecting multiple definitions of robustness and identifying failures.
Outcome: The proposed models are robust against unseen or challenging scenarios, but are still fragile and brittle to out-of-domain data and adversarial attacks.
GAMBIT: A Gamified Jailbreak Framework for Multimodal Large Language Models (2026.acl-long)

Copied to clipboard

Challenge: Existing attacks focus on increasing the complexity of the modified visual task and do not explicitly leverage the model’s own reasoning incentives.
Approach: They propose a framework that decomposes and reassembles harmful visual semantics and constructs a gamified scene that drives the model to explore, reconstruct intent and answer as part of winning the game.
Outcome: Experiments on reasoning and non-reasoning MLLMs show that the proposed framework outperforms baseline models on both vision and text.
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation (2026.acl-long)

Copied to clipboard

Challenge: Multimodal large language models have advanced rapidly, yet most remain English-centric . scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of non-English image–text supervision.
Approach: They propose a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone.
Outcome: The proposed framework achieves competitive performance with a fully multimodally trained model using less than 2% of the text data.
SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network (2024.acl-long)

Copied to clipboard

Challenge: SpikeVoice performs high-quality Text-To-Speech (TTS) via SNN . major obstacle to using SNN for such generative tasks lies in the demand for models to grasp long-term dependencies.
Approach: They propose a brain-inspired Spiking Neural Network (SNN) which performs high-quality Text-To-Speech (TTS) via SNN and explores the potential of SNN to "speak".
Outcome: The proposed model achieves comparable results to Artificial Neural Networks (ANN) with only 10.5% energy consumption of ANN.
Digging out Discrimination Information from Generated Samples for Robust Visual Question Answering (2023.findings-acl)

Copied to clipboard

Challenge: Existing methods to solve this problem rely on additional annotations and generate negative samples .
Approach: They propose a method to Dig out Discrimination information from Generated samples to address these limitations.
Outcome: The proposed method improves on the visual question-answering datasets.
Text Counterfactuals via Latent Optimization and Shapley-Guided Search (2021.emnlp-main)

Copied to clipboard

Challenge: Using latent optimization and Shapley values, we generate a set of minimal modifications to the text to change the classifier's prediction.
Approach: They propose to generate a counterfactual by making minimal modifications to the text to change the model's prediction.
Outcome: The proposed approach achieves favorable performance compared to white-box and black-box baselines using human and automatic evaluations.
Position Really Matters: Towards a Holistic Approach for Prompt Tuning (2025.findings-naacl)

Copied to clipboard

Challenge: Prompt tuning is effective in extracting knowledge from foundation models, but its effectiveness is uncertain.
Approach: They propose a parametric prompt tuning strategy that dynamically determines different factors of prompts based on specific tasks or instances.
Outcome: The proposed approach improves performance across a wide range of tasks including NLP, vision recognition, and vision-language tasks.
Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data (2023.findings-acl)

Copied to clipboard

Challenge: a recent study shows that self-training can improve upon fully supervised baselines in low-resource settings for several sequence-to-sequence tasks.
Approach: They propose to use pseudo-labeling to label unsupervised data and add it to the training pool.
Outcome: The proposed setup improves on the unsupervised data by using pseudo-labeling . the proposed setup provides 0.4% absolute WER and 2.1 BLEU points for En–De .
Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022.coling-1)

Copied to clipboard

Challenge: PTLMs are used to extract knowledge from text on demand.
Approach: They compare visual-linguistic and language-only visual-language models in a zero-shot commonsense question answering inference task.
Outcome: The proposed models are highly promising on certain types of commonsense knowledge associated with the visual world.
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)

Copied to clipboard

Challenge: This survey provides a comprehensive overview of the LLM-driven personality scenario.
Approach: This survey provides a comprehensive overview of the LLM-driven personality scenario.
Outcome: The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps.
Hybrid Self-evolving Structured Memory for Computer-Use Agents (2026.findings-acl)

Copied to clipboard

Challenge: despite advances in vision–language models, real-world computer-use tasks remain challenging due to long-horizon workflows, diverse interfaces, and frequent intermediate errors.
Approach: They propose a graph-based memory that couples discrete symbolic nodes with continuous trajectory embeddings.
Outcome: The proposed system outperforms closed-source models in Qwen2.5-VL-7B and Gemini2.5-Pro-Vision on desktop and mobile platforms.
Towards Infinite-Long Prefix in Transformer (2025.emnlp-main)

Copied to clipboard

Challenge: Prefix Learning is an empirically efficient and effective method for language models . but the theoretical understandings are limited on the performance of such methods .
Approach: They propose a method that can train an ultra-long prefix in a stylized setting using the Neural Tangent Kernel framework.
Outcome: The proposed method can achieve superior performance on vision, natural language, and math data.
LATTE: Learning to Think with Vision Specialists (2025.emnlp-main)

Copied to clipboard

Challenge: Open-source vision-language models excel on simple question-answering tasks, but struggle with complex questions that require both perception and reasoning.
Approach: They propose a family of vision-language models that have LeArned to Think wiTh vision spEcialists by offloading perception to state-of-the-art vision models.
Outcome: The proposed model achieves 4-5% gains over baselines across 6 benchmarks covering both perception and reasoning abilities.
CODIS: Benchmarking Context-dependent Visual Comprehension for Multimodal Large Language Models (2024.acl-long)

Copied to clipboard

Challenge: Multimodal large language models have demonstrated promising results in a variety of tasks that combine vision and language.
Approach: They propose a benchmark to assess the ability of models to use contextual information in free-form text to enhance visual comprehension.
Outcome: The proposed model fails to extract and utilize contextual information to improve understanding of images.
Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions (2023.emnlp-main)

Copied to clipboard

Challenge: Learning and using abstract perceptual concepts is under-investigated in the language-and-vision field.
Approach: They propose a framework for testing category-level perceptual grounding in multi-modal language models by using separate neural networks to generate and interpret descriptions of visual categories.
Outcome: The proposed framework compares prototype- and interpretation-based representations with the performance of the generation model and the interpretation model, which is an indicator of perceptual grounding.
Extending Logic Explained Networks to Text Classification (2022.emnlp-main)

Copied to clipboard

Challenge: Recent studies have proposed explainable-by-design neural models providing logic explanations for their predictions, but these models favour global explanations, while local ones tend to be noisy and verbose.
Approach: They propose to use LENp to improve local explanations by perturbing input words to improve sensitivity and faithfulness of local explanation.
Outcome: The proposed model provides better local explanations than LIME and is more user-friendly than Lime as attested by a human survey.
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2023.acl-long)

Copied to clipboard

Challenge: Experimental results show zero-shot performance on unseen multimodal tasks . instruction tuning has yet to be explored for vision and multimodal task.
Approach: They propose a multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories.
Outcome: The proposed model performs well on unseen multimodal tasks and is highly scalable.
Refer360∘: A Referring Expression Recognition Dataset in 360∘ Images (2020.acl-main)

Copied to clipboard

Challenge: Refer360° is a large-scale referring expression recognition dataset consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes.
Approach: They propose a large-scale referring expression recognition dataset, Refer360°, consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes.
Outcome: The proposed dataset contains 17,137 instruction sequences and ground-truth actions for referring expression recognition in 360° scenes.
Dealing with Semantic Underspecification in Multimodal NLP (2023.acl-long)

Copied to clipboard

Challenge: a linguistic signal can leave underspecified information, such as gender and number . this problem is a crucial feature that boosts language's storage and processing efficiency .
Approach: They argue that intelligent systems must deal with semantic underspecification . it is a feature that boosts language's storage and processing efficiency, they argue . they argue that the problem is not a bug but a problem that could negatively affect performance .
Outcome: a new paper shows that systems that aim at mastering language must deal with semantic underspecification . it shows that human speakers can integrate semantically-underspecified linguistic signals with non-linguistic information .
ModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities (2024.emnlp-main)

Copied to clipboard

Challenge: Large vision-language models have been widely used but stereotypical biases are unexplored.
Approach: They propose a framework to SCAN stereotypical bias within large vision-language models . they examine stereotype biases with respect to gender and race in three scenarios .
Outcome: The proposed framework can reduce stereotypical biases in large vision-language models . the currently popular models show significant stereotype biase .
When Language Models Fall in Love: Animacy Processing in Transformer Language Models (2023.emnlp-main)

Copied to clipboard

Challenge: Animacy is not always expressed directly in language, but it manifests indirectly in English . atypically animate entities are easier to remember and prioritized in visual processing .
Approach: They find that LMs behave much like humans when presented with entities whose animacy is typical.
Outcome: The proposed model can learn about animacy even when presented with atypically animate entities.
MISP-Meeting: A Real-World Dataset with Multimodal Cues for Long-form Meeting Transcription and Summarization (2025.acl-long)

Copied to clipboard

Challenge: Existing systems that can recognize spoken content, extract key information, and produce concise summaries are lacking in meeting transcription and summarization.
Approach: They propose a multimodal dataset that integrates information from speech, vision, and text modalities to facilitate automatic meeting transcription and summarization (AMTS).
Outcome: The proposed dataset reduces the character error rate (CER) by 36.60% to 20.27% and improves speech recognition and large language models.
Curriculum Consistency Learning for Conditional Sentence Generation (2024.emnlp-main)

Copied to clipboard

Challenge: Consistency learning (CL) has proven to be a valuable technique for improving the robustness of conditional sentence generation models.
Approach: They propose a strategy that guides models to learn consistency in alignment with their current capacity to differentiate between features.
Outcome: The proposed strategy delivers +2.0 accuracy point improvement compared with vanilla IT and +0.7 COMET scores over traditional CL methods in MT tasks.
Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames (2025.emnlp-main)

Copied to clipboard

Challenge: Disjoint-3DQA evaluates the spatial reasoning ability of embodied AI assistants based on egocentric video . it aims to catalyze future research at the intersection of vision, language, and embodie .
Approach: They propose a generative QA benchmark that evaluates the ability of embodied AI assistants to integrate spatial cues across time by asking object pairs that are not co-visible in the same frame.
Outcome: The proposed benchmark compares seven state-of-the-art VLMs and finds that they lag behind human performance by 28%, with steeper declines as the temporal gap widens.
Representation Potentials of Foundation Models for Multimodal Alignment: A Survey (2025.emnlp-main)

Copied to clipboard

Challenge: foundation models learn highly transferable representations through large-scale pretraining on diverse data.
Approach: They examine the representation potentials of foundation models by examining their latent capacity to capture task-specific information within a single modality while providing a transferable basis for alignment and unification across modalities.
Outcome: The foundation models exhibit remarkable similarities across architectures and modalities, the authors show . the models can capture task-specific information within a single modality while providing a transferable basis for alignment and unification across modality.
LVLMs are Bad at Overhearing Human Referential Communication (2025.emnlp-main)

Copied to clipboard

Challenge: a crucial skill for embodied AI agents working with humans is grounding in referential communication.
Approach: They use large vision language models to overhear spontaneous conversations between humans . they find that current LVLMs fail to show consistent performance improvement .
Outcome: The proposed models fail to show consistent performance improvement over previous models . the authors release the results to facilitate future research .
OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoning (2026.findings-acl)

Copied to clipboard

Challenge: Existing evaluation frameworks for multimodal large language models suffer from limitations . modality shortcuts and biased reasoning paths are common in such models .
Approach: a new benchmark evaluates omni-modal multi-hop reasoning using 6,144 questions . authors propose OMHBench to address these limitations by comparing modalities .
Outcome: OMHBench evaluates omni-modal multi-hop reasoning on 6,144 questions with balanced reasoning paths . evaluation of 13 state-of-the-art models shows performance gap exists between MLLMs and open-source models .
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture (2024.emnlp-main)

Copied to clipboard

Challenge: FoodieQA is a manually curated, fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China.
Approach: They evaluate vision–language Models and large language models on unseen food images and corresponding questions.
Outcome: The proposed dataset evaluates vision–language Models and large language models on unseen food images and corresponding questions.
Attack as Defense: Safeguarding Large Vision-Language Models from Jailbreaking by Adversarial Attacks (2025.findings-emnlp)

Copied to clipboard

Challenge: adversarial vulnerabilities in vision-language systems pose a challenge to reliability of large systems . typographic manipulations and adversarial perturbations can bypass language model defenses .
Approach: They propose a method that embeds perturbations in vision to disrupt attacks . they use cross-modal interactions to enhance adversarial robustness through perturbations .
Outcome: The proposed approach reduces attack success rates for typographic attacks and adversarial perturbations by integrating visual defenses into the model.
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding (2026.findings-acl)

Copied to clipboard

Challenge: Existing omni-multimodal large language models lack incomplete modality support or lack autonomous proactive monitoring.
Approach: They propose a real-time omni-multimodal assistant for unified reactive and proactive interaction that decouples response initiation from generation to ensure precise triggering without task conflict.
Outcome: The proposed model achieves state-of-the-art performance on proactive tasks while competing in reactive settings.
Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories (2024.emnlp-main)

Copied to clipboard

Challenge: a recent study examines the visual representation of concrete concepts using images from Bing and YFCC.
Approach: They examine the variability in visual representations by using images of concrete and abstract concepts from Bing and YFCC.
Outcome: The proposed model can distinguish between concrete and abstract concepts using basic visual features, the authors show . their model outperforms other models in the nearest neighbor analysis, but it is more complex and requires more visual features .
Grounding Multilingual Multimodal LLMs With Cultural Knowledge (2025.emnlp-main)

Copied to clipboard

Challenge: a new data-centric approach could address cultural gaps in multimodal large language models . despite being trained on billions of image-text pairs, today's models are biased towards English and Western data.
Approach: They propose a data-centric approach that directly grounds MLLMs in cultural knowledge.
Outcome: The proposed approach outperforms open-source models on cultural-focused benchmarks without degrading results on mainstream vision–language tasks.
SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Models (2026.acl-long)

Copied to clipboard

Challenge: Recent advances in multimodal large language models have increased their vulnerability to adversarial manipulation.
Approach: They propose to target audio-only adversarial attacks on multimodal audio–video–language models . they show that attacks can be successful at low perceptual distortions .
Outcome: The proposed models achieve up to 96% success rate under realistic conditions . the proposed models are more robust to noise than to noise and distortion than to speech recognition systems .
Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models (2025.findings-emnlp)

Copied to clipboard

Challenge: ELENA is a framework for embodied emotion analysis using large vision language models . ELEna uses attention maps and a persistent bias towards the facial region .
Approach: They propose a framework that utilizes large vision language models to generate ELENA . they propose to use attention maps to describe emotional reactions from body parts .
Outcome: The proposed framework outperforms baseline models without fine-tuning . it uses large vision language models to generate embodied emotion narratives .
Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baseline (2026.acl-long)

Copied to clipboard

Challenge: Existing facial forgery detection methods focus on binary classification or pixel-level localization, providing little semantic insight into the nature of the manipulation.
Approach: They propose a multimodal task that localizes forged regions and generates natural language explanations grounded in editing process.
Outcome: The proposed task localizes forged regions and generates natural language explanations grounded in editing process.
MAviS: A Multimodal Conversational Assistant For Avian Species (2025.emnlp-main)

Copied to clipboard

Challenge: Existing multimodal large language models face challenges when it comes to specialized topics like avian species.
Approach: They propose a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species.
Outcome: The proposed model outperforms the baseline MiniCPM-o-2.6 by a large margin.
Coarse-to-Fine Multimodal Information Selection for Video Speaking Style Recognition with Large Language Models (2026.findings-acl)

Copied to clipboard

Challenge: Video speaking style recognition (VSSR) aims to classify conversations into different types . integrating all multimodal data yields suboptimal results, authors say .
Approach: They propose a framework that allows users to obtain multimodal data via coarse-to-fine selection . they propose to use visual captions and textual dialogues to integrate multimodal information .
Outcome: The proposed framework outperforms existing training-free approaches and most training-based methods on multiple datasets.
Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer (2025.acl-long)

Copied to clipboard

Challenge: Foundational models and their checkpoints have advanced deep learning, boosting performance across applications.
Approach: They propose a method for pruning fine-tuned models by calculating differences between them and original model.
Outcome: The proposed method can improve performance across vision, NLP, and multi-modal benchmarks.
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning (2025.emnlp-main)

Copied to clipboard

Challenge: Existing multimodal large language models are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations.
Approach: They propose a large-scale multi-turn multimodal dialogue dataset that uses rules and GPT assistance to generate a multi-turned multimodal dialog dataset.
Outcome: The proposed dataset is a strong benchmark for multi-turn multimodal dialogue learning . it features complex dialogues with contextual dependencies that force models to track, ground, and recall information across multiple turns and disparate visual regions.
MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation (2026.acl-long)

Copied to clipboard

Challenge: Existing RAG solutions for large language models are limited by context windows limiting their ability to process long-form, domain-specific content.
Approach: They propose a multimodal knowledge graph-based RAG that enables cross-modal reasoning . their method incorporates visual cues into the construction of knowledge graphs, retrieval phase, and answer generation process .
Outcome: Experimental results show that the proposed approach outperforms existing approaches on textual and multimodal benchmarks.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations