| Challenge: | Unlike autoregressive captioning, the strength of the visual learning signal in MDC does not depend on each token’s position in the sequence, reducing the need for auxiliary objectives. |
| Approach: | a decoder conditioned on visual features is trained to reconstruct the original text. |
| Outcome: | masked diffusion captioning (MDC) is a form of image-conditioned captioning that can be applied to visual tasks. |
Similar Papers
Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing image captioning models require large trainable parameters to bridge visual and textual representations. |
| Approach: | They propose a lightweight image captioning network in combination with continuous diffusion that injects prefix image embeddings into denoising process of diffusion model. |
| Outcome: | The proposed method generates diverse captions with relatively less parameters while maintaining fluency and relevance compared with other models. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |
Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction (2024.acl-long)
Copied to clipboard
| Challenge: | EVLGen is a framework for visual-language pre-training with high computational demands. |
| Approach: | They propose a streamlined framework for the pre-training of visually conditioned language generation models with high computational demands. |
| Outcome: | The proposed framework accelerates training of vision-language models by a factor of 5 without compromising performance. |
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (D19-1)
Copied to clipboard
| Challenge: | a masked language model is used to train a model to predict subsets of mangled words . a parallel decoding algorithm can be used to generate translations in a constant number of iterations. |
| Approach: | They propose a model and a parallel decoding algorithm which train a machine to predict any subset of target words . they introduce conditional masked language models (CMLMs) which are trained with a mangled language model objective . |
| Outcome: | The proposed model improves state-of-the-art performance levels for non-autoregressive and parallel decoding models by over 4 BLEU on average. |
Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach (D19-1)
Copied to clipboard
| Challenge: | Recent work on image captioning has made impressive progress . however, the results are limited and the model is difficult to train . |
| Approach: | They propose a semi-supervised framework for training an image captioning model by assigning pseudo-labels to unpaired samples via Generative Adversarial Networks. |
| Outcome: | The proposed framework is compared to baselines when the number of paired samples is scarce. |
Choosing What to Mask: More Informed Masking for Multimodal Machine Translation (2023.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved remarkable results on several NLP tasks. |
| Approach: | They propose three new masking strategies for cross-lingual visual pre-training that focus on learning different linguistic patterns. |
| Outcome: | The proposed methods outperform the baseline model and achieve state-of-the-art accuracy on the Portuguese-English MMT task. |
Revealing the Attention Floating Mechanism in Masked Diffusion Models (2026.findings-acl)
Copied to clipboard
Xin Dai, Pengcheng Huang, Zhenghao Liu, Shuo Wang, Yukun Yan, Chaojun Xiao, Yu Gu, Ge Yu, Maosong Sun
| Challenge: | Masked diffusion models (MDMs) leverage bidirectional attention and a denoising process. |
| Approach: | They investigate the attention behaviors of Masked diffusion models by revealing the phenomenon of Attention Floating. |
| Outcome: | The proposed model doubles the performance of autoregressive models in knowledge-intensive tasks. |
Pretrained Image-Text Models are Secretly Video Captioners (2025.naacl-short)
Copied to clipboard
| Challenge: | Current video captioning methods often incorporate intricate designs tailored to video inputs. |
| Approach: | They adapt an image-based captioning model to address dynamic video sequences without modifications. |
| Outcome: | The proposed model outperforms specialised captioning systems on major benchmarks. |
Using Visual Feature Space as a Pivot Across Languages (2020.findings-emnlp)
Copied to clipboard
| Challenge: | We show that models trained to generate textual captions in more than one language can leverage their jointly trained feature space during inference to pivot across languages. |
| Approach: | They show that models trained to generate captions in more than one language can leverage their jointly trained feature space during inference to pivot across languages. |
| Outcome: | The proposed approach improves quality of captions in German and English by leveraging captions from a second language. |
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving model interpretability require prior information or human annotations as additional inputs. |
| Approach: | They propose a variational word mask method to automatically learn task-specific important words and reduce irrelevant information on classification, which ultimately improves model interpretability. |
| Outcome: | The proposed method improves model prediction accuracy and interpretability on seven datasets. |