Papers by Dong-Jin Kim
Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach (D19-1)
Copied to clipboard
| Challenge: | Recent work on image captioning has made impressive progress . however, the results are limited and the model is difficult to train . |
| Approach: | They propose a semi-supervised framework for training an image captioning model by assigning pseudo-labels to unpaired samples via Generative Adversarial Networks. |
| Outcome: | The proposed framework is compared to baselines when the number of paired samples is scarce. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |
IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-only training methods overlook the modality gap between using text data during training and employing images during inference. |
| Approach: | They propose a novel approach that aligns text features with visually relevant features to mitigate the modality gap between using text data during training and employing images during inference. |
| Outcome: | The proposed method outperforms the state-of-the-art methods in image captioning and video captioning by a significant margin compared to training with text data. |
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes end-to-end models but suffer from limitations . prior work focused on generating captions from long video streams . |
| Approach: | They propose a saliency-aware framework that localizes events and generates captions for each event. |
| Outcome: | The proposed framework achieves state-of-the-art results on YouCook2 and ViTT. |