Style-Aware Contrastive Learning for Multi-Style Image Captioning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing multi-style image captioning methods focus on visual content and style . existing methods overlook the relationship between linguistic style and visual content. |
| Approach: | They propose a style-aware visual encoder with contrastive learning to mine potential visual content relevant to style and a triplet contrast objective to distinguish whether the image, style and caption matched. |
| Outcome: | The proposed method achieves state-of-the-art performance and an extensive analysis to verify its effectiveness. |
Similar Papers
StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples (2025.naacl-long)
Copied to clipboard
Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, Chris Callison-Burch
| Challenge: | Existing methods for embedding text are limited by the imperfect nature of data acquired under such assumptions. |
| Approach: | They propose a new approach to training stronger content-independent style embeddings using a synthetic dataset of near-exact paraphrases with controlled style variations. |
| Outcome: | The proposed model outperforms existing methods in real-world benchmarks and outperformed leading style representations in downstream applications. |
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval (2023.acl-long)
Copied to clipboard
| Challenge: | Contrastive learning is the dominant paradigm for learning text representations from parallel text, but finding negative examples can be expensive in terms of compute or manual effort. |
| Approach: | They propose a generative model for learning multilingual text embeddings which encourages source separation in multilingual contexts by an approximation. |
| Outcome: | The proposed model outperforms both a strong contrastive and generative baseline on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval. |
Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task (D19-64)
Copied to clipboard
| Challenge: | Existing methods to learn multimodal multilingual embeddings for text and image retrieval tasks are limited to English. |
| Approach: | They propose a new approach to learn multimodal multilingual embeddings for matching images and captions in two languages by combing two existing objective functions and adapting alignment between existing languages. |
| Outcome: | The proposed model achieves state-of-the-art in retrieval and caption-caption tasks while adapting existing language alignments. |
Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task (D19-66)
Copied to clipboard
| Challenge: | Existing methods to learn multimodal multilingual embeddings for text and image retrieval tasks are limited to English. |
| Approach: | They propose a new approach to learn multimodal multilingual embeddings for matching images and captions in two languages by combing two existing objective functions and adapting alignment between existing languages. |
| Outcome: | The proposed model achieves state-of-the-art in retrieval and caption-caption tasks while adapting existing language alignments. |
Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning (2025.acl-short)
Copied to clipboard
| Challenge: | Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. |
| Approach: | They propose to use image captions to implicitly align text representations between languages to make them usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
| Outcome: | The proposed approach is usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text style transfer require style-labeled training data, but use only labeled data at inference time. |
| Approach: | They propose a method that uses readily-available unlabeled text to train style transfer . they use a style vector to condition a decoder to perform style transfer using unlabelled text . |
| Outcome: | The proposed method is competitive on sentiment transfer, even compared to models trained fully on labeled data. |
FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on multimodal fashion tasks has been limited by the data in individual benchmarks or has leveraged generic vision-and-language pre-training but have not taken advantage of the characteristics of fashion data. |
| Approach: | They propose a fashion-specific pre-training framework based on weakly-supervised triplets constructed from fashion image-text pairs. |
| Outcome: | The proposed framework is based on weakly-supervised triplets constructed from fashion image-text pairs and is competitive on a diverse set of fashion tasks. |
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Transductive Learning for Unsupervised Text Style Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for style transfer are based on an inductive learning approach, which represents the style as embeddings, decoder parameters, or discriminator parameters and directly applies these general rules to the test cases. |
| Approach: | They propose a retrieval-based context-aware style representation that involves top-K relevant sentences in the target style in the transfer process. |
| Outcome: | The proposed method outperforms several strong baselines and is general and effective to the task of unsupervised style transfer. |
A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles” (2021.eacl-main)
Copied to clipboard
| Challenge: | Fig. 1 shows how style-transferred multi-modal features can be used in sentiment analysis and emotion recognition. |
| Approach: | They propose to use adaptive normalization to impose style onto text to learn richer representations for multi-modal utterances. |
| Outcome: | The proposed model achieves performance on par with state-of-the-art but using less than a third of the model parameters. |