Papers by Po-Yao Huang
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines zero-shot cross-lingual transfer of vision-language models . we study multilingual text-to-video search in non-English languages without annotations . |
| Approach: | They propose a Transformer-based model that learns contextual multilingual multimodal embeddings . they propose 'zero-shot cross-lingual transfer' to improve multilingual search . |
| Outcome: | The proposed model outperforms baselines on multilingual text-to-video search and multilingual image search on VTT and VATEX. |
Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations (D19-1)
Copied to clipboard
| Challenge: | Recent studies have advanced learning VSE under the monolingual setup. |
| Approach: | They propose a model with diverse multi-head attention to learn grounded multilingual multimodal representations by leveraging visual object detection. |
| Outcome: | The proposed model performs well in German-Image and English-Image matching tasks and in the Semantic Textual Similarity task with English descriptions of visual content. |
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024.acl-long)
Copied to clipboard
| Challenge: | VoiceCraft is a token-infilling neural codec language model for speech editing and zero-shot text-to-speech evaluation. |
| Approach: | They introduce a token infilling neural codec language model that performs on speech editing and zero-shot text-to-speech tasks. |
| Outcome: | The proposed model outperforms previous models on speech editing and zero-shot text-to-speech tasks. |
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021.findings-acl)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
| Challenge: | Existing methods for multimodal video understanding are task-specific, limiting their use for retrieval-style end tasks. |
| Approach: | They propose a task-agnostic multimodal pre-training approach that can accept video or text input, or both, for a variety of end tasks. |
| Outcome: | The proposed approach outperforms existing methods on a wider range of tasks while maintaining separability. |
Generating Hashtags for Short-form Videos with Guided Signals (2023.acl-long)
Copied to clipboard
Tiezheng Yu, Hanchao Yu, Davis Liang, Yuning Mao, Shaoliang Nie, Po-Yao Huang, Madian Khabsa, Pascale Fung, Yi-Chia Wang
| Challenge: | Short-form video hashtag recommendation (SVHR) is a classification or ranking problem that selects hashtags from a set of limited candidates. |
| Approach: | They propose a short-form video hashtag recommendation task that better represents how hashtags are created naturally by retrieving relevant hashtags from a large-scale hashtag pool as extra guidance signals. |
| Outcome: | The proposed model outperforms strong classification baselines on two short-form video datasets and the guidance signals boost the performance by 8.11 and 2.17 absolute ROUGE-1 scores on average. |
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020.acl-main)
Copied to clipboard
| Challenge: | Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora. |
| Approach: | They propose to utilize visual content for disambiguation and promoting latent space alignment in unsupervised machine translation by using multimodal back-translation and pseudo visual pivoting. |
| Outcome: | The proposed model improves over state-of-the-art methods and generalizes well when images are not available at the testing time. |
Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)
Copied to clipboard
Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
| Challenge: | Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown. |
| Approach: | They propose an approach to edit and re-align alt-texts associated with images using human annotation. |
| Outcome: | The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks. |
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021.emnlp-main)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
| Challenge: | Recent work adopts a "pre-training + fine-tuning" approach for zero-shot transfer to end tasks without fine- tuning. |
| Approach: | They propose a contrastive approach to pre-train a transformer model for zero-shot video and text understanding without using any labels on downstream tasks. |
| Outcome: | The proposed model outperforms supervised approaches on downstream tasks and outperformed previous approaches. |