Papers with VTT
Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks (2023.findings-eacl)
Copied to clipboard
Pedro Rodriguez, Mahmoud Azab, Becka Silvert, Renato Sanchez, Linzy Labson, Hardik Shah, Seungwhan Moon
| Challenge: | Existing benchmarks for text-to-video retrieval are incomplete, resulting in false negatives . a recent state-of-the-art model gains 25% recall points, but this is not the case for TVR. |
| Approach: | They propose to retire video captioning datasets as TVR benchmarks . they propose to annotate and release additional caption-video pairs to mitigate this flaw . |
| Outcome: | The proposed method fails to accurately reflect reality, despite lack of purpose-built benchmarks. |
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for video captioning consider a sequence of frames and biases towards focused objects. |
| Approach: | They propose an Object-Oriented Non-Autoregressive approach to video captioning . it performs three steps: 1) identify the focused objects and predict their locations . 2) generate related attribute words and relation words of these focused objects to form a draft caption . |
| Outcome: | The proposed method achieves competitive results with the state-of-the-art methods but with higher diversity and faster inference speed. |
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in text-video retrieval neglect the crucial user perspective, leading to discrepancies between user queries and content retrieved. |
| Approach: | They propose a novel, training-free pipeline that leverages Large Language Models for iterative feedback learning. |
| Outcome: | Experimental results show that MERLIN significantly outperforms existing systems in video retrieval. |
Semi-Supervised Learning for Video Captioning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing video captioning algorithms are heavily dependent on supervised training data. |
| Approach: | They propose to train the video captioning model on labeled and unlabeled data jointly in a semi-supervised learning manner. |
| Outcome: | The proposed model outperforms state-of-the-art semi-supervised learning approaches on VATEX, MSR-VTT and MSVD datasets. |
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines zero-shot cross-lingual transfer of vision-language models . we study multilingual text-to-video search in non-English languages without annotations . |
| Approach: | They propose a Transformer-based model that learns contextual multilingual multimodal embeddings . they propose 'zero-shot cross-lingual transfer' to improve multilingual search . |
| Outcome: | The proposed model outperforms baselines on multilingual text-to-video search and multilingual image search on VTT and VATEX. |
MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for video retrieval rely on embedding-based full-corpus scanning, but there is a bottleneck in semantic asymmetry and computational redundancy. |
| Approach: | They propose a multi-agent framework that rethinks retrieval as cooperative reasoning . they parse raw videos into a structured semantic library, enabling explicit attribute-level indexing . |
| Outcome: | The proposed framework bridges the granularity mismatch gap by parsing raw videos into a structured semantic library . it employs a Logic-aware Debate mechanism with a strict veto protocol . the proposed framework achieves competitive performance without task-specific fine-tuning . |