| Challenge: | Existing techniques for weakly-supervised spatio-temporally grounding natural sentence in video are lacking . |
| Approach: | They propose a weakly-supervised task for spatially grounding sentences in video . they extract instances from video and encode them using attentive interactor . results demonstrate superiority of their proposed task over baseline approaches . |
| Outcome: | The proposed model outperforms baseline approaches in a weakly-supervised task . it can characterize reliable instance-sentence pairs and penalize unreliable ones . |
Similar Papers
Temporally Grounding Natural Sentence in Video (D18-1)
Copied to clipboard
| Challenge: | Existing methods for grounding natural sentences in video are limited to a single pass. |
| Approach: | They propose a Temporal GroundNet (TGN) method that captures the evolving fine-grained frame-by-word interactions between video and sentence to ground the segment corresponding to the sentence. |
| Outcome: | The proposed method significantly improves on the state-of-the-art methods on three public datasets and shows significant improvements in performance. |
Weakly-Supervised Spoken Video Grounding via Semantic Interaction Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work on spoken video grounding challenges extracting semantic information from speech . previous studies focused on textual queries, but recent work focuses on spoken queries . |
| Approach: | They propose a framework for weakly-supervised spoken video grounding to represent cross-modal semantics without expensive temporal annotations. |
| Outcome: | The proposed framework is more efficient than existing methods. |
Weakly-Supervised Temporal Article Grounding (2022.emnlp-main)
Copied to clipboard
Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, Shih-Fu Chang
| Challenge: | Existing VG models make unrealistic assumptions about how to ground video segments . a recent study has shown that video grounding can be useful for downstream applications . |
| Approach: | They propose a new task: Weakly-Supervised temporal Article Grounding (WSAG) given an article and a relevant video, WSAG aims to localize all "groundable" sentences to the video. |
| Outcome: | The proposed method is simple but effective, and it can be used in real-world applications. |
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on temporal sentence grounding rely on expensive video-query paired annotations . despite this, there are no ground-truth annotations in the current work . |
| Approach: | They propose to use paired video-query and segment boundary annotations to generate temporal sentence grounding without training. |
| Outcome: | The proposed model outperforms existing unsupervised methods and beats supervised ones on two challenging datasets. |
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing video QA datasets only contain QA pairs without labels for key clips or regions needed to answer the question. |
| Approach: | They propose a framework that grounds evidence in both spatial and temporal domains to answer questions about videos using bounding boxes. |
| Outcome: | The proposed framework can produce interpretable spatio-temporal attention visualizations. |
Towards Parameter-Efficient Integration of Pre-Trained Language Models In Temporal Video Grounding (2023.findings-acl)
Copied to clipboard
Erica Kido Shimomoto, Edison Marrese-Taylor, Hiroya Takamura, Ichiro Kobayashi, Hideki Nakayama, Yusuke Miyao
| Challenge: | Recent studies have improved query inputs with pre-trained language models, but the effects of this integration are unclear. |
| Approach: | They propose to integrate query sentences with pre-trained language models to train TVG models. |
| Outcome: | The proposed model integrates query sentences with pre-trained language models at cost of more expensive training. |
Learning to Focus on the Foreground for Temporal Sentence Grounding (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for temporal sentence grounding do not capture subtle details of small objects. |
| Approach: | They propose a detection-free framework for temporal sentence grounding that learns to locate foreground regions related to the query in consecutive frames. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on three challenging datasets. |
Video-Text Prompting for Weakly Supervised Spatio-Temporal Video Grounding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods extract each candidate tube feature independently by cropping objects from video frame feature, discarding all contextual information such as position change and inter-entity relationship. |
| Approach: | They propose to use video-text prompts to construct candidate feature instead of cropping tube region from feature map . they also propose negative contrastive samples whose candidate object is erased instead of being highlighted . |
| Outcome: | The proposed methods surpass existing weakly-supervised methods by a great margin . they draw visual markers over objects tubes as video prompts . |
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020.emnlp-main)
Copied to clipboard
| Challenge: | a representative pretraining model is fit to a diverse YouTube8M dataset . a priori, this domain is relatively easy for instructional videos . |
| Approach: | They fit a representative pretraining model to a YouTube8M dataset and examine its success and failure cases. |
| Outcome: | The proposed model can be trained on more diverse video corpora and achieve high performance on many video understanding tasks. |
Reasoning Step-by-Step: Temporal Sentence Localization in Videos via Deep Rectification-Modulation Network (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for temporal sentence localization in videos focus on visual content, but they are insufficient to model complex video contents. |
| Approach: | They propose a deep rectification-modulation network to correct attention misalignment . they use sentence information to capture frame-to-frame relation . |
| Outcome: | The proposed method achieves state-of-the-art performance on three public datasets. |