Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on improving performance with the assumption of independently identical data distribution while ignoring out-of-distribution data. |
| Approach: | They propose a scene-robust NLVL problem and a generalizable framework to learn a robust model. |
| Outcome: | The proposed model learns generalizable domain-invariant representations by alignment and decomposition. |
Similar Papers
Natural Language Video Localization with Learnable Moment Proposals (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video moment localization have poor performance due to predefined rules. |
| Approach: | They propose a model with a fixed set of learnable moment proposals with 'border-aware loss' they propose to localize the video moment corresponding to the query by locating the start and end timestamps in an untrimmed video. |
| Outcome: | The proposed model outperforms state-of-the-art models on two challenging benchmarks. |
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to visual-language understanding lack unified tokenization for images and videos . lack of unified visual representations makes it difficult to learn multi-modal interactions from poor projection layers. |
| Approach: | They propose to unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. |
| Outcome: | The proposed model outperforms Video-ChatGPT on image benchmarks and on 9 image benchmark benchmarks. |
WSLLN:Weakly Supervised Natural Language Localization Networks (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn correspondence between visual segments and texts require temporal coordinates for training, which leads to high costs of annotation. |
| Approach: | They propose weakly supervised language localization networks to detect events in untrimmed videos . they train with only video-sentence pairs without accessing to temporal locations of events . |
| Outcome: | Experiments on ActivityNet Captions and DiDeMo show that WSLLN performs state-of-the-art. |
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Deep neural networks excel at learning from labeled data, but learning from unlabeled data remains a challenge. |
| Approach: | They review neural unsupervised domain adaptation techniques which do not require labeled target domain data. |
| Outcome: | The proposed techniques are more challenging yet widely applicable. |
Span-based Localizing Network for Natural Language Video Localization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to NLVL are either ranking tasks or regressing the target video span. |
| Approach: | They propose a video span localizing network to solve a natural language video localization task using a span-based QA approach. |
| Outcome: | The proposed network outperforms the state-of-the-art methods on three benchmark datasets. |
What’s in a Domain? Learning Domain-Robust Text Representations using Adversarial Training (N18-2)
Copied to clipboard
| Challenge: | a key roadblock is application to new domains, unseen in training. |
| Approach: | They propose a method to optimise in- and out-of-domain accuracy by combing domain-specific and domain-general components with adversarial training for domain. |
| Outcome: | The proposed method improves on domain adaptation and domain-adversarial training. |
Learning to Model Multimodal Semantic Alignment for Story Visualization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Story visualization aims to generate sequence of images to narrate each sentence in a multi-sentence story . current methods face semantic misalignment because of their fixed architecture and diversity of input modalities . |
| Approach: | They propose to use a GAN-based generative model to match semantic levels between text and image representations to solve the semantic misalignment problem. |
| Outcome: | Experiments show that the proposed approach improves image quality and story consistency compared with state-of-the-art methods. |
Deep Generative Model for Joint Alignment and Word Representation (N18-1)
Copied to clipboard
| Challenge: | EmbedAlign model embeds words in their complete observed context and learns by marginalisation of latent lexical alignments. |
| Approach: | They exploit translation as a distributional context and embed words as posterior probability densities, rather than point estimates, which allows them to compare words in context using a measure of overlap between distributions. |
| Outcome: | The proposed model performs on a range of lexical semantics tasks and achieves competitive results on benchmarks including natural language inference, paraphrasing, and text similarity. |
Generating Structured Pseudo Labels for Noise-resistant Zero-shot Video Sentence Localization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot pipelines generate event proposals and then generate a pseudo query for each event proposal. |
| Approach: | They propose a Structure-based Pseudo Label generation (SPL) that generates free-form interpretable pseudo queries before constructing query-dependent event proposals. |
| Outcome: | The proposed method learns with only video data without any annotation . it generates free-form interpretable pseudo queries before constructing query-dependent event proposals . |
MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction (2023.acl-long)
Copied to clipboard
| Challenge: | Natural language video localization (NLVL) aims to localize a temporal moment from an untrimmed video that semantically corresponds to a given text query. |
| Approach: | They propose a proposal-based solution that generates proposals and selects the best matching proposal. |
| Outcome: | The proposed solution is faster than existing approaches on three public datasets. |