Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag (D19-1)
Copied to clipboard
| Challenge: | Existing models of video captioning use a network and semantics are mixed into one feature. |
| Approach: | They propose an Adaptive Semantic Guidance Network which instantiates whole video semantics to different POS-aware semantics with supervision of part of speech (POS) tag. |
| Outcome: | Extensive experiments show that the proposed model is more efficient than state-of-the-art models. |
Similar Papers
Grounding language acquisition by training semantic parsers using captioned videos (D18-1)
Copied to clipboard
| Challenge: | a new method for parsing sentences using captioned videos is being developed . we use video clips to ground the semantics of language, but without annotations . |
| Approach: | They develop a semantic parser that is trained in a grounded setting using captioned videos . they use a corpus of sentences paired with videos without other annotations to train it . |
| Outcome: | The proposed parser recovers the meaning of English sentences despite no annotations . learning a grounded semantic parsers can expand the range of data that parseurs can be trained on . |
VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)
Copied to clipboard
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang
| Challenge: | Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations. |
| Approach: | They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models . |
| Outcome: | The proposed approach is effective across three video captioning models. |
Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens (N18-1)
Copied to clipboard
| Challenge: | Recent work has used attention weights to visualize the focus of neural models in input data. |
| Approach: | They propose to use attention-based visualization techniques to infer token-level labels from a network trained only on sentence-level labeling. |
| Outcome: | The proposed approach outperforms gradient-based methods on four datasets and is expected to outperfect supervised methods. |
Incorporating Background Knowledge into Video Description Generation (D18-1)
Copied to clipboard
| Challenge: | Existing methods for video captioning focus on generating generic descriptions that lack contextual knowledge. |
| Approach: | They propose a method that uses video meta-data to retrieve topically related news documents for a video and extracts the events and named entities from these documents. |
| Outcome: | The proposed model is based on a news video dataset and is evaluated on it. |
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train models on unlabeled web videos are noisy and temporally misaligned . authors propose a method that adds captions and constrained attention loss to improve performance . |
| Approach: | They propose a method that adds captions from video frames as auxiliary text input to provide visual cues for learning better video and language associations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on video-and-language tasks . it adds captions and constrained attention loss to improve model performance . |
Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks (N18-2)
Copied to clipboard
| Challenge: | Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods. |
| Approach: | They propose to integrate semantic representations into neural machine translation by injecting a semantic bias into sentence encoders and achieving improvements in BLEU scores. |
| Outcome: | The proposed representations achieve better BLEU scores over the linguistic-agnostic and syntax-aware versions on the English–German language pair. |
Mask-Align: Self-Supervised Neural Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Word alignment is an important task in many natural language processing tasks. |
| Approach: | They propose a self-supervised word alignment model that takes advantage of the full context on the target side. |
| Outcome: | The proposed model outperforms previous unsupervised models and obtains state-of-the-art results on four language pairs. |
MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Prediction (2026.acl-long)
Copied to clipboard
| Challenge: | Recent research has attempted to transfer reinforcement learning paradigms to Video Large Language Models (MLLMs) but these methods lack explicit supervision for intrinsic temporal coherence and inter-frame correlations. |
| Approach: | They propose a novel post-training objective: Masked Video Prediction (MVP) that requires the model to reconstruct a masked continuous segment from a set of challenging distractors and employs Group Relative Policy Optimization (GRPO) with a fine-grained reward function to enhance the model's understanding of video context and temporal properties. |
| Outcome: | The proposed model improves video reasoning capabilities by reinforcing temporal reasoning and causal understanding. |
Pretrained Image-Text Models are Secretly Video Captioners (2025.naacl-short)
Copied to clipboard
| Challenge: | Current video captioning methods often incorporate intricate designs tailored to video inputs. |
| Approach: | They adapt an image-based captioning model to address dynamic video sequences without modifications. |
| Outcome: | The proposed model outperforms specialised captioning systems on major benchmarks. |
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes end-to-end models but suffer from limitations . prior work focused on generating captions from long video streams . |
| Approach: | They propose a saliency-aware framework that localizes events and generates captions for each event. |
| Outcome: | The proposed framework achieves state-of-the-art results on YouCook2 and ViTT. |