| Challenge: | Existing methods to generate video descriptions fail to mention objects and actions in videos . |
| Approach: | They propose an LSTM-based sequence-to-sequence model with semantic attention mechanism for video description generation that includes external fine-grained visual information. |
| Outcome: | The proposed model can selectively focus on external fine-grained visual information and have a better quality of video descriptions. |
Similar Papers
End-to-end Dense Video Captioning as Sequence Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for dense video captioning use a two-stage generative process . but, more complex tasks are not able to fully utilize this powerful paradigm . |
| Approach: | They propose to model two subtasks of dense video captioning as one sequence generation task and predict the events and the corresponding descriptions. |
| Outcome: | Experiments on YouCook2 and ViTT show that the proposed model can be used on any video platform. |
Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing (P18-1)
Copied to clipboard
| Challenge: | Existing methods for semantic parsing are difficult to design and learn, especially in wideopen domains. |
| Approach: | They propose a neural semantic parsing approach which models semantic par- sing as an end-to-end semantic graph generation process. |
| Outcome: | The proposed model achieves state-of-the-art performance on Overnight dataset and gets competitive performance on Geo and Atis datasets. |
A hierarchical approach to vision-based language generation: from simple sentences to complex natural language (2020.coling-main)
Copied to clipboard
| Challenge: | Automating video to language translation is a challenging problem, but it is unclear what the neural network learns . |
| Approach: | They propose a hierarchical approach to automatically describing videos in natural language . they propose generating video descriptions as sequences of simple sentences followed by a more complex and fluent description in natural languages. |
| Outcome: | The proposed approach generates video descriptions as sequences of simple sentences, followed by a more complex and fluent description in natural language. |
S2SPMN: A Simple and Effective Framework for Response Generation with Relevant Information (D18-1)
Copied to clipboard
| Challenge: | Existing work on how to generate relevant and informative responses is focusing on how dialogue systems generate information from large dialogue corpus. |
| Approach: | They propose to use dialogue corpus to generate relevant responses by using prototypes to extract semantic information from PMN. |
| Outcome: | The proposed model outperforms classical and strong baseline models in generating relevant and informative responses. |
Incorporating Background Knowledge into Video Description Generation (D18-1)
Copied to clipboard
| Challenge: | Existing methods for video captioning focus on generating generic descriptions that lack contextual knowledge. |
| Approach: | They propose a method that uses video meta-data to retrieve topically related news documents for a video and extracts the events and named entities from these documents. |
| Outcome: | The proposed model is based on a news video dataset and is evaluated on it. |
A Graph-to-Sequence Model for AMR-to-Text Generation (P18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic formalism that encodes the meaning of a sentence as a rooted, directed graph. |
| Approach: | They propose a neural graph-to-sequence model that leverages LSTM to encode a linearized AMR structure. |
| Outcome: | The proposed model outperforms existing methods on a benchmark. |
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models for story generation suffer from repetition, logic conflicts, and lack of long-range coherence . |
| Approach: | They propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories by multi-task learning. |
| Outcome: | The proposed model can generate more reasonable stories than state-of-the-art models, compared with existing models, showing that it can capture useful semantic and syntactic features. |
Fine-grained Contrastive Learning for Definition Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Recent pre-trained transformer-based definition generation models lack effective representation learning to contain full semantic components of the given word, leading to under-specific definitions. |
| Approach: | They propose a novel contrastive learning method that encourages the model to capture more detailed semantic representations from the definition sequence encoding. |
| Outcome: | The proposed method could generate more specific definitions compared with state-of-the-art models. |
Controlling the Focus of Pretrained Language Generation Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing mechanisms to control the model's focus are not available for pretrained transformer-based language generation models. |
| Approach: | They propose to augment a pretrained model with trainable "focus vectors" that are directly applied to the model's embeddings while the model itself is kept fixed. |
| Outcome: | The proposed model is able to generate relevant outputs from user-selected highlights while keeping the model fixed. |
Grouped-Attention for Content-Selection and Content-Plan Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent neural data-to-text generation models explicitly learn content-plan given a set of attributes as input. |
| Approach: | They propose a neural content-planner that captures local and global contexts . they use a token-level attention constrained within each input attribute . |
| Outcome: | The proposed model outperforms competitors by 4.92%, 4.70%, and 16.56% on real-world datasets. |