| Challenge: | Existing scripts for everyday tasks are presented in a linear manner, which does not reflect the flexibility displayed by people executing tasks in real life. |
| Approach: | They propose to use loosely aligned videos to train a non-sequential graph script induction task by using a multimodal framework to ground procedural videos to WikiHow textual steps. |
| Outcome: | The proposed model outperforms the WikiHow linear baseline by 48.76% . it can predict future steps given a partial step sequence and generate explicit graph scripts . |
Similar Papers
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on temporal sentence grounding rely on expensive video-query paired annotations . despite this, there are no ground-truth annotations in the current work . |
| Approach: | They propose to use paired video-query and segment boundary annotations to generate temporal sentence grounding without training. |
| Outcome: | The proposed model outperforms existing unsupervised methods and beats supervised ones on two challenging datasets. |
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020.emnlp-main)
Copied to clipboard
| Challenge: | a representative pretraining model is fit to a diverse YouTube8M dataset . a priori, this domain is relatively easy for instructional videos . |
| Approach: | They fit a representative pretraining model to a YouTube8M dataset and examine its success and failure cases. |
| Outcome: | The proposed model can be trained on more diverse video corpora and achieve high performance on many video understanding tasks. |
Benchmarking Hierarchical Script Knowledge (N19-1)
Copied to clipboard
| Challenge: | Understanding procedural language requires reasoning about hierarchical and temporal relations between events. |
| Approach: | They propose a hierarchical script learning dataset and a cloze task to match video captions with missing procedural details. |
| Outcome: | The proposed model matches video captions with missing procedural details to find out if they can understand the language. |
Constructing Procedural Graphs with Multiple Dependency Relations: A New Dataset and Baseline (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to structure procedural knowledge focus on representing descriptive knowledge but ignore another commonsense knowledge-Procedural Knowledge. |
| Approach: | They propose to generate flow graphs from procedural documents by extracting sequential dependency between sentences and missing two important dependencies in procedural document. |
| Outcome: | The proposed method can generate flow graphs from unstructured documents with syntactic information and discourse structures. |
Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn effective alignment between vision and language features are insufficient in practice due to complicated multi-step reasoning. |
| Approach: | They propose an iterative alignment network which iterates inter- and intra-modal features within multiple steps for more accurate grounding. |
| Outcome: | The proposed model performs better than the state-of-the-arts on three challenging benchmarks. |
Unsupervised Semantic Frame Induction using Triclustering (P18-2)
Copied to clipboard
| Challenge: | Recent work on frame-semantics has enabled the development of wide-coverage frame parsers using supervised learning. |
| Approach: | They propose to use dependency triples to perform unsupervised frame induction on a Web-scale corpus. |
| Outcome: | The proposed approach performs state-of-the-art on a FrameNet-derived dataset and performs on par with competitive methods on . verb class clustering task. |
Structured Self-Supervised Pretraining for Commonsense Knowledge Graph Completion (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches focus on generating concepts that have direct and obvious relationships with existing concepts and lack an ability to generate unobvious concepts. |
| Approach: | They propose a general graph-to-paths pretraining framework that leverages high-order structures in CKGs to capture high-level relationships between concepts. |
| Outcome: | The proposed framework can capture high-order relationships between concepts in four special cases: long path, path-to-path, router, and graph-node-path. |
Modeling Event-Pair Relations in External Knowledge Graphs for Script Reasoning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods focus on graph triples with event overlap, but ignore more supportive triples . Script reasoning relies on understanding the relationship between two events . |
| Approach: | They propose a model to learn the inferential relations between events from the whole eventuality KG . they propose 'script adapter' to extend the model to infer the associated relations between an event chain and a subsequent event candidate. |
| Outcome: | The proposed model is compared with baselines using external KG or not on a script reasoning task. |
Relevant CommonSense Subgraphs for “What if...” Procedural Reasoning (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge graphs and commonsense are used to learn causal reasoning over procedural text. |
| Approach: | They propose a multi-hop graph reasoning model to efficiently extract a commonsense subgraph with the most relevant information from a large knowledge graph and predict the causal answer by reasoning over the representations obtained from the commonsen subgraph and contextual interactions between the questions and context. |
| Outcome: | The proposed model achieves state-of-the-art on WIQA benchmark and is comparable to previous models. |
proScript: Partially Ordered Scripts Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Scripts represent structured commonsense knowledge about prototypical events in everyday situations/scenarios such as bake a cake. |
| Approach: | They collect 6.4k crowdsourced partially ordered scripts and develop models that combine language generation and graph structure prediction to generate scripts. |
| Outcome: | The proposed models perform well on two tasks: edge prediction and script generation. |