| Challenge: | Prior SRL benchmarks often miss implicit arguments, leading to incomplete understanding. |
| Approach: | They propose a dataset that necessitates inferring implicit and explicit arguments from contextual information in multimodal cooking procedures. |
| Outcome: | The proposed dataset achieves a 17% relative improvement in F1-score for what-implicit and a 14.7% improvement for where/with-implicative semantic roles over GPT-4o. |
Similar Papers
PizzaCommonSense: A Dataset for Commonsense Reasoning about Intermediate Steps in Cooking Recipes (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Understanding procedural texts is essential for enabling machines to follow instructions and reason about tasks. |
| Approach: | They propose a corpus of cooking recipes enriched with descriptions of intermediate steps . they propose enabling machines to follow instructions and reason about tasks . |
| Outcome: | The proposed model achieves only 26% human-evaluated preference for generations . pizzaCommonsense is a benchmark for the reasoning capabilities of large language models . |
A vision-grounded dataset for predicting typical locations for verbs (L18-1)
Copied to clipboard
| Challenge: | Existing models for inferring location from text are often underestimating the probability of the most typical role fillers. |
| Approach: | They propose a dataset which contains thematic fit judgments for 2,000 verb/location pairs. |
| Outcome: | The proposed dataset can be used to evaluate text-based, vision-based or multimodal inference systems for the typicality of an event's location. |
Order-Based Pre-training Strategies for Procedural Text Understanding (2024.naacl-short)
Copied to clipboard
| Challenge: | Procedural text is difficult to understand due to the changing attributes of entities in the context. |
| Approach: | They propose sequence-based pre-training methods to enhance procedural understanding in natural language processing by using ordered instructions to guide individuals through a task. |
| Outcome: | The proposed methods improve on two datasets in the datasets NPN-Cooking and ProPara domains respectively. |
Benchmarking Hierarchical Script Knowledge (N19-1)
Copied to clipboard
| Challenge: | Understanding procedural language requires reasoning about hierarchical and temporal relations between events. |
| Approach: | They propose a hierarchical script learning dataset and a cloze task to match video captions with missing procedural details. |
| Outcome: | The proposed model matches video captions with missing procedural details to find out if they can understand the language. |
PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on whether large language models are capable of planning or executing plans. |
| Approach: | They propose an abductive reasoning task using wikiHow to test the effectiveness of small models over large models. |
| Outcome: | The proposed task demonstrates the effectiveness of small models over large models in most scenarios. |
A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | a web-based algorithm can be used to align instructions for different tasks . video instructions can be noisy and contain far more information than textual instructions. |
| Approach: | They propose an algorithm that learns pairwise alignments between different recipes . they then use a graph algorithm to derive a joint alignment between multiple video and text recipes based on the same recipe. |
| Outcome: | The proposed algorithm learns pairwise alignments between different recipes for the same dish. |
Grounded Multimodal Procedural Entity Recognition for Procedural Documents: A New Dataset and Baseline (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to extract procedural knowledge from documents focus on text-only settings, which is insufficient for entity disambiguation. |
| Approach: | They propose a model to detect the entity and the corresponding bounding box groundings in images. |
| Outcome: | The proposed model detects the entity and the corresponding bounding box groundings in image (i.e., visual entities) it is based on a dataset of a WikiHow 1 and EHow 2 document and the results are compared with existing models. |
CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained vision-language models perform well in cross-modal tasks, including referring expression comprehension. |
| Approach: | They propose a method that enables VL models to reason with implicit text . they propose to use a dataset to align the text with objects in the images . |
| Outcome: | The proposed method improves performance 37.94% on referring expression comprehension task. |
Grounded Semantic Role Labelling from Synthetic Multimodal Data for Situated Robot Commands (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing symbolic parsers lack flexibility to operate in complex, dynamic environments. |
| Approach: | They propose a framework that combines frame semantics with perceptual grounding to enable robots to interpret commands via multimodal logical forms. |
| Outcome: | The proposed framework produces over 11,000 image-command pairs and lowers the cost of manual parsers. |
Learning Semantic Role Labeling from Compatible Label Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has shown that cross-task interaction helps, but only explored multitask learning so far. |
| Approach: | They propose a framework that jointly models VerbNet and PropBank labels as one sequence and enforcing Semlink constraints during decoding improves the overall F1 . |
| Outcome: | The proposed model outperforms the prior best in-domain model by 3.5 (VerbNet) and 0.8 (PropBank). |