What Action Causes This? Towards Naive Physical Action-Effect Prediction (P18-1)
Copied to clipboard
| Challenge: | a new task on naive physical action-effect prediction addresses the relationship between concrete actions and their effects on the state of the physical world as depicted by images. |
| Approach: | They propose a task that harnesses web image data to facilitate action-effect prediction. |
| Outcome: | The proposed approach harnesses web image data through distant supervision to facilitate learning for action-effect prediction. |
Similar Papers
Learning the Effects of Physical Actions in a Multi-modal Environment (2023.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are trained on large corpora of disembodied texts. |
| Approach: | They propose a multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). They extend an LLM to model latent representations of objects to better predict action outcomes in an environment. |
| Outcome: | The proposed model can capture commonsense when augmented with visual information and generalize and learn commonsensical reasoning better. |
Reasoning about Actions and State Changes by Injecting Commonsense Knowledge (D18-1)
Copied to clipboard
| Challenge: | Recent work has shown impressive progress in comprehending procedural text, but their predictions can be inconsistent or highly improbable. |
| Approach: | They propose to incorporate global constraints and bias reading with corpora-based preferences to improve the predicted effects of actions in a paragraph. |
| Outcome: | The proposed model significantly outperforms earlier models on a benchmark dataset for procedural text comprehension (+8% relative gain) it avoids nonsensical predictions that earlier models make, and it is more robust than previous models. |
Making Large Language Models into World Models with Precondition and Effect Knowledge (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are not inherently designed to model real-world dynamics, but can be induced to perform two critical world model functions: determining the applicability of an action based on a given world state and predicting the resulting world state upon action execution. |
| Approach: | They propose to use Large Language Models to model world states and preconditions . they validate that precondition and effect knowledge generated by LLMs aligns with human understanding of world dynamics . |
| Outcome: | The proposed model can predict valid actions and state transitions, thereby replicating existing models. |
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts (N18-2)
Copied to clipboard
| Challenge: | a novel approach to understanding narratives involves modelling the interaction between characters and actions . we propose role-playing games as a testbed for inferring interactions between characters in narratives . |
| Approach: | They propose role-playing games as a testbed for learning latent ties between characters and actions . they propose to combine character and action descriptions from online discussion forums . |
| Outcome: | The proposed model can capture interactions between characters and actions in narratives . it can predict actions better when character attributes are taken into account . |
Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text (D19-1)
Copied to clipboard
| Challenge: | XPAD is a new model that predicts actions' effects and their dependencies based on background knowledge . previous work on extracting sequences of actions from text has focused on identifying why they are the way they are . |
| Approach: | They propose a new model that biases effect predictions towards those that explain more of the actions in the paragraph and are more plausible with respect to background knowledge. |
| Outcome: | The proposed model outperforms existing systems on explaining actions by predicting dependencies while maintaining the performance on the original task in ProPara. |
Harry Potter and the Action Prediction Challenge from Natural Language (N19-1)
Copied to clipboard
| Challenge: | Using textual descriptions of scenes, we explore the challenge of action prediction from textual description. |
| Approach: | They propose a testbed to approximate whether text inference can be used to predict upcoming actions from textual descriptions of scenes. |
| Outcome: | The proposed model performs best for frequent actions and large scene descriptions, but logistic regression fails on infrequent actions. |
EvEntS ReaLM: Event Reasoning of Entity States via Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to model event implications fail to reason about the world, despite their knowledge of physical attributes. |
| Approach: | They propose to use a model prompting technique to prompt models of event implications by targeting their understanding of physical attributes. |
| Outcome: | The proposed model prompting technique is especially useful for unseen attributes or when only limited data is available. |
SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge Refinement (2025.acl-short)
Copied to clipboard
Runnan Fang, Xiaobin Wang, Yuan Liang, Shuofei Qiao, Jialong Wu, Zekun Xi, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen
| Challenge: | Using Large Language Models (LLMs)-based agents can enhance their understanding of environments and tasks. |
| Approach: | They propose a framework that allows agents to synthesize possible scenarios with multi-step action invocation within the action space and perform Monte Carlo Tree Search exploration to refine their action knowledge in the current environment. |
| Outcome: | The proposed framework synthesizes possible scenarios with multi-step action invocation within the action space and performs Monte Carlo Tree Search exploration to refine action knowledge in the current environment. |
Identifying Physical Object Use in Sentences (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior research has focused on learning the prototypical functions of physical objects . but many sentences refer to objects even when they are not used . |
| Approach: | They propose a task that determines whether a physical object mentioned in a sentence was used or likely will be used. |
| Outcome: | The proposed model exploits data augmentation methods and FrameNet to fine-tune a pre-trainedmodel. |
Improving Natural Language Interaction with Robots Using Advice (N19-1)
Copied to clipboard
| Challenge: | Recent studies focus on learning models for physically grounded language understanding tasks such as the blocks world domain. |
| Approach: | They propose a protocol for including advice, high-level observations about the task, which can help constrain the agent’s prediction. |
| Outcome: | The proposed approach can be extended to include advice, high-level observations about the task, and reduce the effort involved in supplying the advice. |