Papers by Ruoyao Wang
Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text game agents are often modeled using reinforcement learning, but their performance is limited. |
| Approach: | They propose a self-supervised behavior cloning transformer for text games . they explore trajectories that lead to reward within the games and then train small models . their approach consistently uncovers generalizable training data, achieving 90% performance of supervised systems across three benchmark text games. |
| Outcome: | The proposed model achieves 90% performance on three text games. |
FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework (2022.acl-long)
Copied to clipboard
Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan Stroud, Rada Mihalcea
| Challenge: | Existing video understanding evaluation frameworks that use fill-in-the-blanks do not reflect real-world tasks. |
| Approach: | They propose to use fill-in-the-blanks as a video understanding evaluation framework and introduce a novel dataset that collects multiple perspectives on the same video. |
| Outcome: | The proposed framework does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation; (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. |
Behavior Cloned Transformers are Neurosymbolic Reasoners (2023.eacl-main)
Copied to clipboard
| Challenge: | In injecting actions from symbolic modules into the action space of a behavior cloned transformer agent increases performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22% . contemporary agents struggle on tasks such as navigation, admetic and other tasks that humans make use of external tools. |
| Approach: | They propose to inject actions from symbolic modules into the action space of a behavior cloned transformer agent to increase performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22% . |
| Outcome: | The proposed method improves performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by 22%, allowing an agent to reach the highest possible performance on unseen games. |
ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023.emnlp-main)
Copied to clipboard
| Challenge: | We show that language models can generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. |
| Approach: | They propose a corpus of 32 reasoning-focused text games expressed as hundreds of lines of Python code to facilitate this task. |
| Outcome: | The proposed games can generate runnable games on unseen topics in 28% of cases. |
ScienceWorld: Is your Agent Smarter than a 5th Grader? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models cannot reason about or explain learned science concepts in novel contexts, despite transformer-based progress in question-answering and scientific text processing . |
| Approach: | They propose a benchmark to test agents’ scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curriculum. |
| Outcome: | The proposed model outperforms a model trained for 100k steps in a standard elementary school science curriculum. |
Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Science (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems generate hypothesis, run experiments, analyze data, and write or review papers, but they are costly and impractical. |
| Approach: | They propose a challenge dataset for determining the feasibility of hypotheses framed as claims and a temporally-filtered claim verification task using backtesting to test the validity of claims. |
| Outcome: | The proposed model performs well on retrieval augmented generation and code generation while performing 50% of the task. |
LifeQA: A Real-life Dataset for Video Question Answering (2020.lrec-1)
Copied to clipboard
Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, Rada Mihalcea
| Challenge: | Existing video question answering datasets consist of movies and TV shows, but they are not representative of our day-to-day lives. |
| Approach: | They propose a benchmark dataset for video question answering that focuses on day-to-day situations. |
| Outcome: | The proposed dataset analyzes the challenging but realistic aspects of LifeQA . it consists of video clips and over 2.3k multiple-choice questions . |
Can Language Models Serve as Text-Based World Simulators? (2024.acl-short)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have pointed towards an alternative approach by leveraging the huge amount of knowledge contained in their pre-training datasets. |
| Approach: | They build and use a benchmark to quantify how well text-based simulators can serve as text-driven world simulators. |
| Outcome: | The proposed benchmark aims to quantify how well language models can serve as world simulators. |