Papers by Jonathan Stroud
FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework (2022.acl-long)
Copied to clipboard
Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan Stroud, Rada Mihalcea
| Challenge: | Existing video understanding evaluation frameworks that use fill-in-the-blanks do not reflect real-world tasks. |
| Approach: | They propose to use fill-in-the-blanks as a video understanding evaluation framework and introduce a novel dataset that collects multiple perspectives on the same video. |
| Outcome: | The proposed framework does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation; (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. |
LifeQA: A Real-life Dataset for Video Question Answering (2020.lrec-1)
Copied to clipboard
Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, Rada Mihalcea
| Challenge: | Existing video question answering datasets consist of movies and TV shows, but they are not representative of our day-to-day lives. |
| Approach: | They propose a benchmark dataset for video question answering that focuses on day-to-day situations. |
| Outcome: | The proposed dataset analyzes the challenging but realistic aspects of LifeQA . it consists of video clips and over 2.3k multiple-choice questions . |