Papers by Marcus Rohrbach
A Dataset for Telling the Stories of Social Media Videos (D18-1)
Copied to clipboard
| Challenge: | Existing datasets focused on pre-selected human activities, whereas social media videos contain a great diversity of topics. |
| Approach: | They propose a large-scale dataset for video description as a new challenge for multi-sentence video description. |
| Outcome: | The proposed dataset contains 20k videos with 123k sentences, temporally aligned to the video. |
Predicting Implicit Arguments in Procedural Video Instructions (2025.acl-long)
Copied to clipboard
| Challenge: | Prior SRL benchmarks often miss implicit arguments, leading to incomplete understanding. |
| Approach: | They propose a dataset that necessitates inferring implicit and explicit arguments from contextual information in multimodal cooking procedures. |
| Outcome: | The proposed dataset achieves a 17% relative improvement in F1-score for what-implicit and a 14.7% improvement for where/with-implicative semantic roles over GPT-4o. |
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication (P19-1)
Copied to clipboard
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, Devi Parikh
| Challenge: | a goal-driven collaborative drawing task combines language, perception, and actions in a partially observable environment . et al., 1990: 138K messages exchanged between human players. |
| Approach: | They propose a goal-driven collaborative task that combines language, perception, and action . they collect a clip art dataset and use it to build an image-drawing game between two agents . |
| Outcome: | The proposed task integrates language, perception, and action in a virtual world . it is based on a dataset of 10K dialogs and 138K messages exchanged between humans . |
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (N19-1)
Copied to clipboard
| Challenge: | Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image. |
| Approach: | They construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset and use it to benchmark performance of standard visual dialog models. |
| Outcome: | The proposed model is based on a large diagnostic dataset for studying multi-round reasoning in visual dialog. |
VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating AFC systems are limited in terms of task scope, modalities, domain, language diversity, realism, or coverage of misinformation types. |
| Approach: | They propose to use Verified Theses and Statements (VeriTaS) to evaluate AFC systems that are static and subject to data leakage as claims enter pretraining corpora. |
| Outcome: | The proposed system is robust under large-scale pretraining of foundation models and can be updated in the future. |