Papers by Hammad Ayyubi
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models (2023.findings-emnlp)
Copied to clipboard
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing approaches to decompose VL reasoning rely on domain-specific sub-question decomposing models. |
| Approach: | They propose a framework that iteratively decomposes VL reasoning using large language models. |
| Outcome: | The proposed framework outperforms existing models on multiple VL reasoning tasks. |
Weakly-Supervised Temporal Article Grounding (2022.emnlp-main)
Copied to clipboard
Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, Shih-Fu Chang
| Challenge: | Existing VG models make unrealistic assumptions about how to ground video segments . a recent study has shown that video grounding can be useful for downstream applications . |
| Approach: | They propose a new task: Weakly-Supervised temporal Article Grounding (WSAG) given an article and a relevant video, WSAG aims to localize all "groundable" sentences to the video. |
| Outcome: | The proposed method is simple but effective, and it can be used in real-world applications. |
DELOC: Document Element Localizer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to edit PDFs using natural language are ineffective at grounding the edit location effectively. |
| Approach: | They propose a system to ground PDF edit request spatially using a model to predict the edit location in the PDF. |
| Outcome: | The proposed system outperforms existing Multimodal Large Language Models and specialized models on DocEdit. |
VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)
Copied to clipboard
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang
| Challenge: | Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations. |
| Approach: | They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models . |
| Outcome: | The proposed approach is effective across three video captioning models. |
PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction (2025.findings-naacl)
Copied to clipboard
| Challenge: | PuzzleGPT is a modular and iterative puzzlesolving method for predicting time and location from images. |
| Approach: | They propose to formalize this ability into core skills and implement it using different modules in an expert pipeline called PuzzleGPT. |
| Outcome: | The proposed method outperforms large VLMs and finetuned models on TARA and WikiTilo and rivals or surpasses finetuned models. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |