Papers by Vernon Toh
PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns (2024.findings-acl)
Copied to clipboard
| Challenge: | recognizing patterns and abstracting concepts are key to general intelligence, we show . state-of-the-art large multimodal models struggle to generalize well to simple abstract patterns . |
| Approach: | They evaluate large multimodal models with abstract patterns based on colors, numbers, sizes, and shapes. |
| Outcome: | The proposed model fails to generalize well to simple abstract patterns, the study shows . the model fails on single-concept puzzles, despite its sophistication . |
Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Visual-Language-Action models lack the ability to generate actionable policies tailored to specific robotic embodiments. |
| Approach: | They propose an embodied multimodal action model with Grounded Chain of Thought and Look-ahead Spatial Reasoning that enhances spatial reasoning and task planning. |
| Outcome: | The proposed model improves on existing baselines in tasks requiring spatial reasoning and grounding reasoning. |
Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming methods generate adversarial attacks to identify vulnerabilities, but they face slow performance, limited categorical diversity, and high resource demands. |
| Approach: | They propose a method that generates multiple adversarial prompt mutations per iteration and ranks them using scoring functions. |
| Outcome: | The proposed method achieves a 95% attack success rate and reduces time to a 90% ASR by 15.2%. |
AlgoPuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Algorithmic Multimodal Puzzles (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets focused on visual question-answering focus on visual, language, and algorithmic knowledge . a new study examines the performance of multimodal language models in solving algorithmic puzzles . |
| Approach: | They propose a dataset to test the capabilities of multimodal language models in solving algorithmic puzzles. |
| Outcome: | The proposed dataset is generated automatically from human code. |