Papers by Sachit Menon
Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown promising results in arithmetic and symbolic reasoning by expressing intermediate reasoning in text as a chain of thought, yet struggle to extend this capability to answer text queries that are easily solved by visual reasoning. |
| Approach: | They propose a method to unlock the visual reasoning capabilities of multimodal large language models by using a metaphorical ‘whiteboard’ to draw out reasoning steps as images and return these images back to the model for further processing. |
| Outcome: | The proposed method shows that it can be used on four difficult tasks that involve visual and spatial reasoning with no demonstrations or specialized modules. |