Papers by Derek Hoiem
WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior work to control software has used reinforcement learning (RL), requiring many demonstrations and trials to learn simple interaction tasks. |
| Approach: | They propose a Large Language Model to automatically perform web software tasks using click, scroll, and text in- put operations using filtered Document Object Models as observations. |
| Outcome: | The proposed method performs better on the MiniWob++ benchmark with only one in-context example. |
Visual Program Distillation with Template-Based Augmentation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting visual programming to specialized tasks or domains remains challenging due to high annotation and inference costs. |
| Approach: | They propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. |
| Outcome: | The proposed method can generate high-quality visual programs with no human-generated annotations with a relatively small amount of question/answer data. |
ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation (2023.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art vision-language models have limited performance in structural knowledge extraction, such as relations between objects. |
| Approach: | They propose to leverage the inherent structure of programming language to depict visual structural information in a well-organized structured format. |
| Outcome: | The proposed framework improves visual structural knowledge extraction on visual structure prediction tasks. |