Papers by Michal Shlapentokh-Rothman
WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior work to control software has used reinforcement learning (RL), requiring many demonstrations and trials to learn simple interaction tasks. |
| Approach: | They propose a Large Language Model to automatically perform web software tasks using click, scroll, and text in- put operations using filtered Document Object Models as observations. |
| Outcome: | The proposed method performs better on the MiniWob++ benchmark with only one in-context example. |
Visual Program Distillation with Template-Based Augmentation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting visual programming to specialized tasks or domains remains challenging due to high annotation and inference costs. |
| Approach: | They propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. |
| Outcome: | The proposed method can generate high-quality visual programs with no human-generated annotations with a relatively small amount of question/answer data. |