WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. |
| Approach: | They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. |
| Outcome: | The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups. |
Similar Papers
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization (2025.acl-long)
Copied to clipboard
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu
| Challenge: | Existing studies focus on building text-only agents in synthetic environments where the reward signals are clearly defined. |
| Approach: | They propose a multimodal web agent that can autonomously conduct real-world exploration and improve itself after each iteration. |
| Outcome: | The proposed agent improves itself after each iteration, demonstrating strong performance across multiple test sets. |
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024.acl-long)
Copied to clipboard
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, Daniel Fried
| Challenge: | Existing benchmarks focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. |
| Approach: | They propose a benchmark to assess the performance of multimodal web agents . they use visual and textual inputs to process and interpret natural language instructions . |
| Outcome: | a new benchmark assesses the performance of multimodal agents on visually grounded tasks . the benchmark identifies limitations of text-only agents and offers insights towards building stronger agents for the web . |
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents (2025.findings-acl)
Copied to clipboard
Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, Ahmed Hassan Awadallah
| Challenge: | Recent success in large multimodal models (LMMs) has sparked promising applications of agents capable of autonomously completing complex web tasks. |
| Approach: | They propose a scalable recipe to synthesize the largest and most diverse trajectory-level dataset to date. |
| Outcome: | The proposed model synthesizes the largest and most diverse trajectory-level dataset to date, with 94K successful multimodal web trajectories, 720K screenshots, and 33M web elements. |
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations (2025.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art multimodal web agents can perform many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). |
| Approach: | They propose to build multimodal web agents for few-shot adaptability using human demonstrations to improve their generalization and adaptability. |
| Outcome: | The proposed framework enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations. |
WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback (2025.findings-emnlp)
Copied to clipboard
Minda Hu, Tianqing Fang, Jianshu Zhang, Jun-Yu Ma, Zhisong Zhang, Jingyan Zhou, Hongming Zhang, Haitao Mi, Dong Yu, Irwin King
| Challenge: | Web agents powered by Large Language Models lack the ability to perform in uncertain web environments. |
| Approach: | They propose to reconstruct web agents' reasoning skills into chain-of-thought rationales by fine-tuning their LLM backbone into a web-based model. |
| Outcome: | The proposed approach significantly improves the agent self-improving benchmark OpenWebVoyager, demonstrating that it can be used to improve the agent's reasoning skills. |
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current language models and retrieval-augmented LMs are limited in their ability to perform tasks on the web. |
| Approach: | They propose a benchmark to evaluate language agents built on top of language models . they propose 'AssistantBench' which includes 214 tasks that can be automatically evaluated . |
| Outcome: | The proposed agent outperforms existing agents in a new benchmark for language agents on the web. |
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)
Copied to clipboard
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
OpenWebAgent: An Open Toolkit to Enable Web Agents on Large Language Models (2024.acl-demos)
Copied to clipboard
Iat Long Iong, Xiao Liu, Yuxuan Chen, Hanyu Lai, Shuntian Yao, Pengbo Shen, Hao Yu, Yuxiao Dong, Jie Tang
| Challenge: | OpenWebAgent integrates large language models and large multimodal models to improve web automation. |
| Approach: | They propose to integrate large language models and large multimodal models into an open toolkit to optimize web automation. |
| Outcome: | The open toolkit integrates both large language models (LLMs) and large multimodal models (LMMs) it enables the development of powerful, task-oriented web agents, significantly enhancing user experience and operational efficiency on the web. |
MMInA: Benchmarking Multihop Multimodal Internet Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks fail to assess embodied agents in a realistic, evolving environment for compositional Internet tasks. |
| Approach: | They propose a multihop and multimodal benchmark to evaluate embodied agents for compositional Internet tasks. |
| Outcome: | The proposed protocol significantly improves the performance of both the single-hop and multihop web browsing abilities. |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |