WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation (2025.emnlp-main)
Copied to clipboard
Rabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Spandana Gella, Sai Rajeswar
| Challenge: | Existing benchmarks focus on specific aspects of web tasks but lack comprehensive coverage. |
| Approach: | They propose a multilingual benchmark that evaluates three core web tasks: (1) website visual question answering, (2) code editing involving HTML/CSS/JavaScript, and (3) mockup-to-code generation. |
| Outcome: | The proposed model performs well on basic information extraction, but struggles with reasoning and grounding, editing code to preserve functionality, and generating design-to-code that maintains hierarchy and supports multilingual content. |
Similar Papers
WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models focus on webpage generation outcomes. |
| Approach: | They propose a multi-view evaluation framework to evaluate MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code. |
| Outcome: | The proposed framework evaluates MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
A Culturally-diverse Multilingual Multimodal Video Benchmark & Model (2025.emnlp-main)
Copied to clipboard
Bhuiyan Sanjid Shafique, Ashmal Vayani, Muhammad Maaz, Hanoona Abdul Rasheed, Dinura Dissanayake, Mohammed Irfan Kurpath, Yahya Hmaiti, Go Inoue, Jean Lahoud, Md. Safirur Rashid, Shadid Intisar Quasem, Maheen Fatima, Franco Vidal, Mykola Maslych, Ketan Pravin More, Sanoojan Baliah, Hasindri Watawana, Yuhao Li, Fabian Farestam, Leon Schaller, Roman Tymtsiv, Simon Weber, Hisham Cholakkal, Ivan Laptev, Shin’ichi Satoh, Michael Felsberg, Mubarak Shah, Salman Khan, Fahad Shahbaz Khan
| Challenge: | Large multimodal models have gained attention for their effectiveness to understand and generate descriptions of visual content. |
| Approach: | They propose a multilingual Video LMM benchmark to evaluate video LMMs across 14 languages . they also introduce a machine translated multilingual video training set . |
| Outcome: | The proposed video LMM benchmark is designed to evaluate video Lmms across 14 languages including Arabic, Bengali, Chinese, English, French, German, Hindi, Japanese, Russian, Sinhala, Spanish, Swedish, Tamil, and Urdu. |
IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web (2025.findings-acl)
Copied to clipboard
Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Shaosheng Cao, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, Zhoujun Li
| Challenge: | Existing models have been introduced to improve image comprehension, but there is no robust benchmark for imagetoweb conversion. |
| Approach: | They propose a benchmark to assess imagetoweb conversion proficiency of large multimodal models . they propose to measure layout information of web pages by parsing the Document Object Model tree . |
| Outcome: | The proposed benchmark measures the layout information of web pages—i.e., the positional relationships between elements—which has been overlooked by prior work. |
mHumanEval - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluations focus on English-to-Python conversion tasks with limited test cases . code generation from low-resource language prompts remains largely unexplored . |
| Approach: | They propose a benchmark that supports prompts in over 200 natural languages . they provide expert human translations for 15 diverse natural languages (NLs) |
| Outcome: | The HumanEval Benchmark is the most widely used code generation benchmark . it provides expert human translations for 15 diverse natural languages . |
TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-rich image understanding benchmarks lack scale and fragmented scenarios . a new full-image structured output format is proposed to enable fine-grained evaluation of perception and reasoning capabilities. |
| Approach: | They propose a large-scale, multilingual benchmark that includes over 100,000 annotations and 22,000 question-answer pairs. |
| Outcome: | The proposed framework provides a comprehensive platform for developing and evaluating next-generation multimodal AI systems. |
P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs (2025.emnlp-main)
Copied to clipboard
Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou
| Challenge: | Recent advances in large language models showcase varied multilingual capabilities across tasks . previous assessments focused on fundamental natural language processing (NLP) or isolated capability-specific tasks. |
| Approach: | They propose a multilingual multitask benchmark to assess multilingual capabilities . they use a large-scale benchmark covering fundamental and capability-specialized datasets . |
| Outcome: | The proposed benchmark compares models and tasks across languages and tasks and examines knowledge transfer from English to other languages. |
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024.acl-long)
Copied to clipboard
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, Daniel Fried
| Challenge: | Existing benchmarks focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. |
| Approach: | They propose a benchmark to assess the performance of multimodal web agents . they use visual and textual inputs to process and interpret natural language instructions . |
| Outcome: | a new benchmark assesses the performance of multimodal agents on visually grounded tasks . the benchmark identifies limitations of text-only agents and offers insights towards building stronger agents for the web . |
M5 – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models and their multimodal counterparts have shown significant performance disparities across different languages and cultural contexts. |
| Approach: | They propose to evaluate LLMs on diverse vision-language tasks within a multilingual and multicultural context using M5 benchmark. |
| Outcome: | The proposed benchmarks highlight task-agnostic performance disparities between languages and cultural contexts. |
MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing chart understanding benchmarks focus on single-chart tasks, neglecting multi-hop reasoning required to extract and integrate information from multiple charts. |
| Approach: | They propose a benchmark that evaluates MLLMs’ capabilities in four key areas: direct question answering, parallel question answering and comparative reasoning. |
| Outcome: | The proposed benchmark evaluates MLLMs’ capabilities in four key areas: direct question answering, parallel question answering and comparative reasoning. |