IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation

Challenge:	Generated infographics may appear correct at first glance but contain easily overlooked issues, such as distorted data encoding or incorrect textual content.
Approach:	They propose to evaluate reliability of text-to-infographic generation using IGenBench . they employ multimodal large language models to verify each question .
Outcome:	The proposed framework decomposes reliability verification into atomic yes/no questions based on a taxonomy of 10 question types.

Similar Papers

Overview

Challenge

Approach

Outcome

T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation (2026.findings-acl)

Copied to clipboard

Kaiyue Sun, Rongyao Fang, Chengqi Duan, Xian Liu, Aoxue Li, Xihui Liu

Multimodality

Challenge:	Text-to-image (T2I) generative models have demonstrated exceptional capability in synthesizing high-quality images from textual prompts.
Approach:	They propose a benchmark to explore the knowledge-driven reasoning capabilities of T2I models.
Outcome:	The proposed benchmark examines the knowledge-driven reasoning capabilities of T2I models.

R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation (2025.emnlp-main)

Copied to clipboard

Kaijie Chen, Zihao Lin, Zhiyang Xu, Ying Shen, Yuguang Yao, Joy Rimchala, Jiaxin Zhang, Lifu Huang

Multimodality

Challenge:	Reasoning is a fundamental capability underpinning text-to-image (T2I) generation.
Approach:	They propose a benchmark to rigorously assess reasoning-driven T2I generation.
Outcome:	Experiments with 16 representative T2I models show limited reasoning performance . a strong pipeline-based framework decouples reasoning and generation .

Infogen: Generating Complex Statistical Infographics from Documents (2025.acl-long)

Copied to clipboard

Akash Ghosh, Aparna Garimella, Pritika Ramu, Sambaran Bandyopadhyay, Sriparna Saha

Multimodality

Challenge:	Existing efforts to generate simple charts have focused on generating simple infographics from text-heavy documents.
Approach:	They propose to generate statistical infographics composed of multiple sub-charts that are contextually accurate, insightful, and visually aligned.
Outcome:	The proposed framework outperforms both open-source and closed LLMs in text-to-statistical infographic generation.

T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts (2025.acl-long)

Copied to clipboard

Ziwei Huang, Wanggui He, Quanyu Long, Yandi Wang, Haoyuan Li, Zhelun Yu, Fangxun Shu, Weilong Dai, Hao Jiang, Fei Wu, Leilei Gan

Multimodality

Challenge:	Existing studies on text-to-image (T2I) models focus on text alignment, image quality, and object composition capabilities.
Approach:	They propose a T2I-FactualBench benchmark to evaluate the factuality of knowledge-intensive concept generation.
Outcome:	The proposed framework evaluates the factuality of knowledge-intensive concept generation tasks.

Holistic Evaluation for Interleaved Text-and-Image Generation (2024.emnlp-main)

Copied to clipboard

Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, Lifu Huang

Multimodality

Challenge:	Existing evaluation benchmarks do not support arbitrarily interleaved images and text for both inputs and outputs.
Approach:	They propose to use a benchmark to evaluate interleaved text-and-image generation . they define five evaluation aspects for InterleavatedEval, a reference-free metric .
Outcome:	The proposed benchmarks cover a limited number of domains and use cases and lack comparableity-based metrics.

Chart-to-Text: A Large-Scale Benchmark for Chart Summarization (2022.acl-long)

Copied to clipboard

Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty

Text Generation

Challenge:	Inferring key insights from charts can be challenging and time-consuming.
Approach:	They propose a task where the goal is to explain a chart and summarize key takeaways from it in natural language.
Outcome:	The proposed model produces fluent summaries but suffers from hallucinations and factual errors . the proposed model is compared with other models and can be used to generate BLEU scores .

AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)

Copied to clipboard

Yifei Li, Xiang Yue, Zeyi Liao, Huan Sun

Challenge:	generative search engines enhance the reliability of large language model responses by providing cited evidence.
Approach:	They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not .
Outcome:	The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation.

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (2021.findings-emnlp)

Copied to clipboard

Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, Dongwon Lee

Language Models

Challenge:	Recent advances in generative language models have enabled machines to generate realistic texts.
Approach:	They propose a benchmark environment to test the 'Turing Test' problem for neural text generation methods.
Outcome:	The proposed benchmark environment is based on 200K human- or machine-generated samples across 20 labels Human, GPT-1, GTP-2_small, GTT-2_medium, GPG-2_large, GGT-2_PyTorch, GGP-3, GROVER_base, griover_large and GRover_mega.

Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024.emnlp-main)

Copied to clipboard

Jianfeng He, Runing Yang, Linlin Yu, Changbin Li, Ruoxi Jia, Feng Chen, Ming Jin, Chang-Tien Lu

Text Generation

Challenge:	Text summarization is a key natural language generation task, but the high cost of inaccurate summaries raises concerns about the reliability of uncertainty estimation on text summarisation (UE-TS) evaluation methods.
Approach:	They propose a UE-TS benchmark that evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets.
Outcome:	The proposed benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable.

AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation (2025.findings-naacl)

Copied to clipboard

Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan, Debjit Pal

Language Models

Challenge:	Assertions have been the de facto collateral for hardware for over a decade.
Approach:	They propose a benchmark to evaluate LLMs’ effectiveness for assertion generation quantitatively.
Outcome:	The proposed benchmark compares state-of-the-art LLMs with existing benchmarks and shows that they generate higher fractions of functionally correct assertions.