REFINESUMM: Self-Refining MLLM for Generating a Multimodal Summarization Dataset (2024.acl-long)
Copied to clipboard
| Challenge: | generating accurate and faithful multimodal summaries is challenging due to lack of appropriate multimodal datasets . large language models excel at synthesizing key information from diverse sources, but lack of adequate multimodal data sets for fine-tuning . |
| Approach: | They propose a dataset specifically designed for image-text multimodal summarization . they generate summaries from Wikipedia sections and corresponding images and evaluate them . |
| Outcome: | The proposed dataset improves summary quality by training a critic model on human annotations and using its predictions to remove low-quality summaries. |
Similar Papers
MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances have enabled MLLMs to tackle complex challenges such as mathematical reasoning and multimodal understanding. |
| Approach: | They propose a multimodal refinement benchmark to evaluate the refinement capabilities of Multimodal Large Language Models (MLLMs) the benchmark categorizes errors into six error types to highlight areas for improvement in effective reasoning enhancement. |
| Outcome: | The proposed framework evaluates the refinement capabilities of multimodal large language models across six scenarios. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal summarization with multimodal output (MSMO) has attracted increasing research interest . evaluation is an emerging yet underexplored research topic . |
| Approach: | They propose a framework that studies three research questions of MSMO evaluation . they propose an automatic evaluation metric and a meta-evaluation benchmark dataset . |
| Outcome: | The proposed evaluation metric and human-annotated meta-evaluation benchmark are used to assess the quality of evaluation metrics and show the framework is effective. |
Self-Improvement in Multimodal Large Language Models: A Survey (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using data and data, self-improvement for Large Language Models has improved model capabilities without significantly increasing costs. |
| Approach: | This survey provides a comprehensive overview of self-improvement for Large Language Models . it includes commonly used evaluations and downstream applications . |
| Outcome: | The authors provide a comprehensive overview of self-improvement in Multimodal LLMs. |
MCiteBench: A Multimodal Benchmark for Generating Text with Citations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work focuses on generating citations for text-only content . experimental results reveal MLLMs struggle to ground outputs reliably when handling multimodal input . |
| Approach: | They propose a benchmark to assess the ability of MLLMs to generate text with citations in multimodal contexts. |
| Outcome: | The proposed benchmark assesses the ability of MLLMs to generate text with citations in multimodal contexts. |
Towards Unified Multimodal Large Language Models: A survey (2026.findings-acl)
Copied to clipboard
| Challenge: | unified multimodal large language models (MLLMs) are emerging but lack a systematic framework to connect them and situate current trends within a broader landscape. |
| Approach: | They present a systematic review of unified Multimodal Large Language Models . they outline the foundational concepts and prerequisites for understanding them . |
| Outcome: | The present review provides a systematic and systematic overview of unified MLLMs . it discusses persistent challenges and identify promising directions for future research . |
Train a Unified Multimodal Data Quality Classifier with Synthetic Data (2025.findings-emnlp)
Copied to clipboard
Weizhi Wang, Rongmei Lin, Shiyang Li, Colin Lockard, Ritesh Sarkhel, Sanket Lokegaonkar, Jingbo Shang, Xifeng Yan, Nasser Zalmout, Xian Li
| Challenge: | Multimodal Large Language Models are pre-trained on image-text caption data and interleaved document data. |
| Approach: | They propose to train an efficient MLLM as a Unified Mulitmodal Data Quality Classifier to filter image-text caption and interleaved data. |
| Outcome: | The proposed method enables efficient creation of sample-score pairs for caption and interleaved data to train UniFilter. |
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity (2026.findings-acl)
Copied to clipboard
| Challenge: | MLLMs have facilitated multimodal summarization with multimodal outputs, but their evaluation is fragmented . MM-Eval integrates assessments of textual quality, cross-modal alignment, and visual diversity . |
| Approach: | They propose a unified evaluation framework that integrates assessments of textual quality, cross-modal alignment, and visual diversity. |
| Outcome: | The proposed framework improves over heuristic aggregation baselines and provides an interpretable, reference-weak framework for comparative evaluation of multimodal summaries. |
Unveiling Multimodal Processing: Exploring Activation Patterns in Multimodal LLMs for Interpretability and Efficiency (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have remained opaque. |
| Approach: | They propose a method to convert dense MLLMs into fine-grained Mixture-of-Experts architectures. |
| Outcome: | The proposed method outperforms random expert pruning and sparse activation and model pruning. |
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review (2025.findings-acl)
Copied to clipboard
Pei Fu, Tongkun Guan, Zining Wang, Zhentao Guo, Chen Duan, Hao Sun, Boming Chen, Qianyi Jiang, Jiayao Ma, Kai Zhou, Junfeng Luo
| Challenge: | Recent advances in vision-language models have unified perception and understanding tasks within Visual Question Answering paradigms. |
| Approach: | They propose to outline timeline, architecture, and pipeline of nearly all TIU MLLMs and review their performance on mainstream benchmarks. |
| Outcome: | The proposed models perform well on mainstream benchmarks and are compared with other models. |